背景
正好最近项目可能要用到ZooKeeper,于是把之前的ZK资料整理一下,大家有兴趣的就看一看。
基本概念介绍
ZooKeeper 是一个分布式的,开放源码的分布式应用程序协同服务。 ZooKeeper 的设计目标是将那些复杂且容易出错的分布式一致性服 务封装起来,构成一个高效可靠的原语集,并以一系列简单易用 的接口提供给用户使用。以下简称ZK。
典型应用场景
-
配置管理,类似一个数据库
-
DNS服务
-
组成员管理
-
分布式锁
由于ZK的数据都存放在内存里,数据量大多维持在几百兆,而数据库数据几十GB也是常见。
ZK集群的搭建
开发和测试强烈建议采用docker-compose的方式,非常方便。生产的话,需要采用多台独立的机器来做集群实现,否则的话就没有必要使用ZK了。
version: '3.1'
services:
zoo1:
image: zookeeper
restart: always
hostname: zoo1
ports:
- 2181:2181
environment:
ZOO_MY_ID: 1
ZOO_SERVERS: server.1=0.0.0.0:2888:3888;2181 server.2=zoo2:2888:3888;2181 server.3=zoo3:2888:3888;2181
zoo2:
image: zookeeper
restart: always
hostname: zoo2
ports:
- 2182:2181
environment:
ZOO_MY_ID: 2
ZOO_SERVERS: server.1=zoo1:2888:3888;2181 server.2=0.0.0.0:2888:3888;2181 server.3=zoo3:2888:3888;2181
zoo3:
image: zookeeper
restart: always
hostname: zoo3
ports:
- 2183:2181
environment:
ZOO_MY_ID: 3
ZOO_SERVERS: server.1=zoo1:2888:3888;2181 server.2=zoo2:2888:3888;2181 server.3=0.0.0.0:2888:3888;2181
基于Java实现ZK的交互
依赖的库
dependencies {
implementation "org.apache.zookeeper:zookeeper:3.5.5"
implementation "org.apache.curator:curator-recipes:4.2.0"
implementation "org.apache.curator:curator-x-discovery:4.2.0"
implementation "org.apache.curator:curator-x-discovery-server:4.2.0"
testImplementation "junit:junit:4.12"
testImplementation "com.google.truth:truth:1.0"
}
原始API
基本测试用例
package org.yao;
import com.google.common.collect.ImmutableList;
import org.apache.zookeeper.AsyncCallback;
import org.apache.zookeeper.CreateMode;
import org.apache.zookeeper.KeeperException;
import org.apache.zookeeper.Op;
import org.apache.zookeeper.OpResult;
import org.apache.zookeeper.Transaction;
import org.apache.zookeeper.WatchedEvent;
import org.apache.zookeeper.Watcher;
import org.apache.zookeeper.ZooDefs;
import org.apache.zookeeper.ZooKeeper;
import org.junit.After;
import org.junit.Before;
import org.junit.Test;
import java.util.List;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.ThreadLocalRandom;
import static com.google.common.truth.Truth.assertThat;
public class ZooKeeperTests {
private String pathPrefix = "/multi";
private ZooKeeper zk;
private CountDownLatch startLatch;
private CountDownLatch closeLatch;
private AsyncCallback.MultiCallback callback;
private String path1 = pathPrefix + "1";
private String path2 = pathPrefix + "2";
private byte[] data1 = {0x1};
private byte[] data2 = {0x2};
@Before
public void setUp() throws Exception {
//注册回调,在回调后,就停止等待
startLatch = new CountDownLatch(1);
callback =
(int rc, String path, Object ctx, List<OpResult> opResults) -> {
assertThat(rc).isEqualTo(KeeperException.Code.OK.intValue());
System.out.printf("delete multi executed");
closeLatch.countDown();
};
zk = new ZooKeeper("localhost", 2181, new DefaultWatcher());
startLatch.await();
}
@After
public void tearDown() throws Exception {
//清理zk的链接
closeLatch.await();
zk.close();
}
@Test
public void testMulti() throws Exception {
closeLatch = new CountDownLatch(1);
// Create two znodes
Op createOp1 = Op.create(path1, data1, ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
Op createOp2 = Op.create(path2, data2, ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
// Synchronous API
zk.multi(ImmutableList.of(createOp1, createOp2));
System.out.println("create multi executed");
assertThat(zk.getData(path1, false, null)).isEqualTo(data1);
assertThat(zk.getData(path2, false, null)).isEqualTo(data2);
// Delete two znodes
Op deleteOp1 = Op.delete(path1, -1);
Op deleteOp2 = Op.delete(path2, -1);
// Asynchronous API
zk.multi(ImmutableList.of(deleteOp1, deleteOp2), callback, null);
}
@Test
public void testTransaction() throws Exception {
closeLatch = new CountDownLatch(1);
// Create two znodes
Transaction tx = zk.transaction();
tx.create(path1, data1, ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
tx.create(path2, data2, ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
// Synchronous API
tx.commit();
System.out.println("transaction committed");
assertThat(zk.getData(path1, false, null)).isEqualTo(data1);
assertThat(zk.getData(path2, false, null)).isEqualTo(data2);
// Delete two znodes
tx = zk.transaction();
tx.delete(path1, -1);
tx.delete(path2, -1);
// Asynchronous API
tx.commit(callback, null);
}
@Test
public void testTransactionWithCheck() throws Exception {
closeLatch = new CountDownLatch(0);
{
Transaction tx = zk.transaction();
tx.create(path1, data1, ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
tx.create(path2, data2, ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
tx.check(path1, 0);
tx.check(path2, 0);
tx.commit();
}
{
Transaction tx = zk.transaction();
tx.check(path1, 0);
tx.check(path2, 0);
tx.delete(path1, 0);
tx.delete(path2, 0);
tx.commit();
}
}
/**
* getChildren does not list descendants recursively.
*/
@Test
public void testGetChilren() throws Exception {
closeLatch = new CountDownLatch(0);
List<String> paths = zk.getChildren("/a", false);
System.out.printf("child paths: %s\n", paths);
}
class DefaultWatcher implements Watcher {
@Override
public void process(WatchedEvent event) {
if (event.getType() == Event.EventType.None
&& event.getState() == Event.KeeperState.SyncConnected) {
System.out.println("zookeeper client connected");
startLatch.countDown();
}
}
}
}
Curator的API
Curator介绍
Curator 是ZooKeeper 的 Java一种客户端库。Curator 目标是简化 ZK的使用。在未使用Curator的代码都要自己处理 ConnectionLossException 。并且提供了锁,服务发现等封装的较为完毕的实现,避免大家重复造轮子。
测试用例
package org.yao;
import org.apache.curator.RetryPolicy;
import org.apache.curator.framework.CuratorFramework;
import org.apache.curator.framework.CuratorFrameworkFactory;
import org.apache.curator.framework.api.CuratorEvent;
import org.apache.curator.retry.ExponentialBackoffRetry;
import org.apache.zookeeper.CreateMode;
import org.apache.zookeeper.WatchedEvent;
import org.apache.zookeeper.Watcher;
import org.junit.After;
import org.junit.Before;
import org.junit.Test;
import java.util.concurrent.CountDownLatch;
import static com.google.common.truth.Truth.assertThat;
/**
* Example code to demonstrate the usage of Curator client and framework.
*/
public class CuratorTests {
private CuratorFramework client;
private String connectString = "localhost:2181";
private RetryPolicy retryPolicy;
@Before
public void setUp() {
retryPolicy = new ExponentialBackoffRetry(1000, 3);
client = CuratorFrameworkFactory.newClient(connectString, retryPolicy);
/*
// Fluent style
client =
CuratorFrameworkFactory.builder()
.connectString(connectString)
.retryPolicy(retryPolicy)
.build();
*/
// Start client
client.start();
}
@After
public void tearDown() {
client.close();
}
// create -> getData -> delete in synchronous mode
@Test
public void testSyncOp() throws Exception {
String path = "/one";
byte[] data = {'1'};
client.create().withMode(CreateMode.PERSISTENT).forPath(path, data);
byte[] actualData = client.getData().forPath(path);
assertThat(data).isEqualTo(actualData);
client.delete().forPath(path);
client.close();
}
// create -> getData -> delete in asynchronous mode
@Test
public void testAsyncOp() throws Exception {
String path = "/two";
final byte[] data = {'2'};
final CountDownLatch latch = new CountDownLatch(1);
// Use listener only for callbacks
client
.getCuratorListenable()
.addListener(
(CuratorFramework c, CuratorEvent event) -> {
switch (event.getType()) {
case CREATE:
System.out.printf("znode '%s' created\n", event.getPath());
// 2. getData
c.getData().inBackground().forPath(event.getPath());
break;
case GET_DATA:
System.out.printf("got the data of znode '%s'\n", event.getPath());
assertThat(event.getData()).isEqualTo(data);
// 3. Delete
c.delete().inBackground().forPath(path);
break;
case DELETE:
System.out.printf("znode '%s' deleted\n", event.getPath());
latch.countDown();
break;
}
});
// 1. create
client.create().withMode(CreateMode.PERSISTENT).inBackground().forPath(path, data);
latch.await();
client.close();
}
@Test
public void testWatch() throws Exception {
String path = "/three";
byte[] data = {'3'};
byte[] newData = {'4'};
CountDownLatch latch = new CountDownLatch(1);
// Use listener only for watches
client
.getCuratorListenable()
.addListener(
(CuratorFramework c, CuratorEvent event) -> {
switch (event.getType()) {
case WATCHED:
WatchedEvent we = event.getWatchedEvent();
System.out.println("watched event: " + we);
if (we.getType() == Watcher.Event.EventType.NodeDataChanged
&& we.getPath().equals(path)) {
// 4. watch triggered
System.out.printf("got the event for the triggered watch\n");
byte[] actualData = c.getData().forPath(path);
assertThat(actualData).isEqualTo(newData);
}
latch.countDown();
break;
}
});
// 1. create
client.create().withMode(CreateMode.PERSISTENT).forPath(path, data);
// 2. getData and register a watch
byte[] actualData = client.getData().watched().forPath(path);
assertThat(actualData).isEqualTo(data);
// 3. setData
client.setData().forPath(path, newData);
latch.await();
// 5. delete
client.delete().forPath(path);
}
@Test
public void testCallbackAndWatch() throws Exception {
String path = "/four";
byte[] data = {'4'};
byte[] newData = {'5'};
CountDownLatch latch = new CountDownLatch(2);
// Use listener for both callbacks and watches
client
.getCuratorListenable()
.addListener(
(CuratorFramework c, CuratorEvent event) -> {
switch (event.getType()) {
case CREATE:
// 2. callback for create
System.out.printf("znode '%s' created\n", event.getPath());
// 3. getData and register a watch
assertThat(client.getData().watched().forPath(path)).isEqualTo(data);
// 4. setData
client.setData().forPath(path, newData);
latch.countDown();
break;
case WATCHED:
WatchedEvent we = event.getWatchedEvent();
System.out.println("watched event: " + we);
if (we.getType() == Watcher.Event.EventType.NodeDataChanged
&& we.getPath().equals(path)) {
// 5. watch triggered
System.out.printf("got the event for the triggered watch\n");
assertThat(c.getData().forPath(path)).isEqualTo(newData);
}
latch.countDown();
break;
}
});
// 1. create
client.create().withMode(CreateMode.PERSISTENT).inBackground().forPath(path, data);
latch.await();
// 6. delete
client.delete().forPath(path);
}
}
分布式锁的测试用例
package com.zew.learn;
import org.apache.curator.framework.CuratorFramework;
import org.apache.curator.framework.CuratorFrameworkFactory;
import org.apache.curator.framework.recipes.locks.InterProcessMutex;
import org.apache.curator.retry.ExponentialBackoffRetry;
import org.apache.curator.utils.CloseableUtils;
import org.junit.Test;
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicBoolean;
import static org.junit.Assert.assertTrue;
import static org.junit.Assert.fail;
public class CuratorLockTests {
private final String connectString = "10.0.4.162:2181";
private final String lockPath = "/locks";
private static final int QTY = 5;
private static final int REPETITIONS = QTY * 10;
@Test
public void lockTest() {
//建立一个QTY个线程的线程池来模拟多个终端并发的场景
ExecutorService service = Executors.newFixedThreadPool(QTY);
final FakeLimitedResource resource = new FakeLimitedResource();
try {
for (int i = 0; i < QTY; ++i) {
final int index = i;
Callable<Void> task = () -> {
//创建curator的client
CuratorFramework client = CuratorFrameworkFactory.newClient(connectString, new ExponentialBackoffRetry(1000, 3));
try {
client.start();
ExampleClientThatLocks example = new ExampleClientThatLocks(client, lockPath, resource, "Client " + index);
for (int j = 0; j < REPETITIONS; ++j) {
example.doWork(10, TimeUnit.SECONDS);
}
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
} catch (Exception e) {
e.printStackTrace();
// log or do something
} finally {
CloseableUtils.closeQuietly(client);
}
return null;
};
service.submit(task);
}
service.shutdown();
service.awaitTermination(10, TimeUnit.MINUTES);
} catch (InterruptedException e) {
e.printStackTrace();
}
assertTrue(true);
}
static class FakeLimitedResource {
private final AtomicBoolean inUse = new AtomicBoolean(false);
//模拟有限资源的使用情况,隔了一个随机耗时
public void use() throws InterruptedException {
if (!inUse.compareAndSet(false, true)) {
fail("只有一个客户端可以使用这个锁,跑到这里就说明zk让多个客户端拿到了锁");
throw new IllegalStateException("只有一个客户端可以使用这个锁");
}
try {
Thread.sleep((long) (3 * Math.random()));
} finally {
inUse.set(false);
}
}
}
/**
* 负责锁定的类
*/
static class ExampleClientThatLocks {
private final InterProcessMutex lock;
private final FakeLimitedResource resource;
private final String clientName;
public ExampleClientThatLocks(CuratorFramework client,
String lockPath,
FakeLimitedResource resource,
String clientName) {
this.resource = resource;
this.clientName = clientName;
//创建锁,虽然client不一样,但lockPath一样就能起到锁定的效果
lock = new InterProcessMutex(client, lockPath);
}
/**
* 实际锁的核心代码就是下面几句话
* @param time 时间长度
* @param unit 时间单位
* @throws Exception 抛出的异常
*/
public void doWork(long time, TimeUnit unit) throws Exception {
//尝试获取锁,如果超过时间还不能获取就抛异常
if (!lock.acquire(time, unit)) {
throw new IllegalStateException(clientName + " 无法获取锁");
}
try {
System.out.println(clientName + " 拿到了锁");
resource.use();
} finally {
System.out.println(clientName + " 释放锁");
lock.release(); // always release the lock in a finally block
}
}
}
}
服务发现测试用例
package org.yao;
import org.apache.curator.framework.CuratorFramework;
import org.apache.curator.framework.CuratorFrameworkFactory;
import org.apache.curator.retry.RetryOneTime;
import org.apache.curator.utils.CloseableUtils;
import org.apache.curator.x.discovery.ServiceDiscovery;
import org.apache.curator.x.discovery.ServiceDiscoveryBuilder;
import org.apache.curator.x.discovery.ServiceInstance;
import org.apache.curator.x.discovery.ServiceProvider;
import org.junit.Test;
import static com.google.common.truth.Truth.assertThat;
public class ServiceDiscoveryTests {
private String connectString = "localhost:2181";
/** Shows the basic usage for curator-x-discovery. */
@Test
public void testBasics() throws Exception {
CuratorFramework client = null;
ServiceDiscovery<String> discovery = null;
ServiceProvider<String> provider = null;
String serviceName = "test";
String basePath = "/services";
try {
client = CuratorFrameworkFactory.newClient(connectString, new RetryOneTime(1));
client.start();
ServiceInstance<String> instance1 =
ServiceInstance.<String>builder().payload("plant").name(serviceName).port(10064).build();
ServiceInstance<String> instance2 =
ServiceInstance.<String>builder().payload("animal").name(serviceName).port(10065).build();
System.out.printf("instance1 id: %s\n", instance1.getId());
System.out.printf("instance2 id: %s\n", instance2.getId());
discovery =
ServiceDiscoveryBuilder.builder(String.class)
.basePath(basePath)
.client(client)
.thisInstance(instance1)
.build();
discovery.start();
discovery.registerService(instance2);
provider = discovery.serviceProviderBuilder().serviceName(serviceName).build();
provider.start();
assertThat(provider.getInstance().getId()).isNotEmpty();
assertThat(provider.getAllInstances()).containsExactly(instance1, instance2);
client.delete().deletingChildrenIfNeeded().forPath(basePath);
} finally {
CloseableUtils.closeQuietly(provider);
CloseableUtils.closeQuietly(discovery);
CloseableUtils.closeQuietly(client);
}
}
}
参考资料
- ZooKeeper实战与源码剖析
- curator.apache.org/
- github.com/apache/cura…