ZooKeeper极简介绍和入门

261 阅读5分钟

背景

正好最近项目可能要用到ZooKeeper,于是把之前的ZK资料整理一下,大家有兴趣的就看一看。

基本概念介绍

ZooKeeper 是一个分布式的,开放源码的分布式应用程序协同服务。 ZooKeeper 的设计目标是将那些复杂且容易出错的分布式一致性服 务封装起来,构成一个高效可靠的原语集,并以一系列简单易用 的接口提供给用户使用。以下简称ZK。

典型应用场景

  • 配置管理,类似一个数据库

  • DNS服务

  • 组成员管理

  • 分布式锁

    由于ZK的数据都存放在内存里,数据量大多维持在几百兆,而数据库数据几十GB也是常见。

ZK集群的搭建

开发和测试强烈建议采用docker-compose的方式,非常方便。生产的话,需要采用多台独立的机器来做集群实现,否则的话就没有必要使用ZK了。

version: '3.1'

services:
  zoo1:
    image: zookeeper
    restart: always
    hostname: zoo1
    ports:
      - 2181:2181
    environment:
      ZOO_MY_ID: 1
      ZOO_SERVERS: server.1=0.0.0.0:2888:3888;2181 server.2=zoo2:2888:3888;2181 server.3=zoo3:2888:3888;2181

  zoo2:
    image: zookeeper
    restart: always
    hostname: zoo2
    ports:
      - 2182:2181
    environment:
      ZOO_MY_ID: 2
      ZOO_SERVERS: server.1=zoo1:2888:3888;2181 server.2=0.0.0.0:2888:3888;2181 server.3=zoo3:2888:3888;2181

  zoo3:
    image: zookeeper
    restart: always
    hostname: zoo3
    ports:
      - 2183:2181
    environment:
      ZOO_MY_ID: 3
      ZOO_SERVERS: server.1=zoo1:2888:3888;2181 server.2=zoo2:2888:3888;2181 server.3=0.0.0.0:2888:3888;2181

基于Java实现ZK的交互

依赖的库

dependencies {
  implementation "org.apache.zookeeper:zookeeper:3.5.5"
  implementation "org.apache.curator:curator-recipes:4.2.0"
  implementation "org.apache.curator:curator-x-discovery:4.2.0"
  implementation "org.apache.curator:curator-x-discovery-server:4.2.0"

  testImplementation "junit:junit:4.12"
  testImplementation "com.google.truth:truth:1.0"
}

原始API

基本测试用例

package org.yao;

import com.google.common.collect.ImmutableList;
import org.apache.zookeeper.AsyncCallback;
import org.apache.zookeeper.CreateMode;
import org.apache.zookeeper.KeeperException;
import org.apache.zookeeper.Op;
import org.apache.zookeeper.OpResult;
import org.apache.zookeeper.Transaction;
import org.apache.zookeeper.WatchedEvent;
import org.apache.zookeeper.Watcher;
import org.apache.zookeeper.ZooDefs;
import org.apache.zookeeper.ZooKeeper;
import org.junit.After;
import org.junit.Before;
import org.junit.Test;

import java.util.List;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.ThreadLocalRandom;

import static com.google.common.truth.Truth.assertThat;

public class ZooKeeperTests {
  private String pathPrefix = "/multi";
  private ZooKeeper zk;
  private CountDownLatch startLatch;
  private CountDownLatch closeLatch;
  private AsyncCallback.MultiCallback callback;

  private String path1 = pathPrefix + "1";
  private String path2 = pathPrefix + "2";
  private byte[] data1 = {0x1};
  private byte[] data2 = {0x2};

  @Before
  public void setUp() throws Exception {
    //注册回调,在回调后,就停止等待
    startLatch = new CountDownLatch(1);
    callback =
        (int rc, String path, Object ctx, List<OpResult> opResults) -> {
          assertThat(rc).isEqualTo(KeeperException.Code.OK.intValue());
          System.out.printf("delete multi executed");
          closeLatch.countDown();
        };
    zk = new ZooKeeper("localhost", 2181, new DefaultWatcher());
    startLatch.await();
  }

  @After
  public void tearDown() throws Exception {
      //清理zk的链接
    closeLatch.await();
    zk.close();
  }

  @Test
  public void testMulti() throws Exception {
    closeLatch = new CountDownLatch(1);

    // Create two znodes
    Op createOp1 = Op.create(path1, data1, ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
    Op createOp2 = Op.create(path2, data2, ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);

    // Synchronous API
    zk.multi(ImmutableList.of(createOp1, createOp2));
    System.out.println("create multi executed");

    assertThat(zk.getData(path1, false, null)).isEqualTo(data1);
    assertThat(zk.getData(path2, false, null)).isEqualTo(data2);

    // Delete two znodes
    Op deleteOp1 = Op.delete(path1, -1);
    Op deleteOp2 = Op.delete(path2, -1);

    // Asynchronous API
    zk.multi(ImmutableList.of(deleteOp1, deleteOp2), callback, null);
  }

  @Test
  public void testTransaction() throws Exception {
    closeLatch = new CountDownLatch(1);

    // Create two znodes
    Transaction tx = zk.transaction();
    tx.create(path1, data1, ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
    tx.create(path2, data2, ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);

    // Synchronous API
    tx.commit();
    System.out.println("transaction committed");

    assertThat(zk.getData(path1, false, null)).isEqualTo(data1);
    assertThat(zk.getData(path2, false, null)).isEqualTo(data2);

    // Delete two znodes
    tx = zk.transaction();
    tx.delete(path1, -1);
    tx.delete(path2, -1);

    // Asynchronous API
    tx.commit(callback, null);
  }

  @Test
  public void testTransactionWithCheck() throws Exception {
    closeLatch = new CountDownLatch(0);

    {
      Transaction tx = zk.transaction();
      tx.create(path1, data1, ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
      tx.create(path2, data2, ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
      tx.check(path1, 0);
      tx.check(path2, 0);
      tx.commit();
    }

    {
      Transaction tx = zk.transaction();
      tx.check(path1, 0);
      tx.check(path2, 0);
      tx.delete(path1, 0);
      tx.delete(path2, 0);
      tx.commit();
    }
  }


  /**
   * getChildren does not list descendants recursively.
   */
  @Test
  public void testGetChilren() throws Exception {
    closeLatch = new CountDownLatch(0);
    List<String> paths = zk.getChildren("/a", false);
    System.out.printf("child paths: %s\n", paths);
  }

  class DefaultWatcher implements Watcher {
    @Override
    public void process(WatchedEvent event) {
      if (event.getType() == Event.EventType.None
          && event.getState() == Event.KeeperState.SyncConnected) {
        System.out.println("zookeeper client connected");
        startLatch.countDown();
      }
    }
  }
}

Curator的API

Curator介绍

Curator 是ZooKeeper 的 Java一种客户端库。Curator 目标是简化 ZK的使用。在未使用Curator的代码都要自己处理 ConnectionLossException 。并且提供了锁,服务发现等封装的较为完毕的实现,避免大家重复造轮子。

测试用例

package org.yao;

import org.apache.curator.RetryPolicy;
import org.apache.curator.framework.CuratorFramework;
import org.apache.curator.framework.CuratorFrameworkFactory;
import org.apache.curator.framework.api.CuratorEvent;
import org.apache.curator.retry.ExponentialBackoffRetry;
import org.apache.zookeeper.CreateMode;
import org.apache.zookeeper.WatchedEvent;
import org.apache.zookeeper.Watcher;
import org.junit.After;
import org.junit.Before;
import org.junit.Test;

import java.util.concurrent.CountDownLatch;

import static com.google.common.truth.Truth.assertThat;

/**
 * Example code to demonstrate the usage of Curator client and framework.
 */
public class CuratorTests {
  private CuratorFramework client;
  private String connectString = "localhost:2181";
  private RetryPolicy retryPolicy;

  @Before
  public void setUp() {
    retryPolicy = new ExponentialBackoffRetry(1000, 3);
    client = CuratorFrameworkFactory.newClient(connectString, retryPolicy);

    /*
    // Fluent style
    client =
        CuratorFrameworkFactory.builder()
            .connectString(connectString)
            .retryPolicy(retryPolicy)
            .build();
    */

    // Start client
    client.start();
  }

  @After
  public void tearDown() {
    client.close();
  }

  // create -> getData -> delete in synchronous mode
  @Test
  public void testSyncOp() throws Exception {
    String path = "/one";
    byte[] data = {'1'};
    client.create().withMode(CreateMode.PERSISTENT).forPath(path, data);

    byte[] actualData = client.getData().forPath(path);
    assertThat(data).isEqualTo(actualData);

    client.delete().forPath(path);

    client.close();
  }


  // create -> getData -> delete in asynchronous mode
  @Test
  public void testAsyncOp() throws Exception {
    String path = "/two";
    final byte[] data = {'2'};
    final CountDownLatch latch = new CountDownLatch(1);

    // Use listener only for callbacks
    client
        .getCuratorListenable()
        .addListener(
            (CuratorFramework c, CuratorEvent event) -> {
              switch (event.getType()) {
                case CREATE:
                  System.out.printf("znode '%s' created\n", event.getPath());
                  // 2. getData
                  c.getData().inBackground().forPath(event.getPath());
                  break;
                case GET_DATA:
                  System.out.printf("got the data of znode '%s'\n", event.getPath());
                  assertThat(event.getData()).isEqualTo(data);
                  // 3. Delete
                  c.delete().inBackground().forPath(path);
                  break;
                case DELETE:
                  System.out.printf("znode '%s' deleted\n", event.getPath());
                  latch.countDown();
                  break;
              }
            });

    // 1. create
    client.create().withMode(CreateMode.PERSISTENT).inBackground().forPath(path, data);

    latch.await();

    client.close();
  }

  @Test
  public void testWatch() throws Exception {
    String path = "/three";
    byte[] data = {'3'};
    byte[] newData = {'4'};
    CountDownLatch latch = new CountDownLatch(1);

    // Use listener only for watches
    client
        .getCuratorListenable()
        .addListener(
            (CuratorFramework c, CuratorEvent event) -> {
              switch (event.getType()) {
                case WATCHED:
                  WatchedEvent we = event.getWatchedEvent();
                  System.out.println("watched event: " + we);
                  if (we.getType() == Watcher.Event.EventType.NodeDataChanged
                      && we.getPath().equals(path)) {
                    // 4. watch triggered
                    System.out.printf("got the event for the triggered watch\n");
                    byte[] actualData = c.getData().forPath(path);
                    assertThat(actualData).isEqualTo(newData);
                  }
                  latch.countDown();
                  break;
              }
            });

    // 1. create
    client.create().withMode(CreateMode.PERSISTENT).forPath(path, data);
    // 2. getData and register a watch
    byte[] actualData = client.getData().watched().forPath(path);
    assertThat(actualData).isEqualTo(data);

    // 3. setData
    client.setData().forPath(path, newData);
    latch.await();

    // 5. delete
    client.delete().forPath(path);
  }

  @Test
  public void testCallbackAndWatch() throws Exception {
    String path = "/four";
    byte[] data = {'4'};
    byte[] newData = {'5'};
    CountDownLatch latch = new CountDownLatch(2);

    // Use listener for both callbacks and watches
    client
        .getCuratorListenable()
        .addListener(
            (CuratorFramework c, CuratorEvent event) -> {
              switch (event.getType()) {
                case CREATE:
                  // 2. callback for create
                  System.out.printf("znode '%s' created\n", event.getPath());
                  // 3. getData and register a watch
                  assertThat(client.getData().watched().forPath(path)).isEqualTo(data);
                  // 4. setData
                  client.setData().forPath(path, newData);
                  latch.countDown();
                  break;
                case WATCHED:
                  WatchedEvent we = event.getWatchedEvent();
                  System.out.println("watched event: " + we);
                  if (we.getType() == Watcher.Event.EventType.NodeDataChanged
                      && we.getPath().equals(path)) {
                    // 5. watch triggered
                    System.out.printf("got the event for the triggered watch\n");
                    assertThat(c.getData().forPath(path)).isEqualTo(newData);
                  }
                  latch.countDown();
                  break;
              }
            });

    // 1. create
    client.create().withMode(CreateMode.PERSISTENT).inBackground().forPath(path, data);

    latch.await();

    // 6. delete
    client.delete().forPath(path);
  }
}

分布式锁的测试用例

package com.zew.learn;

import org.apache.curator.framework.CuratorFramework;
import org.apache.curator.framework.CuratorFrameworkFactory;
import org.apache.curator.framework.recipes.locks.InterProcessMutex;
import org.apache.curator.retry.ExponentialBackoffRetry;
import org.apache.curator.utils.CloseableUtils;
import org.junit.Test;

import java.util.concurrent.Callable;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicBoolean;

import static org.junit.Assert.assertTrue;
import static org.junit.Assert.fail;

public class CuratorLockTests {
    private final String connectString = "10.0.4.162:2181";
    private final String lockPath = "/locks";
    private static final int QTY = 5;
    private static final int REPETITIONS = QTY * 10;


    @Test
    public void lockTest() {
        //建立一个QTY个线程的线程池来模拟多个终端并发的场景
        ExecutorService service = Executors.newFixedThreadPool(QTY);
        final FakeLimitedResource resource = new FakeLimitedResource();
        try {
            for (int i = 0; i < QTY; ++i) {
                final int index = i;
                Callable<Void> task = () -> {
                    //创建curator的client
                    CuratorFramework client = CuratorFrameworkFactory.newClient(connectString, new ExponentialBackoffRetry(1000, 3));
                    try {
                        client.start();

                        ExampleClientThatLocks example = new ExampleClientThatLocks(client, lockPath, resource, "Client " + index);
                        for (int j = 0; j < REPETITIONS; ++j) {
                            example.doWork(10, TimeUnit.SECONDS);
                        }
                    } catch (InterruptedException e) {
                        Thread.currentThread().interrupt();
                    } catch (Exception e) {
                        e.printStackTrace();
                        // log or do something
                    } finally {
                        CloseableUtils.closeQuietly(client);
                    }
                    return null;
                };
                service.submit(task);
            }

            service.shutdown();
            service.awaitTermination(10, TimeUnit.MINUTES);
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
        assertTrue(true);
    }

    static class FakeLimitedResource {
        private final AtomicBoolean inUse = new AtomicBoolean(false);
        //模拟有限资源的使用情况,隔了一个随机耗时
        public void use() throws InterruptedException {
            if (!inUse.compareAndSet(false, true)) {
                fail("只有一个客户端可以使用这个锁,跑到这里就说明zk让多个客户端拿到了锁");
                throw new IllegalStateException("只有一个客户端可以使用这个锁");

            }

            try {
                Thread.sleep((long) (3 * Math.random()));
            } finally {
                inUse.set(false);
            }
        }
    }

    /**
     * 负责锁定的类
     */
    static class ExampleClientThatLocks {
        private final InterProcessMutex lock;
        private final FakeLimitedResource resource;
        private final String clientName;

        public ExampleClientThatLocks(CuratorFramework client,
                                      String lockPath,
                                      FakeLimitedResource resource,
                                      String clientName) {
            this.resource = resource;
            this.clientName = clientName;
            //创建锁,虽然client不一样,但lockPath一样就能起到锁定的效果
            lock = new InterProcessMutex(client, lockPath);
        }
         /**
         * 实际锁的核心代码就是下面几句话
         * @param time 时间长度
         * @param unit 时间单位
         * @throws Exception 抛出的异常
         */
        public void doWork(long time, TimeUnit unit) throws Exception {
            //尝试获取锁,如果超过时间还不能获取就抛异常
            if (!lock.acquire(time, unit)) {
                throw new IllegalStateException(clientName + " 无法获取锁");
            }

            try {
                System.out.println(clientName + " 拿到了锁");
                resource.use();
            } finally {
                System.out.println(clientName + " 释放锁");
                lock.release(); // always release the lock in a finally block
            }
        }
    }
}

服务发现测试用例

package org.yao;

import org.apache.curator.framework.CuratorFramework;
import org.apache.curator.framework.CuratorFrameworkFactory;
import org.apache.curator.retry.RetryOneTime;
import org.apache.curator.utils.CloseableUtils;
import org.apache.curator.x.discovery.ServiceDiscovery;
import org.apache.curator.x.discovery.ServiceDiscoveryBuilder;
import org.apache.curator.x.discovery.ServiceInstance;
import org.apache.curator.x.discovery.ServiceProvider;
import org.junit.Test;

import static com.google.common.truth.Truth.assertThat;

public class ServiceDiscoveryTests {
  private String connectString = "localhost:2181";

  /** Shows the basic usage for curator-x-discovery. */
  @Test
  public void testBasics() throws Exception {
    CuratorFramework client = null;
    ServiceDiscovery<String> discovery = null;
    ServiceProvider<String> provider = null;
    String serviceName = "test";
    String basePath = "/services";

    try {
      client = CuratorFrameworkFactory.newClient(connectString, new RetryOneTime(1));
      client.start();

      ServiceInstance<String> instance1 =
          ServiceInstance.<String>builder().payload("plant").name(serviceName).port(10064).build();
      ServiceInstance<String> instance2 =
          ServiceInstance.<String>builder().payload("animal").name(serviceName).port(10065).build();

      System.out.printf("instance1 id: %s\n", instance1.getId());
      System.out.printf("instance2 id: %s\n", instance2.getId());

      discovery =
          ServiceDiscoveryBuilder.builder(String.class)
              .basePath(basePath)
              .client(client)
              .thisInstance(instance1)
              .build();
      discovery.start();
      discovery.registerService(instance2);

      provider = discovery.serviceProviderBuilder().serviceName(serviceName).build();
      provider.start();

      assertThat(provider.getInstance().getId()).isNotEmpty();
      assertThat(provider.getAllInstances()).containsExactly(instance1, instance2);

      client.delete().deletingChildrenIfNeeded().forPath(basePath);
    } finally {
      CloseableUtils.closeQuietly(provider);
      CloseableUtils.closeQuietly(discovery);
      CloseableUtils.closeQuietly(client);
    }
  }
}

参考资料

  1. ZooKeeper实战与源码剖析
  2. curator.apache.org/
  3. github.com/apache/cura…