Kafka集群

集群搭载

这里只在一台阿里云上部署集群，但配置是差不多的。只要保证这些broker连接的zookeeper是同一个即可

• 将kafka的配置文件复制两份
• 修改端口号

 cp config/server.properties config/server‐1.properties
 cp config/server.properties config/server‐2.properties

修改配置文件,改id端口号和日志目录,其他不变

broker.id = 1
listeners=PLAINTEXT://***:9093
log.dir=/usr/local/data/kafka‐logs‐1

broker.id = 2
listeners=PLAINTEXT://***:9094
log.dir=/usr/local/data/kafka‐logs‐2

在kafka 的目录下将三个broker启动

 bin/kafka‐server‐start.sh ‐daemon config/server.properties
 bin/kafka‐server‐start.sh ‐daemon config/server‐1.properties
 bin/kafka‐server‐start.sh ‐daemon config/server‐2.properties

测试

通过jps查看

jps -l

Application是阿里云进程,QuorymPeerMain是zookeeper进程 在zk的目录下,通过zookeeper的客户端查看注册brokerId,可以看到3个都注册成功

bin/zkCli.sh
ls /brokers/ids

之前也有提到,分区的副本数目不得超过broker的数目.有了集群,才可以创建多个副本.
分区总数为分区数 * 副本数.接下来创建分区数为7.副本数为2的主题server-cluster.
注意不要用下划线,不然会收到警告.因为下划线用于kafka内部主题的命名.

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 2 --partitions 7 --topic server-cluster

然后查看主题的详情,可以看到分区在各个broker上的分布, ISR是保持同步的副本集

bin/kafka-topics.sh --zookeeper localhost:2181 --describe --topic server-cluster

将Id为2的broker关闭(Ctrl + C) ,然后再次查看 可以看到,由于id为2的broker被关闭,ISR中已将其剔除.而且原来leader为2的分区,重新选举1为leader

集群消费

在开始之前,需要在阿里云中将集群的端口开放
可以通过telnet来大致判断一下开放的几个端口是否能通,如下就是通的

测试

直接用之前的代码测试一下就可以,不过配置要修改一下,将集群的broker都加入,并且改一下主题

    public static final String brokerList = "***:9092,***:9093,***:9094";
    public static final String topic = "server-cluster";

随便输入几条消息,生产者效果如下,之前有加入拦截器,所以消费者收到的消息会有前缀"th-dhu-",不了解的看之前的文章 消费者效果如下.由于生产者既没有指定分区,也没有提供key,相当于对有效分区进行轮询.具体在以前的文章有介绍

多消费者测试

依照之前多线程篇的文章稍作改动,进行测试

生产者

public class MultiProducer {
    /** 有多个可以用逗号隔开 */
    public static final String brokerList = "***:9092,***:9093,***:9094";
    public static final String topic = "server-cluster";

    public static Properties initConfig() {
        Properties properties = new Properties();
        properties.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, brokerList);
        properties.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
        properties.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
        properties.put(ProducerConfig.CLIENT_ID_CONFIG, "0");
        properties.put(ProducerConfig.ACKS_CONFIG, "1");
        properties.put(ProducerConfig.RETRIES_CONFIG, 10); //重试次数
        return properties;
    }

    public static void main(String[] args) {
        Properties properties = MultiProducer.initConfig();
        KafkaProducer<String, String> producer = new KafkaProducer<>(properties);
        Scanner scanner = new Scanner(System.in);
        for (int i = 0; i < 5; i++) {
            try {
                String key = scanner.nextLine();
                String value = scanner.nextLine();
                ProducerRecord<String, String> producerRecord = new ProducerRecord<>(topic, key, value);
                producer.send(producerRecord, new Callback() {
                    @Override
                    public void onCompletion(RecordMetadata metadata, Exception exception) {
                        if (exception != null) {
                            //TODO
                            exception.printStackTrace();
                        } else {
                            System.out.println(
                                    metadata.topic() + "-" + metadata.partition() + "-" + metadata.offset() + "发送成功！");

                        }
                    }
                });
            } catch (Exception e) {
                e.printStackTrace();
            }
        }
        producer.close();
    }
}

消费者

import java.util.List;
import java.util.Properties;

import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.common.PartitionInfo;
import org.apache.kafka.common.serialization.StringDeserializer;

public class MultiConsumerDemo1 {
    /** 有多个可以用逗号隔开 */
    /** 有多个可以用逗号隔开 */
    public static final String brokerList = "***:9092,***:9093,***:9094";
    public static final String topic = "server-cluster";
    /** 消费组的名称 */
    public static final String groupId = "kafka-learner-集群消费";

    public static Properties initConfig() {
        Properties properties = new Properties();
        properties.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, brokerList);
        properties.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
        properties.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
        properties.put(ConsumerConfig.GROUP_ID_CONFIG, groupId); //消费组
        properties.put(ConsumerConfig.CLIENT_ID_CONFIG, "0");
        return properties;
    }

    public static void main(String[] args) {
        Properties properties = initConfig();
        KafkaConsumer<String, String> consumer = new KafkaConsumer<>(MultiConsumerDemo1.initConfig());
        List<PartitionInfo> partitionInfos = consumer.partitionsFor(topic);
        final int consumerThreadNum = partitionInfos.size() / 2; //注意不要超过分区的数目, 这里设置成[1，分区数]都可以
        consumer.close();
        for (int i = 0; i < consumerThreadNum; i++) {
            new KafkaConsumerThread1(properties, topic).start();
        }
    }
}

import java.time.Duration;
import java.util.Collections;
import java.util.HashSet;
import java.util.Properties;
import java.util.Set;

import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.common.TopicPartition;

public class KafkaConsumerThread1 extends Thread {
    private KafkaConsumer<String, String> consumer;

    public KafkaConsumerThread1(Properties properties, String topic) {
        this.consumer = new KafkaConsumer<String, String>(properties);
        this.consumer.subscribe(Collections.singletonList(topic));
    }

    @Override
    public void run() {
        try {
            //这里要先poll到有效值才能获取到这个消费者拥有的分区
            Set<TopicPartition> assignment = new HashSet<>();
            while (assignment.isEmpty()) {
                ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
                for (ConsumerRecord<String, String> record : records) {
                    System.out.println(
                            "分区是：" + record.partition() + "分区的key是：" + record.key() + "分区的值是：" + record.value());
                }
                assignment = consumer.assignment();//获取订阅的分区
            }
            String name = "consumer_" + Thread.currentThread().getName();
            StringBuilder partitions = new StringBuilder();
            consumer.assignment().forEach(e -> partitions.append(e.partition()).append("区 "));
            System.out.println(name + "被分到的分区有：" + partitions.toString());
            while (true) {
                ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
                for (ConsumerRecord<String, String> record : records) {
                    System.out.println(
                            "分区是：" + record.partition() + "分区的key是：" + record.key() + "分区的值是：" + record.value());
                }
            }
        } catch (Exception e) {
            e.printStackTrace();
        } finally {
            consumer.close();
        }
    }
}

运行结果

生产者 消费组

架构图

根据消费者分配的情况,画个图,大概是这样 也可以看出来,Kafka并非读写分离. Producer只会往leader里写入,Consumer也只消费Leader里的数据.