docker安装kafka集群

每天进步一点点！

1. 准备条件

三台虚拟机：

ip : 192.168.237.170

ip : 192.168.237.171

ip : 192.168.237.172

2. 安装docker

curl -fssl https://get.docker.com | bash -s docker --mirror Aliyun

2.1. 查看版本

docker -v

2.2. 设置开机启动

systemctl enable docker

2.3. 配置阿里云镜像

sudo tee /etc/docker/daemon.json << EOF
{
  "registry-mirrors": ["https://u5tj7kjq.mirror.aliyuncs.com"]
}
EOF

sudo systemctl daemon-reload
sudo systemctl restart docker

3. 安装docker-compose

# 下载最新版本的 Docker Compose 
sudo curl -L "https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose

# 添加执行权限
sudo chmod +x /usr/local/bin/docker-compose

# 创建符号链接
sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose

# 验证安装是否成功
docker-compose --version

4. 查看防火墙

# 查看防火墙状态
sudo systemctl status firewalld

# 关闭防火墙
sudo systemctl stop firewalld

# 禁止开机自启动
sudo systemctl disable firewalld

5. 安装zookeeper

5.1. 第一台机器安装zookeeper

在第一台机器，IP为192.168.237.170，新建zookeeper-compose.yml文件

cat << EOF >> zookeeper-compose.yml
version: 3
services:
  zookeeper:
    network_mode: host
    image: zookeeper:3.6.3
    container_name: zookeeper-01
    environment:
      - TZ=Asia/Shanghai
      - ZOO_MY_ID=1
      - ZOO_SERVERS=server.1=192.168.237.170:2888:3888;2181 server.2=192.168.237.171:2888:3888;2181 server.3=192.168.237.172:2888:3888;2181
    volumes:
      - /opt/zookeeper/data:/data
      - /opt/zookeeper/datalog:/datalog

EOF

启动zookeeper容器

docker-compose -f zookeeper-compose.yml up -d

查看是否成功

docker ps

ok,第一台zookeeper安装成功！

5.2. 第二台机器安装zookeeper

在第二台机器，IP为192.168.237.171，新建zookeeper-compose.yml文件

cat << EOF >> zookeeper-compose.yml
version: "3"
services:
  zookeeper:
    network_mode: host
    image: zookeeper:3.6.3
    container_name: zookeeper-02
    environment:
      - TZ=Asia/Shanghai
      - ZOO_MY_ID=2
      - ZOO_SERVERS=server.1=192.168.237.170:2888:3888;2181 server.2=192.168.237.171:2888:3888;2181 server.3=192.168.237.172:2888:3888;2181
    volumes:
      - /opt/zookeeper/data:/data
      - /opt/zookeeper/datalog:/datalog

EOF

启动并且查看是否成功

# 启动
docker-compose -f zookeeper-compose.yml up -d

# 查看是否成功
docker ps

5.3. 第三台机器安装zookeeper

在第三台机器，IP为192.168.237.172，新建zookeeper-compose.yml文件

cat << EOF >> zookeeper-compose.yml
version: "3"
services:
  zookeeper:
    network_mode: host
    image: zookeeper:3.6.3
    container_name: zookeeper-03
    environment:
      - TZ=Asia/Shanghai
      - ZOO_MY_ID=3
      - ZOO_SERVERS=server.1=192.168.237.170:2888:3888;2181 server.2=192.168.237.171:2888:3888;2181 server.3=192.168.237.172:2888:3888;2181
   volumes:
      - /opt/zookeeper/data:/data
      - /opt/zookeeper/datalog:/datalog 


EOF

启动并且查看是否成功

# 启动
docker-compose -f zookeeper-compose.yml up -d

# 查看是否成功
docker ps

好了，现在zookeeper集群安好了，接下来准备安装Kafka。

6. 安装Kafka

6.1. 第一台机器安装Kafka

在第一台机器，IP为192.168.237.170，新建kafka-compose.yml文件

cat << EOF >> kafka-compose.yml
version: "3"
services:
  kafka:
    network_mode: host
    image: bitnami/kafka:3.5.1
    container_name: kafka-01
    environment:
      - TZ=Asia/Shanghai
      - KAFKA_BROKER_ID=1
      - KAFKA_CFG_LISTENERS=PLAINTEXT://192.168.237.170:9092
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://192.168.237.170:9092
      - ALLOW_PLAINTEXT_LISTENER=yes
      - KAFKA_CFG_ZOOKEEPER_CONNECT=192.168.237.170:2181,192.168.237.171:2181,192.168.237.172:2181
 volumes:
      - /opt/kafka/data:/bitnami/kafka   

EOF

启动成功

docker-compose -f kafka-compose.yml up -d

如果启动失败，遇到报错如下：

#启动报错mkdir: cannot create directory '/bitnami/kafka/config': Permission denied
#解决
sudo chown -R 1001:1001 /opt/kafka/data 

查看是否成功

docker ps -a

6.2. 第二台机器安装Kafka

在第二台机器，IP为192.168.237.171，新建kafka-compose.yml文件

cat << EOF >> kafka-compose.yml
version: "3"
services:
  kafka:
    network_mode: host
    image: bitnami/kafka:3.5.1
    container_name: kafka-02
    environment:
      - TZ=Asia/Shanghai
      - KAFKA_BROKER_ID=2
      - KAFKA_CFG_LISTENERS=PLAINTEXT://192.168.237.171:9092
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://192.168.237.171:9092
      - ALLOW_PLAINTEXT_LISTENER=yes
      - KAFKA_CFG_ZOOKEEPER_CONNECT=192.168.237.170:2181,192.168.237.171:2181,192.168.237.172:2181
  
    volumes:
      - /opt/kafka/data:/bitnami/kafka

EOF

启动

docker-compose -f kafka-compose.yml up -d

#启动报错mkdir: cannot create directory '/bitnami/kafka/config': Permission denied
#解决
sudo chown -R 1001:1001 /opt/kafka/data 

查看是否成功

docker ps -a

6.3. 第三台机器安装Kafka

在第三台机器，IP为192.168.237.172，新建kafka-compose.yml文件

cat << EOF >> kafka-compose.yml
version: "3"
services:
  kafka:
    network_mode: host
    image: bitnami/kafka:3.5.1
    container_name: kafka-03
    environment:
      - TZ=Asia/Shanghai
      - KAFKA_BROKER_ID=3
      - KAFKA_CFG_LISTENERS=PLAINTEXT://192.168.237.172:9092
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://192.168.237.172:9092
      - ALLOW_PLAINTEXT_LISTENER=yes
      - KAFKA_CFG_ZOOKEEPER_CONNECT=192.168.237.170:2181,192.168.237.171:2181,192.168.237.172:2181
   
    volumes:
      - /opt/kafka/data:/bitnami/kafka


EOF

启动

docker-compose -f kafka-compose.yml up -d

#启动报错mkdir: cannot create directory '/bitnami/kafka/config': Permission denied
#解决
sudo chown -R 1001:1001 /opt/kafka/data 

查看是否成功

docker ps -a

7. 安装Kafka工具

在第一台机器安装：

 docker run -dit -p 9093:8080 --name redpanda \
  -e KAFKA_BROKERS=192.168.237.170:9092,192.168.237.171:9092,192.168.237.172:9092 \
  docker.redpanda.com/vectorized/console:latest

# 报错日志
docker: Error response from daemon: driver failed programming external connectivity on endpoint redpanda (b3caf8a35ad393289e550729e4654020cc788379a1a6686310bc05670b35be8e):  (iptables failed: iptables --wait -t nat -A DOCKER -p tcp -d 0/0 --dport 9093 -j DNAT --to-destination 172.17.0.2:8080 ! -i docker0: iptables: No chain/target/match by that name.
 (exit status 1)).

# 清理iptables规则：有时候残留的iptables规则可能导致端口映射失败。您可以尝试清理Docker相关的iptables规则：
sudo iptables -t nat -F
sudo service docker restart

现在状态

重新执行

docker run -dit -p 9093:8080 --name redpanda \
  -e KAFKA_BROKERS=192.168.237.170:9092,192.168.237.171:9092,192.168.237.172:9092 \
  docker.redpanda.com/vectorized/console:latest

启动成功

访问：http://192.168.237.170:9093/

ok