基于docker安装flink

321 阅读2分钟

首先搭建standalone集群

1、准备配置文件

1.1、复制默认的 flink-conf.yaml 配置文件,并修改以下配置,然后重命名为 jobmanager-conf.yaml 并放在 /home/debo/dockerData/flink 目录下

jobmanager.rpc.address: flink-jobmanager
jobmanager.bind-host: 0.0.0.0
#taskmanager.bind-host: localhost
#taskmanager.host: localhost
rest.address: 0.0.0.0
rest.bind-address: 0.0.0.0

1.2、复制默认的 flink-conf.yaml 配置文件,并修改以下配置,然后重命名为 taskmanager1-conf.yaml 并放在 /home/debo/dockerData/flink 目录下

jobmanager.rpc.address: flink-jobmanager
jobmanager.bind-host: 0.0.0.0
taskmanager.bind-host: 0.0.0.0
taskmanager.host: flink-taskmanager1
rest.address: 0.0.0.0
rest.bind-address: 0.0.0.0

1.3、复制默认的 flink-conf.yaml 配置文件,并修改以下配置,然后重命名为 taskmanager2-conf.yaml 并放在 /home/debo/dockerData/flink 目录下

jobmanager.rpc.address: flink-jobmanager
jobmanager.bind-host: 0.0.0.0
taskmanager.bind-host: 0.0.0.0
taskmanager.host: flink-taskmanager2
rest.address: 0.0.0.0
rest.bind-address: 0.0.0.0

1.4、复制默认的 flink-conf.yaml 配置文件,并修改以下配置,然后重命名为 taskmanager3-conf.yaml 并放在 /home/debo/dockerData/flink 目录下

jobmanager.rpc.address: flink-jobmanager
jobmanager.bind-host: 0.0.0.0
taskmanager.bind-host: 0.0.0.0
taskmanager.host: flink-taskmanager3
rest.address: 0.0.0.0
rest.bind-address: 0.0.0.0

2、执行 docker 命令

依次执行以下命令创建 flink 集群

docker run \
-d -p 8081:8081 \
--name flink-jobmanager \
--network flink-network \
--mount type=bind,src=/home/debo/dockerData/flink/jobmanager-conf.yaml,target=/opt/flink/conf/flink-conf.yaml flink:1.16.3-java8 jobmanager

docker run -d --name flink-taskmanager1 --network flink-network --mount type=bind,src=/home/debo/dockerData/flink/taskmanager1-conf.yaml,target=/opt/flink/conf/flink-conf.yaml flink:1.16.3-java8 taskmanager
docker run -d --name flink-taskmanager2 --network flink-network --mount type=bind,src=/home/debo/dockerData/flink/taskmanager2-conf.yaml,target=/opt/flink/conf/flink-conf.yaml flink:1.16.3-java8 taskmanager
docker run -d --name flink-taskmanager3 --network flink-network --mount type=bind,src=/home/debo/dockerData/flink/taskmanager3-conf.yaml,target=/opt/flink/conf/flink-conf.yaml flink:1.16.3-java8 taskmanager

部署单台flinkstandalone(建议)

编写docker-compose.yml

version: "2.1" 
services:
  jobmanager:
    image: flink:latest
    expose:
      - "6123"
    ports:
      - "8081:8081"
    command: jobmanager
    environment:
      - JOB_MANAGER_RPC_ADDRESS=jobmanager

  taskmanager:
    image: flink:latest
    expose:
      - "6121"
      - "6122"
    depends_on:
      - jobmanager
    command: taskmanager
    links:
      - "jobmanager:jobmanager"
    environment:
      - JOB_MANAGER_RPC_ADDRESS=jobmanager

在上面的配置中,我们定义了两个服务:jobmanager 和 taskmanager。jobmanager 服务负责管理作业,taskmanager 服务负责执行作业。我们将Flink的Web UI端口映射到宿主机的8081端口,以便查看作业状态和日志。

启动Flink集群

现在,我们可以使用Docker Compose来启动Flink集群。在终端中执行以下命令:

docker-compose up

http://localhost:8081/#/overview

单台方式2

拉取

docker pull flink:latest

创建

docker run -d --name flink_container -p 8081:8081 flink:latest

启动

docker start flink_container