容器化部署kettle集群详细教程

集群规划

kettle版本：9.2

Jdk版本：1.8

主机	角色
master	主节点
node1	从节点1
node1	从节点2

目录结构

[lbs@master kettle]$ tree -L 3
.
├── centos7_openjdk8
│   ├── Dockerfile
│   └── init
│       ├── Centos-7.repo
│       ├── nux-dextop-release-0-5.el7.nux.noarch.rpm
│       └── openlogic-openjdk-8u422-b05-linux-x64.tar.gz
└── kettle9.2
    ├── conf
    │   └── carte-config.xml
    ├── docker-compose.yml
    ├── Dockerfile
    ├── init
    │   ├── kettle.tgz
    │   └── repositories.xml
    └── logs
        └── pdi.log

6 directories, 10 files

制作基础系统镜像

1. 下载资源

   mkdir -p centos7_openjdk8/init && cd centos7_openjdk8
   wget  -P ./init https://mirrors.aliyun.com/repo/Centos-7.repo
   wget -P ./init https://privatefilesbucket-community-edition.s3.us-west-2.amazonaws.com/9.4.0.0-343/ce/client-tools/pdi-ce-9.4.0.0-343.zip
   wget -P ./init https://mirrors.coreix.net/li.nux.ro//nux/dextop/el7/x86_64/nux-dextop-release-0-5.el7.nux.noarch.rpm
   wget -P ./init https://builds.openlogic.com/downloadJDK/openlogic-openjdk/8u422-b05/openlogic-openjdk-8u422-b05-linux-x64.tar.gz
   

2. 编写Dockerfile文件

   sudo tee Dockerfile <<'EOF'
   # Using Centos for base image
   FROM centos:centos7
   
   # author label
   LABEL maintainer="liboshuai"
   
   # copy jdk11 to /usr/local, and decompression
   COPY ./init/* /usr/local
   
   # install timezone gcc
   RUN cd /usr/local \
   && mv Centos-7.repo /etc/yum.repos.d \
   && tar -zxvf  openlogic-openjdk-8u412-b08-linux-x64.tar.gz \
   && mv openlogic-openjdk-8u412-b08-linux-x64 openjdk8 \
   && rm -rf  openlogic-openjdk-8u412-b08-linux-x64.tar.gz \
   && ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime \
   && cd /etc/yum.repos.d \
   && sed -i 's/mirrorlist/#mirrorlist/g' /etc/yum.repos.d/CentOS-* \
   && sed -i 's|#baseurl=http://mirror.centos.org|baseurl=http://vault.centos.org|g' /etc/yum.repos.d/CentOS-*  \
   && yum clean all \
   && yum makecache \
   && yum update -y \
   && yum install -y vim unzip zip epel-release \
   && cd /usr/local \
   && rpm -Uvh nux-dextop-release-0-5.el7.nux.noarch.rpm \
   && rm -rf nux-dextop-release-0-5.el7.nux.noarch.rpm \
   && yum install webkitgtk -y \
   && yum clean all
   
   # cd /opt
   WORKDIR /usr/local
   
   # set jdk8 env
   ENV JAVA_HOME=/usr/local/openjdk8
   ENV CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
   ENV PATH=$JAVA_HOME/bin:$PATH
   
   # exec java -version
   CMD ["java","-version"]
   EOF
   

3. 执行镜像制作命令

   docker build -t centos7_openjdk8:v1.0 .
   

制作kettle镜像

1. 下载资源

   mkdir -p kettle9.2/init && cd kettle9.2
   wget -P ./init https://privatefilesbucket-community-edition.s3.us-west-2.amazonaws.com/9.2.0.0-290/ce/client-tools/pdi-ce-9.2.0.0-290.zip
   unzip ./init/pdi-ce-9.2.0.0-290.zip
   wget -P ./data-integration/lib https://repo1.maven.org/maven2/com/mysql/mysql-connector-j/8.0.33/mysql-connector-j-8.0.33.jar  
   

2. 资源配置

   # 1. 修改./data-integration/spoon.sh 中jvm参数添加-Dfile.encoding=UTF-8
   
   # 2. 重命名data-integration目录为kettle，并打包为kettle.tar.gz
   mv data-integration kettle
   tar -zcvf kettle.tar.gz kettle
   

3. 编写Dockerfile文件

   sudo tee kettle9.2/Dockerfile <<'EOF'
   FROM centos7_openjdk8:v1.0
   
   LABEL maintainer="liboshuai"
   
   ADD kettle.tar.gz /usr/local
   
   RUN mkdir -p ~/.kettle
   
   WORKDIR /usr/local/kettle
   VOLUME /root/.kettle
   VOLUME /usr/local/kettle/conf
   VOLUME /usr/local/kettle/logs
   
   ENTRYPOINT [ "./carte.sh"]
   EOF
   

4. 编写docker-compose.yml文件

   sudo tee kettle9.2/docker-compose.yml <<'EOF'
   version: '3'
   
   services:
     kettle:
       build:
         context: .
         dockerfile: Dockerfile
       container_name: kettle
       network_mode: host
       volumes:
         - ./repository:/root/.kettle
         - ./conf:/usr/local/kettle/conf
         - ./logs:/usr/local/kettle/logs
       environment:
         - PENTAHO_DI_JAVA_OPTIONS=-Xms8g -Xmx8g
       command: ["./conf/carte-config.xml"]
       extra_hosts:
         - "master:10.0.0.87"
         - "node1:10.0.0.81"
         - "node2:10.0.0.82"
   EOF
   

集群部署

下面的步骤均建立在三个节点master、node1、node2都完成了上面的步骤。

master节点

登录master主机，下面步骤均在master主机执行

1. 创建目录

   cd kettle9.2 && mkdir -p {conf,logs,repository}
   

2. 编写carte-config.xml配置文件

   sudo tee conf/carte-config.xml <<'EOF'
   <slave_config>
     <slaveserver>
       <name>master1</name>
       <!--master节点宿主机IP或者域名-->
       <hostname>master</hostname>
       <port>29000</port>
       <username>kettle</username>
       <password>Kettle@2024</password>
       <master>Y</master>
     </slaveserver>
   
     <repository>
       <!-- 与spoon导出的repositories.xml中的内容对应 -->
       <name>mysql</name>
       <username>admin</username>
       <password>admin</password>
     </repository>
     
     <!-- 最大日志行数 -->
     <max_log_lines>10000</max_log_lines>
     <!-- 日志行最大存活时间 -->
     <max_log_timeout_minutes>2880</max_log_timeout_minutes> 
     <!-- 自动清除老日志的时间 -->
     <object_timeout_minutes>2880</object_timeout_minutes>
     
   </slave_config>
   EOF
   

3. 将spoon导出的资源库配置文件repositories.xml放入repository目录（也可以在windows的用户home目录）

4. 构建镜像并启动容器

   docker-compose up -d --build
   

node1节点

登录node1主机，下面步骤均在node1主机执行

1. 创建目录

   cd kettle9.2 && mkdir -p {conf,logs}
   

2. 编写carte-config.xml配置文件

   sudo tee conf/carte-config.xml <<'EOF'
   <slave_config>
     <masters>
       <slaveserver>
         <!-- 与master节点的carte-config.xml内容相应 -->
         <name>master1</name>
         <!--master节点宿主机IP或者域名-->
         <hostname>master</hostname>
         <port>29000</port>
         <username>kettle</username>
         <password>Kettle@2024</password>
         <master>Y</master>
       </slaveserver>
     </masters>
   
     <report_to_masters>Y</report_to_masters>
   
     <slaveserver>
       <!-- 节点名称保持唯一 -->
       <name>node1</name>
       <!--node1节点宿主机IP或者域名-->
       <hostname>node1</hostname>
       <port>29000</port>
       <username>kettle</username>
       <password>Kettle@2024</password>
       <master>N</master>
     </slaveserver>
   
     <repository>
       <!-- 与spoon导出的repositories.xml中的内容对应 -->
       <name>mysql</name>
       <username>admin</username>
       <password>admin</password>
     </repository>
   
     <!-- 最大日志行数 -->
     <max_log_lines>10000</max_log_lines>
     <!-- 日志行最大存活时间 -->
     <max_log_timeout_minutes>2880</max_log_timeout_minutes>
     <!-- 自动清除老日志的时间 -->
     <object_timeout_minutes>2880</object_timeout_minutes>
   
   </slave_config>
   EOF
   

3. 构建镜像并启动容器

   docker-compose up -d --build
   

node2节点

登录node2主机，下面步骤均在node2主机执行

1. 创建目录

   cd kettle9.2 && mkdir -p {conf,logs}
   

2. 编写carte-config.xml配置文件

   sudo tee conf/carte-config.xml <<'EOF'
   <slave_config>
     <masters>
       <slaveserver>
         <!-- 与master节点的carte-config.xml内容相应 -->
         <name>master2</name>
         <!--master节点宿主机IP或者域名-->
         <hostname>master</hostname>
         <port>29000</port>
         <username>kettle</username>
         <password>Kettle@2024</password>
         <master>Y</master>
       </slaveserver>
     </masters>
   
     <report_to_masters>Y</report_to_masters>
   
     <slaveserver>
       <!-- 节点名称保持唯一 -->
       <name>node2</name>
       <!--node2节点宿主机IP或者域名-->
       <hostname>node2</hostname>
       <port>29000</port>
       <username>kettle</username>
       <password>Kettle@2024</password>
       <master>N</master>
     </slaveserver>
   
     <repository>
       <!-- 与spoon导出的repositories.xml中的内容对应 -->
       <name>mysql</name>
       <username>admin</username>
       <password>admin</password>
     </repository>
   
     <!-- 最大日志行数 -->
     <max_log_lines>10000</max_log_lines>
     <!-- 日志行最大存活时间 -->
     <max_log_timeout_minutes>2880</max_log_timeout_minutes>
     <!-- 自动清除老日志的时间 -->
     <object_timeout_minutes>2880</object_timeout_minutes>
   
   </slave_config>
   EOF
   

3. 构建镜像并启动容器

   docker-compose up -d --build