Docker插件部署继续，今儿是数据同步框架：canal
一：配置mysql
1：修改my.cnf配置文件，添加如下内容，开启binlog
[mysqld]
# binlog
log-bin=mysql-bin # 开启 binlog
binlog-format=ROW # 选择 ROW 模式
server_id=1 # 配置 MySQL replaction 需要定义，不要和 canal 的 slaveId 重复
修改完成，重启mysql。
打开数据库，运行sql查看是否开始binlog
show variables like 'log_%'
如下图所示：
我这里数据库使用的事MariaDB，具百度上所说的，mysql8链接的时候密码可能会出问题，具体我没遇到过，后期如果遇到会补充。
2：创建canal用户
# 创建用户
CREATE USER canal IDENTIFIED BY 'canal’;
# 授权用户
GRANT ALL PRIVILEGES ON *.* TO 'canal'@'%' ;
# 刷新权限
FLUSH PRIVILEGES;
# 查询用户是否创建成功
SELECT Host, User FROM mysql.user WHERE User='canal';
这里注意一下，我的mysql不是使用docker安装的，我这里只记录数据库如何开启bin-log，如果你的mysql也想使用docker安装，这个后期会写到。
二：docker-compose搭建canal-server
1：先创建一个canal容器，将其配置文件拷贝出来
docker run -p 11111:11111 --name canal-server -d canal/canal-server:v1.1.5
2：拷贝配置文件
docker cp canal-server:/home/admin/canal-server/conf /opt/docker/canal-server
3：删除容器
docker stop canal-server
docker rm canal-server
4：docker-compose.yml
version: "3.8"
services:
canal-server:
image: canal/canal-server:v1.1.5
container_name: canal-server
# restart: always
ports:
- "11111:11111"
volumes:
- /opt/docker/canal-server/conf/:/home/admin/canal-server/conf
- /opt/docker/canal-server/logs/:/home/admin/canal-server/logs/
privileged: true
environment:
- canal.auto.scan=true
- canal.destinations=canalServer # canal实例名称，根据自己喜好命名
- canal.instance.mysql.slaveId=1085 # canal作为模仿从站的ID，不可与主站重复
- canal.instance.master.address=1.15.157.156:3306 #mysql的连接地址
- canal.instance.dbUsername=canal # mysql的用户
- canal.instance.dbPassword=canal # mysql的密码
- canal.instance.connectionCharset=UTF-8
- canal.instance.tsdb.enable=true
- canal.instance.gtidon=false
- canal.instance.parser.parallelThreadSize=16
# - canal.instance.filter.regex=dbname.tablename # 此处为监听的正则表达式过滤，具体参考官网配置，多个用英文逗号隔开",",例如：dbname.tablename1,dbname.tablename2
networks:
- canal-network
networks: # 加入公共网络（没有这个网络，你需要提前创建）
canal-network:
external: true
启动容器：
docker compose -d up
如果你修改了docker-compose.yml对应的参数之后，使用如下命令即可重启而不用删除已存在的容器：
docker compose up --force-recreate -d
5：查看容器是否启动成功
容器启动成功之后，可以查看容器的日志看容器是否启动成功：
DOCKER_DEPLOY_TYPE=VM
==> INIT /alidata/init/02init-sshd.sh
==> EXIT CODE: 0
==> INIT /alidata/init/fix-hosts.py
==> EXIT CODE: 0
==> INIT DEFAULT
Starting sshd: [ OK ]
Starting crond: [ OK ]
==> INIT DONE
==> RUN /home/admin/app.sh
==> START ...
start canal ...
start canal successful
==> START SUCCESSFUL ...
看到上方这串代码，说明canal启动成功！
容器启动成功之后，可以查看对应的日志文件，查看数据库链接是否成功，上文中我们绑定的数据卷/opt/docker/canal/logs/example/example.log
[main] INFO c.a.otter.canal.instance.spring.CanalInstanceWithSpring - start CannalInstance for 1-example
[main] WARN c.a.o.canal.parse.inbound.mysql.dbsync.LogEventConvert - --> init table filter : ^.*..*$
[main] WARN c.a.o.canal.parse.inbound.mysql.dbsync.LogEventConvert - --> init table black filter : ^mysql.slave_.*$
[main] INFO c.a.otter.canal.instance.core.AbstractCanalInstance - start successful....
[destination = example , address = /1.15.157.156:3306 , EventParser] WARN c.a.o.c.p.inbound.mysql.rds.RdsBinlogEventParserProxy - ---> begin to find start position,
it will be long time for reset or first position
[destination = example , address = /1.15.157.156:3306 , EventParser] WARN c.a.o.c.p.inbound.mysql.rds.RdsBinlogEventParserProxy - prepare to find start position just show master status
[destination = example , address = /1.15.157.156:3306 , EventParser] WARN c.a.o.c.p.inbound.mysql.rds.RdsBinlogEventParserProxy - --->
find start position successfully, EntryPosition[included=false,journalName=mysql-bin.000001,position=146986,serverId=1,gtid=,timestamp=1734849915000] cost : 1028ms , the next step is binlog dump
说明我们数据库也链接成功。
三：docker-compose搭建canal-adapter
1：先创建一个canal容器，将其配置文件拷贝出来
docker run -d --name canal-adapter slpcat/canal-adapter:v1.1.5
2：拷贝配置文件
docker cp canal-adapter:/opt/canal-adapter/conf /opt/docker/canal-adapter
docker cp canal-adapter:/opt/canal-adapter/logs /opt/docker/canal-adapter
3：删除容器
docker stop canal-adapter
docker rm canal-adapter
4：配置canal将数据同步到elasticsearch
（1）：application.yml
文件位置如下图所示：
内容如下：
server:
port: 8081
spring:
jackson:
date-format: yyyy-MM-dd HH:mm:ss
time-zone: GMT+8
default-property-inclusion: non_null
canal.conf:
# ==================================================================================
mode: tcp #tcp kafka rocketMQ rabbitMQ # 我这里使用tcp模式
# ==================================================================================
flatMessage: true
zookeeperHosts:
syncBatchSize: 1000
retries: 0
timeout:
accessKey:
secretKey:
consumerProperties:
# canal tcp consumer
canal.tcp.server.host: 39.99.144.212:11111 # 修改的部分
canal.tcp.zookeeper.hosts:
canal.tcp.batch.size: 500
canal.tcp.username:
canal.tcp.password:
# kafka consumer
kafka.bootstrap.servers: 127.0.0.1:9092
kafka.enable.auto.commit: false
kafka.auto.commit.interval.ms: 1000
kafka.auto.offset.reset: latest
kafka.request.timeout.ms: 40000
kafka.session.timeout.ms: 30000
kafka.isolation.level: read_committed
kafka.max.poll.records: 1000
# rocketMQ consumer
rocketmq.namespace:
rocketmq.namesrv.addr: 127.0.0.1:9876
rocketmq.batch.size: 1000
rocketmq.enable.message.trace: false
rocketmq.customized.trace.topic:
rocketmq.access.channel:
rocketmq.subscribe.filter:
# rabbitMQ consumer
rabbitmq.host:
rabbitmq.virtual.host:
rabbitmq.username:
rabbitmq.password:
rabbitmq.resource.ownerId:
# ==================================================================================
# 数据库配置
srcDataSources:
defaultDS:
url: jdbc:mysql://1.15.157.156:3306/vueapi?useUnicode=true
username: canal
password: canal
canalAdapters:
- instance: canalServer # canal instance Name or mq topic name # 这里需要对应canal-server中的配置
# ==================================================================================
groups:
- groupId: g1
outerAdapters:
- name: logger
# - name: rdb
# key: mysql1
# properties:
# jdbc.driverClassName: com.mysql.jdbc.Driver
# jdbc.url: jdbc:mysql://127.0.0.1:3306/mytest2?useUnicode=true
# jdbc.username: root
# jdbc.password: 121212
# - name: rdb
# key: oracle1
# properties:
# jdbc.driverClassName: oracle.jdbc.OracleDriver
# jdbc.url: jdbc:oracle:thin:@localhost:49161:XE
# jdbc.username: mytest
# jdbc.password: m121212
# - name: rdb
# key: postgres1
# properties:
# jdbc.driverClassName: org.postgresql.Driver
# jdbc.url: jdbc:postgresql://localhost:5432/postgres
# jdbc.username: postgres
# jdbc.password: 121212
# threads: 1
# commitSize: 3000
# - name: hbase
# properties:
# hbase.zookeeper.quorum: 127.0.0.1
# hbase.zookeeper.property.clientPort: 2181
# zookeeper.znode.parent: /hbase
# ==================================================================================
# 这里是我配置的
- name: es7
hosts: 1.15.157.156:9200 # 127.0.0.1:9200 for rest mode
properties:
mode: rest # or transport
security.auth: elastic:camellia # only used for rest mode
cluster.name: elasticsearch
# ==================================================================================
# - name: kudu
# key: kudu
# properties:
# kudu.master.address: 127.0.0.1 # ',' split multi address
我配置的部分 都有注释，参照即可
（2）：mytest_user.yml
配置宿主机映射的容器内conf目录下es7目录下yml文件canal-adapter会扫描“conf/es7/”所有以.yml为后缀名的配置文件此处仅保留一个以“mytest_user.yml”命名的文件，具体内容如下：
dataSourceKey: defaultDS
destination: canalServer
groupId: g1
esMapping:
_index: mytest_user # ES索引
_type: _doc # ES类型为文档
_id: _id
# upsert: true
# pk: id
sql: "select * from user" # 源数据表
# objFields:
# _labels: array:;
# etlCondition: "where a.c_time>={}"
commitBatch: 3000
（3）：docker-compose.yml
version: "3.8"
services:
canal-adapter:
image: slpcat/canal-adapter:v1.1.5 # 所使用的的镜像
container_name: canal-adapter # 容器名
restart: always
volumes: # 数据卷挂载
- /opt/docker/canal-adapter/conf:/opt/canal-adapter/conf
- /opt/docker/canal-adapter/logs:/opt/canal-adapter/logs
privileged: true
networks:
- canal-network
networks: # 加入公共网络（没有这个网络，你需要提前创建）
canal-network:
external: true
启动容器：
docker compose -d up
如果你修改了docker-compose.yml对应的参数之后，使用如下命令即可重启而不用删除已存在的容器：
docker compose up --force-recreate -d
（4）：运行成功如下所示：
[Thread-3] INFO c.a.otter.canal.adapter.launcher.loader.AdapterProcessor - =============> Start to connect destination: canalServer <=============
[main] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterService - ## the canal client adapters are running now ......
[main] INFO org.apache.coyote.http11.Http11NioProtocol - Starting ProtocolHandler ["http-nio-8081"]
[main] INFO org.apache.tomcat.util.net.NioSelectorPool - Using a shared selector for servlet write/read
[main] INFO o.s.boot.web.embedded.tomcat.TomcatWebServer - Tomcat started on port(s): 8081 (http) with context path ''
[main] INFO c.a.otter.canal.adapter.launcher.CanalAdapterApplication - Started CanalAdapterApplication in 3.507 seconds (JVM running for 4.137)
[Thread-3] INFO c.a.otter.canal.adapter.launcher.loader.AdapterProcessor - =============> Subscribe destination: canalServer succeed <=============
（5）：报错：
报错：Outer adapter sync failed! Error sync but ACK!
数据消息能传到adapter，但是同步不到elasticsearch。报错：Outer adapter sync failed! Error sync but ACK!
报错：Caused by: java.net.UnknownHostException: http: Name or service not known
这个问题到目前为止我还没有解决
使用canal将数据同步到elasticsearch就失败了，我也不知道为啥~
5：配置canal将数据同步到mysql
（1）：application.yml
文件位置如下图所示：
文件内容如下所示：
server:
port: 8081
spring:
jackson:
date-format: yyyy-MM-dd HH:mm:ss
time-zone: GMT+8
default-property-inclusion: non_null
canal.conf:
# ==================================================================================
mode: tcp #tcp kafka rocketMQ rabbitMQ # 我这里使用tcp模式
# ==================================================================================
flatMessage: true
zookeeperHosts:
syncBatchSize: 1000
retries: 0
timeout:
accessKey:
secretKey:
consumerProperties:
# canal tcp consumer
canal.tcp.server.host: 39.99.144.212:11111 # 修改的部分
canal.tcp.zookeeper.hosts:
canal.tcp.batch.size: 500
canal.tcp.username:
canal.tcp.password:
# kafka consumer
kafka.bootstrap.servers: 127.0.0.1:9092
kafka.enable.auto.commit: false
kafka.auto.commit.interval.ms: 1000
kafka.auto.offset.reset: latest
kafka.request.timeout.ms: 40000
kafka.session.timeout.ms: 30000
kafka.isolation.level: read_committed
kafka.max.poll.records: 1000
# rocketMQ consumer
rocketmq.namespace:
rocketmq.namesrv.addr: 127.0.0.1:9876
rocketmq.batch.size: 1000
rocketmq.enable.message.trace: false
rocketmq.customized.trace.topic:
rocketmq.access.channel:
rocketmq.subscribe.filter:
# rabbitMQ consumer
rabbitmq.host:
rabbitmq.virtual.host:
rabbitmq.username:
rabbitmq.password:
rabbitmq.resource.ownerId:
# ==================================================================================
# 数据库配置
srcDataSources:
defaultDS:
url: jdbc:mysql://1.15.157.156:3306/vueapi?useUnicode=true
username: canal
password: canal
canalAdapters:
- instance: canalServer # canal instance Name or mq topic name # 这里需要对应canal-server中的配置
# ==================================================================================
groups:
- groupId: g1
outerAdapters:
- name: logger
# ==================================================================================
# 这里是我配置的
- name: es7
hosts: http://1.15.157.156:9200 # 127.0.0.1:9200 for rest mode,如果连接es使用的是rest方式，那么hosts中的ip前要添加http://
properties:
mode: rest # rest or transport
security.auth: elastic:camellia # only used for rest mode
cluster.name: elasticsearch
# ==================================================================================
# - name: kudu
# key: kudu
# properties:
# kudu.master.address: 127.0.0.1 # ',' split multi address
# ==================================================================================
# 这里是我配置的
# mysql同步数据库适配器
# 指定为rdb类型同步
- name: rdb
key: mysql1 # 指定adapter的唯一key, 与表映射配置中outerAdapterKey对应
properties:
jdbc.driverClassName: com.mysql.jdbc.Driver # 数据库驱动
jdbc.url: jdbc:mysql://39.99.144.212:3306/vueapi?useUnicode=true # 数据库连接地址
jdbc.username: root
jdbc.password: xxxxxxxx
# ==================================================================================
（2）：mytest_user.yml
文件位置如下图所示：
文件内容如下图所示：
dataSourceKey: defaultDS
destination: canalServer
groupId: g1
outerAdapterKey: mysql1 # 对应application.yml中配置的key
concurrent: true
dbMapping:
database: vueapi
table: user
targetTable: vueapi.user
targetPk:
id: id
mapAll: true # 是否整表映射, 要求源表和目标表字段名一模一样 (如果targetColumns也配置了映射,则以targetColumns配置为准)
# targetColumns:
# id:
# name:
# role_id:
# c_time:
# test1:
etlCondition: "where c_time>={}"
commitBatch: 3000 # 批量提交的大小
（3）：docker-compose.yml
version: "3.8"
services:
canal-adapter:
image: slpcat/canal-adapter:v1.1.5 # 所使用的的镜像
container_name: canal-adapter # 容器名
restart: always
volumes: # 数据卷挂载
- /opt/docker/canal-adapter/conf:/opt/canal-adapter/conf
- /opt/docker/canal-adapter/logs:/opt/canal-adapter/logs
privileged: true
networks:
- canal-network
networks: # 加入公共网络（没有这个网络，你需要提前创建）
canal-network:
external: true
启动容器：
docker compose -d up
如果你修改了docker-compose.yml对应的参数之后，使用如下命令即可重启而不用删除已存在的容器：
docker compose up --force-recreate -d
（4）运行成功如下所示：
[main] INFO com.alibaba.druid.pool.DruidDataSource - {dataSource-2} inited
[main] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterLoader - Load canal adapter: rdb succeed
[main] INFO c.alibaba.otter.canal.connector.core.spi.ExtensionLoader - extension classpath dir: /opt/canal-adapter/plugin
[main] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterLoader - Start adapter for canal-client mq topic: canalServer-g1 succeed
[Thread-3] INFO c.a.otter.canal.adapter.launcher.loader.AdapterProcessor - =============> Start to connect destination: canalServer <=============
[main] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterService - ## the canal client adapters are running now ......
[main] INFO org.apache.coyote.http11.Http11NioProtocol - Starting ProtocolHandler ["http-nio-8081"]
[main] INFO org.apache.tomcat.util.net.NioSelectorPool - Using a shared selector for servlet write/read
[main] INFO o.s.boot.web.embedded.tomcat.TomcatWebServer - Tomcat started on port(s): 8081 (http) with context path ''
[main] INFO c.a.otter.canal.adapter.launcher.CanalAdapterApplication - Started CanalAdapterApplication in 3.82 seconds (JVM running for 4.805)
[Thread-3] INFO c.a.otter.canal.adapter.launcher.loader.AdapterProcessor - =============> Subscribe destination: canalServer succeed <=============
上图中顺便测试了一下，使用canal将数据同步至另一个数据库是没有问题的。
四：使用docker-compose搭建canal-server和canal-adapter
这个就是将两个docker-compose文件整合一下。
Docker-compose.yml
version: "3.8"
services:
canal-server:
image: canal/canal-server:v1.1.5
container_name: canal-server
restart: always
ports:
- "11111:11111"
volumes:
- /opt/docker/canal-server/conf/:/home/admin/canal-server/conf
- /opt/docker/canal-server/logs/:/home/admin/canal-server/logs/
privileged: true
environment:
- canal.auto.scan=true
- canal.destinations=canalServer # canal实例名称，根据自己喜好命名
- canal.instance.mysql.slaveId=1085 # canal作为模仿从站的ID，不可与主站重复
- canal.instance.master.address=1.15.157.156:3306 #mysql的连接地址
- canal.instance.dbUsername=canal # mysql的用户
- canal.instance.dbPassword=canal # mysql的密码
- canal.instance.connectionCharset=UTF-8
- canal.instance.tsdb.enable=true
- canal.instance.gtidon=false
- canal.instance.parser.parallelThreadSize=16
# - canal.instance.filter.regex=dbname.tablename # 此处为监听的正则表达式过滤，具体参考官网配置，多个用英文逗号隔开",",例如：dbname.tablename1,dbname.tablename2
networks:
- canal-network
canal-adapter:
image: slpcat/canal-adapter:v1.1.5 # 所使用的的镜像
container_name: canal-adapter # 容器名
restart: always
volumes: # 数据卷挂载
- /opt/docker/canal-adapter/conf:/opt/canal-adapter/conf
- /opt/docker/canal-adapter/logs:/opt/canal-adapter/logs
privileged: true
depends_on:
- canal-server
links:
- canal-server
networks:
- canal-network
networks: # 加入公共网络（没有这个网络，你需要提前创建）
canal-network:
external: true
canal的配置参考上方的配置即可，我这里就是简单的使用，并不复杂。
有好的建议，请在下方输入你的评论。
