手把手带你玩转ArgoCD --- debezium 部署

事件监控项目需要从不同的系统中获取不同的事件源，然后统一整理显示。由于各个系统都有使用mysql记录各自的事件信息，同时为了不侵入各系统进行事件搜集，故决定使用debezium进行事件读取搜集。本文将介绍整个部署流程。

安装strimzi

直接下载这个网页文件，用argocd纳管就行。 strimzi.io/install/lat…

部署KafkaConnect

下载debezium-connector-mysql的jar包

curl https://repo1.maven.org/maven2/io/debezium/debezium-connector-mysql/2.0.0.Final/debezium-connector-mysql-2.0.0.Final-plugin.tar.gz --output debezium-connector-mysql-2.0.0.Final-plugin.tar.gz

编写Dockerfile，制作镜像

tienbm90.medium.com/change-data…

FROM quay.io/strimzi/kafka:0.32.0-kafka-3.3.1
USER root:root  
RUN mkdir -p /opt/kafka/plugins/debezium  
COPY ./debezium-connector-mysql/ /opt/kafka/plugins/debezium/  
USER 1001

配置KafkaConnect的配置文件, 并用argocd纳管

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: debezium-connect-cluster
  annotations:
    strimzi.io/use-connector-resources: "true"
spec:
  version: 3.3.1
  image: xxx/debezium-connector-mysql:v0.0.1
  replicas: 1
  bootstrapServers: debezium-cluster-kafka-bootstrap:9092
  config:
    config.providers: secrets
    config.providers.secrets.class: io.strimzi.kafka.KubernetesSecretConfigProvider
    group.id: connect-cluster
    offset.storage.topic: connect-cluster-offsets
    config.storage.topic: connect-cluster-configs
    status.storage.topic: connect-cluster-status
    # -1 means it will use the default replication factor configured in the broker
    config.storage.replication.factor: -1
    offset.storage.replication.factor: -1
    status.storage.replication.factor: -1

问题1:

org.apache.kafka.common.config.ConfigException: Topic 'connect-cluster-offsets' supplied via the 'offset.storage.topic' property is required to have 'cleanup.policy=compact' to guarantee consistency and durability of source connector offsets, but found the topic currently has 'cleanup.policy=delete'. Continuing would likely result in eventually losing source connector offsets and problems restarting this Connect cluster in the future. Change the 'offset.storage.topic' property in the Connect worker configurations to use a topic with 'cleanup.policy=compact'.

connect-cluster-offsets connect-cluster-configs connect-cluster-status 把这三个topic修改下属性'cleanup.policy=compact'即可

部署KafkaConnector

配置KafkaConnect的配置文件

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnector
metadata:
  name: debezium-connector-mysql
  labels:
    strimzi.io/cluster: debezium-connect-cluster
spec:
  class: io.debezium.connector.mysql.MySqlConnector
  tasksMax: 1
  config:
    tasks.max: 1
    database.hostname: 10.10.10.10
    database.port: 3306
    database.user: xxxxxx
    database.password: xxxxxx
    database.server.id: 1
    topic.prefix: xxxxxx
    database.include.list: xxxxxx
    table.include.list: xxxxxx.yyyyyyy
    schema.history.internal.kafka.bootstrap.servers: debezium-cluster-kafka-bootstrap:9092
    schema.history.internal.kafka.topic: schemahistory.xxxxxx.yyyyyyy
    include.schema.changes: true

手动添加topic-"schemahistory.xxxxxx.yyyyyyy" 并配置cleanup.policy为compact

cleanup.policy
支持日志按保存时间删除，或者日志按key压缩（kafka connect时需要使用compact模式）

用argocd纳管, 进行同步

具体属性配置参考官网文档： debezium.io/documentati…

使用kafka connect api

常用命令

查看连接器状态
curl -k https://xxxxxx.com/connectors/debezium-connector-mysql/status

curl -s -X PUT  https://xxxxxx.com/connectors/debezium-connector-mysql/resume

curl -s -X POST  https://xxxxxx.com/connectors/debezium-connector-mysql/tasks/0/restart

curl -X DELETE https://xxxxxx.com/connectors/debezium-connector-mysql

kafka.apache.org/documentati…