Debezium使用

docker-compose.yml

services:
  kafka:
    image: quay.io/debezium/kafka:${DEBEZIUM_VERSION}
    ports:
      - 9092:9092
      - 9093:9093
    environment:
      - CLUSTER_ID=oh-sxaDRTcyAr6pFRbXyzA
      - NODE_ID=1
      - NODE_ROLE=combined
      - KAFKA_CONTROLLER_QUORUM_VOTERS=1@kafka:9093
      - KAFKA_LISTENERS=PLAINTEXT://kafka:9092,CONTROLLER://kafka:9093
      - KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092

  kafka-ui:
    image: ghcr.io/kafbat/kafka-ui:latest
    ports:
      - 8080:8080
    environment:
      TZ: Asia/Shanghai
      LANG: C.UTF-8
      KAFKA_CLUSTERS_0_NAME: kafka
      KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: 172.20.2.15:9092
      SERVER_SERVLET_CONTEXT_PATH: /kafkaui
      AUTH_TYPE: "LOGIN_FORM"
      SPRING_SECURITY_USER_NAME: kafkaAdmin
      SPRING_SECURITY_USER_PASSWORD: Kafka@2025
    depends_on:
      - kafka

  connect:
    image: quay.io/debezium/connect:${DEBEZIUM_VERSION}
    ports:
     - 8083:8083
    depends_on:
     - kafka
    environment:
     - BOOTSTRAP_SERVERS=kafka:9092
     - GROUP_ID=1
     - CONFIG_STORAGE_TOPIC=my_connect_configs
     - OFFSET_STORAGE_TOPIC=my_connect_offsets
     - STATUS_STORAGE_TOPIC=my_connect_statuses

三个容器分别是debezium本身、debezium存储配置的connectors以及同步的数据需要的kafka，查看kafka数据的ui工具kafka-ui。

添加connector

POST http://172.20.2.15:8083/connectors

{
  "name": "postgres-connector",
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "database.hostname": "172.20.2.15",
    "database.port": "5452",
    "database.user": "postgres",
    "database.password": "Matrix@2025",
    "database.dbname": "postgres",
    "database.server.name": "pg_server",
    "slot.name": "debezium_slot",
    "plugin.name": "pgoutput",
    "publication.autocreate.mode": "filtered",  // 仅自动为已配置的表创建 publication
    "table.include.list": "public.hyd_grid_info",  // 要监听的表,多个表用逗号隔开
    "database.history.kafka.bootstrap.servers": "172.20.2.15:9092",
    "database.history.kafka.topic": "schema-changes.pg"
  }
}

修改connector

PUT http://172.20.2.15:8083/connectors/postgres-connector/config 接口中的postgres-connector 是路径参数，表示要修改的connector

{
  "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
  "database.hostname": "172.20.2.15",
  "database.port": "5452",
  "database.user": "postgres",
  "database.password": "Matrix@2025",
  "database.dbname": "postgres",
  "database.server.name": "pg_server",
  "slot.name": "debezium_slot",
  "plugin.name": "pgoutput",
  "publication.autocreate.mode": "filtered",
  "table.include.list": "public.hyd_grid_info,zhy-cloud.st_pptn_r",
  "topic.prefix": "fulfillment",
  "database.history.kafka.bootstrap.servers": "172.20.2.15:9092",
  "database.history.kafka.topic": "schema-changes.pg"
}

查看connector列表

GET http://172.20.2.15:8083/connectors/

connectors配置

1.postgres配置

修改/usr/share/postgresql/postgresql.conf 配置,添加如下内容：

wal_level = logical
max_wal_senders = 10
max_replication_slots = 10

postgres命令SHOW wal_level; 如果返回logical，说明修改成功。

2.mysql配置

connector参考 配置：

{
  "connector.class": "io.debezium.connector.mysql.MySqlConnector",
  "transforms.addStaticField.static.value": "平台",
  "database.user": "ot",
  "database.server.id": "1",
  "transforms": "addStaticField",
  "schema.history.internal.kafka.bootstrap.servers": "172.2.2.5:9092",
  "database.port": "3306",
  "include.schema.changes": "true",
  "topic.prefix": "fullfillment",
  "schema.history.internal.kafka.topic": "fullfillment.schema",
  "database.hostname": "172.20.2.5",
  "database.password": "De@2025",
  "transforms.addStaticField.static.field": "projectName",
  "name": "mysql-connector",
  "table.include.list": "devops.test",
  "transforms.addStaticField.type": "org.apache.kafka.connect.transforms.InsertField$Value",
  "schema.history.internal.consumer.security.protocol": "SASL_PLAINTEXT",
  "schema.history.internal.consumer.sasl.mechanism": "PLAIN",
  "schema.history.internal.consumer.sasl.jaas.config": "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"ucer\" password=\"ucer2025\";",
  "schema.history.internal.producer.security.protocol": "SASL_PLAINTEXT",
  "schema.history.internal.producer.sasl.mechanism": "PLAIN",
  "schema.history.internal.producer.sasl.jaas.config": "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"ucer\" password=\"ucer2025\";"
}