3. Spring Cloud Stream-kafka暂停和恢复消费整合示例介绍简单、高效、稳定地使用Spring

掘金2024年度人气创作者打榜中，快来帮我打榜吧～

介绍

简单、高效、稳定地使用Spring Cloud Stream。
KafkaConsumer 提供了对消费速度进行控制的方法，在有些应用场景下我们可能需要暂停某些分区的消费而先消费其他分区，当达到一定条件时再恢复这些分区的消费。
本文章介绍在生产环境中使用spring cloud stream实现kafka的暂停和恢复消费。

版本说明

kafka server version：2.5.x
kafka client version：2.5.1
spring boot version: 2.3.12.RELEASE
spring cloud version：Hoxton.SR12
spring cloud stream version: 3.0.13.RELEASE
spring cloud stream binder kafka version: 3.0.13.RELEASE
java version：1.8

其他版本的完整代码示例，请访问 github.com/codebaorg/S…

如果这篇文章帮助到了你，欢迎评论、点赞、转发。

依赖

Maven


<parent>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-parent</artifactId>
    <version>2.3.12.RELEASE</version>
    <relativePath/>
</parent>

<properties>
    <spring-cloud.version>Hoxton.SR12</spring-cloud.version>
</properties>

<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-dependencies</artifactId>
            <version>${spring-cloud.version}</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
    </dependencies>
</dependencyManagement>

<dependencies>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter</artifactId>
    </dependency>

    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-stream</artifactId>
    </dependency>

    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-stream-binder-kafka</artifactId>
    </dependency>

</dependencies>

工程和配置

完整代码地址：github.com/codebaorg/S…

application.yaml 配置

spring cloud stream kafka相关配置内容示例如下，其中localhost:9092、test-prod.*-topic,foo、test-prod-group、min-partition-count、replication-factor替换为你的配置：

spring:
  cloud:
    stream:
      default:
        producer:
          error-channel-enabled: true # 开启生产者默认错误信息收集的channel

      kafka:
        binder:
          brokers: localhost:9092
          auto-create-topics: true # 开启自动创建主题
          min-partition-count: 3 # 单个主题的分区数
          replication-factor: 3 # 单个主题的副本数，这个是同时包含主从副本的数量
          configuration:
            acks: -1 # 见配置项说明
            reconnect.backoff.max.ms: 120000 # 见配置项说明

        bindings:
          my-prod-input:
            consumer:
              auto-commit-offset: false # 消费者关闭自动提交offset
              destination-is-pattern: true # 开启正则匹配topic

      bindings:
        my-prod-input:
          destination: test-prod.*-topic,foo # 消费多个主题，用逗号隔开
          group: test-prod-group
          consumer:
            batch-mode: true # 开启批量消费

消费者暂停kafka消费

MyProdSink.class 用于定义input binder的相关信息，其中INPUT的值和application.yaml中的spring.cloud.stream.bindings的值保持一致。

import org.springframework.cloud.stream.annotation.Input;
import org.springframework.messaging.SubscribableChannel;

public interface MyProdSink {

    String INPUT = "my-prod-input";

    @Input(INPUT)
    SubscribableChannel input();

}

kafka消费端暂停消费是按照主题分区维度来暂停的，不是一下就能暂停完毕的，而是逐渐暂停完毕的。具体的过程是：在消费者消费到某个主题分区的消息时才进行这个主题分区的暂停消费，如果消费者一直没有消费到某个主题分区，则是无法暂停这个主题分区的消费。

注意：当需要暂停消费时，不要使用return语句，来达到不执行业务代码的效果，因为这样做会造成消息丢失的问题。


import org.apache.kafka.clients.consumer.Consumer;
import org.apache.kafka.common.TopicPartition;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.cloud.stream.annotation.EnableBinding;
import org.springframework.cloud.stream.annotation.StreamListener;
import org.springframework.kafka.support.Acknowledgment;
import org.springframework.kafka.support.KafkaHeaders;
import org.springframework.messaging.handler.annotation.Header;
import org.springframework.messaging.handler.annotation.Payload;
import org.springframework.stereotype.Component;

import java.util.Collections;
import java.util.List;
import java.util.Set;
import java.util.concurrent.atomic.AtomicInteger;


@Component
@EnableBinding(MyProdSink.class)
public class MyProdConsumer {

    private static final Logger LOGGER = LoggerFactory.getLogger(MyProdConsumer.class);

    private final AtomicInteger counter = new AtomicInteger(0);

    @StreamListener(MyProdSink.INPUT)
    public void consume(
            @Payload List<Object> payloads,
            @Header(KafkaHeaders.RECEIVED_TOPIC) List<String> topics,
            @Header(KafkaHeaders.RECEIVED_PARTITION_ID) List<Integer> partitionIds,
            @Header(KafkaHeaders.GROUP_ID) String groupId,
            @Header(KafkaHeaders.CONSUMER) Consumer<?, ?> consumer,
            @Header(KafkaHeaders.ACKNOWLEDGMENT) Acknowledgment acknowledgment
    ) {
        LOGGER.info("consume payloads size: {}", payloads.size());
        boolean doPause = true;
        // pause
        pause(topics, partitionIds, consumer, doPause);

        // 注意：万万不能写下面这样的return代码
        // if (doPause) { 
        //    return;
        // }
        
        // 假设此处是业务处理代码
        for (int i = 0; i < payloads.size(); i++) {
            byte[] bytes = (byte[]) payloads.get(i);
            LOGGER.info("payload:{} from topic:{}, partitionId:{}, groupId:{}", new String(bytes), topics.get(i), partitionIds.get(i), groupId);
        }

        // manually ack
        acknowledgment.acknowledge();
        LOGGER.info("consumer message total:{}", counter.addAndGet(payloads.size()));

    }

    private void pause(List<String> topics, List<Integer> partitionIds, Consumer<?, ?> consumer, boolean pause) {
        LOGGER.info("pause begin--{}", pause);
        if (!pause) {
            return;
        }

        Set<TopicPartition> paused = consumer.paused();
        LOGGER.info("pause--Consumer.paused.size:{}", paused.size());
        for (int i = 0; i < topics.size(); i++) {
            String topic = topics.get(i);
            Integer partitionId = partitionIds.get(i);
            final TopicPartition topicPartition = new TopicPartition(topic, partitionId);
            // 当前 Kafka Consumer 暂停的 TopicPartition 不包含当前 TopicPartition 时，才进行暂停
            if (!paused.contains(topicPartition)) {
                consumer.pause(Collections.singletonList(topicPartition));
                LOGGER.info("pause TopicPartition:{} switch to paused.paused.size:{}", topicPartition, consumer.paused().size());
            }
        }

    }

}

消费者恢复kafka消费


import org.apache.kafka.clients.consumer.Consumer;
import org.apache.kafka.common.TopicPartition;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.context.ApplicationListener;
import org.springframework.context.annotation.Bean;
import org.springframework.kafka.event.ListenerContainerIdleEvent;
import org.springframework.stereotype.Component;

import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.stream.Collectors;


@Component
public class KafkaListener {

    private static final Logger LOGGER = LoggerFactory.getLogger(KafkaListener.class);

    @Bean
    public ApplicationListener<ListenerContainerIdleEvent> idleListener() {
        boolean resume = true;

        return event -> {
            Consumer<?, ?> consumer = event.getConsumer();
            Set<TopicPartition> pausedSet = consumer.paused();

            if (resume && !pausedSet.isEmpty()) {
                Map<String, List<TopicPartition>> topicPartitionGroupedByTopic = pausedSet.stream()
                        .collect(Collectors.groupingBy(TopicPartition::topic, Collectors.toList()));
                // 将暂停的主题分区全部恢复消费，当然也可根据自身业务进行定制化的恢复逻辑
                topicPartitionGroupedByTopic.forEach((topic, toResumeList) -> {
                    consumer.resume(toResumeList);
                    LOGGER.info("resume Consumer:{} switch to resumed.topic={}", toResumeList, topic);
                });
            }

        };
    }

}

如果这篇文章帮助到了你，欢迎评论、点赞、转发。