Kafka 进阶使用

1. 消息分区与负载均衡

通过自定义分区器实现将消息按用户 ID 分配到指定分区的逻辑。

示例：自定义分区器

import org.apache.kafka.clients.producer.Partitioner;
import org.apache.kafka.common.Cluster;

import java.util.Map;

public class UserPartitioner implements Partitioner {

    @Override
    public int partition(String topic, Object key, byte[] keyBytes,
                         Object value, byte[] valueBytes, Cluster cluster) {
        int numPartitions = cluster.partitionCountForTopic(topic);
        int partition = key.hashCode() % numPartitions;
        return Math.abs(partition); // 确保分区号为正数
    }

    @Override
    public void close() {}

    @Override
    public void configure(Map<String, ?> configs) {}
}

配置自定义分区器

在 application.properties 中配置：

spring.kafka.producer.properties.partitioner.class=com.example.UserPartitioner

生产消息：发送到自定义分区

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.stereotype.Service;

@Service
public class KafkaPartitionProducer {

    @Autowired
    private KafkaTemplate<String, String> kafkaTemplate;

    public void sendMessage(String userId, String message) {
        kafkaTemplate.send("user-topic", userId, message);
        System.out.println("Message sent to user " + userId + ": " + message);
    }
}

运行结果

消息将根据用户 ID (userId) 计算分区，确保同一用户的消息进入同一分区。

---

2. 事务性消息

Kafka 生产者支持事务，保证消息的原子性。

示例：生产者事务

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.stereotype.Service;

@Service
public class KafkaTransactionalProducer {

    @Autowired
    private KafkaTemplate<String, String> kafkaTemplate;

    public void sendTransactionalMessages() {
        kafkaTemplate.executeInTransaction(kafkaOperations -> {
            kafkaOperations.send("transaction-topic", "key1", "message1");
            kafkaOperations.send("transaction-topic", "key2", "message2");
            if (true) throw new RuntimeException("Transaction failed!"); // 模拟异常
            return null;
        });
    }
}

配置事务支持

spring:
  kafka:
    producer:
      transaction-id-prefix: tx-  # 必须设置事务前缀

运行结果

如果事务中有异常，消息将不会提交，确保原子性。

---

3. 消费者手动提交偏移量

控制消费者的偏移量提交，确保在消息成功处理后再提交。

示例：手动提交偏移量

import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.kafka.support.Acknowledgment;
import org.springframework.stereotype.Service;

@Service
public class ManualOffsetConsumer {

    @KafkaListener(topics = "manual-offset-topic", groupId = "my-group")
    public void consume(ConsumerRecord<String, String> record, Acknowledgment acknowledgment) {
        try {
            // 业务逻辑处理
            System.out.println("Processing message: " + record.value());
            // 手动提交偏移量
            acknowledgment.acknowledge();
        } catch (Exception e) {
            System.out.println("Processing failed: " + record.value());
        }
    }
}

配置关闭自动提交

spring:
  kafka:
    consumer:
      enable-auto-commit: false

运行结果

消费者在成功处理消息后，才手动提交偏移量，避免未成功处理的消息被认为已处理。

---

4. 多线程消费

在 Kafka 消费者中实现多线程以提高消费性能。

示例：多线程消费

import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor;
import org.springframework.stereotype.Service;

@Service
public class MultiThreadConsumer {

    private final ThreadPoolTaskExecutor executor;

    public MultiThreadConsumer() {
        this.executor = new ThreadPoolTaskExecutor();
        this.executor.setCorePoolSize(5);  // 核心线程数
        this.executor.setMaxPoolSize(10); // 最大线程数
        this.executor.initialize();
    }

    @KafkaListener(topics = "multi-thread-topic", groupId = "multi-thread-group")
    public void consume(String message) {
        executor.execute(() -> {
            System.out.println("Processing in thread " + Thread.currentThread().getName() + ": " + message);
        });
    }
}

运行结果

消息处理由线程池的多个线程并行执行，提高了消费吞吐量。

---

5. 死信队列（DLQ）

配置死信队列，处理消费失败的消息。

示例：死信队列

import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.stereotype.Service;

@Service
public class DLQConsumer {

    @KafkaListener(topics = "dlq-topic", groupId = "dlq-group")
    public void consume(String message) {
        throw new RuntimeException("Simulating failure: " + message); // 模拟消费失败
    }
}

配置死信队列

spring:
  kafka:
    listener:
      ack-mode: record
    consumer:
      enable-auto-commit: false

配置 DeadLetterPublishingRecoverer：

import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.kafka.listener.DeadLetterPublishingRecoverer;
import org.springframework.kafka.listener.SeekToCurrentErrorHandler;

@Configuration
public class DLQConfig {

    @Bean
    public SeekToCurrentErrorHandler errorHandler(KafkaTemplate<Object, Object> template) {
        return new SeekToCurrentErrorHandler(
            new DeadLetterPublishingRecoverer(template), 3  // 最大重试次数
        );
    }
}

---

6. 消息过滤

消费特定的消息，过滤掉不需要的消息。

示例：消息过滤器

import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.stereotype.Service;

@Service
public class FilteredConsumer {

    @KafkaListener(topics = "filter-topic", groupId = "filter-group",
                   containerFactory = "filteredKafkaListenerContainerFactory")
    public void consumeFilteredMessages(ConsumerRecord<String, String> record) {
        System.out.println("Filtered message: " + record.value());
    }
}

配置过滤器

import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.kafka.config.ConcurrentKafkaListenerContainerFactory;

@Configuration
public class FilterConfig {

    @Bean
    public ConcurrentKafkaListenerContainerFactory<String, String> filteredKafkaListenerContainerFactory() {
        ConcurrentKafkaListenerContainerFactory<String, String> factory =
                new ConcurrentKafkaListenerContainerFactory<>();
        factory.setRecordFilterStrategy(record -> {
            // 仅保留包含 "important" 的消息
            return !record.value().contains("important");
        });
        return factory;
    }
}

运行结果

只有包含 "important" 的消息会被消费。

---

7. 配置日志审计

通过 Kafka 拦截器记录生产消息的日志。

示例：生产者拦截器

import org.apache.kafka.clients.producer.ProducerInterceptor;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.clients.producer.RecordMetadata;

import java.util.Map;

public class LoggingInterceptor implements ProducerInterceptor<String, String> {

    @Override
    public ProducerRecord<String, String> onSend(ProducerRecord<String, String> record) {
        System.out.println("Producing message: " + record.value());
        return record;
    }

    @Override
    public void onAcknowledgement(RecordMetadata metadata, Exception exception) {
        System.out.println("Acknowledged: " + metadata.offset());
    }

    @Override
    public void close() {}

    @Override
    public void configure(Map<String, ?> configs) {}
}

配置拦截器

spring.kafka.producer.properties.interceptor.classes=com.example.LoggingInterceptor

运行结果

每次生产消息时会记录日志，例如消息内容、分区号等。

---