这是我参与11月更文挑战的第28天，活动详情查看：2021最后一次更文挑战
假设一个 Kafka 消费者，从 PackageEvents 主题轮询事件。
服务类（Package service）负责将消费的事件存储到数据库中。
注意：这里代替数据库的地方，可以是API，也可以是第三方应用调用。
如何在服务级别处理异常，其中异常可以作为验证服务或在持久化到数据库中时使用，也可以在调用 API 时使用。
Kafka Consumer:
为了创建一个监听某个主题的消费者，我们在 spring boot 应用程序中的一个方法上使用@KafkaListener (topics = {“packages-received”})。
这里的“packages-received”是轮询消息的主题。
@KafkaListener(topics = {"packages-received"})
public void packagesListener(ConsumerRecord<String,PackageInfoEvent> packageInfoEvent){
log.info("Received event to persist packageInfoEvent :{}", packageInfoEvent.value());
}
通常，Kafka Listener 通过“kafkaListenerFactory”bean 获取属性文件中指定的所有属性，例如 groupId、key 和 value 序列化器信息。 简单来说，“kafkaListenerFactory”bean 是配置 Kafka 监听器的关键。 如果我们需要配置 Kafka 侦听器配置覆盖默认行为，您需要创建“kafkaListenerFactory”bean 并设置所需的配置。
这就是我们将用来为 Kafka Listener/consumer 设置错误处理、重试和恢复的工具
@Bean
ConcurrentKafkaListenerContainerFactory<?, ?> kafkaListenerContainerFactory(
ConcurrentKafkaListenerContainerFactoryConfigurer configurer,
ObjectProvider<ConsumerFactory<Object, Object>> kafkaConsumerFactory) {
ConcurrentKafkaListenerContainerFactory<Object, Object> factory = new ConcurrentKafkaListenerContainerFactory<>();
configurer.configure(factory, kafkaConsumerFactory);
return factory;
}
Kafka Retry:
一般来说，服务层引起的运行时异常，这些是由于您尝试访问的服务（DB，API）已关闭或有问题引起的异常。
这些例外是在以后尝试时可以成功的例外。
以下代码片段显示了如何使用 RetryTemplate 配置重试。
作为回报，RetryTemplate 设置了重试策略，该策略指定您想要重试的最大尝试次数以及您想要重试的异常和不重试的异常。
public class ConsumerConfig {
@Bean
ConcurrentKafkaListenerContainerFactory<?, ?> kafkaListenerContainerFactory(
ConcurrentKafkaListenerContainerFactoryConfigurer configurer,
ObjectProvider<ConsumerFactory<Object, Object>> kafkaConsumerFactory) {
ConcurrentKafkaListenerContainerFactory<Object, Object> factory = new ConcurrentKafkaListenerContainerFactory<>();
configurer.configure(factory, kafkaConsumerFactory);
factory.setRetryTemplate(retryTemplate());
return factory;
}
private RetryTemplate retryTemplate() {
RetryTemplate retryTemplate = new RetryTemplate();
/* here retry policy is used to set the number of attempts to retry and what exceptions you wanted to try and what you don't want to retry.*/
retryTemplate.setRetryPolicy(getSimpleRetryPolicy());
return retryTemplate;
}
private SimpleRetryPolicy getSimpleRetryPolicy() {
Map<Class<? extends Throwable>, Boolean> exceptionMap = new HashMap<>();
// the boolean value in the map determines whether exception should be retried
exceptionMap.put(IllegalArgumentException.class, false);
exceptionMap.put(TimeoutException.class, true);
return new SimpleRetryPolicy(3,exceptionMap,true);
}
}
当事件失败时，即使在重试最大重试次数的某些异常之后，恢复阶段也会启动。
重试和恢复齐头并进，
如果重试次数用完，则恢复将测试事件异常是否可恢复并采取必要的恢复步骤，例如将其放回重试主题或将其保存到数据库以供稍后尝试。
如果事件异常不可恢复，它只是将其传递给错误处理程序。我们将在这里稍后讨论错误处理。
Kafka Recovery :
ConcurrentKafkaListenerContainerFactory 上有一个方便的方法 setRecoveryCallBack() ，它接受 Retry 上下文参数，
在这里我们获得了上下文（在尝试了最大重试之后），它包含有关事件的信息。
@Slf4j
public class ConsumerConfig {
@Bean
ConcurrentKafkaListenerContainerFactory<?, ?> kafkaListenerContainerFactory(
ConcurrentKafkaListenerContainerFactoryConfigurer configurer,
ObjectProvider<ConsumerFactory<Object, Object>> kafkaConsumerFactory) {
ConcurrentKafkaListenerContainerFactory<Object, Object> factory = new ConcurrentKafkaListenerContainerFactory<>();
configurer.configure(factory, kafkaConsumerFactory);
factory.setRetryTemplate(retryTemplate());
factory.setRecoveryCallback((context -> {
if(context.getLastThrowable().getCause() instanceof RecoverableDataAccessException){
//here you can do your recovery mechanism where you can put back on to the topic using a Kafka producer
} else{
// here you can log things and throw some custom exception that Error handler will take care of.
throw new RuntimeException(context.getLastThrowable().getMessage());
}
return null;
}));
return factory;
}
private RetryTemplate retryTemplate() {
RetryTemplate retryTemplate = new RetryTemplate();
/* here retry policy is used to set the number of attempts to
retry and what exceptions you wanted to try and
what you don't want to retry.*/
retryTemplate.setRetryPolicy(getSimpleRetryPolicy());
return retryTemplate;
}
private SimpleRetryPolicy getSimpleRetryPolicy() {
Map<Class<? extends Throwable>, Boolean> exceptionMap = new HashMap<>();
exceptionMap.put(IllegalArgumentException.class, false);
exceptionMap.put(TimeoutException.class, true);
return new SimpleRetryPolicy(3,exceptionMap,true);
}
}
Kafka Error handling:
对于消费事件过程中的任何异常，都会在org.springframework.kafka.listener包中的Kafka“LoggingErrorHandler.class”记录一个错误，
LoggingErrorHandler 实现了“ErrorHandler”接口。
我们可以通过实现“ErrorHandler”接口来实现我们自己的错误处理程序。
@Slf4j
public class ConsumerConfig {
@Bean
ConcurrentKafkaListenerContainerFactory<?, ?> kafkaListenerContainerFactory(
ConcurrentKafkaListenerContainerFactoryConfigurer configurer,
ObjectProvider<ConsumerFactory<Object, Object>> kafkaConsumerFactory) {
ConcurrentKafkaListenerContainerFactory<Object, Object> factory = new ConcurrentKafkaListenerContainerFactory<>();
configurer.configure(factory, kafkaConsumerFactory);
factory.setErrorHandler(((exception, data) -> {
/* here you can do you custom handling, I am just logging it same as default Error handler does
If you just want to log. you need not configure the error handler here. The default handler does it for you.
Generally, you will persist the failed records to DB for tracking the failed records. */
log.error("Error in process with Exception {} and the record is {}", exception, data);
}));
return factory;
}
}
代码片段所有策略协同工作
package com.pack.events.consumer.config;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.ObjectProvider;
import org.springframework.boot.autoconfigure.kafka.ConcurrentKafkaListenerContainerFactoryConfigurer;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.dao.RecoverableDataAccessException;
import org.springframework.kafka.config.ConcurrentKafkaListenerContainerFactory;
import org.springframework.kafka.core.ConsumerFactory;
import org.springframework.retry.policy.SimpleRetryPolicy;
import org.springframework.retry.support.RetryTemplate;
import java.util.HashMap;
import java.util.Map;
import java.util.concurrent.TimeoutException;
@Configuration
@Slf4j
public class ConsumerConfig {
@Bean
ConcurrentKafkaListenerContainerFactory<?, ?> kafkaListenerContainerFactory(
ConcurrentKafkaListenerContainerFactoryConfigurer configurer,
ObjectProvider<ConsumerFactory<Object, Object>> kafkaConsumerFactory) {
ConcurrentKafkaListenerContainerFactory<Object, Object> factory = new ConcurrentKafkaListenerContainerFactory<>();
configurer.configure(factory, kafkaConsumerFactory);
factory.setRetryTemplate(retryTemplate());
factory.setRecoveryCallback((context -> {
if(context.getLastThrowable().getCause() instanceof RecoverableDataAccessException){
//here you can do your recovery mechanism where you can put back on to the topic using a Kafka producer
} else{
// here you can log things and throw some custom exception that Error handler will take care of ..
throw new RuntimeException(context.getLastThrowable().getMessage());
}
return null;
}));
factory.setErrorHandler(((exception, data) -> {
/* here you can do you custom handling, I am just logging it same as default Error handler does
If you just want to log. you need not configure the error handler here. The default handler does it for you.
Generally, you will persist the failed records to DB for tracking the failed records. */
log.error("Error in process with Exception {} and the record is {}", exception, data);
}));
return factory;
}
private RetryTemplate retryTemplate() {
RetryTemplate retryTemplate = new RetryTemplate();
/* here retry policy is used to set the number of attempts to retry and what exceptions you wanted to try and what you don't want to retry.*/
retryTemplate.setRetryPolicy(getSimpleRetryPolicy());
return retryTemplate;
}
private SimpleRetryPolicy getSimpleRetryPolicy() {
Map<Class<? extends Throwable>, Boolean> exceptionMap = new HashMap<>();
exceptionMap.put(IllegalArgumentException.class, false);
exceptionMap.put(TimeoutException.class, true);
return new SimpleRetryPolicy(3,exceptionMap,true);
}
}
```
```
