RabbitMQ的发布确认机制，可以很好的保证生产者消息不丢失。在实际开发过程中，比如消息发送方重启服务、RabbitMQ 服务重启或者 RabbitMQ 集群不可用等情况发生时，都会导致消息投递失败、消息丢失。那么我们怎么才能保证 RabbitMQ 消息的可靠投递呢？这里就要说到消息发布确认策略。
一、RabbitMQ Client 中关于发布确认相关API
发布确认默认是没有开启的，如果要开启需要调用方法 confirmSelect。每当你要想使用发布确认，都必须在Channel 上调用该方法。
1. 单个发布确认消息
这是一种简单的同步确认发布的方式，也就是发布一个消息之后只有当这个消息被确认发布之后，后续的消息才能继续发布。这种确认方式有一个最大的缺点就是：发布速度特别的慢。
1.1 消息生产者
public class SingleAckProducer {
private static String exchangeName = "exchange.prod.ack";
public static void main(String[] args) throws Exception {
Channel channel = RabbitChannelUtil.getChannel();
channel.exchangeDeclare(exchangeName, BuiltinExchangeType.TOPIC, false, false, null);
String body = "singe ack message";
String ackRoutingKey = "rk.prod.ack";
// 开启发布确认
channel.confirmSelect();
for (int i = 1; i <= 10; i++) {
String msg = i + ".".concat(body);
channel.basicPublish(exchangeName, ackRoutingKey, null, msg.getBytes(StandardCharsets.UTF_8));
// 消息服务器返回 false 或者 超时未返回，生产者可以重发消息
boolean ackResult = channel.waitForConfirms();
if (ackResult) {
System.out.println("消息：" + body + "，发送成功！！");
}
}
}
}
1.2 消息消费者
public class SingleAckConsumer {
public static String queueName = "prod.ack.queue";
private static String exchangeName = "exchange.prod.ack";
public static void main(String[] args) throws Exception {
Channel channel = RabbitChannelUtil.getChannel();
channel.exchangeDeclare(exchangeName, BuiltinExchangeType.TOPIC);
channel.queueDeclare(queueName, false, false, true, null);
String ackRoutingKey = "rk.prod.ack";
channel.queueBind(queueName, exchangeName, ackRoutingKey);
System.out.println("SingleAckConsumer 等待接收消息......");
DefaultConsumer consumer = new DefaultConsumer(channel) {
@Override
public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties, byte[] body) throws IOException {
System.out.println("SingleAckConsumer 接受到消息 body : " + new String(body));
}
};
channel.basicConsume(queueName, true, consumer);
}
}
启动运行 SingleAckProducer、SingleAckConsumer 后，可以看出它确实是同步确认的。
2. 批量发布确认消息
这种发布确认方式也是一种同步确认机制，只不过它是一批一批的消息并确认，弥补了单个消息确认的速度慢的问题。但是它也有自己的不足之处：当消息发布出现问题时，是无法确认哪个消息有问题，只能针对整个批次的消息进行补偿处理。
/**
* 批量消息发布确认
*/
public class BatchAckProducer {
private static String exchangeName = "exchange.prod.ack";
public static void main(String[] args) throws Exception {
Channel channel = RabbitChannelUtil.getChannel();
channel.exchangeDeclare(exchangeName, BuiltinExchangeType.TOPIC, false, false, null);
String body = "singe ack message";
String ackRoutingKey = "rk.prod.ack";
channel.confirmSelect(); // 开启发布确认
int batchAckSize = 10; // 每次批量确认10个消息
int notBatchAckSize = 0; // 为确认消息个数
for (int i = 1; i <= 30; i++) {
String msg = i + ".".concat(body);
channel.basicPublish(exchangeName, ackRoutingKey, null, msg.getBytes(StandardCharsets.UTF_8));
notBatchAckSize++;
// 批量确认逻辑
if (notBatchAckSize == batchAckSize) {
channel.waitForConfirms();
notBatchAckSize = 0;
}
// 为了保证还有剩余消息没有确认，再次确认一下子
if (notBatchAckSize > 0) {
channel.waitForConfirms();
}
}
}
}
3. 异步发布确认消息
异步发布确认的整体思路
/**
* 异步发布确认
*/
public class AsyncAckProducer {
private static String exchangeName = "exchange.prod.ack";
public static void main(String[] args) throws Exception {
Channel channel = RabbitChannelUtil.getChannel();
channel.exchangeDeclare(exchangeName, BuiltinExchangeType.TOPIC, false, false, null);
String body = "singe ack message";
String ackRoutingKey = "rk.prod.ack";
channel.confirmSelect(); // 开启发布确认
/**
* 定义一个安全有序的高性能哈希表，目的：
* 1.轻松的将序号和消息绑定
* 2.支持根据序号批量的删除消息条目
* 3.支持并发访问
*/
ConcurrentSkipListMap<Long, String> ackMap = new ConcurrentSkipListMap<>();
ConfirmCallback ackCallback = new ConfirmCallback() {
/**
* deliveryTag : 消息标识号
* multiple : true 表示可以确认小于等于当前序列号的消息；false 表示只能确认当前序列号的消息
*/
@Override
public void handle(long deliveryTag, boolean multiple) {
System.out.println("消息回调,deliveryTag : " + deliveryTag + ", multiple : " + multiple);
if (multiple) {
// 清除当前比当前序列号小的所有消息
ConcurrentNavigableMap<Long, String> confirmMap = ackMap.headMap(deliveryTag, true);
confirmMap.clear();
return;
}
// 只清除当前序列号的对应的消息
ackMap.remove(deliveryTag);
}
};
ConfirmCallback nackCallback = new ConfirmCallback() {
@Override
public void handle(long deliveryTag, boolean multiple) {
String msg = ackMap.get(deliveryTag);
System.out.println("消息 : " + msg + " 未被确认，消息 tag : " + deliveryTag);
}
};
/**
* 添加一个异步确认下消息监听器
* 1.确认收到消息的回调,ackCallback
* 2.未收到消息的回调,nackCallback
*/
channel.addConfirmListener(ackCallback, nackCallback);
for (int i = 1; i <= 30; i++) {
String msg = i + ".".concat(body);
// 获取下一个消息的序列号，其实就是 deliveryTag ，用来将消息和序列号绑定
long nextPublishSeqNo = channel.getNextPublishSeqNo();
ackMap.put(nextPublishSeqNo, msg);
// 发布消息
channel.basicPublish(exchangeName, ackRoutingKey, null, msg.getBytes(StandardCharsets.UTF_8));
}
}
}
3.1 异步发布确认消息
- 未确认的消息落库，异步定时任务定时重新投递。
- 可以把未确认的消息放到一个基于内存的能被发布线程访问的队列，比如说用
ConcurrentLinkedQueue这个队列在confirm callbacks与发布线程之间进行消息的传递。
4. 关于rabbitmq.client 中API的总结
- 单个消息发布确认、批量消息发布确认 这种思想在实际开发中并不常用，因为它是同步操作，在性能方面很难保证。
- 异步发布确认要保证消息的可靠投递，对未确认的消息进行落库，异步定时任务定时重新投递。
二、SpringBoot + RabbitMQ 实现发布确认
发布确认方案如下，这种处理方式的目的就是确保消息可靠投递。
1. 配置 rabbitmq 连接信息
@Configuration
public class RabbitMqConfig {
@Resource
private ExchangeCallback callback;
@Value("${spring.rabbitmq.addresses}")
private String address;
@Value("${spring.rabbitmq.username}")
private String username;
@Value("${spring.rabbitmq.password}")
private String password;
public static final String exchange = "exchange.prod.ack";
public static final String queue = "prod.ack.queue";
public static final String routingKey = "rk.prod.ack";
// 生产者发送消息连接
@Bean("connectionFactory")
public ConnectionFactory connectionFactory() {
CachingConnectionFactory connectionFactory = new CachingConnectionFactory();
connectionFactory.setAddresses(address);
connectionFactory.setShuffleAddresses(true);
connectionFactory.setUsername(username);
connectionFactory.setPassword(password);
// 发布消息成功到交换机后触发回调
connectionFactory.setPublisherConfirmType(CachingConnectionFactory.ConfirmType.CORRELATED);
return connectionFactory;
}
@Bean(name = "rabbitAdmin")
public RabbitAdmin rabbitAdmin() {
//需要传入
RabbitAdmin rabbitAdmin = new RabbitAdmin(connectionFactory());
rabbitAdmin.setAutoStartup(true);
return rabbitAdmin;
}
// 生产者发送消息配置
@Bean(name = "rabbitTemplate")
public AmqpTemplate rabbitTemplate(@Qualifier("connectionFactory") ConnectionFactory connectionFactory) {
RabbitTemplate template = new RabbitTemplate(connectionFactory);
template.setExchange(exchange);
template.setUsePublisherConnection(true);
template.setMessageConverter(new Jackson2JsonMessageConverter());
// 注入回调
template.setConfirmCallback(callback);
return template;
}
// 声明队列
@Bean(name = "ackQueue")
public Queue ackQueue(@Qualifier("rabbitAdmin") RabbitAdmin rabbitAdmin) {
Queue queue = new Queue(RabbitMqConfig.queue, true);
rabbitAdmin.declareQueue(queue);
return queue;
}
// 交换机配置
@Bean(name = "ackExchange")
TopicExchange ackExchange() {
return ExchangeBuilder.topicExchange(exchange).durable(true).build();
}
// 绑定RoutingKey
@Bean
public Binding attRestDayCalculateQueueBinding(@Qualifier("ackQueue") Queue queue, @Qualifier("ackExchange") TopicExchange exchange) {
return BindingBuilder.bind(queue).to(exchange).with(routingKey);
}
// 消费者连接工厂配置
@Bean("rabbitListenerFactory")
public RabbitListenerContainerFactory<?> rabbitListenerContainerFactory(@Qualifier("connectionFactory") ConnectionFactory connectionFactory) {
SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();
factory.setMessageConverter(new Jackson2JsonMessageConverter());
// 消费者自动确认消息
factory.setAcknowledgeMode(AcknowledgeMode.AUTO);
factory.setConnectionFactory(connectionFactory);
factory.setPrefetchCount(1);
return factory;
}
}
- 在
spring boot项目中我建议用@Configuration来配置queue、exchange和routingKey的绑定关系，避免使用手动在rabbitmq后台创建绑定关系，这种方式容易出错，而且不方便其他同事查看 - 如果在消息发送方配置队列绑定关系，启动时会报如下错误
2022-07-27 22:18:53.663 WARN 4088 --- [ntContainer#0-1] o.s.a.r.l.BlockingQueueConsumer : Failed to declare queue: prod.ack.queue
2022-07-27 22:18:53.665 WARN 4088 --- [ntContainer#0-1] o.s.a.r.l.BlockingQueueConsumer : Queue declaration failed; retries left=2
org.springframework.amqp.rabbit.listener.BlockingQueueConsumer$DeclarationException: Failed to declare queue(s):[prod.ack.queue]
at org.springframework.amqp.rabbit.listener.BlockingQueueConsumer.attemptPassiveDeclarations(BlockingQueueConsumer.java:700) ~[spring-rabbit-2.2.11.RELEASE.jar:2.2.11.RELEASE]
at org.springframework.amqp.rabbit.listener.BlockingQueueConsumer.passiveDeclarations(BlockingQueueConsumer.java:584) ~[spring-rabbit-2.2.11.RELEASE.jar:2.2.11.RELEASE]
at org.springframework.amqp.rabbit.listener.BlockingQueueConsumer.start(BlockingQueueConsumer.java:571) ~[spring-rabbit-2.2.11.RELEASE.jar:2.2.11.RELEASE]
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.initialize(SimpleMessageListenerContainer.java:1355) ~[spring-rabbit-2.2.11.RELEASE.jar:2.2.11.RELEASE]
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1200) ~[spring-rabbit-2.2.11.RELEASE.jar:2.2.11.RELEASE]
at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_111]
Caused by: java.io.IOException
at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:129) ~[amqp-client-5.7.3.jar:5.7.3]
at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:125) ~[amqp-client-5.7.3.jar:5.7.3]
at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:147) ~[amqp-client-5.7.3.jar:5.7.3]
at com.rabbitmq.client.impl.ChannelN.queueDeclarePassive(ChannelN.java:1012) ~[amqp-client-5.7.3.jar:5.7.3]
at com.rabbitmq.client.impl.ChannelN.queueDeclarePassive(ChannelN.java:52) ~[amqp-client-5.7.3.jar:5.7.3]
at org.springframework.amqp.rabbit.connection.PublisherCallbackChannelImpl.queueDeclarePassive(PublisherCallbackChannelImpl.java:355) ~[spring-rabbit-2.2.11.RELEASE.jar:2.2.11.RELEASE]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_111]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_111]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_111]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_111]
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory$CachedChannelInvocationHandler.invoke(CachingConnectionFactory.java:1184) ~[spring-rabbit-2.2.11.RELEASE.jar:2.2.11.RELEASE]
at com.sun.proxy.$Proxy82.queueDeclarePassive(Unknown Source) ~[?:?]
at org.springframework.amqp.rabbit.listener.BlockingQueueConsumer.attemptPassiveDeclarations(BlockingQueueConsumer.java:679) ~[spring-rabbit-2.2.11.RELEASE.jar:2.2.11.RELEASE]
... 5 more
Caused by: com.rabbitmq.client.ShutdownSignalException: channel error; protocol method: #method<channel.close>(reply-code=404, reply-text=NOT_FOUND - no queue 'prod.ack.queue' in vhost '/', class-id=50, method-id=10)
at com.rabbitmq.utility.ValueOrException.getValue(ValueOrException.java:66) ~[amqp-client-5.7.3.jar:5.7.3]
at com.rabbitmq.utility.BlockingValueOrException.uninterruptibleGetValue(BlockingValueOrException.java:36) ~[amqp-client-5.7.3.jar:5.7.3]
at com.rabbitmq.client.impl.AMQChannel$BlockingRpcContinuation.getReply(AMQChannel.java:502) ~[amqp-client-5.7.3.jar:5.7.3]
at com.rabbitmq.client.impl.AMQChannel.privateRpc(AMQChannel.java:293) ~[amqp-client-5.7.3.jar:5.7.3]
at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:141) ~[amqp-client-5.7.3.jar:5.7.3]
at com.rabbitmq.client.impl.ChannelN.queueDeclarePassive(ChannelN.java:1012) ~[amqp-client-5.7.3.jar:5.7.3]
at com.rabbitmq.client.impl.ChannelN.queueDeclarePassive(ChannelN.java:52) ~[amqp-client-5.7.3.jar:5.7.3]
at org.springframework.amqp.rabbit.connection.PublisherCallbackChannelImpl.queueDeclarePassive(PublisherCallbackChannelImpl.java:355) ~[spring-rabbit-2.2.11.RELEASE.jar:2.2.11.RELEASE]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_111]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_111]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_111]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_111]
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory$CachedChannelInvocationHandler.invoke(CachingConnectionFactory.java:1184) ~[spring-rabbit-2.2.11.RELEASE.jar:2.2.11.RELEASE]
at com.sun.proxy.$Proxy82.queueDeclarePassive(Unknown Source) ~[?:?]
at org.springframework.amqp.rabbit.listener.BlockingQueueConsumer.attemptPassiveDeclarations(BlockingQueueConsumer.java:679) ~[spring-rabbit-2.2.11.RELEASE.jar:2.2.11.RELEASE]
... 5 more
Caused by: com.rabbitmq.client.ShutdownSignalException: channel error; protocol method: #method<channel.close>(reply-code=404, reply-text=NOT_FOUND - no queue 'prod.ack.queue' in vhost '/', class-id=50, method-id=10)
at com.rabbitmq.client.impl.ChannelN.asyncShutdown(ChannelN.java:522) ~[amqp-client-5.7.3.jar:5.7.3]
at com.rabbitmq.client.impl.ChannelN.processAsync(ChannelN.java:346) ~[amqp-client-5.7.3.jar:5.7.3]
at com.rabbitmq.client.impl.AMQChannel.handleCompleteInboundCommand(AMQChannel.java:182) ~[amqp-client-5.7.3.jar:5.7.3]
at com.rabbitmq.client.impl.AMQChannel.handleFrame(AMQChannel.java:114) ~[amqp-client-5.7.3.jar:5.7.3]
at com.rabbitmq.client.impl.AMQConnection.readFrame(AMQConnection.java:672) ~[amqp-client-5.7.3.jar:5.7.3]
at com.rabbitmq.client.impl.AMQConnection.access$300(AMQConnection.java:48) ~[amqp-client-5.7.3.jar:5.7.3]
at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:599) ~[amqp-client-5.7.3.jar:5.7.3]
... 1 more
当出现这个异常时，请用如下方式进行配置，原因分析参考文章：Failed to declare queue
@Bean(name = "rabbitAdmin")
public RabbitAdmin rabbitAdmin() {
//需要传入
RabbitAdmin rabbitAdmin = new RabbitAdmin(connectionFactory());
rabbitAdmin.setAutoStartup(true);
return rabbitAdmin;
}
// 声明队列
@Bean(name = "ackQueue")
public Queue ackQueue(@Qualifier("rabbitAdmin") RabbitAdmin rabbitAdmin) {
Queue queue = new Queue(RabbitMqConfig.queue, true);
rabbitAdmin.declareQueue(queue);
return queue;
}
2. 交换机回调方法
@Slf4j
@Component
public class ExchangeCallback implements RabbitTemplate.ConfirmCallback {
/**
* 交换机不论是否收到消息都会回调该方法
* @param correlationData 消息数据对象
* @param ack 交换机是否收到消息，true ：交换机已收到消息 ；false ：交换机未收到消息
* @param cause 未收到消息的原因
*/
@Override
public void confirm(CorrelationData correlationData, boolean ack, String cause) {
String id = null != correlationData ? correlationData.getId() : "null";
if (ack) {
log.info("交换机已收到消息, id : {}", id);
return;
}
log.info("交换机未收到消息, id : {} , 原因 : {}", id, cause);
}
}
3. 消息生产者
@Slf4j
@RestController
@RequestMapping("/ack")
public class Producer {
@Resource
private RabbitTemplate rabbitTemplate;
@PostMapping("/send")
public void send() {
CorrelationData data = new CorrelationData("1");
Map<String, String> firstMap = new HashMap<>();
firstMap.put("1", "first msg");
rabbitTemplate.convertAndSend(RabbitMqConfig.exchange, RabbitMqConfig.routingKey, firstMap, data);
CorrelationData data2 = new CorrelationData("2");
Map<String, String> secondMap = new HashMap<>();
secondMap.put("2", "second msg");
rabbitTemplate.convertAndSend(RabbitMqConfig.exchange, RabbitMqConfig.routingKey + ".1", secondMap, data2);
log.info("=========消息发送完成==========");
}
}
4. 消息消费者
@Slf4j
@Component
public class Consumer {
@RabbitHandler
@RabbitListener(queues = RabbitMqConfig.queue, containerFactory = "rabbitListenerFactory")
public void receiveMessage(Message message) {
String msgBody = new String(message.getBody(), StandardCharsets.UTF_8);
log.info("接收到队列 prod.ack.queue 的消息 : {}", msgBody);
}
}
5. 启动类
@SpringBootApplication(scanBasePackages = {"com.sff.delay.example"}, exclude = {RabbitAutoConfiguration.class})
@EnableRabbit
public class RabbtimqDelayQueueApplication {
public static void main(String[] args) {
SpringApplication.run(RabbtimqDelayQueueApplication.class, args);
}
}
6. 运行结果
2022-07-28 22:47:05.613 INFO 1694 --- [tory.publisher1] c.s.d.e.s.a.ExchangeCallback : 交换机已收到消息, id : 1
2022-07-28 22:47:05.636 INFO 1694 --- [nio-8080-exec-1] c.s.d.e.s.a.Producer : =========消息发送完成==========
2022-07-28 22:47:05.648 INFO 1694 --- [ntContainer#0-1] c.s.d.e.s.a.Consumer : 接收到队列 prod.ack.queue 的消息 : {"1":"first msg"}
2022-07-28 22:47:05.665 INFO 1694 --- [tory.publisher1] c.s.d.e.s.a.ExchangeCallback : 交换机已收到消息, id : 2
可以看到id为 1 和 2 的消息都到达了 exchange，但是由于id是 2 的消息绑定的 routingkey= rk.prod.ack.1 ，所以导致消费者无法接受到消息，但是也触发了交换机的回调方法。
