前言
RabbitMq是什么？我觉得是一个消息代理，可以形容为一个送信员，负责接收、存储、转发消息。
在分布式系统中，咱们服务间通信采用的是openFeign的调用，在业务中A服务给B服务发送请求，等待B服务执行完业务返回结果后，才能执行后面的业务，这种调用方式是同步调用。
但是咱们MQ的作用就是将这种通讯，改为异步调用的形式。这样做的好处在于，不需要立刻回应，可以广播发送。换成经典的回答就是，削峰解耦。
市面上常见的MQ有：RabbitMQ、ActiveMQ、RocketMQ、Kafka等。
交换机类型：
- Direct：根据路由键完全匹配去寻找队列
- Topic：根据路由键规则去寻找队列
- Fanout：消息广播，不根据路由键匹配，向所有绑定了的队列发送
- Headers：根据headers属性进行匹配
第一二种已经覆盖大部分业务需求了，不建议使用广播消息，可以使用Topic消息替代。
介绍
- publisher:生产者，发送消息给交换机
- consumer：消费者，从队列接收消息
- queue：队列，存储消息，先进先出的栈结构
- exchange：交换机，消息路由，将消息投进对应的队列
- virtual host：虚拟主机，每个虚拟主机里都有各自的交换机和队列
简单应用
引入依赖
<!--mq-->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-amqp</artifactId>
</dependency>
配置
rabbitmq:
host: 127.0.0.1
port: 5672
username: admin
password: admin
virtual-host: testHost
# 手动提交消息
listener:
simple:
acknowledge-mode: manual
direct:
acknowledge-mode: manual
我这里设置了消息消费后提交方式为手动，这样做的好处在于可以控制消息消费完成的时机，后面会用应用的地方，再详细说。
配置好基础环境后，开始写第一个使用案例，步骤：
- 编写rabbitMq配置和队列交换机绑定关系
- 生产者发送消息到交换机
- 消费者监听队列消费消息
配置和绑定关系
@Configuration
@Slf4j
public class RabbitMQConfig{
/**
* 邮箱交换机
*/
public static final String EMAIL_EXCHANGE = "email_exchange";
/**
* 邮箱队列
*/
public static final String EMAIL_QUEUE = "email_queue";
/**
* 邮箱路由key
*/
public static final String EMAIL_QUEUE_ROUTING_KEY = "email.#";
@Primary
@Bean(name = "rabbitTemplateEmail")
public RabbitTemplate rabbitTemplate(ConnectionFactory connectionFactory) {
RabbitTemplate serverRabbitTemplate = new RabbitTemplate(connectionFactory);
//确认消息是否到达MQ
serverRabbitTemplate.setConfirmCallback((correlationData, ack, cause) -> {
String correlationDataId = correlationData.getId();
if (ack) {
//ACK
log.debug("消息[{}]投递成功", correlationDataId);
} else {
System.out.println("消息[{}]投递失败,cause:{}");
}
});
//配置return监听处理，消息无法路由到queue触发，此处只打印了相关信息，具体逻辑需自己实现
serverRabbitTemplate.setReturnCallback((message, replyCode, replyText, exchange, routingKey) -> {
System.out.println("=============returnCallback触发。消息路由到queue失败===========");
System.out.println("msg=" + new String(message.getBody()));
System.out.println("replyCode=" + replyCode);
System.out.println("replyText=" + replyText);
System.out.println("exchange=" + exchange);
System.out.println("routingKey=" + routingKey);
});
return serverRabbitTemplate;
}
/**
* 创建邮箱交换机
*/
@Bean("emailExchange")
public Exchange emailExchange() {
return new TopicExchange(EMAIL_EXCHANGE, true, false);
}
/**
* 创建邮箱队列
*/
@Bean("emailQueue")
public Queue emailQueue() {
Map<String, Object> args = new HashMap<>();
return QueueBuilder.durable(EMAIL_QUEUE).withArguments(args).build();
}
/**
* 绑定邮箱交换机和队列
*/
@Bean("emailBinding")
public Binding emailBinding(@Qualifier("emailQueue") Queue queue, @Qualifier("emailExchange")Exchange exchange) {
return BindingBuilder.bind(queue).to(exchange).with(EMAIL_QUEUE_ROUTING_KEY).noargs();
}
消息体包装
@Data
@Builder
@NoArgsConstructor
@AllArgsConstructor
public class MessageStruct implements Serializable {
private static final long serialVersionUID = 392365881428311040L;
private String dataId;
private String module;
private String message;
private Integer delayTime;
//业务类型
private String messageCode;
private Map<String, Object> variables;
}
消息生产者
@Slf4j
public class EmailSendProducer {
private static RabbitTemplate rabbitTemplate;
private static RabbitTemplate getRabbitTemplate() {
if (rabbitTemplate == null) {
rabbitTemplate = SpringUtil.getBean("rabbitTemplateEmail", RabbitTemplate.class);
}
return rabbitTemplate;
}
public static void sendEmailDelayedStartMessage(EmailDraftbox emailDraftbox , Long time) {
//消息体
MessageStruct messageStruct = new MessageStruct();
messageStruct.setDataId(emailDraftbox.getId().toString());
messageStruct.setModule(MessageSendEnum.EMAIL_SEND.getModule());
messageStruct.setMessage(MessageSendEnum.EMAIL_SEND.getType());
messageStruct.setMessageCode(MessageSendEnum.EMAIL_SEND.getMessageCode());
messageStruct.setDelayTime(Integer.valueOf(time.toString()));
CorrelationData correlationData = new CorrelationData();
correlationData.setId(String.valueOf(emailDraftbox.getId())); getRabbitTemplate().convertAndSend(RabbitMqConstant.EMAIL_DELAY_PLUGIN_EXCHANGE, RabbitMqConstant.EMAIL_QUEUE_ROUTING_KEY, messageStruct,correlationData);
log.info("sendEmailDelayedStartMessage:消息发送，发送对象：" + JsonUtil.toJson(messageStruct) + ",发送时间：{}" + DateUtil.now());
}
}
消息常量类
public interface RabbitMqConstant {
/**
* 邮件-交换机
*/
String EMAIL_EXCHANGE = "email.exchange";
/**
* 邮件-队列
*/
String EMAIL_QUEUE = "email.queue";
/**
* 邮件-路由名称
*/
String EMAIL_QUEUE_ROUTING_KEY = "email.eue.routingKey";
消费者
@Component
@Slf4j
@RabbitListener(queues = RabbitMqConstant.EMAIL_QUEUE)
@AllArgsConstructor
public class EmailConsumer {
@Autowired
EmailOutboxServiceImpl emailOutboxService;
@Autowired
EmailDraftboxServiceImpl emailDraftboxService;
@RabbitHandler
public void consumer(MessageStruct messageStruct, Message message, Channel channel) throws Exception {
// 如果手动ACK,消息会被监听消费,但是消息在队列中依旧存在,如果 未配置 acknowledge-mode 默认是会在消费完毕后自动ACK掉
final long deliveryTag = message.getMessageProperties().getDeliveryTag();
try {
EmailDraftbox emailDraftbox = emailDraftboxService.getById(messageStruct.getDataId());
//如果在等待，就执行
if (emailDraftbox != null ) {
if(emailDraftbox.getIsWaiting() == 1 && emailDraftbox.getIsDeleted() == 0){
//如果此时的时间小于发送时间就执行，容错一秒,可以改为消息体里的时间做判断
if (emailDraftbox.getSendingTime().getTime() < new Date().getTime() + 1000) {
//修改为已发送
emailDraftbox.setIsWaiting(0);
emailDraftbox.setBelong(1);
emailDraftboxService.updateById(emailDraftbox);
//收件箱保存
emailOutboxService.sendMsg(emailDraftbox);
}
}
}
channel.basicAck(deliveryTag, false);
} catch (IOException e) {
try {
// 处理失败,重新压入MQ
channel.basicRecover();
} catch (IOException e1) {
e1.printStackTrace();
}
}
}
}
上面的案例模拟了一次业务操作，当生产者按照规定好的路由键发送消息后，消费者监听的队列就会获取消息然后消费。我在消费者消费消息的业务操作做了try catch，如果消费失败就会重新压入栈里。上文也说到了，我之前已经将消息提交的模式改为了手动，所以，如果我不提交消息，队列就会认为你没消费。
这种简单案例生产使用肯定是有问题的，我们的业务消息如果一直压回栈里，会不会出现消息积压的情况呢。
为了应对，我们可以采用死信队列。死信队列是为了处理业务队列没有正常消费消息的队列，来源来自三种场景：
- 消息的消息头设置了过期时间，到达时间后直接投入死信队列；
- 消息到达了队列的最大长度，多余的消息被投入死信队列；
- 消息被消费者拒绝
在上一个案例上进行补充死信队列
配置
@Configuration
@Slf4j
public class RabbitMQConfig{
/**
* 邮箱交换机
*/
public static final String EMAIL_EXCHANGE = "email_exchange";
/**
* 邮箱队列
*/
public static final String EMAIL_QUEUE = "email_queue";
/**
* 邮箱路由key
*/
public static final String EMAIL_QUEUE_ROUTING_KEY = "email.#";
/**
* 死信交换机
*/
public static final String EMAIL_DEAD_LETTER_EXCHANGE = "email_dead_letter_exchange";
/**
* 死信队列 routingKey
*/
public static final String EMAIL_DEAD_LETTER_QUEUE_ROUTING_KEY = "email_dead_letter_queue_routing_key";
/**
* 死信队列
*/
public static final String EMAIL_DEAD_LETTER_QUEUE = "email_dead_letter_queue";
@Primary
@Bean(name = "rabbitTemplateEmail")
public RabbitTemplate rabbitTemplate(ConnectionFactory connectionFactory) {
RabbitTemplate serverRabbitTemplate = new RabbitTemplate(connectionFactory);
//确认消息是否到达MQ
serverRabbitTemplate.setConfirmCallback((correlationData, ack, cause) -> {
String correlationDataId = correlationData.getId();
if (ack) {
//ACK
log.debug("消息[{}]投递成功", correlationDataId);
} else {
System.out.println("消息[{}]投递失败,cause:{}");
}
});
//配置return监听处理，消息无法路由到queue触发，此处只打印了相关信息，具体逻辑需自己实现
serverRabbitTemplate.setReturnCallback((message, replyCode, replyText, exchange, routingKey) -> {
System.out.println("=============returnCallback触发。消息路由到queue失败===========");
System.out.println("msg=" + new String(message.getBody()));
System.out.println("replyCode=" + replyCode);
System.out.println("replyText=" + replyText);
System.out.println("exchange=" + exchange);
System.out.println("routingKey=" + routingKey);
});
return serverRabbitTemplate;
}
/**
* 创建死信交换机
*/
@Bean("emailDeadLetterExchange")
public Exchange emailDeadLetterExchange() {
return new TopicExchange(EMAIL_DEAD_LETTER_EXCHANGE, true, false);
}
/**
* 创建死信队列
*/
@Bean("emailDeadLetterQueue")
public Queue emailDeadLetterQueue() {
return QueueBuilder.durable(EMAIL_DEAD_LETTER_QUEUE).build();
}
/**
* 绑定死信交换机和死信队列
*/
@Bean("emailDeadLetterBinding")
public Binding emailDeadLetterBinding(@Qualifier("emailDeadLetterQueue") Queue queue, @Qualifier("emailDeadLetterExchange")Exchange exchange) {
return BindingBuilder.bind(queue).to(exchange).with(EMAIL_DEAD_LETTER_QUEUE_ROUTING_KEY).noargs();
}
/**
* 创建邮箱交换机
*/
@Bean("emailExchange")
public Exchange emailExchange() {
return new TopicExchange(EMAIL_EXCHANGE, true, false);
}
/**
* 创建邮箱队列
*/
@Bean("emailQueue")
public Queue emailQueue() {
Map<String, Object> args = new HashMap<>();
// 设置死信交换机和死信队列的绑定键
Map<String, Object> args = new HashMap<>();
args.put("x-dead-letter-exchange", EMAIL_DEAD_LETTER_EXCHANGE);
args.put("x-dead-letter-routing-key", EMAIL_DEAD_LETTER_QUEUE_ROUTING_KEY);
// 设置消息过期时间
args.put("x-message-ttl", 10000); // 消息过期时间为10秒
return QueueBuilder.durable(EMAIL_QUEUE).withArguments(args).build();
}
/**
* 绑定邮箱交换机和队列
*/
@Bean("emailBinding")
public Binding emailBinding(@Qualifier("emailQueue") Queue queue, @Qualifier("emailExchange")Exchange exchange) {
return BindingBuilder.bind(queue).to(exchange).with(EMAIL_QUEUE_ROUTING_KEY).noargs();
}
}
发送者不用处理，消费者需要增加一个死信队列消费者
@Component
public class DeadLetterConsumer {
@RabbitListener(queues = RabbitMqConstant.EMAIL_DEAD_LETTER_QUEUE)
public void consume(String message) {
System.out.println("Received dead letter: " + message);
}
}
其他应用
定时发布
邮件的定时发布功能，大家会怎么实现？比如我新建一个邮件，设置三十分钟后发送。
rabbitMq提供了一个延时插件，可以实现延时发送功能。但是非高可用，以下是原文：
Delayed messages are stored in a Mnesia table (also see Limitations below) with a single disk replica on the current node. They will survive a node restart. While timer(s) that triggered scheduled delivery are not persisted, it will be re-initialised during plugin activation on node start. Obviously, only having one copy of a scheduled message in a cluster means that losing that node or disabling the plugin on it will lose the messages residing on that node.
配置
@Configuration
@Slf4j
public class RabbitMQConfig{
@Primary
@Bean(name = "rabbitTemplateEmail")
public RabbitTemplate rabbitTemplate(ConnectionFactory connectionFactory) {
RabbitTemplate serverRabbitTemplate = new RabbitTemplate(connectionFactory);
//确认消息是否到达MQ
serverRabbitTemplate.setConfirmCallback((correlationData, ack, cause) -> {
String correlationDataId = correlationData.getId();
if (ack) {
//ACK
log.debug("消息[{}]投递成功", correlationDataId);
} else {
System.out.println("消息[{}]投递失败,cause:{}");
}
});
//配置return监听处理，消息无法路由到queue触发，此处只打印了相关信息，具体逻辑需自己实现
serverRabbitTemplate.setReturnCallback((message, replyCode, replyText, exchange, routingKey) -> {
System.out.println("=============returnCallback触发。消息路由到queue失败===========");
System.out.println("msg=" + new String(message.getBody()));
System.out.println("replyCode=" + replyCode);
System.out.println("replyText=" + replyText);
System.out.println("exchange=" + exchange);
System.out.println("routingKey=" + routingKey);
});
return serverRabbitTemplate;
}
//使用延时插件实现延时队列
/**
* 声明延迟队列插件-交换机
* @return
*/
@Bean("emailDelayPluginExchange")
public DirectExchange delayPluginExchange(){
DirectExchange exchange = new DirectExchange(RabbitMqConstant.EMAIL_DELAY_PLUGIN_EXCHANGE);
//设置延迟
exchange.setDelayed(true);
return exchange;
}
/**
* 声明延迟队列插件-队列
* @return
*/
@Bean("emailDelayPluginQueue")
public Queue delayPluginQueue(){
return new Queue(RabbitMqConstant.EMAIL_DELAY_PLUGIN_QUEUE);
}
/**
* 声明绑定关系
* @param directExchange
* @param queue
* @return
*/
@Bean
public Binding delayPluginBinding(@Qualifier("emailDelayPluginExchange") DirectExchange directExchange, @Qualifier("emailDelayPluginQueue") Queue queue){
return BindingBuilder.bind(queue).to(directExchange).with(RabbitMqConstant.EMAIL_DELAY_PLUGIN_QUEUE_ROUTING_KEY);
}
消息处理器
public class MessagePostProcessUtil {
/**
*
* @param mode 投递模式
* @param delay 延迟时间
* @return
*/
public static MessagePostProcessor getMessagePostProcessor(MessageDeliveryMode mode, Integer delay){
return message -> {
message.getMessageProperties().setDeliveryMode(mode);
message.getMessageProperties().setDelay(delay);
return message;
};
}
}
消息发送者
@Slf4j
public class EmailSendProducer {
private static RabbitTemplate rabbitTemplate;
private static RabbitTemplate getRabbitTemplate() {
if (rabbitTemplate == null) {
rabbitTemplate = SpringUtil.getBean("rabbitTemplateEmail", RabbitTemplate.class);
}
return rabbitTemplate;
}
public static void sendEmailDelayedStartMessage(EmailDraftbox emailDraftbox , Long time) {
//消息体
MessageStruct messageStruct = new MessageStruct();
messageStruct.setDataId(emailDraftbox.getId().toString());
messageStruct.setModule(MessageSendEnum.EMAIL_SEND.getModule());
messageStruct.setMessage(MessageSendEnum.EMAIL_SEND.getType());
messageStruct.setMessageCode(MessageSendEnum.EMAIL_SEND.getMessageCode());
messageStruct.setDelayTime(Integer.valueOf(time.toString()));
CorrelationData correlationData = new CorrelationData();
correlationData.setId(String.valueOf(emailDraftbox.getId()));
MessagePostProcessor processor = MessagePostProcessUtil.getMessagePostProcessor(MessageDeliveryMode.PERSISTENT, messageStruct.getDelayTime());
getRabbitTemplate().convertAndSend(RabbitMqConstant.EMAIL_DELAY_PLUGIN_EXCHANGE, RabbitMqConstant.EMAIL_DELAY_PLUGIN_QUEUE_ROUTING_KEY, messageStruct, processor,correlationData);
System.out.println("邮件发送定时消息已发送");
log.info("sendEmailDelayedStartMessage:消息发送，发送对象：" + JsonUtil.toJson(messageStruct) + ",发送时间：{}" + DateUtil.now());
}
}
延迟插件的实现是改良版的定时器实现的，使用erlang语言开发(看不懂),当消息量上升时，会出现CPU飙升，并且mq重启时会造成TTL重置。
优化方案：
- 控制消息量，将短期时间的延时置入mq，长期的延时入库，定时扫库将延时消息入mq
- 消息入库，实现消息的高可靠，防止延时时间重置
其实利用死信队列+TTL也可以实现一部分的延时功能，但是有个问题是绕不过的，就是队列是一个先进先出的数据结构，假如前一个消息的过期时间很长，就会导致消息一直不消费，整个延时功能就会全部失效，在特定的业务里可以使用这种方式实现延时效果，但是要创建超级多的队列，实现延时。
递归创建交换机队列
@Configuration
public class RabbitMqConfig implements CommandLineRunner {
private final String exchangeName = "EXCHANGE.TOPIC.SENDTASK.N";
private final String queueName = "QUEUE.DELAY.SENDTASK.N";
private final Integer baseLevel = 20;
private final String exchange = "EXCHANGE.DELAY.DIRECT.SENDTASK";
@Autowired
RabbitAdmin rabbitAdmin;
@Autowired
TopicExchange directExchange;
@Bean
RabbitAdmin initRabbitAdmin(ConnectionFactory connectionFactory) {
return new RabbitAdmin(connectionFactory);
}
/**
* Springboot加载完成后, 调用该方法创建延时队列和交换机
* 以 baseLevel = 3为例, 即Level 0 - Level 3, 最大支持 2^4=16秒延时
*/
@Override
public void run(String... args) {
createDelayQueue(baseLevel);
}
/**
* 首先创建根交换机, 即到期后转投的业务交换机,
* 如: SEND-TASK-Exchange, 该交换机类型必须为TopicExchange
*/
@Bean
TopicExchange initDirectExchange() {
return new TopicExchange(exchange, true, false);
}
/**
* 递归方法, 从根交换机上进行逐层创建队列和交换机并绑定关系
*/
public void createDelayQueue(Integer currentLevel) {
if (currentLevel == 0) {
// 递归退出条件, 创建Level-0队列, 该队列TTL值为1秒, 过期后将投递到根交换机中
Queue queue = QueueBuilder.durable(queueName + currentLevel)
.withArgument("x-message-ttl", 1000)
.withArgument("x-dead-letter-exchange", exchange)
.build();
// 创建Level-0交换机
TopicExchange topicExchange = new TopicExchange(exchangeName + currentLevel, true, false);
// 将Level-0交换机与Level-0队列绑定, 以下方法返回：*.*.*.1.#, 即延迟1秒的消息入Level-0队列
String key = MqUtil.getBindingRoutingKey(1, baseLevel, 1);
Binding binding = BindingBuilder.bind(queue).to(topicExchange).with(key);
// Level-0交换机与根交换机绑定, 以下方法返回：*.*.*.0.#, 即不满足Level-0交换机的说明延时时间为0, 直接入根交换机
String key1 = MqUtil.getBindingRoutingKey(1, baseLevel, 0);
Binding binding1 = BindingBuilder.bind(directExchange).to(topicExchange).with(key1);
rabbitAdmin.declareQueue(queue);
rabbitAdmin.declareExchange(topicExchange);
rabbitAdmin.declareBinding(binding);
rabbitAdmin.declareBinding(binding1);
return;
}
// 递归开始, 若想创建第Level层交换机和队列, 必须先创建其下一层的交换机, 因为Level层的队列和交换机要与下层的交换机绑定
createDelayQueue(currentLevel - 1);
// 创建第一层队列, 请注意：当N>21时, 此处会越界成为负值从而报错,可以使用加L优化范围
Queue queue = QueueBuilder.durable(queueName + currentLevel)
.withArgument("x-message-ttl", (1 << currentLevel) * 1000)
.withArgument("x-dead-letter-exchange", exchangeName + (currentLevel - 1))
.build();
// 创建第一层交换机
TopicExchange topicExchange = new TopicExchange(exchangeName + currentLevel, true, false);
// 将同层的交换机与队列绑定
String key = MqUtil.getBindingRoutingKey(1 << currentLevel, baseLevel, 1);
Binding binding = BindingBuilder.bind(queue).to(topicExchange).with(key);
// 将当前层交换机与下层交换机绑定
String key1 = MqUtil.getBindingRoutingKey(1 << currentLevel, baseLevel, 0);
TopicExchange nextTopicExchange = new TopicExchange(exchangeName + (currentLevel - 1), true, false);
Binding binding1 = BindingBuilder.bind(nextTopicExchange).to(topicExchange).with(key1);
rabbitAdmin.declareQueue(queue);
rabbitAdmin.declareExchange(topicExchange);
rabbitAdmin.declareBinding(binding);
rabbitAdmin.declareBinding(binding1);
}
}
public class MqUtil {
// 调用该方法getMessageRoutingKey(10, 3)。延迟10秒，等级为3
// 返回 1.0.1.0.d
public static String getMessageRoutingKey(int delay, int set) {
String binString = Integer.toBinaryString(delay);
StringBuffer sb = new StringBuffer();
for (int i = 0; i < set + 1 - binString.length(); i++) {
sb.append("0");
}
binString = sb.toString() + binString;
StringBuilder sb1 = new StringBuilder();
for (int j = 0; j < binString.length(); j++) {
sb1.append(binString.charAt(j));
sb1.append(".");
}
sb1.append("d");
return sb1.toString();
}
// 调用该方法getBindingRoutingKey(1, 3, 1)。延迟1秒的队列，入队时的路由键为*.*.*.1.#
// 调用该方法getBindingRoutingKey(1, 3, 1)。延时1秒的队列，不满足的路由键为*.*.*.0.#
public static String getBindingRoutingKey(int delay, int set, int val) {
String binString = Integer.toBinaryString(delay);
StringBuffer sb = new StringBuffer();
for (int i = 0; i < set - binString.length() + 1; i++) {
sb.append("*.");
}
binString = sb.toString() + val + ".#";
return binString.toString();
}
}
交换机
队列
消费者
@Slf4j
@Component
@RabbitListener(queues = "SEND-TASK")
@AllArgsConstructor
public class TestHandle {
@RabbitHandler
public void meetingDelayed(MessageStruct messageStruct, Message message, Channel channel) {
// 如果手动ACK,消息会被监听消费,但是消息在队列中依旧存在,如果 未配置 acknowledge-mode 默认是会在消费完毕后自动ACK掉
final long deliveryTag = message.getMessageProperties().getDeliveryTag();
try {
Date date = new Date();
channel.basicAck(deliveryTag, false);
System.out.println("定时测试完成发送时间：{}" + DateUtil.now());
} catch (IOException e) {
try {
// 处理失败,重新压入MQ
channel.basicRecover();
} catch (IOException e1) {
e1.printStackTrace();
}
}
}
}
消息发送
public static void sendMeetingDelayedMessage(Meeting meeting, int l) {
MessageStruct messageStruct = new MessageStruct();
messageStruct.setDataId(meeting.getId().toString());
messageStruct.setModule(MessageSendEnum.MEETING_SEND_DELAY_FINISH.getModule());
messageStruct.setMessage(MessageSendEnum.MEETING_SEND_DELAY_FINISH.getType());
messageStruct.setMessageCode(MessageSendEnum.MEETING_SEND_DELAY_FINISH.getMessageCode());
CorrelationData correlationData = new CorrelationData();
correlationData.setId(String.valueOf(meeting.getId()));
String messageRoutingKey = MqUtil.getMessageRoutingKey(l, 20);
getRabbitTemplate().convertAndSend("EXCHANGE.TOPIC.SENDTASK.N" + 20,messageRoutingKey,messageStruct,correlationData);
System.out.println("发布会议定时发布消息已发送");
log.info("sendMeetingDelayedMessage:消息发送，发送对象：" + JsonUtil.toJson(messageStruct) + ",发送时间：{}" + DateUtil.now());
}
主要思路就是倒金字塔式的死信队列，使用二进制路由键递归建立交换机队列，实现秒级的延时。
缺点很明显，要创建过多过多的队列交换机，也只能处理一段时间内的延时。
优点是能容纳大量的延迟消息，并且消耗CPU较少，是空间换时间的例子。
消息高可靠
生产使用MQ需要考虑的点就很多了，我对消息的发送接收做了封装，主要的思路就是发送方只考虑发送发信，MQ只考虑传信，消费方需要考虑读信然后根据信上的内容完成任务。
业务中，拿发邮件举例，发送方认为邮件发送出去了，对邮件发送的记录或者其他的业务就可以继续往下进行了，但是此时MQ传信，消费方读信的操作都还未进行呢，这就造成了事情只完成了一半的情况。
消息的高可靠是分布式事务和很多业务操作的基础，所以我来介绍下我的做法。
先摆代码
配置
@Slf4j
@Configuration(proxyBeanMethods = false)
public class RabbitMqConfiguration {
@Primary
@Bean(name = "rabbitTemplateEs")
public RabbitTemplate rabbitTemplate(ConnectionFactory connectionFactory) {
RabbitTemplate serverRabbitTemplate = new RabbitTemplate(connectionFactory);
//确认消息是否到达MQ
serverRabbitTemplate.setConfirmCallback((correlationData, ack, cause) -> {
String correlationDataId = correlationData.getId();
if (ack) {
//ACK
log.debug("消息[{}]投递成功", correlationDataId);
} else {
System.out.println("消息[{}]投递失败,cause:{}");
}
});
//配置return监听处理，消息无法路由到queue触发，此处只打印了相关信息，具体逻辑需自己实现
serverRabbitTemplate.setReturnCallback((message, replyCode, replyText, exchange, routingKey) -> {
System.out.println("=============returnCallback触发。消息路由到queue失败===========");
System.out.println("msg=" + new String(message.getBody()));
System.out.println("replyCode=" + replyCode);
System.out.println("replyText=" + replyText);
System.out.println("exchange=" + exchange);
System.out.println("routingKey=" + routingKey);
});
return serverRabbitTemplate;
}
/**
* 待办列表延迟重试-交换机
*
* @return
*/
@Bean("retryForTodoListExchange")
public DirectExchange retryForTodoListExchange() {
return ExchangeBuilder.directExchange(RabbitMqConstant.todoListRetryExchange)
.durable(true) //持久化
.delayed() //延迟
.build();
}
/**
* 待办列表延迟重试-队列
*
* @return
*/
@Bean("retryForTodoListQueue")
public Queue retryForTodoListQueue() {
Map<String, Object> args = new HashMap<>(2);
// 消息重新投递到业务交换机
args.put("x-dead-letter-exchange", RabbitMqConstant.todoListExchange);
// 消息在交换机上挂载投入队列后1s进入死信队列（即重新进入业务队列处理相应业务）
args.put("x-message-ttl", 1000);
return QueueBuilder.durable(RabbitMqConstant.todoListRetryQueue).withArguments(args).build();
}
/**
* 待办列表延迟重试-声明绑定关系
*
* @param directExchange
* @param queue
* @return
*/
@Bean
public Binding retryForTodoListBinding(@Qualifier("retryForTodoListExchange") DirectExchange directExchange, @Qualifier("retryForTodoListQueue") Queue queue) {
//和业务路由键绑定
return BindingBuilder.bind(queue).to(directExchange).with(RabbitMqConstant.todoListRoutingKey);
}
/**
* 待办列表-交换机
*/
@Bean("todoListExchange")
public DirectExchange todoListExchange() {
return ExchangeBuilder.directExchange(RabbitMqConstant.todoListExchange)
.durable(true) // 交换机持久化
.build();
}
/**
* 待办列表-队列
*/
@Bean("todoListQueue")
public Queue todoListQueue() {
//队列持久化，否则消息持久化无意义
return QueueBuilder.durable(RabbitMqConstant.todoListQueue).build();
}
/**
* 待办列表-绑定
*/
@Bean
public Binding retryBizBinding(@Qualifier("todoListExchange") DirectExchange exchange, @Qualifier("todoListQueue") Queue queue) {
return BindingBuilder.bind(queue).to(exchange).with(RabbitMqConstant.todoListRoutingKey);
}
}
常量
public class MqTaskConstant {
/**
* 发送状态-成功
*/
public static final Integer SEND_STATUS_SUCCESS = 1;
/**
* 发送状态-失败
*/
public static final Integer SEND_STATUS_FAIL = 0;
/**
* 执行状态-未执行
*/
public static final Integer EXECUTE_STATUS_NOT = 0;
/**
* 执行状态-成功
*/
public static final Integer EXECUTE_STATUS_SUCCESS = 1;
/**
* 执行状态-失败
*/
public static final Integer EXECUTE_STATUS_FAIL = 2;
/**
* 执行状态-异常
*/
public static final Integer EXECUTE_STATUS_EXCEPTION = 3;
/**
* 执行状态-重试
*/
public static final Integer EXECUTE_STATUS_RETRY = 4;
}
public interface RabbitMqConstant {
//代办任务队列
String todoListExchange = "todoList.exchange";
String todoListRoutingKey = "todoList.routingKey";
String todoListQueue = "todoList.queue";
//代办任务重试队列
String todoListRetryExchange = "todoList.retry.exchange";
String todoListRetryQueue = "todoList.retry.queue";
}
消息任务表实体类
@Data
@TableName("MQ_TASK")
@EqualsAndHashCode(callSuper = true)
@ApiModel(value = "MqTask对象", description = "消息任务表")
public class MqTask extends BaseEntity {
/**
* 业务数据ID--需确保唯一
*/
private String bizId;
/**
* 业务数据
*/
private String bizData;
/**
* 业务模块
*/
private String module;
/**
* 消息类型
*/
private String messageType;
/**
* 消息编码
*/
private String messageCode;
/**
* 发送状态 0 已发送 1 未发送（发送消息过程中发生异常）
*/
private Integer sendStatus;
/**
* 执行状态 0 未执行 1 执行成功 2 执行失败 3 执行异常 4 重试
*/
private Integer executeStatus;
}
消息任务表的增删改查略
发送工具类
@Slf4j
public class SendMqToEsUtil {
/**
* 发送状态-成功
*/
public static final Integer SEND_STATUS_SUCCESS = 1;
/**
* 发送状态-失败
*/
public static final Integer SEND_STATUS_FAIL = 0;
/**
* 执行状态-未执行
*/
public static final Integer EXECUTE_STATUS_NOT = 0;
private static final RabbitTemplate rabbitTemplate;
private static final IMqTaskService mqTaskService;
//避免重复创建
static {
rabbitTemplate = SpringUtil.getBean("rabbitTemplateWorkflow", RabbitTemplate.class);
mqTaskService = SpringUtil.getBean(IMqTaskService.class);
}
/**
* 发送消息时 生成消息任务
* @param messageStruct
* @param mqTask
* @param sendStatus
* @return
*/
private static MqTask saveMqTask(MessageStruct messageStruct, @Nullable MqTask mqTask, Integer sendStatus) {
if (mqTask == null) {
mqTask = new MqTask();
}
//生成消息唯一id
long snowflakeNextId = IdUtil.getSnowflakeNextId();
mqTask.setId(snowflakeNextId);
//业务id
mqTask.setBizId(messageStruct.getDataId());
mqTask.setBizData(JsonUtil.toJson(messageStruct));
mqTask.setModule(messageStruct.getModule());
mqTask.setMessageType(messageStruct.getMessage());
mqTask.setMessageCode(messageStruct.getMessageCode());
mqTask.setExecuteStatus(EXECUTE_STATUS_NOT);
mqTask.setSendStatus(sendStatus);
mqTaskService.save(mqTask);
return mqTask;
}
/**
* 更改消息状态
*/
private static MqTask updateMqTask(MessageStruct messageStruct, @Nullable MqTask mqTask, Integer sendStatus) {
if (mqTask == null) {
mqTask = new MqTask();
}
mqTask.setSendStatus(sendStatus);
mqTaskService.updateById(mqTask);
return mqTask;
}
/**
* 发送消息通用方法
*
* @param messageStruct 消息体
* @param exchange 交换机名称
* @param routingKey 路由key
*/
private static void sendCommonMqMessage(MessageStruct messageStruct, String exchange, String routingKey) {
//记录消息
MqTask mqTask = saveMqTask(messageStruct, null, SEND_STATUS_SUCCESS);
try {
if (Func.isEmpty(mqTask) && Func.isEmpty(mqTask.getId())) {
throw new ServiceException("消息记录新增失败");
}
CorrelationData correlationData = new CorrelationData();
correlationData.setId(Func.toStr(mqTask.getId()));
//消息持久化
rabbitTemplate.convertAndSend(exchange, routingKey, messageStruct, new MessagePostProcessor() {
@Override
public Message postProcessMessage(Message message) throws AmqpException {
message.getMessageProperties().setDeliveryMode(MessageDeliveryMode.PERSISTENT);
return message;
}
}, correlationData);
log.info("消息发送，发送对象：{},发送时间：{}", JsonUtil.toJson(messageStruct), DateUtil.now());
} catch (Exception e) {
updateMqTask(messageStruct, mqTask, SEND_STATUS_FAIL);
}
}
/**
* 发送待办
*/
public static void sendTodoMessageForES(String messageId) {
MessageStruct messageStruct = new MessageStruct();
messageStruct.setDataId(messageId);
messageStruct.setModule(MessageSendEnum.TODO_LIST_ES_SAVE.getModule());
Map<String, Object> data = new HashMap<>();
data.put("taskId", messageId);
messageStruct.setVariables(data);
messageStruct.setMessage(MessageSendEnum.TODO_LIST_ES_SAVE.getType());
messageStruct.setMessageCode(MessageSendEnum.TODO_LIST_ES_SAVE.getMessageCode());
CorrelationData correlationData = new CorrelationData();
correlationData.setId(messageId);
sendCommonMqMessage(messageStruct, RabbitMqConstant.todoListExchange, RabbitMqConstant.todoListRoutingKey);
}
//发送失败确认
public static void confirmCallback(CorrelationData correlationData, boolean ack, String cause) {
if (!ack) {
// 如果消息没有被确认，可以根据 correlationData 查找对应的 MqTask 并更新状态
MqTask mqTask = mqTaskService.getById(correlationData.getId());
if (mqTask != null) {
updateMqTask(null, mqTask, SEND_STATUS_FAIL);
}
}
}
}
消费者抽象类
@Slf4j
@Component
public abstract class AbstractRetryMessageListener {
/**
* es服务的redis前缀
*/
protected String redis_prefix = "ES:MQ_REDIS_KEY:";
/**
* 延迟重试交换机
*/
protected String delayRetryExchange = "";
/**
* 路由key
*/
protected String bizRoutingKey = "";
/**
* 默认延时时间 1分钟 每次重试设置的延时时间为 重试次数 * 默认延时时间
*/
protected Integer defaultDelayTime = 1000 * 60;
protected Integer maxRetryNum = 5;
/**
* 消息记录
*/
protected IMqTaskService mqTaskService;
protected Class loggerClass;
@Autowired
protected RedisTemplate redisTemplate;
private IMqTaskService getMqTaskService() {
if (mqTaskService == null) {
mqTaskService = SpringUtil.getBean(IMqTaskService.class);
}
return mqTaskService;
}
private Logger getLogger() {
if (loggerClass == null) {
return LoggerFactory.getLogger(this.getClass());
} else {
return LoggerFactory.getLogger(loggerClass);
}
}
//消费消息
public void onMessage(MessageStruct messageStruct, Message message, Channel channel) throws Exception {
//重试次数
int maxRetryTimes = this.maxRetryNum;
//消息id
String messageId = messageStruct.getDataId();
//消息幂等,不继续消费
boolean needConsume = redisTemplate.opsForValue().setIfAbsent(redis_prefix + messageId, "success");
if (!needConsume) {
this.basicAck(channel, message, messageId, Boolean.FALSE);
getLogger().error("MQ重复消费 , messageId: {}, errorMsg: {}", messageId);
return;
}
boolean processResult = false;
try {
getLogger().info("接收到 " + RabbitMqConstant.todoListQueue + " 队列消息 : {}", messageId);
processResult = onMsg(messageStruct, message, messageId);
} catch (Exception e) {
getMqTaskService().updateExecuteStatus(Func.toLong(messageId), MqTaskConstant.EXECUTE_STATUS_EXCEPTION);
String errorMsg = e.getMessage() == null ? "" : e.getMessage();
getLogger().error("MQ消费异常 , messageId: {}, errorMsg: {}", messageId, errorMsg, e);
} finally {
if (processResult) {
// 消息消费成功，手动 ack
this.basicAck(channel, message, messageId, Boolean.TRUE);
getLogger().info(RabbitMqConstant.todoListQueue + " 队列消息 : {} 消费成功", messageId);
} else {
int retryNum = this.getRetryNum(message);
if (retryNum <= maxRetryTimes) {
messageStruct.setDelayTime(defaultDelayTime * retryNum);
getLogger().info("当前消息进行第 {} 次重试,重试设置延时时间{} messageId: {},messageDeliveryTag{}, messageContent{}", retryNum, defaultDelayTime * retryNum, messageId, message.getMessageProperties().getDeliveryTag(), messageStruct.getMessage());
// 消息重新投递到队列中进行二次消费
this.basicPublish(messageStruct, channel, message, messageId);
this.basicAck(channel, message, messageId, Boolean.FALSE);
// 延时时间内不删除redisKey
redisTemplate.expire(redis_prefix + messageId, defaultDelayTime * retryNum, TimeUnit.MILLISECONDS);
} else {
getLogger().error("当前消息体重试操作超过 {} 次, 放弃重试! messageStructContent{},messageId: {}, retryNum: {}", maxRetryTimes, messageStruct.getMessage(), messageId, retryNum);
// 消息消费失败
getMqTaskService().updateExecuteStatus(Func.toLong(messageId), MqTaskConstant.EXECUTE_STATUS_FAIL);
this.basicAck(channel, message, messageId, Boolean.FALSE);
// 消息重试失败，则先删除redisKey,便于重试
redisTemplate.delete(redis_prefix + messageId);
}
}
}
}
/**
* 手动ack
*/
private void basicAck(Channel channel, Message message, String correlationId, Boolean isCallBack) throws Exception {
// 手动ack但是不入队
channel.basicAck(message.getMessageProperties().getDeliveryTag(), false);
// 消费成功就回调
if (isCallBack) {
callback(correlationId);
}
}
/**
* 消息重新投递到队列中去
*/
private void basicPublish(MessageStruct messageStruct, Channel channel, Message message, String correlationId) throws Exception {
// 处于重试状态
getMqTaskService().updateExecuteStatus(Func.toLong(correlationId), MqTaskConstant.EXECUTE_STATUS_RETRY);
MessagePropertiesConverter messagePropertiesConverter = new DefaultMessagePropertiesConverter();
MessageProperties messageProperties = message.getMessageProperties();
// 设置过期时间
messageProperties.setDelay(messageStruct.getDelayTime());
// 设置投递模式
messageProperties.setDeliveryMode(MessageDeliveryMode.PERSISTENT);
AMQP.BasicProperties basicProperties = messagePropertiesConverter.fromMessageProperties(messageProperties, "utf-8");
channel.basicPublish(delayRetryExchange, bizRoutingKey, basicProperties, message.getBody());
}
/**
* 获取消息重试次数
*/
private int getRetryNum(Message message) {
int retryNum = 1;
try {
Map<String, Object> headers = message.getMessageProperties().getHeaders();
if (headers == null || !headers.containsKey("x-death")) {
return retryNum;
}
List<Map<String, Object>> deathList = (List<Map<String, Object>>) headers.get("x-death");
if (deathList == null || deathList.size() == 0) {
return retryNum;
}
Map<String, Object> deathMap = deathList.get(0);
// 获取重试次数
Long deathCount = (Long) deathMap.get("count");
if (deathCount == null || deathCount == 0) {
return retryNum;
}
return deathCount.intValue() + 1;
} catch (Exception e) {
String messageId = message.getMessageProperties().getMessageId();
getLogger().error("获取消息重试次数异常, messageId: {}", messageId, e);
//获取重试次数失败，默认重试次数为最大次数+1，就认为重试失败
return this.maxRetryNum + 1;
}
}
/**
* 消息业务处理
*/
protected abstract boolean onMsg(MessageStruct messageStruct, Message message, String correlationId);
/**
* 获取消息重试次数
*/
protected abstract void getMaxRetryNum();
/**
* 设置消息重试交换机和路由键
*/
protected abstract void setRetryExchangeAndRoutingKey();
/**
* 设置消息重试时间
*/
protected abstract void setRetryTime();
/**
* 回调消息消费成功
*/
private Boolean callback(String correlationId) {
// 消息消费成功更改消息状态
getMqTaskService().updateExecuteStatus(Func.toLong(correlationId), MqTaskConstant.EXECUTE_STATUS_SUCCESS);
// 消息消费成功给redis的key设置过期时间，避免资源占用
redisTemplate.expire(redis_prefix + correlationId, 10 * 60, TimeUnit.SECONDS);
return true;
}
}
业务消费者
@Slf4j
@Component
@AllArgsConstructor
@RabbitListener(queues = RabbitMqConstant.todoListQueue)
public class TodoListConsumer extends AbstractRetryMessageListener {
@RabbitHandler
public void receiveMessage(MessageStruct messageStruct, Message message, Channel channel) throws Exception {
//设置最大重试次数
getMaxRetryNum();
//设置重试交换机和路由键
setRetryExchangeAndRoutingKey();
//设置重试时间
setRetryTime();
//执行
super.onMessage(messageStruct, message, channel);
}
/**
* 业务消息消费
*
* @param messageStruct
* @param message
* @param correlationId
* @return
*/
@Override
protected boolean onMsg(MessageStruct messageStruct, Message message, String correlationId) {
String msgBody = new String(message.getBody(), StandardCharsets.UTF_8);
throw new ServiceException("mq业务异常");
// return true;
}
@Override
protected void getMaxRetryNum() {
this.maxRetryNum = 3;
}
@Override
protected void setRetryExchangeAndRoutingKey() {
this.delayRetryExchange = RabbitMqConstant.todoListRetryExchange;
this.bizRoutingKey = RabbitMqConstant.todoListRoutingKey;
}
@Override
protected void setRetryTime() {
this.defaultDelayTime = 1000 * 10;
}
}
讲一下我的思路，和问题解决
基础搭建
消息体，我把消息发送提炼成统一的消息体，将业务消息放入variables进行发送，其余dataId消息id、module发送者服务、message发送者业务、delayTime延迟时间 、messageCode业务编码都根据常量类设置好。
MQTASK表存在发送方的数据库中，作为业务的发起者，也是最需要关心结果的，在发送时进行了消息的入库，保存了消息内容，初始化了发送状态和执行状态
常量类保存了消息的路由队列信息、业务模块编码、消息状态字典等，置于公共服务中。
配置
保留了消息确认的业务口，当消息投递失败时可以选择回滚业务或者改变消息的发送状态，等待定时的发送重试
serverRabbitTemplate.setConfirmCallback((correlationData, ack, cause) -> {
String correlationDataId = correlationData.getId();
if (ack) {
//ACK
log.debug("消息[{}]投递成功", correlationDataId);
} else {
System.out.println("消息[{}]投递失败,cause:{}");
}
});
在队列创建和交换机创建时都使用了Builder创建，并且设置了持久化
QueueBuilder.durable(RabbitMqConstant.todoListRetryQueue).withArguments(args).build();
ExchangeBuilder.directExchange(RabbitMqConstant.todoListRetryExchange)
.durable(true) //持久化
.delayed() //延迟
.build();
BindingBuilder.bind(queue).to(directExchange).with(RabbitMqConstant.todoListRoutingKey);
将死信队列的路由绑定给业务队列，并且不设置死信队列的消费者，从而实现延时重试机制
发送方
发送方除了完成消息入库之外，就是实现了消息确认机制
当发送失败时，调用方法修改发送状态记录发送失败原因
//发送失败确认
public static void confirmCallback(CorrelationData correlationData, boolean ack, String cause) {
if (!ack) {
// 如果消息没有被确认，可以根据 correlationData 查找对应的 MqTask 并更新状态
MqTask mqTask = mqTaskService.getById(correlationData.getId());
if (mqTask != null) {
updateMqTask(null, mqTask, SEND_STATUS_FAIL);
}
}
}
发送消息时，生成唯一的消息id，用此唯一键作为重复消息的判定
private static void sendCommonMqMessage(MessageStruct messageStruct, String exchange, String routingKey) {
//记录消息
MqTask mqTask = saveMqTask(messageStruct, null, SEND_STATUS_SUCCESS);
try {
if (Func.isEmpty(mqTask) && Func.isEmpty(mqTask.getId())) {
throw new ServiceException("消息记录新增失败");
}
CorrelationData correlationData = new CorrelationData();
correlationData.setId(Func.toStr(mqTask.getId()));
//消息持久化
rabbitTemplate.convertAndSend(exchange, routingKey, messageStruct, new MessagePostProcessor() {
@Override
public Message postProcessMessage(Message message) throws AmqpException {
message.getMessageProperties().setDeliveryMode(MessageDeliveryMode.PERSISTENT);
return message;
}
}, correlationData);
log.info("消息发送，发送对象：{},发送时间：{}", JsonUtil.toJson(messageStruct), DateUtil.now());
} catch (Exception e) {
updateMqTask(messageStruct, mqTask, SEND_STATUS_FAIL);
}
}
消费方
使用死信队列+延迟递增的方法去实现消费重试，将消费失败的消息投递回死信队列
private void basicPublish(MessageStruct messageStruct, Channel channel, Message message, String correlationId) throws Exception {
// 处于重试状态
getMqTaskService().updateExecuteStatus(Func.toLong(correlationId), MqTaskConstant.EXECUTE_STATUS_RETRY);
MessagePropertiesConverter messagePropertiesConverter = new DefaultMessagePropertiesConverter();
MessageProperties messageProperties = message.getMessageProperties();
// 设置过期时间
messageProperties.setDelay(messageStruct.getDelayTime());
// 设置投递模式
messageProperties.setDeliveryMode(MessageDeliveryMode.PERSISTENT);
AMQP.BasicProperties basicProperties = messagePropertiesConverter.fromMessageProperties(messageProperties, "utf-8");
channel.basicPublish(delayRetryExchange, bizRoutingKey, basicProperties, message.getBody());
}
获取重试次数是根据消息头的属性实现的，rabbbitmq消息进入死信队列会增加属性
Map<String, Object> headers = message.getMessageProperties().getHeaders();
消息重复消费是使用redis的senNx方法实现的
boolean needConsume = redisTemplate.opsForValue().setIfAbsent(redis_prefix + messageId, "success");
// 延时时间内不删除redisKey
redisTemplate.expire(redis_prefix + messageId, defaultDelayTime * retryNum, TimeUnit.MILLISECONDS);
这里处于重试状态的消息，在redis的key过期后还是存在有重复消费的风险，也有解决方案就是设置共享锁实现，但是这里的过期时间已经是毫秒级了，我觉得没必要考虑。其实重复消费也可以使用数据库唯一键实现，这里不做赘述了。
到这里就考虑了代码层面的消息可靠消费问题了，搭建集群保证MQ的高可用也不做说明了。
