携手创作，共同成长！这是我参与「掘金日新计划 · 8 月更文挑战」的第9天，点击查看活动详情
上节我们说到了消费消息中的几个校验，接下来继续看看消费消息接下来的流程
1 消费消息
1.2 封装拉取请求和拉取后的回调对象PullCallBack
拉取过程代码如下,重点是我加注释的那两行。若是拉取到了消息，则更新到本地缓存队列中。再提交给消费消息的服务。
case FOUND:
long prevRequestOffset = pullRequest.getNextOffset();
pullRequest.setNextOffset(pullResult.getNextBeginOffset());
long pullRT = System.currentTimeMillis() - beginTimestamp;
DefaultMQPushConsumerImpl.this.getConsumerStatsManager().incPullRT(pullRequest.getConsumerGroup(),
pullRequest.getMessageQueue().getTopic(), pullRT);
long firstMsgOffset = Long.MAX_VALUE;
if (pullResult.getMsgFoundList() == null || pullResult.getMsgFoundList().isEmpty()) {
DefaultMQPushConsumerImpl.this.executePullRequestImmediately(pullRequest);
} else {
firstMsgOffset = pullResult.getMsgFoundList().get(0).getQueueOffset();
DefaultMQPushConsumerImpl.this.getConsumerStatsManager().incPullTPS(pullRequest.getConsumerGroup(),
pullRequest.getMessageQueue().getTopic(), pullResult.getMsgFoundList().size());
//如果有拉取到消息，那么在这里将消息保存到对应的本地缓存队列ProcessQueue中
boolean dispatchToConsume = processQueue.putMessage(pullResult.getMsgFoundList());
//在这里将拉取到的消息提交给ConsumerMessageService服务。
DefaultMQPushConsumerImpl.this.consumeMessageService.submitConsumeRequest(
pullResult.getMsgFoundList(),
processQueue,
pullRequest.getMessageQueue(),
dispatchToConsume);
}
break;
ConsumeMessageService是一个通用的消费服务接口，有两个实现类ConsumeMessageConcurrentlyService(并发消费)和ConsumeMessageOrderlyService（顺序消费）
ConsumeMessageService的核心代码如下：
public interface ConsumeMessageService {
//启动服务
void start();
//关闭服务
void shutdown(long awaitTerminateMillis);
//更新消费线程池的核心线程数
void updateCorePoolSize(int corePoolSize);
//增加一个消费线程池的核心线程数
void incCorePoolSize();
//减少一个消费线程池的核心线程数
void decCorePoolSize();
//获取一个消费线程池的核心线程数
int getCorePoolSize();
//这个方法用于消费已经消费过一次的消息
ConsumeMessageDirectlyResult consumeMessageDirectly(final MessageExt msg, final String brokerName);
//将消息封装成线程池任务，提交给消费服务，消费服务再将消息传递给业务服务进行处理
void submitConsumeRequest(
final List<MessageExt> msgs,
final ProcessQueue processQueue,
final MessageQueue messageQueue,
final boolean dispathToConsume);
void submitPopConsumeRequest(
final List<MessageExt> msgs,
final PopProcessQueue processQueue,
final MessageQueue messageQueue);
}
这里就牵扯两个概念: 消息分发和消费消息
1.2.1 消息分发
ConsumeMessageService通过submitConsumeRequest方法接收到消息消费任务之后，将消息按照固定数量封装成多个ConsumeRequest对象，并发送到消费线程池，等待业务消费。ConsumeMessageOrderlyService会将Pull的所有消息放在另一个本地队列中，然后提交一个ConsumeRequest到消费线程池。
1.2.2 消费消息
消费的逻辑在它的两个实现类中，我们说一下并发消费的过程
@Override
public void run() {
MessageListenerConcurrently listener = ConsumeMessageConcurrentlyService.this.messageListener;
ConsumeConcurrentlyContext context = new ConsumeConcurrentlyContext(messageQueue);
ConsumeConcurrentlyStatus status = null;
defaultMQPushConsumerImpl.tryResetPopRetryTopic(msgs, consumerGroup);
//预处理重试队列的消息
defaultMQPushConsumerImpl.resetRetryAndNamespace(msgs, defaultMQPushConsumer.getConsumerGroup());
ConsumeMessageContext consumeMessageContext = null;
//消费消息之前的hook
if (ConsumeMessageConcurrentlyService.this.defaultMQPushConsumerImpl.hasHook()) {
...
ConsumeMessageConcurrentlyService.this.defaultMQPushConsumerImpl.executeHookBefore(consumeMessageContext);
}
...
try {
...
//处理消费回调
status = listener.consumeMessage(Collections.unmodifiableList(msgs), context);
} catch (Throwable e) {
...
hasException = true;
}
long consumeRT = System.currentTimeMillis() - beginTimestamp;
...
//消费执行之后的hook
if (ConsumeMessageConcurrentlyService.this.defaultMQPushConsumerImpl.hasHook()) {
consumeMessageContext.setStatus(status.toString());
consumeMessageContext.setSuccess(ConsumeConcurrentlyStatus.CONSUME_SUCCESS == status);
ConsumeMessageConcurrentlyService.this.defaultMQPushConsumerImpl.executeHookAfter(consumeMessageContext);
}
...
//处理消费结果
if (!processQueue.isDropped()) {
ConsumeMessageConcurrentlyService.this.processConsumeResult(status, context, this);
} else {
...
}
}
这里可以分为四步
1.2.2.1 预处理
执行消费前的hook和重试消息预处理
消费前的hook是消费前的消息预处理，包含格式校验等。
重试消息预处理：若是消费的消息来自重试队列，这里相对老版本有所更改，之前是将Topic的名称重置为原来的Topic名，并不会使用重试Topic名。目前新版本是将topic名称更改为重试topic名称。
重试消息预处理的代码如下：
public void resetRetryAndNamespace(final List<MessageExt> msgs, String consumerGroup) {
final String groupTopic = MixAll.getRetryTopic(consumerGroup);
for (MessageExt msg : msgs) {
String retryTopic = msg.getProperty(MessageConst.PROPERTY_RETRY_TOPIC);
if (retryTopic != null && groupTopic.equals(msg.getTopic())) {
msg.setTopic(retryTopic);
}
if (StringUtils.isNotEmpty(this.defaultMQPushConsumer.getNamespace())) {
msg.setTopic(NamespaceUtil.withoutNamespace(msg.getTopic(), this.defaultMQPushConsumer.getNamespace()));
}
}
}
1.2.2.2 消费回调
//设置消息开始时间为当前时间
long beginTimestamp = System.currentTimeMillis();
...
try {
if (msgs != null && !msgs.isEmpty()) {
for (MessageExt msg : msgs) {
//将消息列表转为不可修改的List
MessageAccessor.setConsumeStartTimeStamp(msg, String.valueOf(System.currentTimeMillis()));
}
}
//消费回调
//将消息传递给用户编写的业务消费代码进行处理
status = listener.consumeMessage(Collections.unmodifiableList(msgs), context);
} catch (Throwable e) {
...
}
long consumeRT = System.currentTimeMillis() - beginTimestamp;
1.2.2.3 消费结果统计
统计消费耗时
long consumeRT = System.currentTimeMillis() - beginTimestamp;
消费后的hook可以统计与我们业务相关的一些指标
if (ConsumeMessageConcurrentlyService.this.defaultMQPushConsumerImpl.hasHook()) {
consumeMessageContext.setStatus(status.toString());
consumeMessageContext.setSuccess(ConsumeConcurrentlyStatus.CONSUME_SUCCESS == status);
ConsumeMessageConcurrentlyService.this.defaultMQPushConsumerImpl.executeHookAfter(consumeMessageContext);
}
1.2.2.4 消费结果处理
包含消费指标统计，消费重试处理，消费位点处理。
消费指标统计是对消费成功和失败的TPS的统计
switch (status) {
case CONSUME_SUCCESS:
if (ackIndex >= consumeRequest.getMsgs().size()) {
ackIndex = consumeRequest.getMsgs().size() - 1;
}
int ok = ackIndex + 1;
int failed = consumeRequest.getMsgs().size() - ok;
this.getConsumerStatsManager().incConsumeOKTPS(consumerGroup, consumeRequest.getMessageQueue().getTopic(), ok);
this.getConsumerStatsManager().incConsumeFailedTPS(consumerGroup, consumeRequest.getMessageQueue().getTopic(), failed);
break;
case RECONSUME_LATER:
ackIndex = -1;
this.getConsumerStatsManager().incConsumeFailedTPS(consumerGroup, consumeRequest.getMessageQueue().getTopic(),
consumeRequest.getMsgs().size());
break;
default:
break;
}
消费重试处理主要将消费重试次数加1
switch (this.defaultMQPushConsumer.getMessageModel()) {
case BROADCASTING:
for (int i = ackIndex + 1; i < consumeRequest.getMsgs().size(); i++) {
MessageExt msg = consumeRequest.getMsgs().get(i);
log.warn("BROADCASTING, the message consume failed, drop it, {}", msg.toString());
}
break;
case CLUSTERING:
List<MessageExt> msgBackFailed = new ArrayList<MessageExt>(consumeRequest.getMsgs().size());
for (int i = ackIndex + 1; i < consumeRequest.getMsgs().size(); i++) {
MessageExt msg = consumeRequest.getMsgs().get(i);
boolean result = this.sendMessageBack(msg, context);
if (!result) {
msg.setReconsumeTimes(msg.getReconsumeTimes() + 1);
msgBackFailed.add(msg);
}
}
if (!msgBackFailed.isEmpty()) {
consumeRequest.getMsgs().removeAll(msgBackFailed);
this.submitConsumeRequestLater(msgBackFailed, consumeRequest.getProcessQueue(), consumeRequest.getMessageQueue());
}
break;
default:
break;
}
消费位点处理则根据消费结果更新消费位点记录
long offset = consumeRequest.getProcessQueue().removeMessage(consumeRequest.getMsgs());
if (offset >= 0 && !consumeRequest.getProcessQueue().isDropped()) {
this.defaultMQPushConsumerImpl.getOffsetStore().updateOffset(consumeRequest.getMessageQueue(), offset, true);
}
