欢迎关注公众号【sharedCode】致力于主流中间件的源码分析, 个人网站:www.shared-code.com/
官方示例
本文的示例是通过push的方式创建的消费者,push和pull的消费者在启动的流程上源码是差不多的
public static void main(String[] args) throws InterruptedException, MQClientException {
// 创建一个消费者对象
DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("please_rename_unique_group_name_4");
// 初始从哪个位置开始消费
consumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_FIRST_OFFSET);
// 订阅topic
consumer.subscribe("TopicTest", "*");
//注册消息监听器, 这里注册的是普通消息的
consumer.registerMessageListener(new MessageListenerConcurrently() {
@Override
public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs,
ConsumeConcurrentlyContext context) {
System.out.printf("%s Receive New Messages: %s %n", Thread.currentThread().getName(), msgs);
return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
}
});
//启动起来
consumer.start();
System.out.printf("Consumer Started.%n");
}
start
public synchronized void start() throws MQClientException {
switch (this.serviceState) {
case CREATE_JUST:
// 等待创建
log.info("the consumer [{}] start beginning. messageModel={}, isUnitMode={}", this.defaultMQPushConsumer.getConsumerGroup(),
this.defaultMQPushConsumer.getMessageModel(), this.defaultMQPushConsumer.isUnitMode());
// 重置启动状态
this.serviceState = ServiceState.START_FAILED;
// 检查配置
this.checkConfig();
this.copySubscription();
if (this.defaultMQPushConsumer.getMessageModel() == MessageModel.CLUSTERING) {
this.defaultMQPushConsumer.changeInstanceNameToPID();
}
// 获取MQClient工厂,用来创建MQClientInstance
this.mQClientFactory = MQClientManager.getInstance().getAndCreateMQClientInstance(this.defaultMQPushConsumer, this.rpcHook);
// 设置负载均衡器里面的消费组
this.rebalanceImpl.setConsumerGroup(this.defaultMQPushConsumer.getConsumerGroup());
// 消费模式
this.rebalanceImpl.setMessageModel(this.defaultMQPushConsumer.getMessageModel());
// Rebalance算法实现策略
this.rebalanceImpl.setAllocateMessageQueueStrategy(this.defaultMQPushConsumer
.getAllocateMessageQueueStrategy());
// mQClientFactory
this.rebalanceImpl.setmQClientFactory(this.mQClientFactory);
// 初始化获取消息的API工具类
this.pullAPIWrapper = new PullAPIWrapper(
mQClientFactory,
this.defaultMQPushConsumer.getConsumerGroup(), isUnitMode());
this.pullAPIWrapper.registerFilterMessageHook(filterMessageHookList);
// 判断消费者offsetStore是否为空(默认为空),getOffsetStore为过期方法,后期会删除掉
if (this.defaultMQPushConsumer.getOffsetStore() != null) {
this.offsetStore = this.defaultMQPushConsumer.getOffsetStore();
} else {
// 判断消费者的消息类型,有集群消费和广播消费 ,默认集群消费
switch (this.defaultMQPushConsumer.getMessageModel()) {
case BROADCASTING:// 广播消费
this.offsetStore = new LocalFileOffsetStore(this.mQClientFactory, this.defaultMQPushConsumer.getConsumerGroup());
break;
case CLUSTERING: // 集群消费
this.offsetStore = new RemoteBrokerOffsetStore(this.mQClientFactory, this.defaultMQPushConsumer.getConsumerGroup());
break;
default:
break;
}
this.defaultMQPushConsumer.setOffsetStore(this.offsetStore);
}
// 加载offset,集群消费存储则不需要, 广播消费需要
this.offsetStore.load();
// 判断消息监听器,是普通消息,还是顺序消息
if (this.getMessageListenerInner() instanceof MessageListenerOrderly) {
// 顺序消息
this.consumeOrderly = true;
this.consumeMessageService =
new ConsumeMessageOrderlyService(this, (MessageListenerOrderly) this.getMessageListenerInner());
} else if (this.getMessageListenerInner() instanceof MessageListenerConcurrently) {
// 普通消息
this.consumeOrderly = false;
this.consumeMessageService =
new ConsumeMessageConcurrentlyService(this, (MessageListenerConcurrently) this.getMessageListenerInner());
}
//启动消息监听处理服务类
this.consumeMessageService.start();
// 将消费组注册到本地
boolean registerOK = mQClientFactory.registerConsumer(this.defaultMQPushConsumer.getConsumerGroup(), this);
if (!registerOK) {
this.serviceState = ServiceState.CREATE_JUST;
this.consumeMessageService.shutdown();
throw new MQClientException("The consumer group[" + this.defaultMQPushConsumer.getConsumerGroup()
+ "] has been created before, specify another name please." + FAQUrl.suggestTodo(FAQUrl.GROUP_NAME_DUPLICATE_URL),
null);
}
// 启动MQClient
mQClientFactory.start();
log.info("the consumer [{}] start OK.", this.defaultMQPushConsumer.getConsumerGroup());
this.serviceState = ServiceState.RUNNING;
break;
case RUNNING:
case START_FAILED:
case SHUTDOWN_ALREADY:
throw new MQClientException("The PushConsumer service state not OK, maybe started once, "
+ this.serviceState
+ FAQUrl.suggestTodo(FAQUrl.CLIENT_SERVICE_NOT_OK),
null);
default:
break;
}
this.updateTopicSubscribeInfoWhenSubscriptionChanged();
this.mQClientFactory.checkClientInBroker();
this.mQClientFactory.sendHeartbeatToAllBrokerWithLock();
this.mQClientFactory.rebalanceImmediately();
}
步骤说明:
-
检查配置,初始化启动状态
-
获取MQClient工厂,用来创建MQClientInstance
-
设置消息消费模式,分为广播消费和集群消费, 广播消费为: 集群里面任意一台消费者服务,都可以接收到订阅到的消息,。集群消费:生产者发布一个消息,同一个集群里面,只有一台消费者服务可以消费到消息。 默认为集群消费
-
设置Rebalance算法实现策略,默认
AllocateMessageQueueAveragely, 负载均衡的算法后面会单独开一篇文章进行讲解,此处不过多描述。 -
消息offsetStore的选择器,根据消费者的消息类型不同,选择不同的存储策略
集群消费: 指的是生产者发送了一条消息到对应的队列里面去,那么该条消息仅会被队列指定的消费者所消费,如果队列数或者消费者数量发生变化,相应的队列可能会被另外的消费者消费,基于这种特性,因此offset的消费进度是存储在borker端的,因此初始化的是RemoteBrokerOffsetStore, 消息的消费进度是会上报到远程的borker的广播消费:指的是生产者发送了一条消息到broker里面去,只要是订阅了这个topic的,那么所有的消费者都可以消费,因此,基于这种特性,每个消费者的消费进度是存储在消费者的自身所在的服务器上,不是上报到broker里面去的,如果都上传到broker上去,没有任何意义,因为这个消费进度offset仅针对当前消费者, 发生任何变化(如:broker宕机,消费者数量发生变化), 对应的消费者都不需要处理的,因此初始化的是LocalFileOffsetStore存储器,字面上的意思其实就是存在本地文件里面。
-
this.offsetStore.load()加载offset存储器,LocalFileOffsetStore会加载本地的存储文件,RemoteBrokerOffsetStore则是一个空方法, 换句话说,集群消费不需要加载操作, 广播消费是需要的。 -
判断消息监听器, 根据不同的消息监听器,判断是根据顺序消费的模式还是普通消费的模式,这里是顺序消费和普通消费的关键区分点。
-
启动消息监听处理服务类
-
将消费组注册到本地,本地维护了一个
consumerTable的map结构 -
启动MQClient , 用来跟borker打交道的,一般一个rokcetMq集群一个, 多个集群是需要初始化多个的。
下面主要看一下初始化MQClient ,这个不管在生产者还是消费者一边,都是非常核心的实例,在之前的文章中,是有介绍过初始化这个实例,下面再次说一下
public void start() throws MQClientException {
synchronized (this) {
switch (this.serviceState) {
case CREATE_JUST:
this.serviceState = ServiceState.START_FAILED;
// If not specified,looking address from name server
if (null == this.clientConfig.getNamesrvAddr()) {
this.mQClientAPIImpl.fetchNameServerAddr();
}
// 负责网络通信的接口调用启动,
this.mQClientAPIImpl.start();
// 初始化各类定时任务
this.startScheduledTask();
// 开启pullService , 针对consumer
this.pullMessageService.start();
// 启动RebalanceService服务,针对consumer
this.rebalanceService.start();
// 启动一个groupName为CLIENT_INNER_PRODUCER的DefaultMQProducer,
// 用于将消费失败的消息发回broker,消息的topic格式为%RETRY%ConsumerGroupName。
this.defaultMQProducer.getDefaultMQProducerImpl().start(false);
log.info("the client factory [{}] start OK", this.clientId);
this.serviceState = ServiceState.RUNNING;
break;
case RUNNING:
break;
case SHUTDOWN_ALREADY:
break;
case START_FAILED:
throw new MQClientException("The Factory object[" + this.getClientId() + "] has been created before, and failed.", null);
default:
break;
}
}
}