Rocketmq源码解读——Filtersrv
概述
Filtersrv负责自定义代码过滤Consumer从Broker拉去的消息。在没有Filtersrv的情况下，拉取消息很简单，就是直接连接Broker进行拉取即可。如果有了Filtersrv，则拉取是：Consumer ——> Filtersrv ——> Broker
Filtersrv注册到Broker
对应关系：
- 一个Filtersrv对应一个Broker。
- 一个Broker可以对应多个Filtersrv。Filtersrv的高可用可以通过启动多个Filtersrv实现。
- Filtersrv注册失败，主动退出并关闭。
Filtersrv的注册部分（FiltersrvController#initialize）：
this.remotingExecutor =
Executors.newFixedThreadPool(nettyServerConfig.getServerWorkerThreads(),
new ThreadFactoryImpl("RemotingExecutorThread_"));
this.registerProcessor();
//定时向Broker注册Filtersrv
this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {
@Override
public void run() {
FiltersrvController.this.registerFilterServerToBroker();
}
}, 3, 10, TimeUnit.SECONDS);
this.defaultMQPullConsumer.setNamesrvAddr(this.filtersrvConfig.getNamesrvAddr());
return true;
继续看一下这个注册方法：
public void registerFilterServerToBroker() {
try {
//注册到broker，broker的地址是之前配置好的
RegisterFilterServerResponseHeader responseHeader =
this.filterServerOuterAPI.registerFilterServerToBroker(
this.filtersrvConfig.getConnectWhichBroker(), this.localAddr());
//一个pull consumer拉取消息使用
this.defaultMQPullConsumer.getDefaultMQPullConsumerImpl().getPullAPIWrapper()
.setDefaultBrokerId(responseHeader.getBrokerId());
if (null == this.brokerName) {
this.brokerName = responseHeader.getBrokerName();
}
} catch (Exception e) {
//注册失败就退出了
System.exit(-1);
}
}
Consumer从Filtersrv拉取消息
Consumer拉取使用过滤类方式订阅的消费消息时，从Broker对应的Filtersrv列表随机选择一个拉取消息。如果选择不到Filtersrv，则无法拉取消息。因此，Filtersrv一定要做高可用。
看下pullKernelImpl方法：
// 若订阅topic使用过滤类，使用filtersrv获取消息
String brokerAddr = findBrokerResult.getBrokerAddr();
if (PullSysFlag.hasClassFilterFlag(sysFlagInner)) {
brokerAddr = computPullFromWhichFilterServer(mq.getTopic(), brokerAddr);
}
继续看一下computPullFromWhichFilterServer是怎么做的：
private String computPullFromWhichFilterServer(final String topic, final String brokerAddr)
throws MQClientException {
ConcurrentMap<String, TopicRouteData> topicRouteTable = this.mQClientFactory.getTopicRouteTable();
if (topicRouteTable != null) {
TopicRouteData topicRouteData = topicRouteTable.get(topic);
List<String> list = topicRouteData.getFilterServerTable().get(brokerAddr);
if (list != null && !list.isEmpty()) {
return list.get(randomNum() % list.size());
}
}
throw new MQClientException(...);
}
获取Filtersrv地址。如果有多个Filtersrv，随机选择一个。
Filtersrv从Broker拉取消息
Filtersrv可以理解成一个Consumer，向Broker拉取消息时，实际使用的DefaultMQPullConsumer（Filtersrv#start方法中会启动一个DefaultMQPullConsumer）的方法和逻辑。
看下DefaultRequestProcessor的pullMessageForward方法，这个方法是中转方法，Consumer的拉取消息会通过Filtersrv转发，有点长我们分开看：
//初始化拉取的request
final RemotingCommand response = RemotingCommand.createResponseCommand(PullMessageResponseHeader.class);
final PullMessageResponseHeader responseHeader = (PullMessageResponseHeader) response.readCustomHeader();
final PullMessageRequestHeader requestHeader =
(PullMessageRequestHeader) request.decodeCommandCustomHeader(PullMessageRequestHeader.class);
final FilterContext filterContext = new FilterContext();
filterContext.setConsumerGroup(requestHeader.getConsumerGroup());
response.setOpaque(request.getOpaque());
DefaultMQPullConsumer pullConsumer = this.filtersrvController.getDefaultMQPullConsumer();
这里就主要是初始化一些变量。继续看：
//校验Topic过滤类是否完整
final FilterClassInfo findFilterClass =
this.filtersrvController.getFilterClassManager()
.findFilterClass(requestHeader.getConsumerGroup(), requestHeader.getTopic());
if (null == findFilterClass) {
response.setCode(ResponseCode.SYSTEM_ERROR);
response.setRemark("Find Filter class failed, not registered");
return response;
}
if (null == findFilterClass.getMessageFilter()) {
response.setCode(ResponseCode.SYSTEM_ERROR);
response.setRemark("Find Filter class failed, registered but no class");
return response;
}
这里主要校验对应的Topic的过滤类是否完整。
//设置下次请求从Broker主节点拉取消息。
responseHeader.setSuggestWhichBrokerId(MixAll.MASTER_ID);
MessageQueue mq = new MessageQueue();
mq.setTopic(requestHeader.getTopic());
mq.setQueueId(requestHeader.getQueueId());
mq.setBrokerName(this.filtersrvController.getBrokerName());
long offset = requestHeader.getQueueOffset();
int maxNums = requestHeader.getMaxMsgNums();
final PullCallback pullCallback = new PullCallback() {
@Override
public void onSuccess(PullResult pullResult) {
responseHeader.setMaxOffset(pullResult.getMaxOffset());
responseHeader.setMinOffset(pullResult.getMinOffset());
responseHeader.setNextBeginOffset(pullResult.getNextBeginOffset());
response.setRemark(null);
switch (pullResult.getPullStatus()) {
case FOUND:
//如果拉取到消息
response.setCode(ResponseCode.SUCCESS);
List<MessageExt> msgListOK = new ArrayList<MessageExt>();
//过滤消息
try {
for (MessageExt msg : pullResult.getMsgFoundList()) {
boolean match = findFilterClass.getMessageFilter().match(msg, filterContext);
if (match) {
msgListOK.add(msg);
}
}
if (!msgListOK.isEmpty()) {
returnResponse(requestHeader.getConsumerGroup(), requestHeader.getTopic(), ctx, response, msgListOK);
return;
} else {
response.setCode(ResponseCode.PULL_RETRY_IMMEDIATELY);
}
} catch (Throwable e) {
final String error =
String.format("do Message Filter Exception, ConsumerGroup: %s Topic: %s ",
requestHeader.getConsumerGroup(), requestHeader.getTopic());
log.error(error, e);
response.setCode(ResponseCode.SYSTEM_ERROR);
response.setRemark(error + RemotingHelper.exceptionSimpleDesc(e));
returnResponse(requestHeader.getConsumerGroup(), requestHeader.getTopic(), ctx, response, null);
return;
}
break;
case NO_MATCHED_MSG:
response.setCode(ResponseCode.PULL_RETRY_IMMEDIATELY);
break;
case NO_NEW_MSG:
response.setCode(ResponseCode.PULL_NOT_FOUND);
break;
case OFFSET_ILLEGAL:
response.setCode(ResponseCode.PULL_OFFSET_MOVED);
break;
default:
break;
}
returnResponse(requestHeader.getConsumerGroup(), requestHeader.getTopic(), ctx, response, null);
}
@Override
public void onException(Throwable e) {
response.setCode(ResponseCode.SYSTEM_ERROR);
response.setRemark("Pull Callback Exception, " + RemotingHelper.exceptionSimpleDesc(e));
returnResponse(requestHeader.getConsumerGroup(), requestHeader.getTopic(), ctx, response, null);
return;
}
};
// 拉取消息
pullConsumer.pullBlockIfNotFound(mq, null, offset, maxNums, pullCallback);
return null;
拉取过程是使用回调的方式来进行过滤的。
Consumer上传过滤类
使用过滤类的时候，应该使用subscribe(String topic, String fullClassName, String filterClassSource)方法。
订阅后，发送心跳到Broker，上传过滤类源码到Filtersrv。
public void sendHeartbeatToAllBrokerWithLock() {
if (this.lockHeartbeat.tryLock()) {
try {
this.sendHeartbeatToAllBroker();
this.uploadFilterClassSource();
} catch (final Exception e) {
} finally {
this.lockHeartbeat.unlock();
}
} else {
}
}
看下上传过滤类代码的部分uploadFilterClassSource：
private void uploadFilterClassSource() {
Iterator<Entry<String, MQConsumerInner>> it = this.consumerTable.entrySet().iterator();
while (it.hasNext()) {
Entry<String, MQConsumerInner> next = it.next();
MQConsumerInner consumer = next.getValue();
//push方式才会过滤
if (ConsumeType.CONSUME_PASSIVELY == consumer.consumeType()) {
Set<SubscriptionData> subscriptions = consumer.subscriptions();
for (SubscriptionData sub : subscriptions) {
if (sub.isClassFilterMode() && sub.getFilterClassSource() != null) {
final String consumerGroup = consumer.groupName();
final String className = sub.getSubString();
final String topic = sub.getTopic();
final String filterClassSource = sub.getFilterClassSource();
try {
//上传源码
this.uploadFilterClassToAllFilterServer(consumerGroup, className, topic, filterClassSource);
} catch (Exception e) {
}
}
}
}
}
}
