前言
Namesrv主要作用就是为消息生产者和消息消费者提供关于topic的路由信息，Namesrv就需要具备路由基础信息管理，Broker节点管理等功能。Namesrv本身是一个无状态的节点，本文我们一起来看Broker是如何将信息注册到Namesrv上。
路由元信息介绍
路由元信息保存在org.apache.rocketmq.namesrv.routeinfo.RouteInfoManager，它存储的信息见下面信息。
// 保存Topic和队列信息，也叫路由信息
private final Map<String/* topic */, Map<String/*brokerName*/, QueueData>> topicQueueTable;
// 存储brokerName和队列信息
private final Map<String/* brokerName */, BrokerData> brokerAddrTable;
// 集群和broker关系
private final Map<String/* clusterName */, Set<String/* brokerName */>> clusterAddrTable;
// 在线的broker和Broker信息对应关系
private final Map<BrokerAddrInfo/* brokerAddr */, BrokerLiveInfo> brokerLiveTable;
// 过滤服务器信息
private final Map<BrokerAddrInfo/* brokerAddr */, List<String>/* Filter Server */> filterServerTable;
// topic,Queue映射关系表
private final Map<String/* topic */, Map<String/*brokerName*/, TopicQueueMappingInfo>> topicQueueMappingInfoTable;
路由注册
RocketMQ中路由注册是通过Broker主动与namesrv保持心跳。Broker启动后每隔30秒集群中所有namesrv发送心跳，namesrv发送心跳，namesrv收到Broker心跳时会更新
Broker发送心跳包
Broker发送心跳的代码是在org.apache.rocketmq.broker.BrokerController#start方法中，BrokerController启动之后会以10s的时间间隔向namesrv发送心跳包
public void start() throws Exception {
// ...
// 向namesrv发送心跳
scheduledFutures.add(this.scheduledExecutorService.scheduleAtFixedRate(new AbstractBrokerRunnable(this.getBrokerIdentity()) {
@Override
public void run0() {
try {
// ...
BrokerController.this.registerBrokerAll(true, false, brokerConfig.isForceRegister());
} catch (Throwable e) {
BrokerController.LOG.error("registerBrokerAll Exception", e);
}
}
}, 1000 * 10, Math.max(10000/*10s*/, Math.min(brokerConfig.getRegisterNameServerPeriod(), 60000/*60s*/)), TimeUnit.MILLISECONDS));
// ...
}
每次发送心跳包，broker会遍历所有namesrv，循环遍历，逐个发送心跳包。心跳包的header中保存当前broker的信息，body保存topic信息。
public List<RegisterBrokerResult> registerBrokerAll(
// ...
final BrokerIdentity brokerIdentity) {
final List<RegisterBrokerResult> registerBrokerResultList = new CopyOnWriteArrayList<>();
// 获取所有nameServerAddress
List<String> nameServerAddressList = this.remotingClient.getAvailableNameSrvList();
final RegisterBrokerRequestHeader requestHeader = new RegisterBrokerRequestHeader();
// 设置RegisterBrokerRequestHeader
final CountDownLatch countDownLatch = new CountDownLatch(nameServerAddressList.size());
// 遍历所有namesrvAddr
for (final String namesrvAddr : nameServerAddressList) {
brokerOuterExecutor.execute(new AbstractBrokerRunnable(brokerIdentity) {
@Override
public void run0() {
try {
// 发送心跳
RegisterBrokerResult result = registerBroker(namesrvAddr, oneway, timeoutMills, requestHeader, body);
// ...
} finally {
countDownLatch.countDown();
}
}
});
}
// ...
return registerBrokerResultList;
}
namesrv接收心跳包
namesrv中处理网络请求通过org.apache.rocketmq.namesrv.processor.DefaultRequestProcessor类，如果请求根据请求code不同，会调用不同方法处理不同的请求，当requestCode为RequestCode.REGISTER_BROKER时，请求会转发到DefaultRequestProcessor#registerBroker处理broker的心跳包。
public class DefaultRequestProcessor implements NettyRequestProcessor {
@Override
public RemotingCommand processRequest(ChannelHandlerContext ctx,
RemotingCommand request) throws RemotingCommandException {
switch (request.getCode()) {
// 注册broker
case RequestCode.REGISTER_BROKER:
return this.registerBroker(ctx, request);
// ...
}
}
}
DefaultRequestProcessor#registerBroker处理心跳包主要包含下面三个步骤
- 将requestBody用CRC32算法加密，与requestHeader中的CRC32值进行比较，判断请求数据是否正确
- 用json反序列化解析topic信息
- 调用
RouteInfoManager#registerBroker注册broker。
CRC32检测原理
CRC检验原理实际上就是在一个p位二进制数据序列之后附加一个r位二进制检验码(序列)，从而构成一个总长为n＝p＋r位的二进制序列；附加在数据序列之后的这个检验码与数据序列的内容之间存在着某种特定的关系。如果因干扰等原因使数据序列中的某一位或某些位发生错误，这种特定关系就会被破坏。因此，通过检查这一关系，就可以实现对数据正确性的检验。
public RemotingCommand registerBroker(ChannelHandlerContext ctx,
RemotingCommand request) throws RemotingCommandException {
//校验crc32
if (!checksum(ctx, request, requestHeader)) {
response.setCode(ResponseCode.SYSTEM_ERROR);
response.setRemark("crc32 not match");
return response;
}
// ...
// 解析requestBody中的路由信息(json反序列化)
final RegisterBrokerBody registerBrokerBody = extractRegisterBrokerBodyFromRequest(request, requestHeader);
topicConfigWrapper = registerBrokerBody.getTopicConfigSerializeWrapper();
filterServerList = registerBrokerBody.getFilterServerList();
// ...
// 刷新broker路由信息
RegisterBrokerResult result = this.namesrvController.getRouteInfoManager().registerBroker(/* ...*/);
// ...
return response;
}
下面我们来看一下RouteInfoManager是如何注册路由的，首先需要加写锁，同一时刻只能处理一个broker的心跳包，获取到锁后，会将brokerName放入到当前集群的brokerNameSet中。
this.lock.writeLock().lockInterruptibly();
// 1.初始化或者更新集群中brokerNames信息
Set<String> brokerNames = ConcurrentHashMapUtils.computeIfAbsent((ConcurrentHashMap<String, Set<String>>) this.clusterAddrTable, clusterName, k -> new HashSet<>());
brokerNames.add(brokerName);
第二步从brokerAddrTable获取当前broker的BrokerData，如果不存在，则说明是第一次注册当前broker信息，然后创建BrokerData放入到brokerAddrTable中，如果存在则说明不是第一次创建BrokerData，直接获取即可。获取到BrokerData，则需要更新BrokerData里保存的brokerAddrsMap中当前心跳BrokerId对应的地址。如果brokerAddrsMap存在当前的地址，但是BrokerId又不相同，说明主从切换导致当前BrokerId变化了，因此需要先从brokerAddrsMap删除旧数据，然后再将最新的Broker信息put到brokerAddrsMap中。
BrokerData brokerData = this.brokerAddrTable.get(brokerName);
if (null == brokerData) { // 如果brokerData不存在则说明是第一次创建
registerFirst = true;
brokerData = new BrokerData(clusterName, brokerName, new HashMap<>());
this.brokerAddrTable.put(brokerName, brokerData);
}
Map<Long, String> brokerAddrsMap = brokerData.getBrokerAddrs();
boolean isMinBrokerIdChanged = false;
long prevMinBrokerId = 0;
if (!brokerAddrsMap.isEmpty()) {
prevMinBrokerId = Collections.min(brokerAddrsMap.keySet());
}
// 如果当前brokerId小于最小，说明主从切换了
if (brokerId < prevMinBrokerId) {
isMinBrokerIdChanged = true;
}
// ...
// 如果主从切换了,要先删除brokerAddrsMap中旧的broker信息
brokerAddrsMap.entrySet().removeIf(item -> null != brokerAddr && brokerAddr.equals(item.getValue()) && brokerId != item.getKey());
// 如果namesrv中的broker信息版本比当前心跳的版本更新，说明心跳乱了，namesrv会把broker从brokerLiveTable中删除，并返回失败
String oldBrokerAddr = brokerAddrsMap.get(brokerId);
if (null != oldBrokerAddr && !oldBrokerAddr.equals(brokerAddr)) {
BrokerLiveInfo oldBrokerInfo = brokerLiveTable.get(new BrokerAddrInfo(clusterName, oldBrokerAddr));
if (null != oldBrokerInfo) {
long oldStateVersion = oldBrokerInfo.getDataVersion().getStateVersion();
long newStateVersion = topicConfigWrapper.getDataVersion().getStateVersion();
// 如果当前心跳的版本比之前的版本更旧，则会将当前broker从存活的列表中删除
if (oldStateVersion > newStateVersion) {
clusterName, brokerName, brokerId, oldBrokerAddr, oldStateVersion, brokerAddr, newStateVersion);
brokerLiveTable.remove(new BrokerAddrInfo(clusterName, brokerAddr));
return result;
}
}
}
// ...
// 将当前broker信息更新到brokerAddrsMap中
String oldAddr = brokerAddrsMap.put(brokerId, brokerAddr);
第三步更新路由元信息，这里的路由原信息包括TopicConfig以及TopicQueueMappingInfo两部分。
在namesrv中，每个topic只有一个TopicConfig元信息，它保存在topicQueueTable的map中，topicQueueTable的key是topicName。TopicConfig的属性包括topic的topic名称(topicName)，读队列数量(readQueueNums)，写队列数量(writeQueueNums)以及当前topic的读写权限(perm)，TopicConfig
private String topicName; // topic名称
private int readQueueNums = defaultReadQueueNums; // 读队列数量，默认16
private int writeQueueNums = defaultWriteQueueNums;// 写队列数量，默认16
private int perm = PermName.PERM_READ | PermName.PERM_WRITE;// 读写权限，默认可读可写。
TopicQueueMappingInfo是broker中topic队列映射信息，它保存在topicQueueMappingInfoTable中。它包含的信息如下所示
public class TopicQueueMappingInfo extends RemotingSerializable {
// 冗余字段，topic名称
String topic;
// 所有队列数量
int totalQueues;
// brokername
String bname;
// 时间戳
long epoch;
// 是否是脏数据
boolean dirty;
// 路由逻辑id和物理id映射关系表
protected ConcurrentMap<Integer/*logicId*/, Integer/*physicalId*/> currIdMap = new ConcurrentHashMap<>();
}
TopicConfig与TopicQueueMappingInfo更新逻辑如下所示，只有第一次创建或者TopicConfig信息更新了，并且当前心跳是主节点，才能更新TopicConfig。
// 3.更新路由元数据
if (null != topicConfigWrapper && (isMaster || isPrimeSlave)) {
ConcurrentMap<String, TopicConfig> tcTable =
topicConfigWrapper.getTopicConfigTable();
if (tcTable != null) {
for (Map.Entry<String, TopicConfig> entry : tcTable.entrySet()) {
// 第一次创建或者是topic配置更新了，才可以更新topic的QueueData
if (registerFirst || this.isTopicConfigChanged(clusterName, brokerAddr,
topicConfigWrapper.getDataVersion(), brokerName,
entry.getValue().getTopicName())) {
final TopicConfig topicConfig = entry.getValue();
if (isPrimeSlave) {// 主节点才能更新topic读写配置
topicConfig.setPerm(topicConfig.getPerm() & (~PermName.PERM_WRITE));
}
this.createAndUpdateQueueData(brokerName, topicConfig);
}
}
}
// 如果当前心跳broker中的topic信息变化了，则更新topicQueueMappingInfoTable中当前brokerName的TopicQueueMappingInfo
if (this.isBrokerTopicConfigChanged(clusterName, brokerAddr, topicConfigWrapper.getDataVersion()) || registerFirst) {
TopicConfigAndMappingSerializeWrapper mappingSerializeWrapper = TopicConfigAndMappingSerializeWrapper.from(topicConfigWrapper);
Map<String, TopicQueueMappingInfo> topicQueueMappingInfoMap = mappingSerializeWrapper.getTopicQueueMappingInfoMap();
for (Map.Entry<String, TopicQueueMappingInfo> entry : topicQueueMappingInfoMap.entrySet()) {
if (!topicQueueMappingInfoTable.containsKey(entry.getKey())) {
topicQueueMappingInfoTable.put(entry.getKey(), new HashMap<>());
}
topicQueueMappingInfoTable.get(entry.getKey()).put(entry.getValue().getBname(), entry.getValue());
}
}
}
第四步，更新brokerLiveTable中当前broker的存活信息
BrokerAddrInfo brokerAddrInfo = new BrokerAddrInfo(clusterName, brokerAddr);
BrokerLiveInfo prevBrokerLiveInfo = this.brokerLiveTable.put(brokerAddrInfo,
new BrokerLiveInfo(
System.currentTimeMillis(),
timeoutMillis == null ? DEFAULT_BROKER_CHANNEL_EXPIRED_TIME : timeoutMillis,
topicConfigWrapper == null ? new DataVersion() : topicConfigWrapper.getDataVersion(),
channel,
haServerAddr));
第五步，更新过滤server信息
if (filterServerList != null) {
if (filterServerList.isEmpty()) {
this.filterServerTable.remove(brokerAddrInfo);
} else {
this.filterServerTable.put(brokerAddrInfo, filterServerList);
}
}
将上面namesrv心跳包处理过程整理成时序图如下所示
总结
Namesrv管理路由信息的类是RouteInfoManager，Broker会每隔30s遍历所有namesrv，将心跳信息发送给每个Namesrv。Namesrv收到心跳包后，更新RouteInfoManager中的路由信息。
