RocketMq源码分析(四):Broker启动流程

relax_peng
438 阅读8分钟

一.概述和入口

broker启动入口在 BrokerStartup.main()方法中

public static void main(String[] args) {
    start(createBrokerController(args));
}

分为两步:

  • 1.创建brokerController
  • 2.启动brokerController 下面我们来一步步分析

二.创建BrokerController

/**
 * 创建 brokerController
 * 创建四大核心配置和 消费量管理/topic管理/消息处理等
 */
public static BrokerController createBrokerController(String[] args) {
....省略一些解析命令/tls安全相关
final BrokerController controller = new BrokerController(
    brokerConfig,
    nettyServerConfig,
    nettyClientConfig,
    messageStoreConfig);
// remember all configs to prevent discard
controller.getConfiguration().registerConfig(properties);

/**
 * 初始化
 */
boolean initResult = controller.initialize();
if (!initResult) {
    controller.shutdown();
    System.exit(-3);
}
/**  添加钩子函数,在关闭的时候进行触发,发送shutdown消息 */
Runtime.getRuntime().addShutdownHook(new Thread(new Runnable() {
    private volatile boolean hasShutdown = false;
    private AtomicInteger shutdownTimes = new AtomicInteger(0);

    @Override
    public void run() {
        synchronized (this) {
            log.info("Shutdown hook was invoked, {}", this.shutdownTimes.incrementAndGet());
            if (!this.hasShutdown) {
                this.hasShutdown = true;
                long beginTime = System.currentTimeMillis();
                /**这里会发一条注销消息给nameServer */
                controller.shutdown();
                long consumingTimeTotal = System.currentTimeMillis() - beginTime;
                log.info("Shutdown hook over, consuming total time(ms): {}", consumingTimeTotal);
            }
        }
    }
}, "ShutdownHook"));

上面代码做四件事情

  • 命令解析(和nameserver启动类似)
  • new BrokerController()
  • 初始化brokerController
  • 添加钩子函数

接下来这几个步骤逐步讲解

2.1 命令的解析

直接解析启动参数中 -c -n -p -m 的一些命令,不是很重要

2.2 创建brokerController

final BrokerController controller = new BrokerController(
    brokerConfig,
    nettyServerConfig,
    nettyClientConfig,
    messageStoreConfig);

继续进入BrokerController构造方法,可以看到很多重要的成员变量

public BrokerController(
    final BrokerConfig brokerConfig,
    final NettyServerConfig nettyServerConfig,
    final NettyClientConfig nettyClientConfig,
    final MessageStoreConfig messageStoreConfig
) {
    /**
     * 4个核心配置信息
     */
    this.brokerConfig = brokerConfig;
    this.nettyServerConfig = nettyServerConfig;
    this.nettyClientConfig = nettyClientConfig;
    this.messageStoreConfig = messageStoreConfig;
    this.setStoreHost(new InetSocketAddress(this.getBrokerConfig().getBrokerIP1(), getListenPort()));
     this.brokerStatsManager = messageStoreConfig.isEnableLmq() ? new LmqBrokerStatsManager(this.brokerConfig.getBrokerClusterName(), this.brokerConfig.isEnableDetailStat()) : new BrokerStatsManager(this.brokerConfig.getBrokerClusterName(), this.brokerConfig.isEnableDetailStat());
    /**
     * 管理consumer消费消息的offset
     * 管理consumer消费消息位置的偏移量，偏移量表示消费者组消费该topic消息的位置，后面再消费时，就从该位置后消费，避免重复消费消息，也避免了漏消费消息。
     */
    this.consumerOffsetManager = messageStoreConfig.isEnableLmq() ? new LmqConsumerOffsetManager(this) : new ConsumerOffsetManager(this);

    /** 管理topic配置 就是用来管理topic配置的，如topic名称，topic队列数量 */
    this.topicConfigManager = messageStoreConfig.isEnableLmq() ? new LmqTopicConfigManager(this) : new TopicConfigManager(this);
    this.topicQueueMappingManager = new TopicQueueMappingManager(this);
    /**
     * 处理 consumer 拉消息请求的
     */
    this.pullMessageProcessor = new PullMessageProcessor(this);
    /**
     * 消息送达的监听器 当生产者的消息送达时，由该监听器监听
     */
    this.messageArrivingListener = new NotifyMessageArrivingListener(this.pullRequestHoldService, this.popMessageProcessor, this.notificationProcessor);
if (nettyClientConfig != null) {
    /**
     *  往外发消息的组件
     * 如向NameServer发送注册/注销消息
     */
    this.brokerOuterAPI = new BrokerOuterAPI(nettyClientConfig);
}

可以看到在brokerController实例化的时候,会构建很多对象,上面代码只是粘贴一部分,具体用到的时候,会在后面文章中讲解,这里混一个脸熟

2.3 初始化brokerController

构建好brokerController后,会进行初始化,代码是 controller.initialize()

public boolean initialize() throws CloneNotSupportedException {
    // 加载配置文件中的配置
    boolean result = this.topicConfigManager.load();
    result = result && this.topicQueueMappingManager.load();
    result = result && this.consumerOffsetManager.load();
    result = result && this.subscriptionGroupManager.load();
    result = result && this.consumerFilterManager.load();
    result = result && this.consumerOrderInfoManager.load();

    if (result) {
        try {
            // 消息存储管理组件，管理磁盘上的消息
            DefaultMessageStore defaultMessageStore = new DefaultMessageStore(this.messageStoreConfig, this.brokerStatsManager, this.messageArrivingListener, this.brokerConfig);
            defaultMessageStore.setTopicConfigTable(topicConfigManager.getTopicConfigTable());

            // 启用了DLeger,就创建DLeger相关组件
            if (messageStoreConfig.isEnableDLegerCommitLog()) {
                DLedgerRoleChangeHandler roleChangeHandler = new DLedgerRoleChangeHandler(this, defaultMessageStore);
                ((DLedgerCommitLog) defaultMessageStore.getCommitLog()).getdLedgerServer().getdLedgerLeaderElector().addRoleChangeHandler(roleChangeHandler);
            }

            // broker统计组件
            this.brokerStats = new BrokerStats(defaultMessageStore);
            //load plugin
            MessageStorePluginContext context = new MessageStorePluginContext(this, messageStoreConfig, brokerStatsManager, messageArrivingListener);
            this.messageStore = MessageStoreFactory.build(context, defaultMessageStore);
            this.messageStore.getDispatcherList().addFirst(new CommitLogDispatcherCalcBitMap(this.brokerConfig, this.consumerFilterManager));
            if (this.brokerConfig.isEnableControllerMode()) {
                this.replicasManager = new ReplicasManager(this);
            }
            if (messageStoreConfig.isTimerWheelEnable()) {
                this.timerCheckpoint = new TimerCheckpoint(BrokerPathConfigHelper.getTimerCheckPath(messageStoreConfig.getStorePathRootDir()));
                TimerMetrics timerMetrics = new TimerMetrics(BrokerPathConfigHelper.getTimerMetricsPath(messageStoreConfig.getStorePathRootDir()));
                this.timerMessageStore = new TimerMessageStore(messageStore, messageStoreConfig, timerCheckpoint, timerMetrics, brokerStatsManager);
                this.timerMessageStore.registerEscapeBridgeHook(msg -> escapeBridge.putMessage(msg));
                this.messageStore.setTimerMessageStore(this.timerMessageStore);
            }
        } catch (IOException e) {
            result = false;
            LOG.error("BrokerController#initialize: unexpected error occurs", e);
        }
    }
    if (messageStore != null) {
        registerMessageStoreHook();
    }

    // 加载磁盘上的记录，如commitLog写入的位置、消费者主题/队列的信息
    result = result && this.messageStore.load();

    if (messageStoreConfig.isTimerWheelEnable()) {
        result = result && this.timerMessageStore.load();
    }

    //scheduleMessageService load after messageStore load success
    result = result && this.scheduleMessageService.load();

    for (BrokerAttachedPlugin brokerAttachedPlugin : brokerAttachedPlugins) {
        if (brokerAttachedPlugin != null) {
            result = result && brokerAttachedPlugin.load();
        }
    }

    if (result) {
        // todo 处理 nettyServer
        initializeRemotingServer();

        // 初始化一些线程池,如consumer pull线程
        initializeResources();

        // 注册处理器  TODO 每个code对应一个processor,后续生产者发消息的时候,根据code找到对应的processor进行处理数据
        registerProcessor();

        //todo 启动定时任务 如broker与nameserver的心跳
        initializeScheduledTasks();

        //初始化一些事务
        initialTransaction();

这一块逻辑比较多,主要步骤

  • 1.加载配置文件
  • 2.消息管理组件/统计组件的初始化
  • 3.创建nettyServer
  • 4.netty的handler上注册不同code对应的processor
  • 5.启动一些定时任务,保持broker和nameserver的一些通信 下面我们来分析3-4-5这核心的三步

1.创建nettyServer

入口是 initializeRemotingServer(); 会创建两个netty服务端

protected void initializeRemotingServer() throws CloneNotSupportedException {
    this.remotingServer = new NettyRemotingServer(this.nettyServerConfig,  
...省略
    this.fastRemotingServer = new NettyRemotingServer(fastConfig, this.clientHousekeepingService);
}

看NettyRemotingServer,就是构建了bootstrap和bossGroup/workGroup等一些线程池,代码如下

public NettyRemotingServer(final NettyServerConfig nettyServerConfig,
    final ChannelEventListener channelEventListener) {
    super(nettyServerConfig.getServerOnewaySemaphoreValue(), nettyServerConfig.getServerAsyncSemaphoreValue());
    this.serverBootstrap = new ServerBootstrap();
    this.nettyServerConfig = nettyServerConfig;
    this.channelEventListener = channelEventListener;
    // 创建了一个名为publicExecutor线程池
    this.publicExecutor = buildPublicExecutor(nettyServerConfig);
    //创建boosGroup
    this.eventLoopGroupBoss = buildBossEventLoopGroup();
    // 创建workGroup
    this.eventLoopGroupSelector = buildEventLoopGroupSelector();

    loadSslContext();
}

2.netty的handler上注册不同code对应的processor

入口 registerProcessor()

public void registerProcessor() {
    /*
     * SendMessageProcessor
     */
    sendMessageProcessor.registerSendMessageHook(sendMessageHookList);
    sendMessageProcessor.registerConsumeMessageHook(consumeMessageHookList);

    this.remotingServer.registerProcessor(RequestCode.SEND_MESSAGE, sendMessageProcessor, this.sendMessageExecutor);
    this.remotingServer.registerProcessor(RequestCode.SEND_MESSAGE_V2, sendMessageProcessor, this.sendMessageExecutor);
    this.remotingServer.registerProcessor(RequestCode.SEND_BATCH_MESSAGE, sendMessageProcessor, this.sendMessageExecutor);
    this.remotingServer.registerProcessor(RequestCode.CONSUMER_SEND_MSG_BACK, sendMessageProcessor, this.sendMessageExecutor);
    //TODO 10 就是处理生产者发送的code码
    this.fastRemotingServer.registerProcessor(RequestCode.SEND_MESSAGE, sendMessageProcessor, this.sendMessageExecutor);
    //TODO 310 就是处理生产者发送的code码

进入 registerProcessor方法

public void registerProcessor(int requestCode, NettyRequestProcessor processor, ExecutorService executor) {
    ExecutorService executorThis = executor;
    if (null == executor) {
        executorThis = this.publicExecutor;
    }
    //TODO 把process和线程池封装为pair对象
    Pair<NettyRequestProcessor, ExecutorService> pair = new Pair<>(processor, executorThis);
    // 放入map中,每一个code对应一个pair
    this.processorTable.put(requestCode, pair);
}

在第三篇文章中,我们看到nameserver在接受请求的时候,netty中的处理消息的handler,也是通过不同的code找到对应的pair进行不同的处理,这里就是设置pair

注意: 如果code没有找到对应的pair,会使用默认的pair

3.启动一些定时任务,保持broker和nameserver的一些通信

入口initializeScheduledTasks()方法 这个方法里面会启动很多的定时任务,这里举例一些

BrokerController.this.getBrokerStats().record();
BrokerController.this.consumerOffsetManager.persist();
BrokerController.this.consumerFilterManager.persist();
BrokerController.this.consumerOrderInfoManager.persist();
BrokerController.this.protectBroker();
BrokerController.this.printMasterAndSlaveDiff();
BrokerController.this.brokerOuterAPI.refreshMetadata();
BrokerController.this.brokerOuterAPI.updateNameServerAddressList(BrokerController.this.brokerConfig.getNamesrvAddr());

2.4 添加钩子函数,关闭时触发

/**  添加钩子函数,在关闭的时候进行触发,发送shutdown消息 */
Runtime.getRuntime().addShutdownHook(new Thread(new Runnable() {
    private volatile boolean hasShutdown = false;
    private AtomicInteger shutdownTimes = new AtomicInteger(0);

    @Override
    public void run() {
        synchronized (this) {
            log.info("Shutdown hook was invoked, {}", this.shutdownTimes.incrementAndGet());
            if (!this.hasShutdown) {
                this.hasShutdown = true;
                long beginTime = System.currentTimeMillis();
                /**这里会发一条注销消息给nameServer */
                controller.shutdown();
                long consumingTimeTotal = System.currentTimeMillis() - beginTime;
                log.info("Shutdown hook over, consuming total time(ms): {}", consumingTimeTotal);
            }
        }
    }
}, "ShutdownHook"));

项目关闭的时候,会调用brokerController的shutdown方法,会做一些资源的释放和向nameserver移除自己的注册信息,如下:

protected void shutdownBasicService() {

    shutdown = true;

    //获取所有的 nameServer，遍历发送注销消息
    this.unregisterBrokerAll();

    if (this.shutdownHook != null) {
        this.shutdownHook.beforeShutdown(this);
    }

    if (this.remotingServer != null) {
        this.remotingServer.shutdown();
    }

    if (this.fastRemotingServer != null) {
        this.fastRemotingServer.shutdown();
    }

    if (this.brokerStatsManager != null) {
        this.brokerStatsManager.shutdown();
    }

    if (this.clientHousekeepingService != null) {
        this.clientHousekeepingService.shutdown();
    }

三.启动brokerController

入口代码是controller.start();

public void start() throws Exception {

    this.shouldStartTime = System.currentTimeMillis() + messageStoreConfig.getDisappearTimeAfterStart();

    if (messageStoreConfig.getTotalReplicas() > 1 && this.brokerConfig.isEnableSlaveActingMaster() || this.brokerConfig.isEnableControllerMode()) {
        isIsolated = true;
    }
    // broker对外发放消息的组件，向nameServer上报存活消息时使用了它，也是一个netty服务,todo 目前来看,这里还没和nameServ建立连接,因为这里是一个client,调用接口的时候,才触发 bootstrap.connect()
    if (this.brokerOuterAPI != null) {
        this.brokerOuterAPI.start();
    }

    /**
     * 启动一些基础服务
     * 比如: 启动消息存储/接受producer发消息的netty
     * todo messageStore.start() messageStore:从commitLog文件读取数据,存入consumerQueue文件中
     */
    startBasicService();

    /**
     * todo 注册broker
     */
    if (!isIsolated && !this.messageStoreConfig.isEnableDLegerCommitLog() && !this.messageStoreConfig.isDuplicationEnable()) {
        changeSpecialServiceStatus(this.brokerConfig.getBrokerId() == MixAll.MASTER_ID);
        this.registerBrokerAll(true, false, true);
    }

    /**
     * Math.max(10000, Math.min(brokerConfig.getRegisterNameServerPeriod(), 60000)), TimeUnit.MILLISECONDS) 不能小于10s,不能大于60s
     */
    scheduledFutures.add(this.scheduledExecutorService.scheduleAtFixedRate(new AbstractBrokerRunnable(this.getBrokerIdentity()) {
        @Override
        public void run2() {
            try {
                if (System.currentTimeMillis() < shouldStartTime) {
                    BrokerController.LOG.info("Register to namesrv after {}", shouldStartTime);
                    return;
                }
                if (isIsolated) {
                    BrokerController.LOG.info("Skip register for broker is isolated");
                    return;
                }
                BrokerController.this.registerBrokerAll(true, false, brokerConfig.isForceRegister());
            } catch (Throwable e) {
                BrokerController.LOG.error("registerBrokerAll Exception", e);
            }
        }
    }, 1000 * 10, Math.max(10000, Math.min(brokerConfig.getRegisterNameServerPeriod(), 60000)), TimeUnit.MILLISECONDS));
    // brokerConfig.isEnableSlaveActingMaster()默认是false
    if (this.brokerConfig.isEnableSlaveActingMaster()) {
        /**
         * todo 定时任务: broker每1秒钟向 nameserver 发送心跳
         */
        scheduleSendHeartbeat();

        scheduledFutures.add(this.syncBrokerMemberGroupExecutorService.scheduleAtFixedRate(new AbstractBrokerRunnable(this.getBrokerIdentity()) {
            @Override
            public void run2() {
                try {
                    BrokerController.this.syncBrokerMemberGroup();
                } catch (Throwable e) {
                    BrokerController.LOG.error("sync BrokerMemberGroup error. ", e);
                }
            }
        }, 1000, this.brokerConfig.getSyncBrokerMemberGroupPeriod(), TimeUnit.MILLISECONDS));
    }


    /**
     * todo broker向nameserv发送心跳
     * brokerConfig.isEnableControllerMode() 参数默认值也是false
     */
    if (this.brokerConfig.isEnableControllerMode()) {
        scheduleSendHeartbeat();
    }

    if (brokerConfig.isSkipPreOnline()) {
        startServiceWithoutCondition();
    }
}

3.1 启动netty客户端 brokderOutAPI

代码入口 this.brokerOuterAPI.start();

public void start() {
    if (this.defaultEventExecutorGroup == null) {
        this.defaultEventExecutorGroup = new DefaultEventExecutorGroup(
            nettyClientConfig.getClientWorkerThreads(),
            new ThreadFactory() {

                private AtomicInteger threadIndex = new AtomicInteger(0);

                @Override
                public Thread newThread(Runnable r) {
                    return new Thread(r, "NettyClientWorkerThread_" + this.threadIndex.incrementAndGet());
                }
            });
    }
    Bootstrap handler = this.bootstrap.group(this.eventLoopGroupWorker).channel(NioSocketChannel.class)
        .option(ChannelOption.TCP_NODELAY, true)
        .option(ChannelOption.SO_KEEPALIVE, false)
        .option(ChannelOption.CONNECT_TIMEOUT_MILLIS, nettyClientConfig.getConnectTimeoutMillis())
        .handler(new ChannelInitializer<SocketChannel>() {
            @Override
            public void initChannel(SocketChannel ch) throws Exception {
                ChannelPipeline pipeline = ch.pipeline();
                if (nettyClientConfig.isUseTLS()) {
                    if (null != sslContext) {
                        pipeline.addFirst(defaultEventExecutorGroup, "sslHandler", sslContext.newHandler(ch.alloc()));
                        LOGGER.info("Prepend SSL handler");
                    } else {
                        LOGGER.warn("Connections are insecure as SSLContext is null!");
                    }
                }
                ch.pipeline().addLast(
                    nettyClientConfig.isDisableNettyWorkerGroup() ? null : defaultEventExecutorGroup,
                    new NettyEncoder(),
                    new NettyDecoder(),
                    // todo netty的客户端,添加心跳监测机制
                    new IdleStateHandler(0, 0, nettyClientConfig.getClientChannelMaxIdleTimeSeconds()),
                    // TODO netty添加连接的handler,主要功能是感应到connect/disconnect/close/exception/心跳事件后,往监听器中发送一条事件消息
                    new NettyConnectManageHandler(),
                    new NettyClientHandler());
            }
        });

同nameserver一样,因为这块代码也是公用的

3.2 启动其他的一些基础服务

代码入口 startBasicService()

image.png 里面会启动很多组件,比较重要的是启动netty的服务端,继续进入

public void start() {
    this.defaultEventExecutorGroup = new DefaultEventExecutorGroup(
        nettyServerConfig.getServerWorkerThreads(),
        new ThreadFactory() {

            private final AtomicInteger threadIndex = new AtomicInteger(0);

            @Override
            public Thread newThread(Runnable r) {
                return new Thread(r, "NettyServerCodecThread_" + this.threadIndex.incrementAndGet());
            }
        });

    // 初始化 netty的四个handler,后面会添加到netty中的pipeline中
    prepareSharableHandlers();

    serverBootstrap.group(this.eventLoopGroupBoss, this.eventLoopGroupSelector)
        .channel(useEpoll() ? EpollServerSocketChannel.class : NioServerSocketChannel.class)
        .option(ChannelOption.SO_BACKLOG, 1024)
        .option(ChannelOption.SO_REUSEADDR, true)
        .childOption(ChannelOption.SO_KEEPALIVE, false)
        .childOption(ChannelOption.TCP_NODELAY, true)
            //TODO 绑定ip和端口
        .localAddress(new InetSocketAddress(this.nettyServerConfig.getBindAddress(),
            this.nettyServerConfig.getListenPort()))
        .childHandler(new ChannelInitializer<SocketChannel>() {
            @Override
            public void initChannel(SocketChannel ch) {
                ch.pipeline()
                        //handshakeHandler  处理握手操作，用来判断tls的开启状态
                    .addLast(defaultEventExecutorGroup, HANDSHAKE_HANDLER_NAME, handshakeHandler)

                        /**
                         * 批量添加五个handler
                         * encoder/NettyDecoder：处理报文的编解码操作
                         * IdleStateHandler：处理心跳
                         * connectionManageHandler：处理连接请求
                         * serverHandler：处理读写请求=> TODO 用来处理broker注册消息、producer/consumer获取topic消息的
                         */
                    .addLast(defaultEventExecutorGroup,
                        encoder,
                        new NettyDecoder(),
                        new IdleStateHandler(0, 0,
                            nettyServerConfig.getServerChannelMaxIdleTimeSeconds()),
                        connectionManageHandler,
                        //serverHandler：处理读写请求
                        serverHandler
                    );
            }
        });

    addCustomConfig(serverBootstrap);

这里和nameserver启动的netty服务端是一样的,因为这一块代码是在remoting模块中,broker和nameserver是共用这一块的

3.3 开始定时任务,把broker注册到nameserver

入口registerBrokerAll() 继续进入

/**
 * 这里会判断是否需要进行注册
 * needRegister(): 这个方法调用了brokerOuterAPI.needRegister(...)来判断broker是否发生了变化，只要一个NameServer上发生了变化，就说明需要进行注册操作。
 */
if (forceRegister || needRegister(this.brokerConfig.getBrokerClusterName(),
    this.getBrokerAddr(),
    this.brokerConfig.getBrokerName(),
    this.brokerConfig.getBrokerId(),
    this.brokerConfig.getRegisterBrokerTimeoutMills(),
    this.brokerConfig.isInBrokerContainer())) {
    // 进行注册操作
    doRegisterBrokerAll(checkOrderConfig, oneway, topicConfigWrapper);
}

最后调用接口,发起注册

// 注册
List<RegisterBrokerResult> registerBrokerResultList = this.brokerOuterAPI.registerBrokerAll(
    this.brokerConfig.getBrokerClusterName(),
    this.getBrokerAddr(),
    this.brokerConfig.getBrokerName(),
    this.brokerConfig.getBrokerId(),
    this.getHAServerAddr(),
    // 这个对象里就包含了当前broker的版本信息
    topicConfigWrapper,
    this.filterServerManager.buildNewFilterServerList(),
    oneway,
    this.brokerConfig.getRegisterBrokerTimeoutMills(),
    this.brokerConfig.isEnableSlaveActingMaster(),
    this.brokerConfig.isCompressedRegister(),
    this.brokerConfig.isEnableSlaveActingMaster() ? this.brokerConfig.getBrokerNotActiveTimeoutMillis() : null,
    this.getBrokerIdentity());

四.总结

总的来说，启动流程分为3个：

  1. 解析配置文件，这一步会解析各种配置，并将其赋值到对应的对象中
  2. BrokerController创建及初始化：创建了BrokerController对象，并进行初始化操作，所谓的初始化，就是加载配置文件中配置、创建线程池、注册请求处理器、启动定时任务等
  3. BrokerController启动：这一步是启动broker的核心组件，如messageStore(消息存储)、remotingServer(netty服务，用来处理producer与consumer请求)、brokerOuterAPI(netty服务，用来向nameServer上报当前broker信息)等。

在分析启动过程中，重点分析了两类消息的发送：

  1. 在ShutdownHook中，broker会向nameServer发送注销消息，这表明在broker关闭前，nameServer会清除当前broker的注册信息

  2. broker启动后，会启动一个定时任务，定期判断是否需要向nameServer注册，判断是否需要注册时，会向nameServer发送code为QUERY_DATA_VERSION的消息，从nameServer得到当前broker的版本号，该版本号与本地版本号不一致，就表示需要向broker重新注册了，即发送注册消息。

avatar
文章
阅读
粉丝