Spring Lifecycle及Tomcat Quartz优雅停机

1,477 阅读3分钟

版本约定:

  • SpringBoot 2.5.2
  • Spring 5.3.8
  • embed Tomcat 9.0.48
  • Quartz 2.3.2

停机

对于JVM来说,停机分为两种,正常停机和强制停机

  • 正常停机
    • 所有非守护线程退出
    • System.exit(0)
    • Ctrl + C
    • kill -15 (SIGTERM)
  • 强制停机
    • kill -9 (SIGKILL)
    • Runtime.halt()
    • 掉电关机

在使用Tomcat的时候,有一条非守护线程container-0,待收到关闭指令后这条线程结束,JVM会关闭
此外,在SpringBoot的SpringApplication里有一个SpringApplicationShutdownHook,会注册一个关闭钩子,确保spring容器清理干净.

Runtime.getRuntime().addShutdownHook(new Thread(this, "SpringApplicationShutdownHook"));

在Spring关闭过程中会调用lifecycleProcessor.onClose(),有必要了解Lifecycle的工作过程

Lifecycle

Spring在2.0的时候引入了Lifecycle,3.0引入SmartLifecycle,后者比前者多了phase的属性,在启动和关闭的时候指定顺序,启动时按从小到大的顺序执行(从Integer.MIN_VALUEInteger.MAX_VALUE, phase相同bean的前后顺序不能确定),关闭时顺序相反(从Integer.MAX_VALUEInteger.MIN_VALUE),Lifecyclephase默认为0
此外,后者比前者的stop多了回调,stop线程无须逐个同步等待stop,通过回调方法通知stop线程结束即可.

以关闭stop为例

private void stopBeans() {
   Map<String, Lifecycle> lifecycleBeans = getLifecycleBeans();
   Map<Integer, LifecycleGroup> phases = new HashMap<>();
   // 遍历每一个Lifecycle ean
   lifecycleBeans.forEach((beanName, bean) -> {
      int shutdownPhase = getPhase(bean);
      LifecycleGroup group = phases.get(shutdownPhase);
      if (group == null) {
         group = new LifecycleGroup(shutdownPhase, this.timeoutPerShutdownPhase, lifecycleBeans, false);
         phases.put(shutdownPhase, group);
      }
      // phase相同的bean归属同一组
      group.add(beanName, bean);
   });
   if (!phases.isEmpty()) {
      List<Integer> keys = new ArrayList<>(phases.keySet());
      // 逆序排列,依次调用stop
      keys.sort(Collections.reverseOrder());
      for (Integer key : keys) {
         phases.get(key).stop();
      }
   }
}

SmartLifecycle回调

以下是SmartLifecycle回调的代码解读,统计组内有多少个bean需要stop的bean,逐个调用并等待,或直到超时跳过.
调用stop时传入Runnable作为回调,实现方可以用另外的线程运行关闭逻辑,结束后通知latch

public void stop() {
   this.members.sort(Collections.reverseOrder());
   // 算出组内有多少个bean需要stop的bean
   CountDownLatch latch = new CountDownLatch(this.smartMemberCount);
   Set<String> countDownBeanNames = Collections.synchronizedSet(new LinkedHashSet<>());
   Set<String> lifecycleBeanNames = new HashSet<>(this.lifecycleBeans.keySet());
   // 同一个组内逐个调用`dostop`
   for (LifecycleGroupMember member : this.members) {
      if (lifecycleBeanNames.contains(member.name)) {
         doStop(this.lifecycleBeans, member.name, latch, countDownBeanNames);
      }
      else if (member.bean instanceof SmartLifecycle) {
         // Already removed: must have been a dependent bean from another phase
         latch.countDown();
      }
   }
   // 等待所有bean的stop完结,或直到超时跳过
   try {
      latch.await(this.timeout, TimeUnit.MILLISECONDS);
      if (latch.getCount() > 0 && !countDownBeanNames.isEmpty() && logger.isInfoEnabled()) {
         logger.info("Failed to shut down " + countDownBeanNames.size() + " bean" +
               (countDownBeanNames.size() > 1 ? "s" : "") + " with phase value " +
               this.phase + " within timeout of " + this.timeout + "ms: " + countDownBeanNames);
      }
   }
}

private void doStop(Map<String, ? extends Lifecycle> lifecycleBeans, final String beanName,
      final CountDownLatch latch, final Set<String> countDownBeanNames) {

   Lifecycle bean = lifecycleBeans.remove(beanName);
   if (bean != null) {
       // 优先处理有依赖的bean
      String[] dependentBeans = getBeanFactory().getDependentBeans(beanName);
      for (String dependentBean : dependentBeans) {
         doStop(lifecycleBeans, dependentBean, latch, countDownBeanNames);
      }
      try {
         if (bean.isRunning()) {
            if (bean instanceof SmartLifecycle) {
               countDownBeanNames.add(beanName);
               // 调用stop,传入Runnable作为回调
               // 实现方可以用另外的线程运行关闭逻辑,结束后通知latch
               ((SmartLifecycle) bean).stop(() -> {
                  latch.countDown();
                  countDownBeanNames.remove(beanName);
               });
            }
            else {
               bean.stop();
            }
         }
         else if (bean instanceof SmartLifecycle) {
            // Don't wait for beans that aren't running...
            latch.countDown();
         }
      }
   }
}

Tomcat优雅停机

SpringBoot从2.3支持优雅关机特性,是由WebServerGracefulShutdownLifecycle提供的能力,相关的还有WebServerStartStopLifecycle

WebServerGracefulShutdownLifecyclephaseInteger.MAX_VALUE, 而WebServerStartStopLifecycleInteger.MAX_VALUE - 1,根据上面的分析,在关闭阶段WebServerGracefulShutdownLifecycle先于WebServerStartStopLifecycle执行.

WebServerGracefulShutdownLifecycle

委托调用webServershutDownGracefully()方法,在Tomcat的实现中,会另起一条线程tomcat-shutdown关闭执行关闭逻辑(Netty和Jetty也是另起线程).
tomcat-shutdown里会停止Tomcat再接收新的请求connector.pause(),待Context的业务处理线程结束后,结束这个阶段.

WebServerStartStopLifecycle

如果在有效时间内未能优雅停机,直接进入到WebServerStartStopLifecycle里,调用abort()终止优雅停机,并关闭Tomcat,包括销毁相关线程http-nio-8080-exec-*等,(对这些线程调用interrupt()),一开始提到的守护线程container-0也会因为stopAwait=true而结束.

public void stop() throws WebServerException {
   synchronized (this.monitor) {
      boolean wasStarted = this.started;
      try {
         this.started = false;
         try {
            if (this.gracefulShutdown != null) {
                // 强制停止
               this.gracefulShutdown.abort();
            }
            // 关闭Tomcat,包括销毁相关线程http-nio-8080-exec-*等
            stopTomcat();
            this.tomcat.destroy();
         }
         catch (LifecycleException ex) {
            // swallow and continue
         }
      }
      catch (Exception ex) {
         throw new WebServerException("Unable to stop embedded Tomcat", ex);
      }
      finally {
         if (wasStarted) {
            containerCounter.decrementAndGet();
         }
      }
   }
}

Quartz优雅停机

quartz在spring管理下的核心是SchedulerFactoryBean, 同样实现了SmartLifecycle,还有DisposableBean,在关闭阶段先scheduler.standby(),再scheduler.shutdown()

@Override
public void stop() throws SchedulingException {
   if (this.scheduler != null) {
      try {
         this.scheduler.standby();
      }
      catch (SchedulerException ex) {
         throw new SchedulingException("Could not stop Quartz Scheduler", ex);
      }
   }
}
@Override
public void destroy() throws SchedulerException {
   if (this.scheduler != null) {
      logger.info("Shutting down Quartz Scheduler");
      this.scheduler.shutdown(this.waitForJobsToCompleteOnShutdown);
   }
}

实际上,shutdown()也会调用standby()

public void shutdown(boolean waitForJobsToComplete) {
    
    if(shuttingDown || closed) {
        return;
    }
    shuttingDown = true;
    // shutdown里也调用了standby
    standby();

    //关掉Scheduler的线程,例如quartzScheduler_QuartzSchedulerThread
    schedThread.halt(waitForJobsToComplete);
    
    notifySchedulerListenersShuttingdown();
    //关掉工作线程,例如quartzScheduler_Worker-*
    resources.getThreadPool().shutdown(waitForJobsToComplete);

值得注意的是,如果使用quartz默认的SimpleThreadPool,默认为非守护线程,无论waitForJobsToComplete是否设置,都需要工作线程全部结束才能退出JVM(如果有死循环,只能强制停机)
如果将SimpleThreadPool设置为守护线程,可能导致工作线程未结束就停机,导致Job或Trigger的状态不正常,此时可以设置waitForJobsToComplete=true来规避