Spring-Quartz企业级理解-调度器的运行

1,282 阅读7分钟

在调度器的初始化文章中说到,由于SchedulerFactoryBean实现了SmartLifecycle接口,将在spring容器启动后,自动执行start()方法

public void start() throws SchedulingException {
    if (this.scheduler != null) {
        try {
            this.startScheduler(this.scheduler, this.startupDelay);
        } catch (SchedulerException var2) {
            throw new SchedulingException("Could not start Quartz Scheduler", var2);
        }
    }
}

可以看到,真正执行启动的是startScheduler方法

protected void startScheduler(final Scheduler scheduler, final int startupDelay) throws SchedulerException {
    if (startupDelay <= 0) {
        this.logger.info("Starting Quartz Scheduler now");
        scheduler.start();
    } else {
        Thread schedulerThread = new Thread() {
            public void run() {
                try {
                    // 这里可以在xml中配置startupDelay,延迟启动StdScheduler.start()方法,主要作用是错开数据库的使用高峰(由于业务初始化时也需要从数据库读取数据)
                    TimeUnit.SECONDS.sleep((long)startupDelay);
                } catch (InterruptedException var3) {
                    Thread.currentThread().interrupt();
                }
                try {
                    scheduler.start();
                } catch (SchedulerException var2) {
                    throw new SchedulingException("Could not start Quartz Scheduler after delay", var2);
                }
            }
        };
        schedulerThread.setName("Quartz Scheduler [" + scheduler.getSchedulerName() + "]");
        schedulerThread.setDaemon(true);
        schedulerThread.start();
    }
}

进入StdScheduler.start()方法,实际执行的是QuartzScheduler.start()方法

public void start() throws SchedulerException {
    sched.start();
}

QuartzScheduler.start()方法关键代码如下

public void start() throws SchedulerException {
    ....
        if (initialStart == null) {
            // 第一次启动应该进入这个分支,jobStore的实际类是LocalDataSourceJobStore(在shcedulerFactory中初始化过)
            initialStart = new Date();
            this.resources.getJobStore().schedulerStarted();            
            startPlugins();
        } else {
            resources.getJobStore().schedulerResumed();
        }
    // 在上一篇的最后说到,在QuartzScheduler的构造方法中初始化并运行了QuartzSchedulerThread schedThread对象
    // 在此将QuartzSchedulerThread的pause字段设置为false
    // QuartzSchedulerThread的构造方法将pause设置为true的,实际上QuartzSchedulerThread.run()的业务会被paused=true阻塞
    schedThread.togglePause(false);
    ....
}

LocalDataSourceJobStore继承了JobStoreSupport类,实际执行的是JobstoreSupport.schedulerStarted()方法

public void schedulerStarted() throws SchedulerException {
    if (isClustered()) {
        // clusterManagementThread是JobStoreSupport的一个内部类,在此执行ClusterManager.initialize()
        clusterManagementThread = new ClusterManager();
        if(initializersLoader != null)
            clusterManagementThread.setContextClassLoader(initializersLoader);
        clusterManagementThread.initialize();
    } else {
        try {
            recoverJobs();
        } catch (SchedulerException se) {
            throw new SchedulerConfigException(
                "Failure occured during job recovery.", se);
        }
    }
    ....
        schedulerRunning = true;
    ....
}

ClusterManager是Thread的子类

public void initialize() {
    this.manage();
    ThreadExecutor executor = getThreadExecutor();
    executor.execute(ClusterManager.this);
}

实际执行的是ClusterManager.manage()方法,该方法的核心是JobStoreSupport.doCheckin()方法

private boolean manage() {
    boolean res = false;
    try {
        res = doCheckin();
        numFails = 0;
        getLog().debug("ClusterManager: Check-in complete.");
    } catch (Exception e) {
        if(numFails % 4 == 0) {
            getLog().error(
                "ClusterManager: Error managing cluster: "
                + e.getMessage(), e);
        }
        numFails++;
    }
    return res;
}

JobStoreSupport.doCheckIn()的作用是检查这个scheduler以及scheduler下挂的trigger和job在数据库中的状态是否正常

protected boolean doCheckin() throws JobPersistenceException {
    boolean transOwner = false;
    boolean transStateOwner = false;
    boolean recovered = false;
    Connection conn = getNonManagedTXConnection();
    try {
        List<SchedulerStateRecord> failedRecords = null;
        if (!firstCheckIn) {
            failedRecords = clusterCheckIn(conn);
            commitConnection(conn);
        }
        // 第一次checkIn(scheduler启动)进入if分支
        if (firstCheckIn || (failedRecords.size() > 0)) {
            // lockHandler对象的类型是StdRowLockSemaphore,这里目的是使用select * for update从qrtz_locks表获得STATE_ACCESS锁(如不存在会insert)
            // TATE_ACCESS锁在commit之前,集群中的其他节点获取锁的时候会阻塞等待
            // 只有先获得STATE_ACCESS锁,才会执行后面的代码
            // 实际执行sql的代码在StdRowLockSemaphore.executeSQL()方法,比较简单,在此不展开
            getLockHandler().obtainLock(conn, "STATE_ACCESS");
            transStateOwner = true;
            // 第一次checkIn(scheduler启动)进入clusterCheckIn(conn)方法
            // clusterCheckIn()方法其实就是把自己的数据插入到qrtz_scheduler_state表(存在则update),并且clusterCheckIn()调用了findFailedInstances()
            // findFailedInstances()的作用是取回所有qrtz_scheduler_state表中长时间未checkIn(认为集群节点已经下线)的scheduler的实例
            failedRecords = (firstCheckIn) ? clusterCheckIn(conn) : findFailedInstances(conn);
            if (failedRecords.size() > 0) {
                getLockHandler().obtainLock(conn, "TRIGGER_ACCESS");
                transOwner = true;
                clusterRecover(conn, failedRecords);
                recovered = true;
            }
        }
        // commit释放数据库锁
        commitConnection(conn);
    } catch (JobPersistenceException e) {
        rollbackConnection(conn);
        throw e;
    } finally {
        // 释放前面获得的线程锁(保存在ThradLocal中)
        try {
            releaseLock(LOCK_TRIGGER_ACCESS, transOwner);
        } finally {
            try {
                releaseLock(LOCK_STATE_ACCESS, transStateOwner);
            } finally {
                cleanupConnection(conn);
            }
        }
    }
    firstCheckIn = false;
    return recovered;
}

上面一段代码,终于到了最关键的地方,JobStoreSupport到底是如何与数据库交互的,请看下面的分析

protected List<SchedulerStateRecord> findFailedInstances(Connection conn)
    throws JobPersistenceException {
    try {
        List<SchedulerStateRecord> failedInstances = new LinkedList<SchedulerStateRecord>();
        boolean foundThisScheduler = false;
        long timeNow = System.currentTimeMillis();
        // 真正与数据库交互的是Delegate对象,这里的实现类是StdJDBCDelegate
        // 真正执行的sql是SELECT * FROM {0}SCHEDULER_STATE WHERE SCHED_NAME = {1} ({0}和{1}是占位符,{0}是表的前缀,{1}是schedulerName)
        List<SchedulerStateRecord> states = getDelegate().selectSchedulerStateRecords(conn, null);
        for(SchedulerStateRecord rec: states) {
            // 第一次启动由于instanceId中的时间戳是实时生成的,应进入else分支
            if (rec.getSchedulerInstanceId().equals(getInstanceId())) {
                foundThisScheduler = true;
                if (firstCheckIn) {
                    failedInstances.add(rec);
                }
            } else {
                // 计算LAST_CHECKIN_TIME + 一定的时间间隔余量是否 < 当前时间,如果成立,说明长时间未更新,认为该scheduler下线了,add进入failedInstances
                if (calcFailedIfAfter(rec) < timeNow) {
                    failedInstances.add(rec);
                }
            }
        }
        if (firstCheckIn) {
            // findOrphanedFailedInstances()的目的是读取qrtz_fired_triggers表中的数据行的INSTANCE_NAME不在qrtz_scheduler_state表的INSTANCE_NAME的数据
            // SELECT DISTINCT INSTANCE_NAME FROM {0}FIRED_TRIGGERS WHERE SCHED_NAME = {1}
            failedInstances.addAll(findOrphanedFailedInstances(conn, states));
        }
        if ((!foundThisScheduler) && (!firstCheckIn)) {
            getLog().warn(
                "This scheduler instance (" + getInstanceId() + ") is still " + 
                "active but was recovered by another instance in the cluster.  " +
                "This may cause inconsistent behavior.");
        }
        return failedInstances;
    } catch (Exception e) {
        lastCheckin = System.currentTimeMillis();
        throw new JobPersistenceException("Failure identifying failed instances when checking-in: "
                                          + e.getMessage(), e);
    }
}

findFailedInstances()方法会找出所有被认为已经下线的scheduler,返回到doCheckIn()方法的failedRecords对象中 如果failedRecords.size() > 0,那么在获取TRIGGER_ACCESS后执行clusterRecover(conn, failedRecords) 一个健康的集群,一般不会进入clusterRecover()方法中,这里可以放到后面再讲 至此,ClusterManager线程的初始化完成,让我们回到另外一个关键的线程QuartzSchedulerThread 在调度器的启动篇的最后,在最后构造QuartzScheduler的时候,初始化QuartzSchedulerThread线程对象并执行,以下是QuartzSchedulerThread的构造方法

QuartzSchedulerThread(QuartzScheduler qs, QuartzSchedulerResources qsRsrcs, boolean setDaemon, int threadPrio) {
    super(qs.getSchedulerThreadGroup(), qsRsrcs.getThreadName());
    this.qs = qs;
    this.qsRsrcs = qsRsrcs;
    this.setDaemon(setDaemon);
    if(qsRsrcs.isThreadsInheritInitializersClassLoadContext()) {
        log.info("QuartzSchedulerThread Inheriting ContextClassLoader of thread: " + Thread.currentThread().getName());
        this.setContextClassLoader(Thread.currentThread().getContextClassLoader());
    }
    this.setPriority(threadPrio);
    // paused = true会阻止run()方法中实际内容的执行
    paused = true;
    halted = new AtomicBoolean(false);
}

QuartzSchedulerThread的run()方法中的实际内容的执行受到两个关键boolean变量的控制 1.paused,初始化为true,在执行完QuartzScheduler.schedulerStarted()方法后,置为false 2.halted,只有在shutdown的时候会变为true

public void run() {
    int acquiresFailed = 0;
    while (!halted.get()) {
        try {
            synchronized (sigLock) {
                // paused初始化为true,会一直执行while循环直到pause=true
                while (paused && !halted.get()) {
                    try {
                        sigLock.wait(1000L);
                    } catch (InterruptedException ignore) {
                    }
                    acquiresFailed = 0;
                }
                if (halted.get()) {
                    break;
                }
            }
            if (acquiresFailed > 1) {
                try {
                    long delay = computeDelayForRepeatedErrors(qsRsrcs.getJobStore(), acquiresFailed);
                    Thread.sleep(delay);
                } catch (Exception ignore) {
                }
            }
            int availThreadCount = qsRsrcs.getThreadPool().blockForAvailableThreads();
            // 总会进入if分支,因为上一步获取数量的时候是阻塞的
            if(availThreadCount > 0) {
                List<OperableTrigger> triggers;
                long now = System.currentTimeMillis();
                clearSignaledSchedulingChange();
                try {
                    // 先获取即将需要运行的trigger,进入展开部分[1]
                    triggers = qsRsrcs.getJobStore().acquireNextTriggers(now + idleWaitTime, Math.min(availThreadCount, qsRsrcs.getMaxBatchSize()), qsRsrcs.getBatchTimeWindow());
                    acquiresFailed = 0;
                    if (log.isDebugEnabled())
                        log.debug("batch acquisition of " + (triggers == null ? 0 : triggers.size()) + " triggers");
                } catch (JobPersistenceException jpe) {
                    if (acquiresFailed == 0) {
                        qs.notifySchedulerListenersError("An error occurred while scanning for the next triggers to fire.", jpe);
                    }
                    if (acquiresFailed < Integer.MAX_VALUE)
                        acquiresFailed++;
                    continue;
                } catch (RuntimeException e) {
                    if (acquiresFailed == 0) {
                        getLog().error("quartzSchedulerThreadLoop: RuntimeException " + e.getMessage(), e);
                    }
                    if (acquiresFailed < Integer.MAX_VALUE)
                        acquiresFailed++;
                    continue;
                }
                if (triggers != null && !triggers.isEmpty()) {
                    now = System.currentTimeMillis();
                    // 按照NEXT_FIRE_TIME升序,所以第一个一定是最早触发的
                    long triggerTime = triggers.get(0).getNextFireTime().getTime();
                    long timeUntilTrigger = triggerTime - now;
                    while(timeUntilTrigger > 2) {
                        synchronized (sigLock) {
                            if (halted.get()) {
                                break;
                            }
                            if (!isCandidateNewTimeEarlierWithinReason(triggerTime, false)) {
                                try {
                                    now = System.currentTimeMillis();
                                    timeUntilTrigger = triggerTime - now;
                                    if(timeUntilTrigger >= 1)
                                        sigLock.wait(timeUntilTrigger);
                                } catch (InterruptedException ignore) {
                                }
                            }
                        }
                        if(releaseIfScheduleChangedSignificantly(triggers, triggerTime)) {
                            break;
                        }
                        now = System.currentTimeMillis();
                        timeUntilTrigger = triggerTime - now;
                    }
                    if(triggers.isEmpty())
                        continue;
                    List<TriggerFiredResult> bndles = new ArrayList<TriggerFiredResult>();
                    boolean goAhead = true;
                    synchronized(sigLock) {
                        goAhead = !halted.get();
                    }
                    if(goAhead) {
                        try {
                            // 更新已经获取的triggers的状态,包括qrtz_triggers表的下次触发时间STATUS字段和NEXT_FIRE_TIME字段
                            // 以及qrtz_fired_triggers表的STATE字段,进入展开部分[2]
                            List<TriggerFiredResult> res = qsRsrcs.getJobStore().triggersFired(triggers);
                            if(res != null)
                                bndles = res;
                        } catch (SchedulerException se) {
                            qs.notifySchedulerListenersError("An error occurred while firing triggers '" + triggers + "'", se);
                            for (int i = 0; i < triggers.size(); i++) {
                                qsRsrcs.getJobStore().releaseAcquiredTrigger(triggers.get(i));
                            }
                            continue;
                        }
                    }
                    for (int i = 0; i < bndles.size(); i++) {
                        TriggerFiredResult result =  bndles.get(i);
                        TriggerFiredBundle bndle =  result.getTriggerFiredBundle();
                        Exception exception = result.getException();
                        if (exception instanceof RuntimeException) {
                            getLog().error("RuntimeException while firing trigger " + triggers.get(i), exception);
                            qsRsrcs.getJobStore().releaseAcquiredTrigger(triggers.get(i));
                            continue;
                        }
                        if (bndle == null) {
                            qsRsrcs.getJobStore().releaseAcquiredTrigger(triggers.get(i));
                            continue;
                        }
                        JobRunShell shell = null;
                        try {
                            shell = qsRsrcs.getJobRunShellFactory().createJobRunShell(bndle);
                            shell.initialize(qs);
                        } catch (SchedulerException se) {
                            qsRsrcs.getJobStore().triggeredJobComplete(triggers.get(i), bndle.getJobDetail(), CompletedExecutionInstruction.SET_ALL_JOB_TRIGGERS_ERROR);
                            continue;
                        }
                        // 执行定时任务的具体内容,通过线程池执行JobRunShell.run()方法,进入展开部分[3]
                        if (qsRsrcs.getThreadPool().runInThread(shell) == false) {
                            getLog().error("ThreadPool.runInThread() return false!");
                            qsRsrcs.getJobStore().triggeredJobComplete(triggers.get(i), bndle.getJobDetail(), CompletedExecutionInstruction.SET_ALL_JOB_TRIGGERS_ERROR);
                        }
                    }
                    continue;
                }
            } else {
                continue;
            }
            long now = System.currentTimeMillis();
            long waitTime = now + getRandomizedIdleWaitTime();
            long timeUntilContinue = waitTime - now;
            synchronized(sigLock) {
                try {
                  if(!halted.get()) {
                    if (!isScheduleChanged()) {
                      sigLock.wait(timeUntilContinue);
                    }
                  }
                } catch (InterruptedException ignore) {
                }
            }

        } catch(RuntimeException re) {
            getLog().error("Runtime error occurred in main trigger firing loop.", re);
        }
    }
    qs = null;
    qsRsrcs = null;
}

展开部分[1] triggers = qsRsrcs.getJobStore().acquireNextTriggers()先获取并更新qrtz_triggers表的statues=acquired,让其他节点不能重复获取trigger

protected List<OperableTrigger> acquireNextTrigger(Connection conn, long noLaterThan, int maxCount, long timeWindow)
    throws JobPersistenceException {
    if (timeWindow < 0) {
      throw new IllegalArgumentException();
    }
    List<OperableTrigger> acquiredTriggers = new ArrayList<OperableTrigger>();
    Set<JobKey> acquiredJobKeysForNoConcurrentExec = new HashSet<JobKey>();
    final int MAX_DO_LOOP_RETRY = 3;
    int currentLoopCount = 0;
    do {
        currentLoopCount ++;
        try {
            // 通过以下sql获取需要即将触发的trigger的name和group
            // NEXT_FIRE_TIME的默认范围是[当前时间-60s,当前时间+30s]
            // SELECT TRIGGER_NAME, TRIGGER_GROUP, NEXT_FIRE_TIME, PRIORITY FROM {0}TRIGGERS
            // WHERE SCHED_NAME = {1} AND TRIGGER_STATE = WAITING AND NEXT_FIRE_TIME <= ? AND 
            // (MISFIRE_INSTR = -1 OR (MISFIRE_INSTR != -1 AND NEXT_FIRE_TIME >= ?)) 
            // ORDER BY NEXT_FIRE_TIME ASC, PRIORITY DESC
            List<TriggerKey> keys = getDelegate().selectTriggerToAcquire(conn, noLaterThan + timeWindow, getMisfireTime(), maxCount);
            if (keys == null || keys.size() == 0)
                return acquiredTriggers;
            long batchEnd = noLaterThan;
            for(TriggerKey triggerKey: keys) {
                // 获取trigger实例
                // SELECT * FROM {0}TRIGGERS WHERE SCHED_NAME = {1} AND TRIGGER_NAME = ? AND TRIGGER_GROUP = ?
                OperableTrigger nextTrigger = retrieveTrigger(conn, triggerKey);
                if(nextTrigger == null) {
                    continue;
                }
                JobKey jobKey = nextTrigger.getJobKey();
                JobDetail job;
                try {
                    // 从qrtz_job_details表读取定时任务的详情
                    // SELECT * FROM {0}JOB_DETAILS WHERE SCHED_NAME = {1} AND JOB_NAME = ? AND JOB_GROUP = ?
                    job = retrieveJob(conn, jobKey);
                } catch (JobPersistenceException jpe) {
                    try {
                        getLog().error("Error retrieving job, setting trigger state to ERROR.", jpe);
                        getDelegate().updateTriggerState(conn, triggerKey, STATE_ERROR);
                    } catch (SQLException sqle) {
                        getLog().error("Unable to set trigger state to ERROR.", sqle);
                    }
                    continue;
                }
                if (job.isConcurrentExectionDisallowed()) {
                    if (acquiredJobKeysForNoConcurrentExec.contains(jobKey)) {
                        continue;
                    } else {
                        acquiredJobKeysForNoConcurrentExec.add(jobKey);
                    }
                }
                Date nextFireTime = nextTrigger.getNextFireTime();
                if (nextFireTime == null) {
                    log.warn("Trigger {} returned null on nextFireTime and yet still exists in DB!",
                        nextTrigger.getKey());
                    continue;
                }
                if (nextFireTime.getTime() > batchEnd) {
                  break;
                }
                // 先校验qrtz_triggers中的状态为WAITING,然后更新状态为ACQUIRED,这是一个原子操作
                // UPDATE {0}TRIGGERS SET TRIGGER_STATE = ACQUIRED 
                // WHERE SCHED_NAME = {1} AND TRIGGER_NAME = ? AND TRIGGER_GROUP = ? 
                // AND TRIGGER_STATE = WAITING
                int rowsUpdated = getDelegate().updateTriggerStateFromOtherState(conn, triggerKey, STATE_ACQUIRED, STATE_WAITING);
                if (rowsUpdated <= 0) {
                    continue;
                }
                nextTrigger.setFireInstanceId(getFiredTriggerRecordId());
                // 插入qrtz_fired_triggers表
                // INSERT INTO {0}FIRED_TRIGGERS 
                // (SCHED_NAME, ENTRY_ID, TRIGGER_NAME, TRIGGER_GROUP, INSTANCE_NAME, 
                // FIRED_TIME, SCHED_TIME, STATE, JOB_NAME, JOB_GROUP, IS_NONCONCURRENT, 
                // REQUESTS_RECOVERY, PRIORITY) 
                // VALUES({1}, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                int ans = getDelegate().insertFiredTrigger(conn, nextTrigger, STATE_ACQUIRED, null);
                if(acquiredTriggers.isEmpty()) {
                    batchEnd = Math.max(nextFireTime.getTime(), System.currentTimeMillis()) + timeWindow;
                }
                acquiredTriggers.add(nextTrigger);
            }
            if(acquiredTriggers.size() == 0 && currentLoopCount < MAX_DO_LOOP_RETRY) {
                continue;
            }
            break;
        } catch (Exception e) {
            throw new JobPersistenceException("Couldn't acquire next trigger: " + e.getMessage(), e);
        }
    } while (true);
    return acquiredTriggers;
}

展开部分[2] 更新trigger为正在执行的状态

protected TriggerFiredBundle triggerFired(Connection conn, OperableTrigger trigger)
    throws JobPersistenceException {
    JobDetail job;
    Calendar cal = null;
    try {
        // 先校验trigger的状态,必须为ACQUIRED
        // SELECT TRIGGER_STATE FROM {0}TRIGGERS WHERE SCHED_NAME = {1} AND TRIGGER_NAME = ? AND TRIGGER_GROUP = ?
        String state = getDelegate().selectTriggerState(conn, trigger.getKey());
        if (!state.equals(STATE_ACQUIRED)) {
            return null;
        }
    } catch (SQLException e) {
        throw new JobPersistenceException("Couldn't select trigger state: " + e.getMessage(), e);
    }
    try {
        // 从qrtz_job_details表取回jobdetail
        // SELECT * FROM {0}JOB_DETAILS WHERE SCHED_NAME = {1} AND JOB_NAME = ? AND JOB_GROUP = ?
        job = retrieveJob(conn, trigger.getJobKey());
        ....
        if (job == null) { return null; }
    } catch (JobPersistenceException jpe) {
        ....
    }
    ....
    try {
        // 将qrtz_fired_triggers表的STATE更新为EXECUTING
        // UPDATE {0}FIRED_TRIGGERS SET 
        // INSTANCE_NAME = ?, FIRED_TIME = ?, SCHED_TIME = ?, 
        // STATE = "EXECUTING", JOB_NAME = ?, JOB_GROUP = ?, IS_NONCONCURRENT = ?, 
        // REQUESTS_RECOVERY = ? WHERE SCHED_NAME = {1} AND ENTRY_ID = ?
        getDelegate().updateFiredTrigger(conn, trigger, "EXECUTING", job);
    } catch (SQLException e) {
        throw new JobPersistenceException("Couldn't insert fired trigger: " + e.getMessage(), e);
    }
    Date prevFireTime = trigger.getPreviousFireTime();
    // 计算下次触发时间
    trigger.triggered(cal);
    // 如果job是可以并行执行的(上一次触发还没运行完,不可以运行下一次触发的内容),那么更新qrtz_triggers表的状态为WAITING
    // 如果job是不可并行执行的(上一次触发还没运行完,可以运行下一次触发的内容),那么更新qrtz_triggers表的状态为BLOCKED
    // 是否可并行执行,取决于qrtz_job_details表的IS_NONCONCURRENT字段,如果给定时任务类打上@DisallowConcurrentExecution注解,那么IS_NONCONCURRENT=1
    String state = "WAITING";
    boolean force = true;
    if (job.isConcurrentExectionDisallowed()) {
        state = "BLOCKED";
        force = false;
        try {
            // UPDATE {0}TRIGGERS SET TRIGGER_STATE = {OLD_STATE}
            // WHERE SCHED_NAME = {1} AND JOB_NAME = ? AND JOB_GROUP = ? AND TRIGGER_STATE = {NEW_STATE}
            getDelegate().updateTriggerStatesForJobFromOtherState(conn, job.getKey(),
                    STATE_BLOCKED, STATE_WAITING);
            getDelegate().updateTriggerStatesForJobFromOtherState(conn, job.getKey(),
                    STATE_BLOCKED, STATE_ACQUIRED);
            getDelegate().updateTriggerStatesForJobFromOtherState(conn, job.getKey(),
                    STATE_PAUSED_BLOCKED, STATE_PAUSED);
        } catch (SQLException e) {
            throw new JobPersistenceException("Couldn't update states of blocked triggers: " + e.getMessage(), e);
        }
    } 
    ....
    // 将qrtz_triggers表的状态置为WAITING(可以并发执行)或者BLOCKED(不可以并发执行)
    // 回看展开[1]中的selectTriggerToAcquire方法,选择接下来触发的trigger的必要条件是状态=WAITING
    // 所以状态=BLOCKED的trigger不会被选中,也就无法触发,释放trigger的操作在展开[3]中叙述
    storeTrigger(conn, trigger, job, true, state, force, false);
    job.getJobDataMap().clearDirtyFlag();
    return new TriggerFiredBundle(job, trigger, cal, trigger.getKey().getGroup()
            .equals(Scheduler.DEFAULT_RECOVERY_GROUP), new Date(), trigger
            .getPreviousFireTime(), prevFireTime, trigger.getNextFireTime());
}

展开部分[3] 在JobRunShell.run()方法结尾,会执行jobStore.triggeredJobComplete()方法,更新数据库中trigger的状态

protected void triggeredJobComplete(Connection conn, OperableTrigger trigger, JobDetail jobDetail,
    CompletedExecutionInstruction triggerInstCode) throws JobPersistenceException {
    try {
        ....
        if (jobDetail.isConcurrentExectionDisallowed()) {
            // qrtz_triggers表的状态从BLOCKED更新为WAITING
            // UPDATE {0}TRIGGERS SET TRIGGER_STATE = ? WHERE SCHED_NAME = {1} AND JOB_NAME = ? AND JOB_GROUP = ? AND TRIGGER_STATE = ?
            getDelegate().updateTriggerStatesForJobFromOtherState(conn, jobDetail.getKey(), STATE_WAITING, STATE_BLOCKED);
            getDelegate().updateTriggerStatesForJobFromOtherState(conn, jobDetail.getKey(), STATE_PAUSED, STATE_PAUSED_BLOCKED);
            signalSchedulingChangeOnTxCompletion(0L);
        }
        ....
    } catch (SQLException e) {
        throw new JobPersistenceException("Couldn't update trigger state(s): " + e.getMessage(), e);
    }
    try {
        // 删除qrtz_fired_triggers表中的数据
        // DELETE FROM {0}FIRED_TRIGGERS WHERE SCHED_NAME = {1} AND ENTRY_ID = ?
        getDelegate().deleteFiredTrigger(conn, trigger.getFireInstanceId());
    } catch (SQLException e) {
        throw new JobPersistenceException("Couldn't delete fired trigger: " + e.getMessage(), e);
    }
}

至此,可以总结一下调度的过程 1.从qrtz_triggers表读取NEXT_FIRE_TIME在时间范围内,并且STATUS=WAITING的触发器 2.在qrtz_triggers表更新获取到的触发器的STATUS=ACQUIRED,在qrtz_fired_triggers表插入数据,STATUS=ACQUIRED 3.更新qrtz_triggers表的STATUS=WAITIING(或者BLOCKED,取决于定时任务能否并发执行),更新qrtz_fired_triggers表的STATUS=EXECUTING 4.执行定时任务,完成后删除qrtz_fired_triggers的数据(如果定时任务不能并发执行,同时更新qrtz_triggers表的STATUS=WAITIING) 可以看出,Quartz的原理并不是轮询每个节点,而是每个节点来竞争触发器,能够竞争到触发器,说明当前节点确实比较空闲,能够腾出资源来赢得竞争,符合负载均衡的定义