xxl-job架构
一.概要
在上一章中,我们介绍了xxl-admin是如果定时查询需要执行的任务,找到该任务后,会封装任务参数,通过配置的策略,从启动的执行器中选择一个具体的url,调用执行器的run方法,进行触发任务,这一篇文章,我们就来详细进行源码分析
二.执行器执行任务
在之前文章中# xxl-job源码分析(二):业务接入方启动过程 中,业务方启动时候,会启动一个netty项目,会在netty中添加四个handler,如图所示
这个类就是处理http请求的handle,当有请求进来的时候,就回触发channelRead方法,对netty不了解的话,可以看我之前写的netty系列文章
netty源码解析
EmbedHttpServerHandler
进入这个handler
protected void channelRead0(final ChannelHandlerContext ctx, FullHttpRequest msg) throws Exception {
// request parse
//final byte[] requestBytes = ByteBufUtil.getBytes(msg.content()); // byteBuf.toString(io.netty.util.CharsetUtil.UTF_8);
// TODO 1.获取请求参数
String requestData = msg.content().toString(CharsetUtil.UTF_8);
//TODO 2.获取请求url =/run
String uri = msg.uri();
//TODO 3.获取请求方式 GET or POST
HttpMethod httpMethod = msg.method();
boolean keepAlive = HttpUtil.isKeepAlive(msg);
String accessTokenReq = msg.headers().get(XxlJobRemotingUtil.XXL_JOB_ACCESS_TOKEN);
// 开启线程执行任务
bizThreadPool.execute(new Runnable() {
@Override
public void run() {
// TODO 4.执行任务
Object responseObj = process(httpMethod, uri, requestData, accessTokenReq);
// to json
String responseJson = GsonTool.toJson(responseObj);
//TODO 5.返回结果(直接返回成功,任务会继续异步执行)
writeResponse(ctx, keepAlive, responseJson);
}
});
}
进入 Object responseObj = process(httpMethod, uri, requestData, accessTokenReq);
public ReturnT<String> run(TriggerParam triggerParam) {
// load old：jobHandler + jobThread TODO 查询这个任务的线程,第一次执行任务,是没有这个任务的
JobThread jobThread = XxlJobExecutor.loadJobThread(triggerParam.getJobId());
IJobHandler jobHandler = jobThread!=null?jobThread.getHandler():null;
String removeOldReason = null;
// valid：jobHandler + jobThread
GlueTypeEnum glueTypeEnum = GlueTypeEnum.match(triggerParam.getGlueType());
if (GlueTypeEnum.BEAN == glueTypeEnum) {
// todo 创建 jobhandler,备注,执行器在启动的时候,会扫描@xxlJob注解修饰的方法,注册到map中,这里直接取出来,IJobHandler就是对jdk的method对象的封装
IJobHandler newJobHandler = XxlJobExecutor.loadJobHandler(triggerParam.getExecutorHandler());
// valid old jobThread
if (jobThread!=null && jobHandler != newJobHandler) {
// change handler, need kill old thread
removeOldReason = "change jobhandler or glue type, and terminate the old job thread.";
jobThread = null;
jobHandler = null;
}
// valid handler
if (jobHandler == null) {
jobHandler = newJobHandler;
if (jobHandler == null) {
return new ReturnT<String>(ReturnT.FAIL_CODE, "job handler [" + triggerParam.getExecutorHandler() + "] not found.");
}
}
} else if (GlueTypeEnum.GLUE_GROOVY == glueTypeEnum) {
// valid old jobThread
if (jobThread != null &&
!(jobThread.getHandler() instanceof GlueJobHandler
&& ((GlueJobHandler) jobThread.getHandler()).getGlueUpdatetime()==triggerParam.getGlueUpdatetime() )) {
// change handler or gluesource updated, need kill old thread
removeOldReason = "change job source or glue type, and terminate the old job thread.";
jobThread = null;
jobHandler = null;
}
// valid handler
if (jobHandler == null) {
try {
IJobHandler originJobHandler = GlueFactory.getInstance().loadNewInstance(triggerParam.getGlueSource());
jobHandler = new GlueJobHandler(originJobHandler, triggerParam.getGlueUpdatetime());
} catch (Exception e) {
logger.error(e.getMessage(), e);
return new ReturnT<String>(ReturnT.FAIL_CODE, e.getMessage());
}
}
}
// executor block strategy
if (jobThread != null) {
ExecutorBlockStrategyEnum blockStrategy = ExecutorBlockStrategyEnum.match(triggerParam.getExecutorBlockStrategy(), null);
if (ExecutorBlockStrategyEnum.DISCARD_LATER == blockStrategy) {
// discard when running
if (jobThread.isRunningOrHasQueue()) {
return new ReturnT<String>(ReturnT.FAIL_CODE, "block strategy effect："+ExecutorBlockStrategyEnum.DISCARD_LATER.getTitle());
}
} else if (ExecutorBlockStrategyEnum.COVER_EARLY == blockStrategy) {
// kill running jobThread
if (jobThread.isRunningOrHasQueue()) {
removeOldReason = "block strategy effect：" + ExecutorBlockStrategyEnum.COVER_EARLY.getTitle();
jobThread = null;
}
} else {
// just queue trigger
}
}
// TODO jobThread 为null的话,就创建一个,并执行线程,是一个死循环,从 triggerQueue中读取参数,并执行
if (jobThread == null) {
jobThread = XxlJobExecutor.registJobThread(triggerParam.getJobId(), jobHandler, removeOldReason);
}
// TODO 把参数放在任务队列线程中的 triggerQueue 中
ReturnT<String> pushResult = jobThread.pushTriggerQueue(triggerParam);
return pushResult;
}
上面这一段业务逻辑比较长,核心是下面几个不走
- 1.找个这个任务对应的JobThread,如果没有,下面会创建
- 2.找个这个JobThread对应的jobHander,如果没有,就从jobHandlerRepository中获取,备注:jobHandlerRepository就是执行器在启动的时候,扫描@xxlJob注解后,放在jobHandlerRepository中,第二篇文章说过了
- 3.jobThread线程会一直死循环, 从triggerQueue中获取参数,执行我们都任务代码
jobThread = XxlJobExecutor.registJobThread(triggerParam.getJobId(), jobHandler, removeOldReason);
- 4.把本次触发的参数放入jobThread的triggerQueue中,让步骤[3]去执行
ReturnT<String> pushResult = jobThread.pushTriggerQueue(triggerParam);
三.jobThread的执行逻辑
这一块代码较长,如下:
public void run() {
// 如果注解中有init方法,就先执行init方法
try {
handler.init();
} catch (Throwable e) {
logger.error(e.getMessage(), e);
}
// execute
while(!toStop){
running = false;
idleTimes++;
TriggerParam triggerParam = null;
try {
// todo 从triggerQueue队列中取值,最多阻塞3s
triggerParam = triggerQueue.poll(3L, TimeUnit.SECONDS);
if (triggerParam!=null) {
running = true;
idleTimes = 0;
triggerLogIdSet.remove(triggerParam.getLogId());
// 记录日志相关,不重要
String logFileName = XxlJobFileAppender.makeLogFileName(new Date(triggerParam.getLogDateTime()), triggerParam.getLogId());
XxlJobContext xxlJobContext = new XxlJobContext(
triggerParam.getJobId(),
triggerParam.getExecutorParams(),
logFileName,
triggerParam.getBroadcastIndex(),
triggerParam.getBroadcastTotal());
// init job context
XxlJobContext.setXxlJobContext(xxlJobContext);
// execute
XxlJobHelper.log("<br>----------- xxl-job job execute start -----------<br>----------- Param:" + xxlJobContext.getJobParam());
//TODO 如果任务配置的时候,有执行时间限制的话,会使用callback方式执行
if (triggerParam.getExecutorTimeout() > 0) {
// limit timeout
Thread futureThread = null;
try {
FutureTask<Boolean> futureTask = new FutureTask<Boolean>(new Callable<Boolean>() {
@Override
public Boolean call() throws Exception {
// init job context
XxlJobContext.setXxlJobContext(xxlJobContext);
handler.execute();
return true;
}
});
futureThread = new Thread(futureTask);
futureThread.start();
Boolean tempResult = futureTask.get(triggerParam.getExecutorTimeout(), TimeUnit.SECONDS);
} catch (TimeoutException e) {
XxlJobHelper.log("<br>----------- xxl-job job execute timeout");
XxlJobHelper.log(e);
// handle result
XxlJobHelper.handleTimeout("job execute timeout ");
} finally {
futureThread.interrupt();
}
} else {
// TODO 调用我们都@xxlJob修饰的任务方法
handler.execute();
}
// valid execute handle data
if (XxlJobContext.getXxlJobContext().getHandleCode() <= 0) {
XxlJobHelper.handleFail("job handle result lost.");
} else {
String tempHandleMsg = XxlJobContext.getXxlJobContext().getHandleMsg();
tempHandleMsg = (tempHandleMsg!=null&&tempHandleMsg.length()>50000)
?tempHandleMsg.substring(0, 50000).concat("...")
:tempHandleMsg;
XxlJobContext.getXxlJobContext().setHandleMsg(tempHandleMsg);
}
XxlJobHelper.log("<br>----------- xxl-job job execute end(finish) -----------<br>----------- Result: handleCode="
+ XxlJobContext.getXxlJobContext().getHandleCode()
+ ", handleMsg = "
+ XxlJobContext.getXxlJobContext().getHandleMsg()
);
} else {
if (idleTimes > 30) {
if(triggerQueue.size() == 0) { // avoid concurrent trigger causes jobId-lost
XxlJobExecutor.removeJobThread(jobId, "excutor idel times over limit.");
}
}
}
} catch (Throwable e) {
if (toStop) {
XxlJobHelper.log("<br>----------- JobThread toStop, stopReason:" + stopReason);
}
// TODO 失败的话,日志中记录失败信息
StringWriter stringWriter = new StringWriter();
e.printStackTrace(new PrintWriter(stringWriter));
String errorMsg = stringWriter.toString();
XxlJobHelper.handleFail(errorMsg);
XxlJobHelper.log("<br>----------- JobThread Exception:" + errorMsg + "<br>----------- xxl-job job execute end(error) -----------");
} finally {
if(triggerParam != null) {
// callback handler info
if (!toStop) {
// TODO 把结果放入callBackQueue 队列中,用于其他线程告诉xxl-admin执行的结果
TriggerCallbackThread.pushCallBack(new HandleCallbackParam(
triggerParam.getLogId(),
triggerParam.getLogDateTime(),
XxlJobContext.getXxlJobContext().getHandleCode(),
XxlJobContext.getXxlJobContext().getHandleMsg() )
);
} else {
// is killed
TriggerCallbackThread.pushCallBack(new HandleCallbackParam(
triggerParam.getLogId(),
triggerParam.getLogDateTime(),
XxlJobContext.HANDLE_CODE_FAIL,
stopReason + " [job running, killed]" )
);
}
}
}
}
// callback trigger request in queue
while(triggerQueue !=null && triggerQueue.size()>0){
TriggerParam triggerParam = triggerQueue.poll();
if (triggerParam!=null) {
// is killed
TriggerCallbackThread.pushCallBack(new HandleCallbackParam(
triggerParam.getLogId(),
triggerParam.getLogDateTime(),
XxlJobContext.HANDLE_CODE_FAIL,
stopReason + " [job not executed, in the job queue, killed.]")
);
}
}
// TODO 执行任务的destroy方法
try {
handler.destroy();
} catch (Throwable e) {
logger.error(e.getMessage(), e);
}
logger.info(">>>>>>>>>>> xxl-job JobThread stoped, hashCode:{}", Thread.currentThread());
}
核心逻辑如下:
- 1.如果注解中有init方法,就先执行init方法
- 2.从triggerQueue队列中取出任务执行的参数,最多阻塞3s
- 3.记录日志
- 4.执行任务
- 4-1.如果任务执行失败,就记录失败信息到日志中
- 4-2 最后把执行结果放在 TriggerCallbackThread线程的callBackQueue队列中,TriggerCallbackThread线程回回调这些结果
- 5.如果任务停止的话,就调用任务的destroy方法
四.jobThrad线程什么时候会被终止
因为每一个任务执行的时候,都会对应一个jobThread,如果这个任务不是周期性的执行,而是只执行一次的话,那么这个线程就需要在一定的时机,关闭线程,释放资源
4.1 循环30次,这个线程没有任务,关闭
在上一章中,执行任务是一个死循环,会记录循环的次数
4.2 项目关闭的时候,会关闭所有的任务的线程池
4.3 调速器主动发送kill请求的时候
五,结果回填
在第四章中,讲解了把执行结果放在 TriggerCallbackThread线程的callBackQueue队列中,TriggerCallbackThread线程回回调这些结果,大家可以去看一下,也比较简单,# xxl-job源码分析(二):业务接入方启动过程的 [ 3.再来看start方法中的 TriggerCallbackThread线程]中也写了
