问题现象

线上网关服务器的内存在运行几天后，内存占用非常大；以至于超过了内存最低空闲率引起报警。

如果不加干预，小内存的服务器会直接内存消耗完而不能响应，造成服务崩溃。

这是在一台线上服务器统计记录的内存消耗数据图表：

可以看到不超过一周，每个nginx worker的内存占用都接近了1G，内存占用是很大的；且每天都在增加，说明有非常明显的内存泄漏问题。

问题排查

开始的时候不清楚什么数据一直在吃内存，经过梳理分析后，有下面几个场景，

内存模型

1. system memory in worker level

• lua code 4M - userdata, object
• runtime memory object, like string, array, pointer

2. share dict in system level
3. lrucache in worker level

可能的问题

1. plugin injection
2. balancer handle
3. syslog output
4. health checker

结合网关的两台开发环境107，144来看，平时一直空跑着，没有业务使用；但是内存占用也是一样的每天都在增长，也是一样的慢慢快占完了内存。由此推断出问题与正常跑业务使用的内存活动行为无关。

而平时空跑的网关服务，只有集群事件的同步和健康检查两个活动行为。集群事件同步是简单的http请求，且每5秒执行一次，http请求都是短链接，基本不消耗内存，且请求完socket就关闭了，资源也可以及时回收掉。健康检查是所有的上游服务都有的行为，且有主动健康和主动不健康两个后台作业服务，且健康检测业务相对比较重，分为tcp, http, https集中检测运行方式。

结论是怀疑健康检测造成的内存泄漏问题比较大。

健康检查器的问题排查

从health checker自身业务逻辑看，业务运行使用了大量的timer来做后台定时作业管理，和处理异步并发的锁行为。

再次检查代码发现checker中有对timer做缓存处理，怀疑缓存的timer造成了内存资源占用着一直无法释放 - 网关的服务容器是openresty, 且网关代码中使用timer都是openresty ngx_lua的ngx.timer.* API; openresty官方有说明ngx.timer.at API是一个模拟的http request来实现的，如果ngx.timer.at一直不结束，那么对应使用的资源会一直不释放，还包括timer本身的模拟tcp链接资源。

印象中网关之前的内存占用一直不大，且没出现过占用内存一直增长的问题(也可能与现在版本迭代一般一周左右一次有关，重新部署后覆盖了问题)，而网关对health checker的依赖包版本近半年来做过更新，从原先的1.0版本升级到了2.0版本。于是就返回去看了下当前的health checker组件的历史版本，发现现在存在着多个版本，而且最近更新和维护的也不是2.0版本，而是一个1.4的子版本1.4.2。

对比health checker 1.4.2与2.0版本的代码发现，两者存在较大差异，且1.4.2版本没有对ngx.timer.at的缓存处理，于是决定立即切换health checker的版本为1.4.2，考虑到问题分析和后面可能会进行调整，便决定拷贝一份1.4.2源码下来单独维护。代码单独维护后，去除了一些不适合实际需要的配置代码，主业务逻辑不变。

这样新的health checker部署后，在测试、预发环境运行一段时间后，发现内存使用率比较小，且没有出现不断快速增长的情况，内存占用相对稳定。于是观察金两周后，把新的代码发布上线了。上线后内存占用情况观察了两天，也是比较理想，与测试和预发环境一样，相对很稳定。

不过，在发布上线前两天，测试环境出现了两起业务方服务节点进行了重启，而重启前后网关没有及时对业务节点进行及时上线/下线处理，造成了业务服务请求过来时，没有服务节点可用，业务不可用的情况出现。而上线前一天预发环境发生了两起类似的问题，都是网关服务部署几天后业务服务运行中进行了重启操作，而健康检测没有及时检测到且做出相应上线下线节点操作，引起了服务不可用。考虑到问题出现具有偶然性，且不是大面积的很多服务都有此类问题，于是没有进行回滚操作；但问题一直在排查。

重要的是线上部署发布后，也出现了两个业务方服务进行重启或重新部署操作，出现了一样的问题。除了紧急通过手动脚本处理变更业务服务的节点状态来上线/下线外，也加紧了问题的排查。从已有的健康检查日志数据来看，这个健康检查器不能及时检测到在不同的网关服务节点并不是都存在的，有的网关节点是好的，有的网关节点是有问题的，具有概率性。

健康检查器的定时作业模块问题排查

周六来公司加班，重新理了下思路。通过分析后在健康检查器中添加日志，观察其作业运行逻辑。开发环境部署运行后确认了，健康检查器的定时器作业不是预先想想的样子-是业务上游服务自身设置的维度，而是网关服务的worker维度。且通过数据观察到，业务服务自身设置的5秒每次健康检测，3秒每次不健康检测频率并没有出现，而是每一个业务服务的健康检测频率都是近似2分18秒一次。

2021/09/04 12:25:31 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (833) healthchecker callback, upstream=833:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:31 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (833) healthchecker callback, upstream=833:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:32 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (686) healthchecker callback, upstream=686:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:32 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (686) healthchecker callback, upstream=686:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:32 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (946) healthchecker callback, upstream=946:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:32 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (946) healthchecker callback, upstream=946:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:32 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (793) healthchecker callback, upstream=793:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:33 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (793) healthchecker callback, upstream=793:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:33 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (882) healthchecker callback, upstream=882:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:33 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (882) healthchecker callback, upstream=882:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:33 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (61) healthchecker callback, upstream=61:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:33 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (61) healthchecker callback, upstream=61:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:33 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (68) healthchecker callback, upstream=68:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:33 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (68) healthchecker callback, upstream=68:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:33 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (802) healthchecker callback, upstream=802:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:33 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (802) healthchecker callback, upstream=802:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:33 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (824) healthchecker callback, upstream=824:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:33 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (824) healthchecker callback, upstream=824:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:33 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (970) healthchecker callback, upstream=970:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:33 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (970) healthchecker callback, upstream=970:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:33 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (892) healthchecker callback, upstream=892:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:33 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (892) healthchecker callback, upstream=892:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:33 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (747) healthchecker callback, upstream=747:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:33 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (747) healthchecker callback, upstream=747:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:33 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (688) healthchecker callback, upstream=688:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:33 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (688) healthchecker callback, upstream=688:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:33 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (749) healthchecker callback, upstream=749:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:33 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (749) healthchecker callback, upstream=749:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:33 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (711) healthchecker callback, upstream=711:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:33 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (711) healthchecker callback, upstream=711:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:33 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (690) healthchecker callback, upstream=690:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:33 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (690) healthchecker callback, upstream=690:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:33 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (691) healthchecker callback, upstream=691:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:33 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (691) healthchecker callback, upstream=691:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:33 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (888) healthchecker callback, upstream=888:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:33 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (888) healthchecker callback, upstream=888:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:33 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (909) healthchecker callback, upstream=909:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:33 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (909) healthchecker callback, upstream=909:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:33 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (953) healthchecker callback, upstream=953:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:33 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (953) healthchecker callback, upstream=953:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:33 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (703) healthchecker callback, upstream=703:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:33 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (703) healthchecker callback, upstream=703:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:33 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (773) healthchecker callback, upstream=773:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:33 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (773) healthchecker callback, upstream=773:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:33 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (960) healthchecker callback, upstream=960:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:33 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (960) healthchecker callback, upstream=960:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:33 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (945) healthchecker callback, upstream=945:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:33 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (945) healthchecker callback, upstream=945:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:33 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (932) healthchecker callback, upstream=932:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:33 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (932) healthchecker callback, upstream=932:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:33 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (930) healthchecker callback, upstream=930:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:33 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (930) healthchecker callback, upstream=930:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:33 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (955) healthchecker callback, upstream=955:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:33 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (955) healthchecker callback, upstream=955:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:33 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (677) healthchecker callback, upstream=677:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:33 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (677) healthchecker callback, upstream=677:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:33 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (938) healthchecker callback, upstream=938:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:33 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (938) healthchecker callback, upstream=938:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:33 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (940) healthchecker callback, upstream=940:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:33 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (940) healthchecker callback, upstream=940:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:33 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (789) healthchecker callback, upstream=789:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:33 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (789) healthchecker callback, upstream=789:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:33 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (821) healthchecker callback, upstream=821:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:33 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (821) healthchecker callback, upstream=821:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:33 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (756) healthchecker callback, upstream=756:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:33 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (756) healthchecker callback, upstream=756:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:33 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (733) healthchecker callback, upstream=733:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:33 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (733) healthchecker callback, upstream=733:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (696) healthchecker callback, upstream=696:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (696) healthchecker callback, upstream=696:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (894) healthchecker callback, upstream=894:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (894) healthchecker callback, upstream=894:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (901) healthchecker callback, upstream=901:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (901) healthchecker callback, upstream=901:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (905) healthchecker callback, upstream=905:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (905) healthchecker callback, upstream=905:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (28) healthchecker callback, upstream=28:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (28) healthchecker callback, upstream=28:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (923) healthchecker callback, upstream=923:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (923) healthchecker callback, upstream=923:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (765) healthchecker callback, upstream=765:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (765) healthchecker callback, upstream=765:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (22) healthchecker callback, upstream=22:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (22) healthchecker callback, upstream=22:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (911) healthchecker callback, upstream=911:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (911) healthchecker callback, upstream=911:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (896) healthchecker callback, upstream=896:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (896) healthchecker callback, upstream=896:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (933) healthchecker callback, upstream=933:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (933) healthchecker callback, upstream=933:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (904) healthchecker callback, upstream=904:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (904) healthchecker callback, upstream=904:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (976) healthchecker callback, upstream=976:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (976) healthchecker callback, upstream=976:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (961) healthchecker callback, upstream=961:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (961) healthchecker callback, upstream=961:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (784) healthchecker callback, upstream=784:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (784) healthchecker callback, upstream=784:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (783) healthchecker callback, upstream=783:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (783) healthchecker callback, upstream=783:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (753) healthchecker callback, upstream=753:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (753) healthchecker callback, upstream=753:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (757) healthchecker callback, upstream=757:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (757) healthchecker callback, upstream=757:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (683) healthchecker callback, upstream=683:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (683) healthchecker callback, upstream=683:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (767) healthchecker callback, upstream=767:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (767) healthchecker callback, upstream=767:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (908) healthchecker callback, upstream=908:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (908) healthchecker callback, upstream=908:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (772) healthchecker callback, upstream=772:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (772) healthchecker callback, upstream=772:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (983) healthchecker callback, upstream=983:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (983) healthchecker callback, upstream=983:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (709) healthchecker callback, upstream=709:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (709) healthchecker callback, upstream=709:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (822) healthchecker callback, upstream=822:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (822) healthchecker callback, upstream=822:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (944) healthchecker callback, upstream=944:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (944) healthchecker callback, upstream=944:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (967) healthchecker callback, upstream=967:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (967) healthchecker callback, upstream=967:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (872) healthchecker callback, upstream=872:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (872) healthchecker callback, upstream=872:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (761) healthchecker callback, upstream=761:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (761) healthchecker callback, upstream=761:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (968) healthchecker callback, upstream=968:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (968) healthchecker callback, upstream=968:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (32) healthchecker callback, upstream=32:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (32) healthchecker callback, upstream=32:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (762) healthchecker callback, upstream=762:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (762) healthchecker callback, upstream=762:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (760) healthchecker callback, upstream=760:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (760) healthchecker callback, upstream=760:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (1000) healthchecker callback, upstream=1000:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (1000) healthchecker callback, upstream=1000:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (880) healthchecker callback, upstream=880:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (880) healthchecker callback, upstream=880:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (769) healthchecker callback, upstream=769:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (769) healthchecker callback, upstream=769:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (1) healthchecker callback, upstream=1:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (1) healthchecker callback, upstream=1:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (792) healthchecker callback, upstream=792:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (792) healthchecker callback, upstream=792:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (972) healthchecker callback, upstream=972:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (972) healthchecker callback, upstream=972:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (780) healthchecker callback, upstream=780:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:25:43 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (780) healthchecker callback, upstream=780:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:13 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (719) healthchecker callback, upstream=719:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:13 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (719) healthchecker callback, upstream=719:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:13 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (705) healthchecker callback, upstream=705:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:13 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (705) healthchecker callback, upstream=705:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:13 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (814) healthchecker callback, upstream=814:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:13 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (814) healthchecker callback, upstream=814:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:13 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (620) healthchecker callback, upstream=620:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:13 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (620) healthchecker callback, upstream=620:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:13 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (823) healthchecker callback, upstream=823:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:13 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (823) healthchecker callback, upstream=823:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:13 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (698) healthchecker callback, upstream=698:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:13 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (698) healthchecker callback, upstream=698:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:13 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (738) healthchecker callback, upstream=738:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:13 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (738) healthchecker callback, upstream=738:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:23 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (608) healthchecker callback, upstream=608:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:23 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (608) healthchecker callback, upstream=608:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:23 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (706) healthchecker callback, upstream=706:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:23 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (706) healthchecker callback, upstream=706:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:23 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (883) healthchecker callback, upstream=883:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:23 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (883) healthchecker callback, upstream=883:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:23 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (966) healthchecker callback, upstream=966:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:23 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (966) healthchecker callback, upstream=966:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:23 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (870) healthchecker callback, upstream=870:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:23 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (870) healthchecker callback, upstream=870:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:23 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (921) healthchecker callback, upstream=921:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:23 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (921) healthchecker callback, upstream=921:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:23 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (956) healthchecker callback, upstream=956:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:23 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (956) healthchecker callback, upstream=956:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:23 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (943) healthchecker callback, upstream=943:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:23 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (943) healthchecker callback, upstream=943:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:23 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (898) healthchecker callback, upstream=898:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:23 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (898) healthchecker callback, upstream=898:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:23 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (770) healthchecker callback, upstream=770:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:23 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (770) healthchecker callback, upstream=770:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:23 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (910) healthchecker callback, upstream=910:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:23 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (910) healthchecker callback, upstream=910:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:23 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (903) healthchecker callback, upstream=903:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:23 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (903) healthchecker callback, upstream=903:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:23 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (805) healthchecker callback, upstream=805:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:23 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (805) healthchecker callback, upstream=805:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:23 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (630) healthchecker callback, upstream=630:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:23 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (630) healthchecker callback, upstream=630:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:23 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (743) healthchecker callback, upstream=743:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:23 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (743) healthchecker callback, upstream=743:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:23 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (851) healthchecker callback, upstream=851:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:23 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (851) healthchecker callback, upstream=851:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:23 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (712) healthchecker callback, upstream=712:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:23 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (712) healthchecker callback, upstream=712:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:23 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (622) healthchecker callback, upstream=622:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:23 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (622) healthchecker callback, upstream=622:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:23 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (27) healthchecker callback, upstream=27:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:23 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (27) healthchecker callback, upstream=27:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:23 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (775) healthchecker callback, upstream=775:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:23 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (775) healthchecker callback, upstream=775:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:23 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (913) healthchecker callback, upstream=913:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:23 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (913) healthchecker callback, upstream=913:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:23 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (748) healthchecker callback, upstream=748:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:23 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (748) healthchecker callback, upstream=748:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:23 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (907) healthchecker callback, upstream=907:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:23 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (907) healthchecker callback, upstream=907:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:23 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (766) healthchecker callback, upstream=766:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:23 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (766) healthchecker callback, upstream=766:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:23 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (891) healthchecker callback, upstream=891:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:23 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (891) healthchecker callback, upstream=891:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:23 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (980) healthchecker callback, upstream=980:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:23 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (980) healthchecker callback, upstream=980:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:24 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (971) healthchecker callback, upstream=971:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:24 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (971) healthchecker callback, upstream=971:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:24 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (751) healthchecker callback, upstream=751:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:24 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (751) healthchecker callback, upstream=751:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:24 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (528) healthchecker callback, upstream=528:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:24 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (528) healthchecker callback, upstream=528:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:24 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (810) healthchecker callback, upstream=810:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:24 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (810) healthchecker callback, upstream=810:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:24 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (30) healthchecker callback, upstream=30:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:24 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (30) healthchecker callback, upstream=30:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:24 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (979) healthchecker callback, upstream=979:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:24 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (979) healthchecker callback, upstream=979:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:24 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (837) healthchecker callback, upstream=837:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:24 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (837) healthchecker callback, upstream=837:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:24 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (714) healthchecker callback, upstream=714:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:24 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (714) healthchecker callback, upstream=714:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:24 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (889) healthchecker callback, upstream=889:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:24 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (889) healthchecker callback, upstream=889:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:24 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (744) healthchecker callback, upstream=744:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:24 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (744) healthchecker callback, upstream=744:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:24 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (720) healthchecker callback, upstream=720:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:24 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (720) healthchecker callback, upstream=720:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:24 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (771) healthchecker callback, upstream=771:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:24 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (771) healthchecker callback, upstream=771:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:24 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (957) healthchecker callback, upstream=957:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:24 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (957) healthchecker callback, upstream=957:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:24 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (724) healthchecker callback, upstream=724:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:24 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (724) healthchecker callback, upstream=724:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:24 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (786) healthchecker callback, upstream=786:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:24 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (786) healthchecker callback, upstream=786:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:24 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (777) healthchecker callback, upstream=777:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:24 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (777) healthchecker callback, upstream=777:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:24 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (682) healthchecker callback, upstream=682:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:24 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (682) healthchecker callback, upstream=682:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:24 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (914) healthchecker callback, upstream=914:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:24 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (914) healthchecker callback, upstream=914:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:24 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (707) healthchecker callback, upstream=707:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:24 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (707) healthchecker callback, upstream=707:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:24 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (695) healthchecker callback, upstream=695:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:24 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (695) healthchecker callback, upstream=695:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:24 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (829) healthchecker callback, upstream=829:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:24 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (829) healthchecker callback, upstream=829:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:24 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (679) healthchecker callback, upstream=679:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:24 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (679) healthchecker callback, upstream=679:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:24 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (958) healthchecker callback, upstream=958:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:24 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (958) healthchecker callback, upstream=958:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:24 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (687) healthchecker callback, upstream=687:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:24 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (687) healthchecker callback, upstream=687:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:24 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (788) healthchecker callback, upstream=788:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:24 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (788) healthchecker callback, upstream=788:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:24 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (871) healthchecker callback, upstream=871:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:24 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (871) healthchecker callback, upstream=871:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:24 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (941) healthchecker callback, upstream=941:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:24 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (941) healthchecker callback, upstream=941:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:24 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (964) healthchecker callback, upstream=964:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:24 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (964) healthchecker callback, upstream=964:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:24 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (816) healthchecker callback, upstream=816:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (816) healthchecker callback, upstream=816:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (838) healthchecker callback, upstream=838:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (838) healthchecker callback, upstream=838:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (721) healthchecker callback, upstream=721:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (721) healthchecker callback, upstream=721:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (951) healthchecker callback, upstream=951:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (951) healthchecker callback, upstream=951:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (924) healthchecker callback, upstream=924:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (924) healthchecker callback, upstream=924:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (23) healthchecker callback, upstream=23:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (23) healthchecker callback, upstream=23:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (977) healthchecker callback, upstream=977:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (977) healthchecker callback, upstream=977:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (808) healthchecker callback, upstream=808:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (808) healthchecker callback, upstream=808:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (885) healthchecker callback, upstream=885:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (885) healthchecker callback, upstream=885:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (809) healthchecker callback, upstream=809:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (809) healthchecker callback, upstream=809:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (489) healthchecker callback, upstream=489:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (489) healthchecker callback, upstream=489:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (927) healthchecker callback, upstream=927:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (927) healthchecker callback, upstream=927:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (692) healthchecker callback, upstream=692:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (692) healthchecker callback, upstream=692:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (973) healthchecker callback, upstream=973:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (973) healthchecker callback, upstream=973:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (916) healthchecker callback, upstream=916:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (916) healthchecker callback, upstream=916:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (928) healthchecker callback, upstream=928:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (928) healthchecker callback, upstream=928:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (962) healthchecker callback, upstream=962:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (962) healthchecker callback, upstream=962:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (952) healthchecker callback, upstream=952:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (952) healthchecker callback, upstream=952:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (697) healthchecker callback, upstream=697:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (697) healthchecker callback, upstream=697:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (936) healthchecker callback, upstream=936:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (936) healthchecker callback, upstream=936:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (826) healthchecker callback, upstream=826:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (826) healthchecker callback, upstream=826:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (685) healthchecker callback, upstream=685:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (685) healthchecker callback, upstream=685:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (986) healthchecker callback, upstream=986:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (986) healthchecker callback, upstream=986:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (899) healthchecker callback, upstream=899:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (899) healthchecker callback, upstream=899:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (912) healthchecker callback, upstream=912:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (912) healthchecker callback, upstream=912:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (710) healthchecker callback, upstream=710:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (710) healthchecker callback, upstream=710:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (806) healthchecker callback, upstream=806:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (806) healthchecker callback, upstream=806:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (636) healthchecker callback, upstream=636:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (636) healthchecker callback, upstream=636:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (942) healthchecker callback, upstream=942:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (942) healthchecker callback, upstream=942:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (774) healthchecker callback, upstream=774:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (774) healthchecker callback, upstream=774:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (874) healthchecker callback, upstream=874:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (874) healthchecker callback, upstream=874:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (33) healthchecker callback, upstream=33:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (33) healthchecker callback, upstream=33:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (934) healthchecker callback, upstream=934:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (934) healthchecker callback, upstream=934:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (616) healthchecker callback, upstream=616:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (616) healthchecker callback, upstream=616:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (807) healthchecker callback, upstream=807:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (807) healthchecker callback, upstream=807:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (842) healthchecker callback, upstream=842:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (842) healthchecker callback, upstream=842:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (702) healthchecker callback, upstream=702:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (702) healthchecker callback, upstream=702:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (746) healthchecker callback, upstream=746:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (746) healthchecker callback, upstream=746:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (29) healthchecker callback, upstream=29:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (29) healthchecker callback, upstream=29:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (750) healthchecker callback, upstream=750:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (750) healthchecker callback, upstream=750:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (752) healthchecker callback, upstream=752:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (752) healthchecker callback, upstream=752:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (713) healthchecker callback, upstream=713:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (713) healthchecker callback, upstream=713:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (755) healthchecker callback, upstream=755:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (755) healthchecker callback, upstream=755:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (758) healthchecker callback, upstream=758:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (758) healthchecker callback, upstream=758:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (832) healthchecker callback, upstream=832:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (832) healthchecker callback, upstream=832:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (776) healthchecker callback, upstream=776:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (776) healthchecker callback, upstream=776:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:26 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (745) healthchecker callback, upstream=745:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:26 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (745) healthchecker callback, upstream=745:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:26 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (929) healthchecker callback, upstream=929:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:26 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (929) healthchecker callback, upstream=929:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:26 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (25) healthchecker callback, upstream=25:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:26 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (25) healthchecker callback, upstream=25:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:26 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (694) healthchecker callback, upstream=694:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:26 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (694) healthchecker callback, upstream=694:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:26 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (937) healthchecker callback, upstream=937:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:26 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (937) healthchecker callback, upstream=937:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:26 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (918) healthchecker callback, upstream=918:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:26 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (918) healthchecker callback, upstream=918:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:26 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (680) healthchecker callback, upstream=680:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:26 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (680) healthchecker callback, upstream=680:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:26 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (939) healthchecker callback, upstream=939:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:26 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (939) healthchecker callback, upstream=939:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:26 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (830) healthchecker callback, upstream=830:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:26 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (830) healthchecker callback, upstream=830:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:26 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (893) healthchecker callback, upstream=893:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:26 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (893) healthchecker callback, upstream=893:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:26 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (897) healthchecker callback, upstream=897:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:26 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (897) healthchecker callback, upstream=897:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:26 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (779) healthchecker callback, upstream=779:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:26 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (779) healthchecker callback, upstream=779:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:26 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (759) healthchecker callback, upstream=759:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:26 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (759) healthchecker callback, upstream=759:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:26 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (982) healthchecker callback, upstream=982:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:26 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (982) healthchecker callback, upstream=982:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:26 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (699) healthchecker callback, upstream=699:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:26 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (699) healthchecker callback, upstream=699:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:26 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (819) healthchecker callback, upstream=819:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:26 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (819) healthchecker callback, upstream=819:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:26 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (926) healthchecker callback, upstream=926:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:26 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (926) healthchecker callback, upstream=926:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:29 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (490) healthchecker callback, upstream=490:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:29 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (490) healthchecker callback, upstream=490:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:39 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (632) healthchecker callback, upstream=632:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:39 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (632) healthchecker callback, upstream=632:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:39 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (498) healthchecker callback, upstream=498:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:39 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (498) healthchecker callback, upstream=498:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:39 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (954) healthchecker callback, upstream=954:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:39 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (954) healthchecker callback, upstream=954:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:39 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (635) healthchecker callback, upstream=635:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:39 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (635) healthchecker callback, upstream=635:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:39 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (919) healthchecker callback, upstream=919:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:39 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (919) healthchecker callback, upstream=919:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:39 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (900) healthchecker callback, upstream=900:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:39 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (900) healthchecker callback, upstream=900:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:39 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (301) healthchecker callback, upstream=301:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:39 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (301) healthchecker callback, upstream=301:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:39 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (787) healthchecker callback, upstream=787:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:39 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (787) healthchecker callback, upstream=787:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:39 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (708) healthchecker callback, upstream=708:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:39 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (708) healthchecker callback, upstream=708:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:39 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (725) healthchecker callback, upstream=725:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:39 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (725) healthchecker callback, upstream=725:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:39 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (969) healthchecker callback, upstream=969:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:39 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (969) healthchecker callback, upstream=969:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:39 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (887) healthchecker callback, upstream=887:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:39 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (887) healthchecker callback, upstream=887:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:39 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (959) healthchecker callback, upstream=959:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:39 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (959) healthchecker callback, upstream=959:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:39 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (790) healthchecker callback, upstream=790:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:39 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (790) healthchecker callback, upstream=790:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:39 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (963) healthchecker callback, upstream=963:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:39 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (963) healthchecker callback, upstream=963:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:39 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (754) healthchecker callback, upstream=754:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:39 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (754) healthchecker callback, upstream=754:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:39 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (624) healthchecker callback, upstream=624:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:39 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (624) healthchecker callback, upstream=624:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:39 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (742) healthchecker callback, upstream=742:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:39 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (742) healthchecker callback, upstream=742:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:39 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (840) healthchecker callback, upstream=840:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:39 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (840) healthchecker callback, upstream=840:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:39 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (947) healthchecker callback, upstream=947:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:39 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (947) healthchecker callback, upstream=947:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:39 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (981) healthchecker callback, upstream=981:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:39 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (981) healthchecker callback, upstream=981:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:39 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (69) healthchecker callback, upstream=69:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:26:39 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (69) healthchecker callback, upstream=69:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (917) healthchecker callback, upstream=917:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (917) healthchecker callback, upstream=917:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (764) healthchecker callback, upstream=764:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (764) healthchecker callback, upstream=764:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (723) healthchecker callback, upstream=723:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (723) healthchecker callback, upstream=723:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (768) healthchecker callback, upstream=768:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (768) healthchecker callback, upstream=768:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (886) healthchecker callback, upstream=886:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (886) healthchecker callback, upstream=886:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (804) healthchecker callback, upstream=804:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (804) healthchecker callback, upstream=804:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (820) healthchecker callback, upstream=820:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (820) healthchecker callback, upstream=820:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (684) healthchecker callback, upstream=684:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (684) healthchecker callback, upstream=684:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (895) healthchecker callback, upstream=895:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (895) healthchecker callback, upstream=895:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (722) healthchecker callback, upstream=722:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (722) healthchecker callback, upstream=722:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (839) healthchecker callback, upstream=839:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (839) healthchecker callback, upstream=839:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (948) healthchecker callback, upstream=948:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (948) healthchecker callback, upstream=948:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (700) healthchecker callback, upstream=700:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (700) healthchecker callback, upstream=700:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (718) healthchecker callback, upstream=718:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (718) healthchecker callback, upstream=718:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (794) healthchecker callback, upstream=794:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (794) healthchecker callback, upstream=794:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (825) healthchecker callback, upstream=825:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (825) healthchecker callback, upstream=825:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (799) healthchecker callback, upstream=799:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (799) healthchecker callback, upstream=799:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (987) healthchecker callback, upstream=987:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (987) healthchecker callback, upstream=987:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (763) healthchecker callback, upstream=763:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (763) healthchecker callback, upstream=763:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (902) healthchecker callback, upstream=902:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (902) healthchecker callback, upstream=902:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (693) healthchecker callback, upstream=693:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (693) healthchecker callback, upstream=693:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (931) healthchecker callback, upstream=931:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (931) healthchecker callback, upstream=931:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (920) healthchecker callback, upstream=920:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (920) healthchecker callback, upstream=920:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (785) healthchecker callback, upstream=785:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (785) healthchecker callback, upstream=785:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (726) healthchecker callback, upstream=726:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (726) healthchecker callback, upstream=726:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (884) healthchecker callback, upstream=884:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (884) healthchecker callback, upstream=884:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (965) healthchecker callback, upstream=965:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (965) healthchecker callback, upstream=965:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (831) healthchecker callback, upstream=831:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (831) healthchecker callback, upstream=831:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (689) healthchecker callback, upstream=689:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (689) healthchecker callback, upstream=689:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (935) healthchecker callback, upstream=935:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (935) healthchecker callback, upstream=935:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (297) healthchecker callback, upstream=297:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (297) healthchecker callback, upstream=297:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (778) healthchecker callback, upstream=778:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (778) healthchecker callback, upstream=778:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (704) healthchecker callback, upstream=704:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (704) healthchecker callback, upstream=704:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (26) healthchecker callback, upstream=26:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (26) healthchecker callback, upstream=26:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (781) healthchecker callback, upstream=781:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (781) healthchecker callback, upstream=781:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (791) healthchecker callback, upstream=791:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (791) healthchecker callback, upstream=791:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (950) healthchecker callback, upstream=950:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (950) healthchecker callback, upstream=950:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (619) healthchecker callback, upstream=619:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (619) healthchecker callback, upstream=619:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (818) healthchecker callback, upstream=818:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (818) healthchecker callback, upstream=818:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (915) healthchecker callback, upstream=915:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (915) healthchecker callback, upstream=915:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (302) healthchecker callback, upstream=302:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (302) healthchecker callback, upstream=302:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (906) healthchecker callback, upstream=906:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (906) healthchecker callback, upstream=906:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (701) healthchecker callback, upstream=701:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:09 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (701) healthchecker callback, upstream=701:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:19 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (601) healthchecker callback, upstream=601:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:19 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (601) healthchecker callback, upstream=601:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:19 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (925) healthchecker callback, upstream=925:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:19 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (925) healthchecker callback, upstream=925:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:19 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (24) healthchecker callback, upstream=24:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:27:19 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (24) healthchecker callback, upstream=24:unhealthy, worker id: 1, context: ngx.timer

隔了两个2分18秒后，

2021/09/04 12:32:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (833) healthchecker callback, upstream=833:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:32:25 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (833) healthchecker callback, upstream=833:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:32:26 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (686) healthchecker callback, upstream=686:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:32:26 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (686) healthchecker callback, upstream=686:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:32:26 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (946) healthchecker callback, upstream=946:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:32:26 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (946) healthchecker callback, upstream=946:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:32:26 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (793) healthchecker callback, upstream=793:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:32:26 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (793) healthchecker callback, upstream=793:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:32:26 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (882) healthchecker callback, upstream=882:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:32:26 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (882) healthchecker callback, upstream=882:unhealthy, worker id: 1, context: ngx.timer

......

2021/09/04 12:34:03 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (906) healthchecker callback, upstream=906:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:34:03 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (906) healthchecker callback, upstream=906:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:34:03 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (701) healthchecker callback, upstream=701:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:34:03 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (701) healthchecker callback, upstream=701:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:34:13 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (601) healthchecker callback, upstream=601:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:34:13 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (601) healthchecker callback, upstream=601:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:34:13 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (925) healthchecker callback, upstream=925:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:34:13 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (925) healthchecker callback, upstream=925:unhealthy, worker id: 1, context: ngx.timer
2021/09/04 12:34:13 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (24) healthchecker callback, upstream=24:healthy, worker id: 1, context: ngx.timer
2021/09/04 12:34:13 [warn] 4736#0: *19 [lua] checker.lua:1131: log(): [healthchecks] (24) healthchecker callback, upstream=24:unhealthy, worker id: 1, context: ngx.timer

当时对这个2分18秒值非常的奇怪，怎么都想不通。什么业务配置或运行处理也没有与2分18秒沾边的逻辑。于是先放了下，看为什么业务服务配置的5秒和3秒检测频率没有执行，通过观察发现timer的数量变化比较有规律，空闲的机器上timer会隔段时间来变小。由于已经看到了实际的后台作业是worker维度，于是对代码中最前面的负责维护worker维度的定时器启动的另一个父定时器进行了清理 - 因为正常都会初始化成功，且成功后该定时器没有一点用处，实际是不需要的。代码提交，重新部署下开发环境，这次在看运行时定时器的数量，发现一直稳定的4个。这时对前面的问题突然就觉得通了。结论如下：

1.4.2的checker是worker级别，1.1.0的timer是同步阻塞的，所以总共有worker数量个checker, 且实际的作业时间间隔为所有上游服务的target列表检查完所需时间。即假如有1000个targets，检查完的时间是50分钟，那么实际检查频率就是50分钟

2分18秒的时间间隔问题出在检查器的定时器模块上，lua-resty-timer中直接使用了ngx.timer.at来处理开始的worker检查作业，多个ngx.timer.at是并发的，而单个ngx.timer.at中的业务代码处理是先后同步进行的，因此一个worker的health checker执行时，是处理所有的上游服务健康检查，这样公共处理完才能进行下一作业时间点的作业，而公共处理完的时间就是2分18秒，就是看到的间隔。

当然这个2分18秒肯定是随着不同机器不同的上游服务数量是变化的。

问题解决

知道了问题后，就好修复了。有两个方案，一个是仍用worker级别的检查器，但是要增加上游服务检查时的并行处理(ngx.thread);另一个是业务服务自身级别的。回头看了下原来的2.0版本的健康检查器，发现这个版本的检查器是业务服务自身级别的，所以运行了半年没有出现过健康检查及时性的问题。

考虑到业务的维护性和易读性，决定采用业务服务自身级别的健康检查器；与2.0版本的一致。但是这样就要想办法解决内存资源占用的问题。而这个问题是出在定时器模块，非健康检查器本身。

于是把定时器模块(lua-resty-timer)拿下来，源码分析并考虑解决的方法。在重新阅读了operesty官方文档后，已经有了思路。这个问题解决需要让ngx.timer运行后结束，不能一个timer一直挂起着。由于健康检查需要一直运行，那么如果ngx.timer结束了，肯定需要再有个逻辑片来重新拉起一个ngx.timer；这样原来的timer结束了，资源就释放了，新的timer隔段时间再释放，再重新拉起一个。这就需要一个新的定时作业来维护处理，而这个维护处理就是简单的状态机维护，逻辑特别简单，也不占用资源。 代码片段如下，

local function renew_timer ()
  for uuid, job in pairs(anchor_registry) do
    if job.renewal and not job.cancel_flag then
      job.born = now()
      job.renewal = nil
      anchor_registry[uuid] = job
      timer_at(0, handler, uuid)
    end
  end
end

具体什么时间决定结束老timer, 再拉起新timer, 是有跑着的所有业务服务的健康检查资源决定，如果定时器特别多，机器资源又不太够，那么可以把时间间隔小一点比如2分钟；相反，怎可以大一点，比如半小时。由此看来，机器占用的最多资源就是这个时间间隔内的健康检查器消耗的内存。

修复后的内存占用情况

写在最后

lua-resty-healthchekcer不同版本的内存消耗情况，以及2.0版本的内存消耗持续增长问题，我想作者应该也知道的。可能大家面对的业务场景不一样，比如云原生sidecar的网关业务上数据非常小，可能就一两个上游服务，这样长期使用估计占用资源也不多，不是很紧迫。

但是能优化不占用资源肯定是最好的，有空会提一个pr出来，把自己的方案贡献出来。

附lua-resty-healthchecker系统设计图

业务模型接口

默认检查配置参数