集成监控Executor的线程池参数

827 阅读2分钟

实现方式采用spring的bean后置装载的接口

代码:

public class ExecutorMetricWapper ``implements BeanPostProcessor {     ``@Override``    ``public Object postProcessAfterInitialization(Object bean, String beanName) ``throws BeansException {``        ``if ((bean ``instanceof Executor)) {``            ``ExecutorServiceMetrics.monitor(Metrics.globalRegistry, (Executor) bean, beanName, Collections.emptyList());``            ``LOGGER.info(``"Executor beanName:{} Has been registered to monitor"``, beanName);``            ``return bean;``        ``}``        ``return bean;``    ``}     ``private static final Logger LOGGER = LoggerFactory.getLogger(ExecutorMetricWapper.``class``);``}

 

效果:

只是公司内的grafana监控图

 

指标的含义:

spring自带的ThreadPoolTaskExecutor线程池(至于spring的Executor为什么参数没有自定实现的Executor线程池全,这问题是由于实现机制的原因)

指标含义推荐expr
```html
executor_idle_seconds_count
| ```html
executor_idle_seconds_sum
```        | 任务耗时        | 同上                                                                                                    |
| ```html
executor_idle_seconds_max
```        | 任务空闲秒数(最大值) | --                                                                                                    |
| ```html
executor_execution_seconds_count
``` | 任务执行次数      | 执行器平均执行秒数:increase(executor_execution_seconds_sum[5m])/increase(executor_execution_seconds_count[5m]) |
| ```html
executor_execution_seconds_sum
```   | 任务耗时        | 同上                                                                                                    |
| ```html
executor_execution_seconds_max
```   | 任务执行秒数(最大值) | --                                                                                                    |

自定义的Executor线程池

| 指标                                         | 含义           | 推荐expr                                                                                                  |
| ------------------------------------------ | ------------ | ------------------------------------------------------------------------------------------------------- |
| ```html
executor_completed_tasks_total
``` | 完成的任务        | 完成的任务/分钟:increase(executor_completed_tasks_total[1m])                                                   |
| ```html
executor_idle_seconds_count
```    | 执行器闲置次数      | 执行器平均闲置秒数:increase(executor_idle_seconds_sum[5m])/increase(executor_idle_seconds_count[5m])             |
| ```html
executor_idle_seconds_sum
```      | 执行器耗时        | 同上                                                                                                      |
| ```html
executor_idle_seconds_max
```      | 执行器空闲秒数(最大值) | --                                                                                                      |
| ```html
executor_active_threads
```        | 执行器活跃的线程数    | --                                                                                                      |
| ```html
executor_seconds_count
```         | 执行器执行次数      | 执行器平均执行秒数:increase(executor_seconds_sum[5m])/increase(executor_seconds_count[5m])                       |
| ```html
executor_seconds_sum
```           | 执行器耗时        | 同上                                                                                                      |
| ```html
executor_seconds_max
```           | 执行器执行秒数(最大值) | --                                                                                                      |
| ```html
executor_queued_tasks
```          | 排队的任务        | 队列占用率:100*(1-executor_queue_remaining_tasks/ (executor_queue_remaining_tasks + executor_queued_tasks )) |
| ```html
executor_queue_remaining_tasks
``` | 排队的剩余任务      | 同上                                                                                                      |
| ```html
executor_pool_size_threads
```     | 执行器的线程池大小    | --                                                                                                      |
| executor_rejected_tasks_total              | 拒绝的任务        | 拒绝的任务/分钟:increase(executor_rejected_tasks_total[1m])                                                    |
| executor_scheduled_cron_total              | 执行器预定的cron   | --                                                                                                      |
| executor_scheduled_once_total              | 执行器预定一次大小    | --                                                                                                      |
| executor_scheduled_repetitively_total      | 执行器重复执行次数    | --                                                                                                      |

参照:INFRA|springboot2.X统一配置面板的grpc统计模块

 

其他的监控也是类似,甚至可以自己来写实现

从这个问题主要是引出监控的事情,要是想要对一个线程池进行matrix监控,那么当前比较low的实现方法如下

 

| `@Service``public` `class` `ThreadPoolMonitor ``implements` `InitializingBean {` `    ``@Resource``(name = ``"consumerExecutor"``)``    ``ThreadPoolTaskExecutor consumerExecutor;``    ``@Resource``    ``ConnectionPool connectionPool;` `    ``private` `static` `final` `Iterable<Tag> consumerExecutorTAG = Collections.singletonList(Tag.of(``"thread.pool.name"``, ``"consumerExecutor"``));``    ``private` `static` `final` `Iterable<Tag> okHttpExecutorTAG = Collections.singletonList(Tag.of(``"thread.pool.name"``, ``"okHttpExecutor"``));``    ``private` `final` `ScheduledExecutorService scheduledExecutor = Executors.newSingleThreadScheduledExecutor();` `    ``private` `Runnable monitor = () -> {``        ``//这里需要捕获异常,尽管实际上不会产生异常,但是必须预防异常导致调度线程池线程失效的问题``        ``try` `{``            ``Metrics.gauge(``"thread.pool.core.size"``, consumerExecutorTAG, consumerExecutor, ThreadPoolTaskExecutor::getCorePoolSize);``            ``Metrics.gauge(``"thread.pool.largest.size"``, consumerExecutorTAG, consumerExecutor, e -> e.getThreadPoolExecutor().getLargestPoolSize());``            ``Metrics.gauge(``"thread.pool.max.size"``, consumerExecutorTAG, consumerExecutor, ThreadPoolTaskExecutor::getMaxPoolSize);``            ``Metrics.gauge(``"thread.pool.active.size"``, consumerExecutorTAG, consumerExecutor, ThreadPoolTaskExecutor::getActiveCount);``            ``Metrics.gauge(``"thread.pool.thread.count"``, consumerExecutorTAG, consumerExecutor, ThreadPoolTaskExecutor::getPoolSize);``            ``Metrics.gauge(``"thread.pool.queue.size"``, consumerExecutorTAG, consumerExecutor, e -> e.getThreadPoolExecutor().getQueue().size());`  `            ``Metrics.gauge(``"thread.pool.connectionCount"``, okHttpExecutorTAG, connectionPool, ConnectionPool::connectionCount);``            ``Metrics.gauge(``"thread.pool.idleConnectionCount"``, okHttpExecutorTAG, connectionPool, ConnectionPool::idleConnectionCount);``        ``} ``catch` `(Exception e) {``            ``//ignore``        ``}``    ``};` `    ``@Override``    ``public` `void` `afterPropertiesSet() ``throws` `Exception {``        ``// 每5秒执行一次``        ``scheduledExecutor.scheduleWithFixedDelay(monitor, ``0``, ``5``, TimeUnit.SECONDS);``    ``}` `}` |
| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |