最近查看ES日志，发现每隔10-30s就有一条key失效[日志1]，通过刷新ES apikey缓存后直接变为了找不到api key[日志2]，说明这个api key之前是存在的，被删除了，通过日志中的key信息无法直接定位到应用，这给排错带来一些麻烦。
既然ES能够知道Key ID,那说明api key串包含了ID信息，通过测试确认API Key由ID:Secrect组成。
将日志中的KEY ID V9xdU5ABS4QXBJP5R-hp进行Base64编码得到如下字符Vjl4ZFU1QUJTNFFYQkpQNVItaHA=,删掉末尾的=填充符，在项目中搜索依然没找到，最后上服务器用grep -r "Vjl4ZFU1QUJTNFFYQkpQNVItaHA"，发现是之前弃用的docker容器ElasticSearch Mysql-Connector同步数据的连接器使用的，停掉后日志没再继续产生。
日志1：
{"@timestamp":"2024-08-14T00:10:57.657Z", "log.level": "WARN", "message":"Authentication using apikey failed - api key [V9xdU5ABS4QXBJP5R-hp] has been invalidated", "ecs.ve
rsion": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch][transport_worker][T#15]","log.logger":"or
g.elasticsearch.xpack.security.authc.ApiKeyAuthenticator","elasticsearch.cluster.uuid":"JePhXaHzT4W3Ksla_-XG9w","elasticsearch.node.id":"KPftHqTCQjCc-u7Fj2D4rw","elasticsea
rch.node.name":"elasticsearch","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2024-08-14T00:10:58.661Z", "log.level": "WARN", "message":"Authentication using apikey failed - api key [V9xdU5ABS4QXBJP5R-hp] has been invalidated", "ecs.ve
rsion": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch][transport_worker][T#15]","log.logger":"or
g.elasticsearch.xpack.security.authc.ApiKeyAuthenticator","elasticsearch.cluster.uuid":"JePhXaHzT4W3Ksla_-XG9w","elasticsearch.node.id":"KPftHqTCQjCc-u7Fj2D4rw","elasticsea
rch.node.name":"elasticsearch","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2024-08-14T00:11:00.665Z", "log.level": "WARN", "message":"Authentication using apikey failed - api key [V9xdU5ABS4QXBJP5R-hp] has been invalidated", "ecs.ve
rsion": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch][transport_worker][T#15]","log.logger":"or
g.elasticsearch.xpack.security.authc.ApiKeyAuthenticator","elasticsearch.cluster.uuid":"JePhXaHzT4W3Ksla_-XG9w","elasticsearch.node.id":"KPftHqTCQjCc-u7Fj2D4rw","elasticsea
rch.node.name":"elasticsearch","elasticsearch.cluster.name":"docker-cluster"}
日志2：
{"@timestamp":"2024-08-14T02:05:07.698Z", "log.level": "WARN", "message":"Authentication using apikey failed - unable to find apikey with id V9xdU5ABS4QXBJP5R-hp", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch][system_critical_read][T#5]","log.logger":"org.elasticsearch.xpack.security.authc.ApiKeyAuthenticator","elasticsearch.cluster.uuid":"JePhXaHzT4W3Ksla_-XG9w","elasticsearch.node.id":"KPftHqTCQjCc-u7Fj2D4rw","elasticsearch.node.name":"elasticsearch","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2024-08-14T02:06:12.694Z", "log.level": "WARN", "message":"Authentication using apikey failed - unable to find apikey with id V9xdU5ABS4QXBJP5R-hp", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch][system_critical_read][T#4]","log.logger":"org.elasticsearch.xpack.security.authc.ApiKeyAuthenticator","elasticsearch.cluster.uuid":"JePhXaHzT4W3Ksla_-XG9w","elasticsearch.node.id":"KPftHqTCQjCc-u7Fj2D4rw","elasticsearch.node.name":"elasticsearch","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2024-08-14T02:06:13.699Z", "log.level": "WARN", "message":"Authentication using apikey failed - unable to find apikey with id V9xdU5ABS4QXBJP5R-hp", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch][system_critical_read][T#1]","log.logger":"org.elasticsearch.xpack.security.authc.ApiKeyAuthenticator","elasticsearch.cluster.uuid":"JePhXaHzT4W3Ksla_-XG9w","elasticsearch.node.id":"KPftHqTCQjCc-u7Fj2D4rw","elasticsearch.node.name":"elasticsearch","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2024-08-14T02:06:15.704Z", "log.level": "WARN", "message":"Authentication using apikey failed - unable to find apikey with id V9xdU5ABS4QXBJP5R-hp", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch][system_critical_read][T#3]","log.logger":"org.elasticsearch.xpack.security.authc.ApiKeyAuthenticator","elasticsearch.cluster.uuid":"JePhXaHzT4W3Ksla_-XG9w","elasticsearch.node.id":"KPftHqTCQjCc-u7Fj2D4rw","elasticsearch.node.name":"elasticsearch","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2024-08-14T02:06:19.709Z", "log.level": "WARN", "message":"Authentication using apikey failed - unable to find apikey with id V9xdU5ABS4QXBJP5R-hp", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch][system_critical_read][T#2]","log.logger":"org.elasticsearch.xpack.security.authc.ApiKeyAuthenticator","elasticsearch.cluster.uuid":"JePhXaHzT4W3Ksla_-XG9w","elasticsearch.node.id":"KPftHqTCQjCc-u7Fj2D4rw","elasticsearch.node.name":"elasticsearch","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2024-08-14T02:06:27.713Z", "log.level": "WARN", "message":"Authentication using apikey failed - unable to find apikey with id V9xdU5ABS4QXBJP5R-hp", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch][system_critical_read][T#5]","log.logger":"org.elasticsearch.xpack.security.authc.ApiKeyAuthenticator","elasticsearch.cluster.uuid":"JePhXaHzT4W3Ksla_-XG9w","elasticsearch.node.id":"KPftHqTCQjCc-u7Fj2D4rw","elasticsearch.node.name":"elasticsearch","elasticsearch.cluster.name":"docker-cluster"}
