前情提要
-
memcached客户端采用的是google的xmemcache客户端
-
xmemcached配置的是KetamaMemcachedSessionLocator,也就是所说的一致性hash。基于这个配置，我们理解的是：在我们将集群中某台memcached节点下线，在达到配置的心跳失败次数以后，会将相关的节点从client的节点集合中踢掉，原来处于该节点上的key,会根据一致性hash算法落到其它节点上。 也就是说对于系统影响应该是（心跳间隔时间+心跳超时时间）* 最大心跳次数
现象
线上环境有12个memcache节点，某天先将其中一台执行下线，准备换新的机器,但是日志一直在报警，持续时间超过我们预期
java.util.concurrent.TimeoutException: Timed out(3000 milliseconds) waiting for operation while connected to x.y.x.m:11211
at net.rubyeye.xmemcached.XMemcachedClient.latchWait(XMemcachedClient.java:2577)
at net.rubyeye.xmemcached.XMemcachedClient.fetch0(XMemcachedClient.java:649)
at net.rubyeye.xmemcached.XMemcachedClient.get0(XMemcachedClient.java:1092)
at net.rubyeye.xmemcached.XMemcachedClient.get$sentryProxy(XMemcachedClient.java:1050)
at net.rubyeye.xmemcached.XMemcachedClient.get(XMemcachedClient.java)
at net.rubyeye.xmemcached.XMemcachedClient.get(XMemcachedClient.java:1061)
at net.rubyeye.xmemcached.XMemcachedClient.get(XMemcachedClient.java:1083)
