背景
A公司购买云产品(或者自建 现在比较少了)redis集群，由于成本问题一开始购买很低的配置，一年后由于业务突飞猛进redis 扛不住了需要进行升级场景如下：
- 集群3主3从多加6个节点 主3从变成6主6从
- 集群3主3从多 比如4c8g,迁移机器新的机器 3主3从8c64g 这种情况客户端怎么保证最小或者无影响的redis迁移升级，这个时候需要开启redis客户端拓扑功能
spring boot redis集群配置
1、配置文件
spring:
# Redis配置
redis:
lettuce:
pool:
# 连接池最大连接数
max-active: 199
# 连接池最大阻塞等待时间（使用负值表示没有限制）
max-wait: -1
# 连接池中的最大空闲连接
max-idle: 8
# 连接池中的最小空闲连接
min-idle: 0
cluster:
nodes:
- 172.20.19.250:7001
- 172.20.19.250:7002
- 172.20.19.250:7003
- 172.20.19.250:7004
- 172.20.19.250:7005
- 172.20.19.250:7006
2、代码配置
@Configuration
public class RedisConfig {
@Resource
private RedisProperties redisProperties;
@Bean
public LettuceConnectionFactory lettuceConnectionFactory() {
RedisClusterConfiguration redisClusterConfiguration = new RedisClusterConfiguration(redisProperties.getCluster().getNodes());
redisClusterConfiguration.setMaxRedirects(3);
redisClusterConfiguration.setPassword("bitnami");
//支持自适应集群拓扑刷新和静态刷新源
ClusterTopologyRefreshOptions clusterTopologyRefreshOptions = ClusterTopologyRefreshOptions.builder()
.enablePeriodicRefresh()
.enableAllAdaptiveRefreshTriggers()
.refreshPeriod(Duration.ofSeconds(5))
.build();
ClusterClientOptions clusterClientOptions = ClusterClientOptions.builder()
.topologyRefreshOptions(clusterTopologyRefreshOptions).build();
//从优先，读写分离，读从可能存在不一致，最终一致性CP
LettuceClientConfiguration lettuceClientConfiguration = LettuceClientConfiguration.builder()
// .readFrom(ReadFrom.SLAVE_PREFERRED)
.clientOptions(clusterClientOptions).build();
return new LettuceConnectionFactory(redisClusterConfiguration, lettuceClientConfiguration);
}
@Bean
public RedisTemplate<String, Object> redisTemplate(LettuceConnectionFactory redisConnectionFactory) {
RedisTemplate<String, Object> redisTemplate = new RedisTemplate<>();
redisTemplate.setConnectionFactory(redisConnectionFactory);
StringRedisSerializer stringRedisSerializer = new StringRedisSerializer();
// key序列化
redisTemplate.setKeySerializer(stringRedisSerializer);
// Hash key序列化
redisTemplate.setHashKeySerializer(stringRedisSerializer);
// fastJson序列化
FastJsonRedisSerializer<Object> fastJsonRedisSerializer = new FastJsonRedisSerializer<>(Object.class);
// value序列化
redisTemplate.setValueSerializer(fastJsonRedisSerializer);
// Hash value序列化
redisTemplate.setHashValueSerializer(fastJsonRedisSerializer);
redisTemplate.afterPropertiesSet();
return redisTemplate;
}
}
ClusterTopologyRefreshOptions 自动拓扑当集群节点信息变化的时候5s会同步一次
docker搭建redis集群
1、 docker compose 172.20.19.250 是我本地ip redis密码bitnami
services:
redis-node-0:
image: bitnami/redis-cluster:7.0.5
ports: #配置了节点应宣布的 IP（REDIS_CLUSTER_ANNOUNCE_IP），用于非动态 IP 环境时，必须要对外映射端口
- 7001:7001 #redis连接用端口
- 17001:17001 #redis之间消息连接的端口 默认是redis连接端口+10000
environment:
- 'REDIS_PASSWORD=bitnami'
- 'REDIS_PORT_NUMBER=7001' #替换原始6379端口
- 'REDIS_CLUSTER_ANNOUNCE_IP=172.20.19.250' #当前redis使用的静态IP(为了开放出来供外部使用这里写宿主机IP or WSL2子系统IP)
- 'REDIS_CLUSTER_DYNAMIC_IPS=no' #标明不使用动态IP
- 'REDIS_NODES=redis-node-0:7001 redis-node-1:7002 redis-node-2:7003 redis-node-3:7004 redis-node-4:7005 redis-node-5:7006'
redis-node-1:
image: bitnami/redis-cluster:7.0.5
ports:
- 7002:7002
- 17002:17002
environment:
- 'REDIS_PASSWORD=bitnami'
- 'REDIS_PORT_NUMBER=7002'
- 'REDIS_CLUSTER_ANNOUNCE_IP=172.20.19.250'
- 'REDIS_CLUSTER_DYNAMIC_IPS=no'
- 'REDIS_NODES=redis-node-0:7001 redis-node-1:7002 redis-node-2:7003 redis-node-3:7004 redis-node-4:7005 redis-node-5:7006'
redis-node-2:
image: bitnami/redis-cluster:7.0.5
ports:
- 7003:7003
- 17003:17003
environment:
- 'REDIS_PASSWORD=bitnami'
- 'REDIS_PORT_NUMBER=7003'
- 'REDIS_CLUSTER_ANNOUNCE_IP=172.20.19.250'
- 'REDIS_CLUSTER_DYNAMIC_IPS=no'
- 'REDIS_NODES=redis-node-0:7001 redis-node-1:7002 redis-node-2:7003 redis-node-3:7004 redis-node-4:7005 redis-node-5:7006'
redis-node-3:
image: bitnami/redis-cluster:7.0.5
ports:
- 7004:7004
- 17004:17004
environment:
- 'REDIS_PASSWORD=bitnami'
- 'REDIS_PORT_NUMBER=7004'
- 'REDIS_CLUSTER_ANNOUNCE_IP=172.20.19.250'
- 'REDIS_CLUSTER_DYNAMIC_IPS=no'
- 'REDIS_NODES=redis-node-0:7001 redis-node-1:7002 redis-node-2:7003 redis-node-3:7004 redis-node-4:7005 redis-node-5:7006'
redis-node-4:
image: bitnami/redis-cluster:7.0.5
ports:
- 7005:7005
- 17005:17005
environment:
- 'REDIS_PASSWORD=bitnami'
- 'REDIS_PORT_NUMBER=7005'
- 'REDIS_CLUSTER_ANNOUNCE_IP=172.20.19.250'
- 'REDIS_CLUSTER_DYNAMIC_IPS=no'
- 'REDIS_NODES=redis-node-0:7001 redis-node-1:7002 redis-node-2:7003 redis-node-3:7004 redis-node-4:7005 redis-node-5:7006'
redis-node-5:
image: bitnami/redis-cluster:7.0.5
ports:
- 7006:7006
- 17006:17006
depends_on:
- redis-node-0
- redis-node-1
- redis-node-2
- redis-node-3
- redis-node-4
environment:
- 'REDISCLI_AUTH=bitnami'
- 'REDIS_PASSWORD=bitnami'
- 'REDIS_PORT_NUMBER=7006'
- 'REDIS_CLUSTER_ANNOUNCE_IP=172.20.19.250'
- 'REDIS_CLUSTER_DYNAMIC_IPS=no'
- 'REDIS_NODES=redis-node-0:7001 redis-node-1:7002 redis-node-2:7003 redis-node-3:7004 redis-node-4:7005 redis-node-5:7006'
- 'REDIS_CLUSTER_REPLICAS=1'
- 'REDIS_CLUSTER_CREATOR=yes'
redisinsight:
container_name: redisinsight
image: redislabs/redisinsight:1.13.1
depends_on:
- redis-node-0
- redis-node-1
- redis-node-2
- redis-node-3
- redis-node-4
- redis-node-5
ports:
- "8001:8001"
2、查看集群信息 cluster node
# 登录
redis-cli -h 172.20.19.250 -p 7006 -a bitnami
cluster nodes
98c423be8d21147cffe3116844b0530699af5499 172.20.19.250:7001@17001 master - 0 1739417505914 12 connected 416-8281 10923-16383
0101410cf2a3ed6d902265fcaae8e9d7f1ea78c4 172.20.19.250:7005@17005 slave 98c423be8d21147cffe3116844b0530699af5499 0 1739417505000 12 connected
475b90b7fc9843074db4a515cd38a695bea61bdf 172.20.19.250:7003@17003 slave 98c423be8d21147cffe3116844b0530699af5499 0 1739417503000 12 connected
20a32ff4d63bf8ed8f57ccd1a5c7304e11afd59e 172.20.19.250:7004@17004 slave 98c423be8d21147cffe3116844b0530699af5499 0 1739417502901 12 connected
42c2c8912b3112a3271401054af31a3a57163995 172.20.19.250:7002@17002 slave 0521d4489a4ad534d5689d22f33a99fed3762857 0 1739417504000 11 connected
0521d4489a4ad534d5689d22f33a99fed3762857 172.20.19.250:7006@17006 myself,master - 0 1739417504000 11 connected 0-415 8282-10922
2、节点迁移 172.20.19.250:7006（0-415 8282-10922）节点数据迁移到172.20.19.250:7001
redis-cli -h 172.20.19.250 -p 7006 -a bitnami --cluster reshard 172.20.19.250:7003 --cluster-slots 16383 --cluster-to 0521d4489a4ad534d5689d22f33a99fed3762857 --cluster-from 98c423be8d21147cffe3116844b0530699af5499 --cluster-yes
迁移后的数据集群只有一个master 172.20.19.250:7006 包含了所有数据
cluster nodes
98c423be8d21147cffe3116844b0530699af5499 172.20.19.250:7001@17001 slave 0521d4489a4ad534d5689d22f33a99fed3762857 0 1739417684000 13 connected
0101410cf2a3ed6d902265fcaae8e9d7f1ea78c4 172.20.19.250:7005@17005 slave 0521d4489a4ad534d5689d22f33a99fed3762857 0 1739417684463 13 connected
475b90b7fc9843074db4a515cd38a695bea61bdf 172.20.19.250:7003@17003 slave 0521d4489a4ad534d5689d22f33a99fed3762857 0 1739417685465 13 connected
20a32ff4d63bf8ed8f57ccd1a5c7304e11afd59e 172.20.19.250:7004@17004 slave 0521d4489a4ad534d5689d22f33a99fed3762857 0 1739417680000 13 connected
42c2c8912b3112a3271401054af31a3a57163995 172.20.19.250:7002@17002 slave 0521d4489a4ad534d5689d22f33a99fed3762857 0 1739417680000 13 connected
0521d4489a4ad534d5689d22f33a99fed3762857 172.20.19.250:7006@17006 myself,master - 0 1739417684000 13 connected 0-16383
3、删除172.20.19.250:7006
删除后再看节点信息 cluster nodes
0101410cf2a3ed6d902265fcaae8e9d7f1ea78c4 172.20.19.250:7005@17005 slave 0521d4489a4ad534d5689d22f33a99fed3762857 0 1739417901000 13 connected
475b90b7fc9843074db4a515cd38a695bea61bdf 172.20.19.250:7003@17003 slave 0521d4489a4ad534d5689d22f33a99fed3762857 0 1739417900000 13 connected
20a32ff4d63bf8ed8f57ccd1a5c7304e11afd59e 172.20.19.250:7004@17004 slave 0521d4489a4ad534d5689d22f33a99fed3762857 0 1739417901174 13 connected
42c2c8912b3112a3271401054af31a3a57163995 172.20.19.250:7002@17002 slave 0521d4489a4ad534d5689d22f33a99fed3762857 0 1739417902178 13 connected
0521d4489a4ad534d5689d22f33a99fed3762857 172.20.19.250:7006@17006 myself,master - 0 1739417901000 13 connected 0-16383
4、查看spring boot客户端客户端在这期间一直有redis 操作中间没有任何报错 只有一warn 日志
[WARN ] 2025-02-13 13:54:36,031 io.lettuce.core.cluster.topology.DefaultClusterTopologyRefresh - Unable to connect to [172.20.19.250:7001]: Connection refused: no further information: /172.20.19.250:7001
[WARN ] 2025-02-13 13:54:41,026 io.lettuce.core.cluster.topology.DefaultClusterTopologyRefresh - Unable to connect to [172.20.19.250:7001]: Connection refused: no further information: /172.20.19.250:7001
[WARN ] 2025-02-13 13:54:46,033 io.lettuce.core.cluster.topology.DefaultClusterTopologyRefresh - Unable to connect to [172.20.19.250:7001]: Connection refused: no further information: /172.20.19.250:7001
[WARN ] 2025-02-13 13:54:51,027 io.lettuce.core.cluster.topology.DefaultClusterTopologyRefresh - Unable to connect to [172.20.19.250:7001]: Connection refused: no further information: /172.20.19.250:7001
[INFO ] 2025-02-13 13:54:53,655 com.wf.game.sdk.log.listener.BaseRedisListener - 用户登录, key=GAME_SDK:LOG:USER:LOGIN0, 读取数据size=2
5、如果不开拓扑功能系统会报错 重新测试 删除的节点是 172.20.19.250:7003
[WARN ] 2025-02-13 11:24:33,065 io.lettuce.core.protocol.ConnectionWatchdog - Cannot reconnect to [172.20.19.250:7003]: Connection refused: no further information: /172.20.19.250:7003
[INFO ] 2025-02-13 11:24:36,019 io.lettuce.core.protocol.ConnectionWatchdog - Reconnecting, last destination was 172.20.19.250:7003
[WARN ] 2025-02-13 11:24:38,059 io.lettuce.core.protocol.ConnectionWatchdog - Cannot reconnect to [172.20.19.250:7003]: Connection refused: no further information: /172.20.19.250:7003
[INFO ] 2025-02-13 11:24:43,314 io.lettuce.core.protocol.ConnectionWatchdog - Reconnecting, last destination was 172.20.19.250:7003
[WARN ] 2025-02-13 11:24:45,369 io.lettuce.core.protocol.ConnectionWatchdog - Cannot reconnect to [172.20.19.250:7003]: Connection refused: no further information: /172.20.19.250:7003
[INFO ] 2025-02-13 11:24:49,516 io.lettuce.core.protocol.ConnectionWatchdog - Reconnecting, last destination was 172.20.19.250:7003
[WARN ] 2025-02-13 11:24:51,548 io.lettuce.core.protocol.ConnectionWatchdog - Cannot reconnect to [172.20.19.250:7003]: Connection refused: no further information: /172.20.19.250:7003
还会有这个错误
org.springframework.dao.QueryTimeoutException: Redis command timed out; nested exception is io.lettuce.core.RedisCommandTimeoutException: Command timed out after 1 minute(s)
at org.springframework.data.redis.connection.lettuce.LettuceExceptionConverter.convert(LettuceExceptionConverter.java:70) ~[spring-data-redis-2.7.14.jar:2.7.14]
at org.springframework.data.redis.connection.lettuce.LettuceExceptionConverter.convert(LettuceExceptionConverter.java:41) ~[spring-data-redis-2.7.14.jar:2.7.14]
at org.springframework.data.redis.PassThroughExceptionTranslationStrategy.translate(PassThroughExceptionTranslationStrategy.java:44) ~[spring-data-redis-2.7.14.jar:2.7.14]
at org.springframework.data.redis.FallbackExceptionTranslationStrategy.translate(FallbackExceptionTranslationStrategy.java:42) ~[spring-data-redis-2.7.14.jar:2.7.14]
at org.springframework.data.redis.connection.lettuce.LettuceConnection.convertLettuceAccessException(LettuceConnection.java:277) ~[spring-data-redis-2.7.14.jar:2.7.14]
at org.springframework.data.redis.connection.lettuce.LettuceConnection.await(LettuceConnection.java:1085) ~[spring-data-redis-2.7.14.jar:2.7.14]
at org.springframework.data.redis.connection.lettuce.LettuceConnection.lambda$doInvoke$4(LettuceConnection.java:938) ~[spring-data-redis-2.7.14.jar:2.7.14]
at org.springframework.data.redis.connection.lettuce.LettuceInvoker$Synchronizer.invoke(LettuceInvoker.java:673) ~[spring-data-redis-2.7.14.jar:2.7.14]
at org.springframework.data.redis.connection.lettuce.LettuceInvoker$DefaultSingleInvocationSpec.get(LettuceInvoker.java:589) ~[spring-data-redis-2.7.14.jar:2.7.14]
at org.springframework.data.redis.connection.lettuce.LettuceScriptingCommands.evalSha(LettuceScriptingCommands.java:122) ~[spring-data-redis-2.7.14.jar:2.7.14]
at org.springframework.data.redis.connection.DefaultedRedisConnection.evalSha(DefaultedRedisConnection.java:1824) ~[spring-data-redis-2.7.14.jar:2.7.14]
at jdk.internal.reflect.GeneratedMethodAccessor20.invoke(Unknown Source) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
at org.springframework.data.redis.core.CloseSuppressingInvocationHandler.invoke(CloseSuppressingInvocationHandler.java:61) ~[spring-data-redis-2.7.14.jar:2.7.14]
at com.sun.proxy.$Proxy172.evalSha(Unknown Source) ~[?:?]
at org.springframework.data.redis.core.script.DefaultScriptExecutor.eval(DefaultScriptExecutor.java:77) ~[spring-data-redis-2.7.14.jar:2.7.14]
