Redission内存泄漏问题排查Redission内存泄漏问题排查，包含日志分析，GC情况分析、Dump文件分析、泄漏

现象

8.31，测试反馈开发者站礼包创建失败，排查发现报错：

{"msg":"内部错误","logId":"CTR-unified-platform-web-tj-staging-44hm4_0831101943571_4982","code":1,"data":null}

排查

日志分析

2023-08-31 10:19:48,836 [http-nio-10201-exec-3] ERROR ErrorHandler.java 40 com.xiaomi.huyu.game.unifiedplatform.web.config.ErrorHandler CTR-unified-platform-web-tj-staging-44hm4_0831101943571_4982 - handle err:

org.springframework.web.util.NestedServletException: Handler dispatch failed; nested exception is java.lang.OutOfMemoryError: Java heap space

根据日志发现，发生了内存溢出

GC情况分析

GC查询命令参考

命令： jstat -gcutil 23484 1000 5

发现老年代一直在gc，内存无法释放，怀疑有内存泄漏

dump内存文件，排查内存泄漏源头

jmap命令dump内存文件

mat工具分析内存快照，排查内存泄漏代码

分析发现Redission框架里的 IdleConnectionWatcher 占用了大量内存，主要集中在ConcurrentHashMap里，没有释放

分析内存泄漏根因

什么情况下会往IdleConnectionWatcher类里的map添加元素？

新的连接添加到连接池中，以便在需要时可以使用

add方法处添加debug断点，分析entry一直添加元素的原因

debug发现，有2处会调用add方法

ClusterConnectionManager类，集群模式下的连接管理器，主要负责维护Redis集群的连接池和节点信息。

public ClusterConnectionManager(ClusterServersConfig cfg, Config config, UUID id) {
    super(config, id);

    if (cfg.getNodeAddresses().isEmpty()) {
        throw new IllegalArgumentException("At least one cluster node should be defined!");
    }

    this.natMapper = cfg.getNatMapper();
    this.config = create(cfg);
    initTimer(this.config);
    
    Throwable lastException = null;
    List<String> failedMasters = new ArrayList<String>();
    for (String address : cfg.getNodeAddresses()) {
        RedisURI addr = new RedisURI(address);
        CompletionStage<RedisConnection> connectionFuture = connectToNode(cfg, addr, addr.getHost());
        try {
            RedisConnection connection = connectionFuture.toCompletableFuture().join();

            if (cfg.getNodeAddresses().size() == 1 && !addr.isIP()) {
                configEndpointHostName = addr.getHost();
            }
            
            clusterNodesCommand = RedisCommands.CLUSTER_NODES;
            if (addr.isSsl()) {
                clusterNodesCommand = RedisCommands.CLUSTER_NODES_SSL;
            }
            
            List<ClusterNodeInfo> nodes = connection.sync(clusterNodesCommand);
            
            StringBuilder nodesValue = new StringBuilder();
            for (ClusterNodeInfo clusterNodeInfo : nodes) {
                nodesValue.append(clusterNodeInfo.getNodeInfo()).append("\n");
            }
            log.info("Redis cluster nodes configuration got from {}:\n{}", connection.getRedisClient().getAddr(), nodesValue);

            lastClusterNode = addr;
            
            CompletableFuture<Collection<ClusterPartition>> partitionsFuture = parsePartitions(nodes);
            Collection<ClusterPartition> partitions = partitionsFuture.join();
            List<CompletableFuture<Void>> masterFutures = new ArrayList<>();
            for (ClusterPartition partition : partitions) {
                if (partition.isMasterFail()) {
                    failedMasters.add(partition.getMasterAddress().toString());
                    continue;
                }
                if (partition.getMasterAddress() == null) {
                    throw new IllegalStateException("Master node: " + partition.getNodeId() + " doesn't have address.");
                }
                // 此处会调用IdleConnectionWatcher的add方法
                CompletableFuture<Void> masterFuture = addMasterEntry(partition, cfg);
                masterFutures.add(masterFuture);
            }

            CompletableFuture<Void> masterFuture = CompletableFuture.allOf(masterFutures.toArray(new CompletableFuture[0]));
            try {
                masterFuture.join();
            } catch (CompletionException e) {
                lastException = e.getCause();
            }
            break;
        } catch (Exception e) {
            if (e instanceof CompletionException) {
                e = (Exception) e.getCause();
            }
            lastException = e;
            log.warn(e.getMessage());
        }
    }

    if (lastPartitions.isEmpty()) {
        stopThreads();
        if (failedMasters.isEmpty()) {
            throw new RedisConnectionException("Can't connect to servers!", lastException);
        } else {
            throw new RedisConnectionException("Can't connect to servers! Failed masters according to cluster status: " + failedMasters, lastException);
        }
    }

    if (cfg.isCheckSlotsCoverage() && lastPartitions.size() != MAX_SLOT) {
        stopThreads();
        if (failedMasters.isEmpty()) {
            throw new RedisConnectionException("Not all slots covered! Only " + lastPartitions.size() + " slots are available. Set checkSlotsCoverage = false to avoid this check.", lastException);
        } else {
            throw new RedisConnectionException("Not all slots covered! Only " + lastPartitions.size() + " slots are available. Set checkSlotsCoverage = false to avoid this check. Failed masters according to cluster status: " + failedMasters, lastException);
        }
    }
    // 定时任务，每5秒扫描一次节点信息变化，问题就出在这里
    scheduleClusterChangeCheck(cfg);
}

定时任务执行逻辑 scheduleClusterChangeCheck

private void scheduleClusterChangeCheck(ClusterServersConfig cfg) {
    monitorFuture = group.schedule(new Runnable() {
        @Override
        public void run() {
            if (configEndpointHostName != null) {
                String address = cfg.getNodeAddresses().iterator().next();
                RedisURI uri = new RedisURI(address);
                AddressResolver<InetSocketAddress> resolver = resolverGroup.getResolver(getGroup().next());
                Future<List<InetSocketAddress>> allNodes = resolver.resolveAll(InetSocketAddress.createUnresolved(uri.getHost(), uri.getPort()));
                allNodes.addListener(new FutureListener<List<InetSocketAddress>>() {
                    @Override
                    public void operationComplete(Future<List<InetSocketAddress>> future) throws Exception {
                        AtomicReference<Throwable> lastException = new AtomicReference<Throwable>(future.cause());
                        if (!future.isSuccess()) {
                            checkClusterState(cfg, Collections.emptyIterator(), lastException);
                            return;
                        }
                        
                        List<RedisURI> nodes = new ArrayList<>();
                        for (InetSocketAddress addr : future.getNow()) {
                            RedisURI address = toURI(uri.getScheme(), addr.getAddress().getHostAddress(), "" + addr.getPort());
                            nodes.add(address);
                        }
                        
                        Iterator<RedisURI> nodesIterator = nodes.iterator();
                        checkClusterState(cfg, nodesIterator, lastException);
                    }
                });
            } else {
                AtomicReference<Throwable> lastException = new AtomicReference<Throwable>();
                List<RedisURI> nodes = new ArrayList<>();
                List<RedisURI> slaves = new ArrayList<>();
                // 获取上一次缓存的节点和分片信息，剔除失败节点
                for (ClusterPartition partition : getLastPartitions()) {
                    if (!partition.isMasterFail()) {
                        nodes.add(partition.getMasterAddress());
                    }

                    Set<RedisURI> partitionSlaves = new HashSet<>(partition.getSlaveAddresses());
                    partitionSlaves.removeAll(partition.getFailedSlaveAddresses());
                    slaves.addAll(partitionSlaves);
                }
                // 顺序随机
                Collections.shuffle(nodes);
                Collections.shuffle(slaves);
                
                // master nodes first
nodes.addAll(slaves);

                Iterator<RedisURI> nodesIterator = nodes.iterator();
                // 检查节点状态
                checkClusterState(cfg, nodesIterator, lastException);
            }
        }

    }, cfg.getScanInterval(), TimeUnit.MILLISECONDS);
}

checkClusterState方法，此处方法的含义是

随机取一个节点，建立连接，发送nodes命令，获取整个集群的节点信息
节点连接失败，取下一个节点尝试

private void checkClusterState(ClusterServersConfig cfg, Iterator<RedisURI> iterator, AtomicReference<Throwable> lastException) {
    if (!iterator.hasNext()) {
        if (lastException.get() != null) {
            log.error("Can't update cluster state", lastException.get());
        }
        scheduleClusterChangeCheck(cfg);
        return;
    }
    if (!getShutdownLatch().acquire()) {
        return;
    }
    RedisURI uri = iterator.next();
    // 连接节点
    CompletionStage<RedisConnection> connectionFuture = connectToNode(cfg, uri, configEndpointHostName);
    connectionFuture.whenComplete((connection, e) -> {
        if (e != null) {
            lastException.set(e);
            getShutdownLatch().release();
            checkClusterState(cfg, iterator, lastException);
            return;
        }
        // 解析并更新节点信息
        updateClusterState(cfg, connection, iterator, uri, lastException);
    });
}

updateClusterState 更新节点信息

private void updateClusterState(ClusterServersConfig cfg, RedisConnection connection, 
        Iterator<RedisURI> iterator, RedisURI uri, AtomicReference<Throwable> lastException) {
    // 发送nodes命令，获取整个集群的节点信息
    RFuture<List<ClusterNodeInfo>> future = connection.async(clusterNodesCommand);
    future.whenComplete((nodes, e) -> {
            if (e != null) {
                closeNodeConnection(connection);
                lastException.set(e);
                getShutdownLatch().release();
                checkClusterState(cfg, iterator, lastException);
                return;
            }

            if (nodes.isEmpty()) {
                log.debug("cluster nodes state got from {}: doesn't contain any nodes", connection.getRedisClient().getAddr());
                getShutdownLatch().release();
                checkClusterState(cfg, iterator, lastException);
                return;
            }

            lastClusterNode = uri;

            StringBuilder nodesValue = new StringBuilder();
            if (log.isDebugEnabled()) {
                for (ClusterNodeInfo clusterNodeInfo : nodes) {
                    nodesValue.append(clusterNodeInfo.getNodeInfo()).append("\n");
                }
                log.debug("cluster nodes state got from {}:\n{}", connection.getRedisClient().getAddr(), nodesValue);
            }
            // 获取节点的分片槽点信息
            CompletableFuture<Collection<ClusterPartition>> newPartitionsFuture = parsePartitions(nodes);
            newPartitionsFuture.whenComplete((newPartitions, ex) -> {
                CompletableFuture<Void> masterFuture = checkMasterNodesChange(cfg, newPartitions);
                // 检查从节点是否有更新，问题出在这里
                checkSlaveNodesChange(newPartitions);
                masterFuture.whenComplete((res, exc) -> {
                    checkSlotsMigration(newPartitions);
                    checkSlotsChange(newPartitions);
                    getShutdownLatch().release();
                    scheduleClusterChangeCheck(cfg);
                });
            });
    });
}

checkSlaveNodesChange

比较缓存中上一次的最新节点分片信息和这一次取到的节点分片信息，进行比较

private void checkSlaveNodesChange(Collection<ClusterPartition> newPartitions) {
    Map<RedisURI, ClusterPartition> lastPartitions = getLastPartitonsByURI();
    for (ClusterPartition newPart : newPartitions) {
        ClusterPartition currentPart = lastPartitions.get(newPart.getMasterAddress());
        if (currentPart == null) {
            continue;
        }

        MasterSlaveEntry entry = getEntry(currentPart.slots().nextSetBit(0));
        // should be invoked first in order to remove stale failedSlaveAddresses
        // 关键点，比较从节点变化，增加或删除信息
Set<RedisURI> addedSlaves = addRemoveSlaves(entry, currentPart, newPart);
        // Do some slaves have changed state from failed to alive?
upDownSlaves(entry, currentPart, newPart, addedSlaves);
    }
}

addRemoveSlaves

比较新旧从节点地址是否一致，不一致，

private Set<RedisURI> addRemoveSlaves(MasterSlaveEntry entry, ClusterPartition currentPart, ClusterPartition newPart) {
    Set<RedisURI> removedSlaves = new HashSet<>(currentPart.getSlaveAddresses());
    removedSlaves.removeAll(newPart.getSlaveAddresses());

    for (RedisURI uri : removedSlaves) {
        currentPart.removeSlaveAddress(uri);

        if (entry.slaveDown(uri, FreezeReason.MANAGER)) {
            log.info("slave {} removed for slot ranges: {}", uri, currentPart.getSlotRanges());
        }
    }
    // 节点地址比较
    Set<RedisURI> addedSlaves = new HashSet<>(newPart.getSlaveAddresses());
    addedSlaves.removeAll(currentPart.getSlaveAddresses());
    for (RedisURI uri : addedSlaves) {
        ClientConnectionsEntry slaveEntry = entry.getEntry(uri);
        if (slaveEntry != null) {
            currentPart.addSlaveAddress(uri);
            entry.slaveUp(uri, FreezeReason.MANAGER);
            log.info("slave: {} added for slot ranges: {}", uri, currentPart.getSlotRanges());
            continue;
        }
        // 关键点，添加新的从节点信息
        CompletableFuture<Void> future = entry.addSlave(uri, false, NodeType.SLAVE, configEndpointHostName);
        future.whenComplete((res, ex) -> {
            if (ex != null) {
                log.error("Can't add slave: " + uri, ex);
                return;
            }

            currentPart.addSlaveAddress(uri);
            entry.slaveUp(uri, FreezeReason.MANAGER);
            log.info("slave: {} added for slot ranges: {}", uri, currentPart.getSlotRanges());
        });
    }
    return addedSlaves;
}

MasterSlaveEntry类是用于维护Redis主从节点信息的类，以便Redisson客户端在进行读写操作时能够正确地选择节点。

public CompletableFuture<Void> addSlave(RedisURI address, boolean freezed, NodeType nodeType, String sslHostname) {
    // 创建客户端
    RedisClient client = connectionManager.createClient(nodeType, address, sslHostname);
    return addSlave(client, freezed, nodeType);
}

private CompletableFuture<Void> addSlave(RedisClient client, boolean freezed, NodeType nodeType) {
    CompletableFuture<InetSocketAddress> addrFuture = client.resolveAddr();
    return addrFuture.thenCompose(res -> {
        // 关键路径
        ClientConnectionsEntry entry = new ClientConnectionsEntry(client,
                config.getSlaveConnectionMinimumIdleSize(),
                config.getSlaveConnectionPoolSize(),
                config.getSubscriptionConnectionMinimumIdleSize(),
                config.getSubscriptionConnectionPoolSize(), connectionManager, nodeType);
        if (freezed) {
            synchronized (entry) {
                entry.setFreezeReason(FreezeReason.SYSTEM);
            }
        }
        return slaveBalancer.add(entry);
    }).whenComplete((r, ex) -> {
        if (ex != null) {
            client.shutdownAsync();
        }
    });
}

ClientConnectionsEntry类是用于维护Redis节点的连接信息的类

public ClientConnectionsEntry(RedisClient client, int poolMinSize, int poolMaxSize, int subscribePoolMinSize, int subscribePoolMaxSize,
        ConnectionManager connectionManager, NodeType nodeType) {
    this.client = client;
    this.freeConnectionsCounter = new AsyncSemaphore(poolMaxSize);
    this.connectionManager = connectionManager;
    this.nodeType = nodeType;
    this.freeSubscribeConnectionsCounter = new AsyncSemaphore(subscribePoolMaxSize);

    if (subscribePoolMaxSize > 0) {
        // IdleConnectionWatcher添加连接信息到map缓存里
        connectionManager.getConnectionWatcher().add(this, subscribePoolMinSize, subscribePoolMaxSize, freeSubscribeConnections, freeSubscribeConnectionsCounter, c -> {
            freeSubscribeConnections.remove(c);
            return allSubscribeConnections.remove(c);
        });
    }
    connectionManager.getConnectionWatcher().add(this, poolMinSize, poolMaxSize, freeConnections, freeConnectionsCounter, c -> {
            freeConnections.remove(c);
            return allConnections.remove(c);
        });
}

到此流程已经结束，看似正常的流程，为何会导致内存泄漏？

结合集群信息发现了问题的真正原因

10.38.200.74:6614,10.38.200.74:6615,10.38.200.43:6608,10.38.200.74:6612,10.38.200.48:6608,10.38.200.74:6610,10.38.200.74:6608,10.38.200.74:6611,10.38.200.74:6616,10.38.200.74:6613

上面的是测试环境的集群连接地址，项目启动初始化获取节点信息的时候，连接的是10.38.200.74:6614这个节点，获取了集群的所有节点信息
其中：master:10.38.200.48:6616 ,slave:空，master:10.38.200.48:6610,slave:空，这2个主节点，没有对应的从节点信息，怀疑是节点挂了
当定时任务扫描时，是打乱集群节点后取第一个节点建立连接获取集群信息，当渠道48:6618，48:6611，48:6609，48:6610，48:6608这几个节点，并发送nodes命令获取集群信息后，获取的节点

master:10.38.200.48:6616 ,10.38.200.74:6616:，master:10.38.200.48:6610,slave:10.38.200.74:6610，

与上面74节点取的集群节点信息不一致，导致了每次和48节点连接，都会发生从节点不一致情况，导致和从节点建立连接并缓存，内存无法释放，最终oom

疑问

为什么从节点发送改变，不去更新ClusterPartition，从而导致缓存里的主从节点和新的一致不一致
只有主节点发生改变，才会更新ClusterPartition信息
为什么连接信息缓存一直不释放

Redission内存泄漏问题排查

现象

排查

日志分析

GC情况分析

dump内存文件，排查内存泄漏源头

分析内存泄漏根因

疑问