堆外内存溢出

oldFarmer
1,790 阅读2分钟

一：启动微服务模块，用浏览器直接访问json数据

image.png 发现我们编写的接口功能是正常的

二：打开jmeter对该接口进行50个线程的压测

image.png 压测一会，发现已经出现大量异常

image.png 异常原因

{"timestamp":"2021-08-17 08:38:02","status":500,"error":"Internal Server Error","message":"Redis exception; nested exception is io.lettuce.core.RedisException: io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 37748736 byte(s) of direct memory (used: 67108864, max: 100663296)","trace":"org.springframework.data.redis.RedisSystemException: Redis exception; nested exception is io.lettuce.core.RedisException: io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 37748736 byte(s) of direct memory (used: 67108864, max: 100663296)

三：打开jvisuanvm，连接上我们的后台服务

虚拟机内存虽然设置的比较小，但是jvm内存没有表现出内存溢出的现象，说明这个报错和jvm没有太大关系 image.png

四：打开后台服务日志

可以看到里面的关键字有OutOfDirectMemoryError，打开后台日志，也有大量的报错日志

io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 37748736 byte(s) of direct memory (used: 67108864, max: 100663296)
	at io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:725) ~[netty-common-4.1.39.Final.jar:4.1.39.Final]
	at io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:680) ~[netty-common-4.1.39.Final.jar:4.1.39.Final]
	at io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:772) ~[netty-buffer-4.1.39.Final.jar:4.1.39.Final]
	at io.netty.buffer.PoolArena$DirectArena.newUnpooledChunk(PoolArena.java:762) ~[netty-buffer-4.1.39.Final.jar:4.1.39.Final]
	at io.netty.buffer.PoolArena.allocateHuge(PoolArena.java:260) ~[netty-buffer-4.1.39.Final.jar:4.1.39.Final]
	at io.netty.buffer.PoolArena.allocate(PoolArena.java:232) ~[netty-buffer-4.1.39.Final.jar:4.1.39.Final]
	at io.netty.buffer.PoolArena.reallocate(PoolArena.java:400) ~[netty-buffer-4.1.39.Final.jar:4.1.39.Final]
	at io.netty.buffer.PooledByteBuf.capacity(PooledByteBuf.java:119) ~[netty-buffer-4.1.39.Final.jar:4.1.39.Final]
	at io.netty.buffer.AbstractByteBuf.ensureWritable0(AbstractByteBuf.java:303) ~[netty-buffer-4.1.39.Final.jar:4.1.39.Final]
	at io.netty.buffer.AbstractByteBuf.ensureWritable(AbstractByteBuf.java:274) ~[netty-buffer-4.1.39.Final.jar:4.1.39.Final]
	at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1111) ~[netty-buffer-4.1.39.Final.jar:4.1.39.Final]
	at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1104) ~[netty-buffer-4.1.39.Final.jar:4.1.39.Final]
	at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1095) ~[netty-buffer-4.1.39.Final.jar:4.1.39.Final]
	at io.lettuce.core.protocol.CommandHandler.channelRead(CommandHandler.java:554) ~[lettuce-core-5.1.8.RELEASE.jar:na]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.39.Final.jar:4.1.39.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.39.Final.jar:4.1.39.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.39.Final.jar:4.1.39.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1421) [netty-transport-4.1.39.Final.jar:4.1.39.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.39.Final.jar:4.1.39.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.39.Final.jar:4.1.39.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) [netty-transport-4.1.39.Final.jar:4.1.39.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.39.Final.jar:4.1.39.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:697) [netty-transport-4.1.39.Final.jar:4.1.39.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:632) [netty-transport-4.1.39.Final.jar:4.1.39.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:549) [netty-transport-4.1.39.Final.jar:4.1.39.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:511) [netty-transport-4.1.39.Final.jar:4.1.39.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918) [netty-common-4.1.39.Final.jar:4.1.39.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.39.Final.jar:4.1.39.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.39.Final.jar:4.1.39.Final]
	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_162]

可以很明显的看到是堆外内存溢出异常了，这个是由于SpringBoot2.0以后默认使用 Lettuce作为操作redis的客户端，它使用 netty进行网络通信，lettuce 的bug导致netty堆外内存溢出

  • 1、SpringBoot2.0以后默认使用 Lettuce作为操作redis的客户端，它使用 netty进行网络通信
  • 2、lettuce 的bug导致netty堆外内存溢出，-Xmx300m netty 如果没有指定堆内存移除，默认使用 -Xmx300m
  • 可以通过指定参数-Dio.netty.maxDirectMemory 进行设置
  • 解决方案 不能使用 -Dio.netty.maxDirectMemory调大内存 这样只是减缓内存溢出的时间。
  • 1、升级 lettuce客户端，2、 切换使用jedis
  • redisTemplate:
  • lettuce、jedis 操作redis的底层客户端，Spring再次封装

五：问题解决步骤

目前的解决办法是切换使用redis，后续会通过升级lettuce客户端来解决这个bug

5.1：切换使用jedis的依赖，并祛除lettuce依赖，核心配置如下

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-data-redis</artifactId>
    <exclusions>
        <exclusion>
            <groupId>io.lettuce</groupId>
            <artifactId>lettuce-core</artifactId>
        </exclusion>
    </exclusions>
</dependency>
<dependency>
    <groupId>redis.clients</groupId>
    <artifactId>jedis</artifactId>
</dependency>

5.2直接重启后台服务

使用jmeter重新进行压测，发现没有再报OutOfDirectMemoryError了，问题解决！！！

最后贴出netty判断堆外内存异常的核心代码如下

private static void incrementMemoryCounter(int capacity) {
    if (DIRECT_MEMORY_COUNTER != null) {
        long newUsedMemory = DIRECT_MEMORY_COUNTER.addAndGet(capacity);
        if (newUsedMemory > DIRECT_MEMORY_LIMIT) {
            DIRECT_MEMORY_COUNTER.addAndGet(-capacity);
            throw new OutOfDirectMemoryError("failed to allocate " + capacity
                    + " byte(s) of direct memory (used: " + (newUsedMemory - capacity)
                    + ", max: " + DIRECT_MEMORY_LIMIT + ')');
        }
    }
}

netty默认取值堆外内存等于-Xmx的代码片段

public static ByteBuffer allocateDirectNoCleaner(int capacity) {
    assert USE_DIRECT_BUFFER_NO_CLEANER;

    incrementMemoryCounter(capacity);
    try {
        return PlatformDependent0.allocateDirectNoCleaner(capacity);
    } catch (Throwable e) {
        decrementMemoryCounter(capacity);
        throwException(e);
        return null;
    }
}
avatar
文章
阅读
粉丝