前言

本文记录了学习Netty源码的过程，正片文章内容包含思路分析和大量源码。 @空歌白石

由于掘金文章长度限制只能将文章分为几部分。

本文主要包含：

Netty简介。Netty源码剖析学习（一）
Netty基本组件。Netty源码剖析学习（一）
Netty服务端启动。Netty源码剖析学习（一）
NioEventLoop。Netty源码剖析学习（一）
Netty如何新建连接。Netty源码剖析学习（一）
Pipeline。Netty源码剖析学习（二）
Netty的内存分配ByteBuf。Netty源码剖析学习（二）
Netty的解码逻辑。Netty源码剖析学习（二）
Netty的编码逻辑。Netty源码剖析学习（二）
Netty性能优化工具类。Netty源码剖析学习（三）
Netty中的设计模式。Netty源码剖析学习（三）

注：本文源码为当前最新版本：netty 4.1

6. pipeline

Netty的大动脉，负责数据的传播。

几个问题：

Netty是如何判断ChannelHandler类型的？
- 使用 instanceof 判断inbound还是outbound
对于ChannelHandler的添加应该遵循什么样的顺序？
- inbound与添加handler的顺序正相关
- outbound与添加handler顺序逆相关
用户手动触发事件传播，不同的触发方式由什么区别？

pipeline的初始化

pipeline在创建Channel的时候被创建
pipeline节点数据结构：ChannelHandlerContext
pipeline中的两大哨兵：head和tail
- head用于开始
- tail负责结束，并且会处理异常等

一个channel对应一个pipeline

protected AbstractChannel(Channel parent, ChannelId id) {
    this.parent = parent;
    this.id = id;
    unsafe = newUnsafe();
    pipeline = newChannelPipeline();
}

protected DefaultChannelPipeline newChannelPipeline() {
    return new DefaultChannelPipeline(this);
}

双向链表的数据结构。

protected DefaultChannelPipeline(Channel channel) {
    this.channel = ObjectUtil.checkNotNull(channel, "channel");
    succeededFuture = new SucceededChannelFuture(channel, null);
    voidPromise =  new VoidChannelPromise(channel, true);

    tail = new TailContext(this);
    head = new HeadContext(this);

    head.next = tail;
    tail.prev = head;
}

ChannelHandlerContext

public interface ChannelHandlerContext extends AttributeMap, ChannelInboundInvoker, ChannelOutboundInvoker {

    Channel channel();

    EventExecutor executor();
   
    // 空歌白石：省略部分代码

ChannelInboundInvoker


public interface ChannelInboundInvoker {

    ChannelInboundInvoker fireChannelRegistered();

    ChannelInboundInvoker fireChannelUnregistered();
   
    // 空歌白石：省略部分代码
}

ChannelOutboundInvoker

public interface ChannelOutboundInvoker {

    ChannelFuture bind(SocketAddress localAddress);

    ChannelFuture connect(SocketAddress remoteAddress);
   
    // 空歌白石：省略部分代码
}

final class TailContext extends AbstractChannelHandlerContext implements ChannelInboundHandler {

    TailContext(DefaultChannelPipeline pipeline) {
        super(pipeline, null, TAIL_NAME, TailContext.class);
        setAddComplete();
    }

final class HeadContext extends AbstractChannelHandlerContext
        implements ChannelOutboundHandler, ChannelInboundHandler {

    private final Unsafe unsafe;

    HeadContext(DefaultChannelPipeline pipeline) {
        super(pipeline, null, HEAD_NAME, HeadContext.class);
        unsafe = pipeline.channel().unsafe();
        setAddComplete();
    }

添加ChannelHandler

判断是否重复添加
创建节点并添加至链表
回调添加完成事件

@Override
public final ChannelPipeline addLast(EventExecutorGroup executor, ChannelHandler... handlers) {
    ObjectUtil.checkNotNull(handlers, "handlers");

    for (ChannelHandler h: handlers) {
        if (h == null) {
            break;
        }
        addLast(executor, null, h);
    }

    return this;
}

@Override
public final ChannelPipeline addLast(EventExecutorGroup group, String name, ChannelHandler handler) {
    final AbstractChannelHandlerContext newCtx;
    synchronized (this) {
        checkMultiplicity(handler);

        newCtx = newContext(group, filterName(name, handler), handler);

        addLast0(newCtx);

        if (!registered) {
            newCtx.setAddPending();
            callHandlerCallbackLater(newCtx, true);
            return this;
        }

        EventExecutor executor = newCtx.executor();
        if (!executor.inEventLoop()) {
            callHandlerAddedInEventLoop(newCtx, executor);
            return this;
        }
    }
    callHandlerAdded0(newCtx);
    return this;
}

private static void checkMultiplicity(ChannelHandler handler) {
    if (handler instanceof ChannelHandlerAdapter) {
        ChannelHandlerAdapter h = (ChannelHandlerAdapter) handler;
        if (!h.isSharable() && h.added) {
            throw new ChannelPipelineException(
                    h.getClass().getName() +
                    " is not a @Sharable handler, so can't be added or removed multiple times.");
        }
        h.added = true;
    }
}

public boolean isSharable() {
    Class<?> clazz = getClass();
    Map<Class<?>, Boolean> cache = InternalThreadLocalMap.get().handlerSharableCache();
    Boolean sharable = cache.get(clazz);
    if (sharable == null) {
        sharable = clazz.isAnnotationPresent(Sharable.class);
        cache.put(clazz, sharable);
    }
    return sharable;
}

private AbstractChannelHandlerContext context0(String name) {
    AbstractChannelHandlerContext context = head.next;
    while (context != tail) {
        if (context.name().equals(name)) {
            return context;
        }
        context = context.next;
    }
    return null;
}

private void callHandlerAdded0(final AbstractChannelHandlerContext ctx) {
    try {
        ctx.callHandlerAdded();
    } catch (Throwable t) {
        boolean removed = false;
        try {
            atomicRemoveFromHandlerList(ctx);
            ctx.callHandlerRemoved();
            removed = true;
        } catch (Throwable t2) {
            if (logger.isWarnEnabled()) {
                logger.warn("Failed to remove a handler: " + ctx.name(), t2);
            }
        }

        if (removed) {
            fireExceptionCaught(new ChannelPipelineException(
                    ctx.handler().getClass().getName() +
                    ".handlerAdded() has thrown an exception; removed.", t));
        } else {
            fireExceptionCaught(new ChannelPipelineException(
                    ctx.handler().getClass().getName() +
                    ".handlerAdded() has thrown an exception; also failed to remove.", t));
        }
    }
}

删除ChannelHandler

权限校验中经常会使用ChannelHandler的删除操作。

找到ChannelHandler节点
标准的链表方式删除
回调删除Handler事件

private AbstractChannelHandlerContext getContextOrDie(ChannelHandler handler) {
    AbstractChannelHandlerContext ctx = (AbstractChannelHandlerContext) context(handler);
    if (ctx == null) {
        throw new NoSuchElementException(handler.getClass().getName());
    } else {
        return ctx;
    }
}

private AbstractChannelHandlerContext context0(String name) {
    AbstractChannelHandlerContext context = head.next;
    while (context != tail) {
        if (context.name().equals(name)) {
            return context;
        }
        context = context.next;
    }
    return null;
}

private AbstractChannelHandlerContext remove(final AbstractChannelHandlerContext ctx) {
    assert ctx != head && ctx != tail;

    synchronized (this) {
   
    // 空歌白石：省略部分代码

        EventExecutor executor = ctx.executor();
        if (!executor.inEventLoop()) {
            executor.execute(new Runnable() {
                @Override
                public void run() {
                    callHandlerRemoved0(ctx);
                }
            });
            return ctx;
        }
    }
    callHandlerRemoved0(ctx);
    return ctx;
}
    // 空歌白石：省略部分代码

private void callHandlerRemoved0(final AbstractChannelHandlerContext ctx) {
    // Notify the complete removal.
    try {
        ctx.callHandlerRemoved();
    } catch (Throwable t) {
        fireExceptionCaught(new ChannelPipelineException(
                ctx.handler().getClass().getName() + ".handlerRemoved() has thrown an exception.", t));
    }
}

Inbound事件的传播

何为InBound事件以及ChannelInBoundHandler
ChannelRead事件的传播
SimpleInBoundHandler处理器

public interface ChannelInboundHandler extends ChannelHandler {
    void channelRegistered(ChannelHandlerContext ctx) throws Exception;
    // 空歌白石：省略部分代码
}

@Skip
@Override
public void channelRead(ChannelHandlerContext ctx, Object msg) throws Exception {
    ctx.fireChannelRead(msg);
}

@Override
public ChannelHandlerContext fireChannelRead(final Object msg) {
    invokeChannelRead(findContextInbound(MASK_CHANNEL_READ), msg);
    return this;
}

private AbstractChannelHandlerContext findContextInbound(int mask) {
    AbstractChannelHandlerContext ctx = this;
    EventExecutor currentExecutor = executor();
    do {
        ctx = ctx.next;
    } while (skipContext(ctx, currentExecutor, mask, MASK_ONLY_INBOUND));
    return ctx;
}

SimpleChannelInboundHandler

SimpleChannelInboundHandler可以自动释放ByteBuf。

public abstract class SimpleChannelInboundHandler<I> extends ChannelInboundHandlerAdapter {
    private final TypeParameterMatcher matcher;
    private final boolean autoRelease;

    protected SimpleChannelInboundHandler() {
        this(true);
    }
   
    // 空歌白石：省略部分代码
}

public static boolean release(Object msg) {
    if (msg instanceof ReferenceCounted) {
        return ((ReferenceCounted) msg).release();
    }
    return false;
}

OutBound事件的传播

何为outBound事件以及ChannelOutBoundHandler
write()事件的传播

推荐使用ctx.channel().write()写outbound，避免用ctx.write();。

public interface ChannelOutboundHandler extends ChannelHandler {
    void bind(ChannelHandlerContext ctx, SocketAddress localAddress, ChannelPromise promise) throws Exception;
   
    // 空歌白石：省略部分代码
}

@Skip
@Override
public void write(ChannelHandlerContext ctx, Object msg, ChannelPromise promise) throws Exception {
    ctx.write(msg, promise);
}
   
    // 空歌白石：省略部分代码


private void write(Object msg, boolean flush, ChannelPromise promise) {
    ObjectUtil.checkNotNull(msg, "msg");
    try {
        if (isNotValidPromise(promise, true)) {
            ReferenceCountUtil.release(msg);
            // cancelled
            return;
        }
    } catch (RuntimeException e) {
        ReferenceCountUtil.release(msg);
        throw e;
    }
   
    // 空歌白石：省略部分代码
}

private AbstractChannelHandlerContext findContextOutbound(int mask) {
    AbstractChannelHandlerContext ctx = this;
    EventExecutor currentExecutor = executor();
    do {
        ctx = ctx.prev;
    } while (skipContext(ctx, currentExecutor, mask, MASK_ONLY_OUTBOUND));
    return ctx;
}

void invokeWrite(Object msg, ChannelPromise promise) {
    if (invokeHandler()) {
        invokeWrite0(msg, promise);
    } else {
        write(msg, promise);
    }
}

异常传播

异常的触发链
- 与Handler的添加顺序有关。
- 与intbound和outbound的类型无关。
- handler不处理异常的话，会有tail节点负责最后的处理职责。
- 由于是由tail节点处理，因此只有inbound的unhandle方法。
异常处理的最佳实践
- 在最后增加一个针对异常的Handler。

private void invokeChannelRead(Object msg) {
    // 空歌白石：省略部分代码
        fireChannelRead(msg);
    // 空歌白石：省略部分代码
}

private void invokeExceptionCaught(final Throwable cause) {
    if (invokeHandler()) {
        try {
            handler().exceptionCaught(this, cause);
        } catch (Throwable error) {
            if (logger.isDebugEnabled()) {
                logger.debug(
                    "An exception {}" +
                    "was thrown by a user handler's exceptionCaught() " +
                    "method while handling the following exception:",
                    ThrowableUtil.stackTraceToString(error), cause);
            } else if (logger.isWarnEnabled()) {
                logger.warn(
                    "An exception '{}' [enable DEBUG level for full stacktrace] " +
                    "was thrown by a user handler's exceptionCaught() " +
                    "method while handling the following exception:", error, cause);
            }
        }
    } else {
        fireExceptionCaught(cause);
    }
}
    // 空歌白石：省略部分代码

static void invokeExceptionCaught(final AbstractChannelHandlerContext next, final Throwable cause) {
    // 空歌白石：省略部分代码
    EventExecutor executor = next.executor();
    // 空歌白石：省略部分代码
}

onUnhandledInboundException只有Inbound的未捕获异常处理，没有outBound的。

protected void onUnhandledInboundException(Throwable cause) {
    try {
        logger.warn(
                "An exceptionCaught() event was fired, and it reached at the tail of the pipeline. " +
                        "It usually means the last handler in the pipeline did not handle the exception.",
                cause);
    } finally {
        ReferenceCountUtil.release(cause);
    }
}
public static boolean release(Object msg) {
    if (msg instanceof ReferenceCounted) {
        return ((ReferenceCounted) msg).release();
    }
    return false;
}

7. Netty的内存分配（ByteBuf）

三个问题：

Netty的内存类型有哪些？
- 堆内堆外以外的类型
如何减少多线程内存分配之间的竞争？
不同大小的内存是如何进行分配的？

byteBuf在netty独立的buffer包中。

主要内容：

内存与内存管理器的抽象
不同规则大小和不同类别的内存的分配策略
内存的回收过程

ByBuf结构

 * <pre>
 *      +-------------------+------------------+------------------+
 *      | discardable bytes |  readable bytes  |  writable bytes  |
 *      |                   |     (CONTENT)    |                  |
 *      +-------------------+------------------+------------------+
 *      |                   |                  |                  |
 *      0      <=      readerIndex   <=   writerIndex    <=    capacity
 * </pre>

/**
* Returns the maximum allowed capacity of this buffer. This value provides an upper
* bound on {@link #capacity()}.
*/
public abstract int maxCapacity();

 * <pre>
 *  BEFORE discardReadBytes()
 *
 *      +-------------------+------------------+------------------+
 *      | discardable bytes |  readable bytes  |  writable bytes  |
 *      +-------------------+------------------+------------------+
 *      |                   |                  |                  |
 *      0      <=      readerIndex   <=   writerIndex    <=    capacity
 *
 *
 *  AFTER discardReadBytes()
 *
 *      +------------------+--------------------------------------+
 *      |  readable bytes  |    writable bytes (got more space)   |
 *      +------------------+--------------------------------------+
 *      |                  |                                      |
 * readerIndex (0) <= writerIndex (decreased)        <=        capacity
 * </pre>

包含5类常用方法

read
set
write
mark
reset

public abstract byte  readByte();

ByteBuf分类

Pooled和Unpooled，预分配内存和非预分配内存，是否池化。
Unsafe和非Unsafe，是否会依赖于sun.Unsafe类。
Heap和Direct
- Heap在堆上分配，会被GC管理
- Direct不受JVM控制，堆外内存，需要手动释放，否则可能会引发OOM

AbstractByteBuf

@Override
public byte readByte() {
    checkReadableBytes0(1);
    int i = readerIndex;
    byte b = _getByte(i);
    readerIndex = i + 1;
    return b;
}

@Override
public ByteBuf setByte(int index, int value) {
    checkIndex(index);
    _setByte(index, value);
    return this;
}
   
    // 空歌白石：省略部分代码

UnpooledHeapByteBuf vs UnpooledDirectByteBuf

public class UnpooledHeapByteBuf extends AbstractReferenceCountedByteBuf {

    private final ByteBufAllocator alloc;
    byte[] array;
    private ByteBuffer tmpNioBuf;

public class UnpooledDirectByteBuf extends AbstractReferenceCountedByteBuf {

    private final ByteBufAllocator alloc;

    ByteBuffer buffer; // accessed by UnpooledUnsafeNoCleanerDirectByteBuf.reallocateDirect()
    private ByteBuffer tmpNioBuf;
    private int capacity;
    private boolean doNotFree;

    /**
     * Creates a new big-endian direct buffer with reasonably small initial capacity, which
     * expands its capacity boundlessly on demand.
     */
    public static ByteBuf directBuffer() {
        return ALLOC.directBuffer();
    }

@Override
public ByteBuf directBuffer() {
    return directBuffer(DEFAULT_INITIAL_CAPACITY, DEFAULT_MAX_CAPACITY);
}
    // 空歌白石：省略部分代码

@Override
protected ByteBuf newDirectBuffer(int initialCapacity, int maxCapacity) {
    final ByteBuf buf;
    if (PlatformDependent.hasUnsafe()) {
        buf = noCleaner ? new InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf(this, initialCapacity, maxCapacity) :
                new InstrumentedUnpooledUnsafeDirectByteBuf(this, initialCapacity, maxCapacity);
    } else {
        buf = new InstrumentedUnpooledDirectByteBuf(this, initialCapacity, maxCapacity);
    }
    return disableLeakDetector ? buf : toLeakAwareBuffer(buf);
}

ByteBufAllocator

负责Netty的内存分配。

public interface ByteBufAllocator {
    // 空歌白石：省略部分代码

    ByteBuf buffer(int initialCapacity, int maxCapacity);
   
    // 空歌白石：省略部分代码
 }

AbstractByteBufAllocator

AbstractByteBufAllocator是对ByteBufAllocator的具体实现。UML继承关系图：

@Override
public ByteBuf buffer() {
    if (directByDefault) {
        return directBuffer();
    }
    return heapBuffer();
}
    // 空歌白石：省略部分代码

PlatformDependent.hasUnsafe()判断是否有Unsafe。

/**
    * Create a direct {@link ByteBuf} with the given initialCapacity and maxCapacity.
    */
protected abstract ByteBuf newDirectBuffer(int initialCapacity, int maxCapacity);

@Override
protected ByteBuf newDirectBuffer(int initialCapacity, int maxCapacity) {
    // 空歌白石：省略部分代码
        buf = directArena.allocate(cache, initialCapacity, maxCapacity);
    } else {
    // 空歌白石：省略部分代码
}

UnpooledByteBufAllocator

@Override
protected ByteBuf newHeapBuffer(int initialCapacity, int maxCapacity) {
    return PlatformDependent.hasUnsafe() ?
            new InstrumentedUnpooledUnsafeHeapByteBuf(this, initialCapacity, maxCapacity) :
            new InstrumentedUnpooledHeapByteBuf(this, initialCapacity, maxCapacity);
}

@Override
protected ByteBuf newDirectBuffer(int initialCapacity, int maxCapacity) {
    final ByteBuf buf;
    if (PlatformDependent.hasUnsafe()) {
        buf = noCleaner ? new InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf(this, initialCapacity, maxCapacity) :
                new InstrumentedUnpooledUnsafeDirectByteBuf(this, initialCapacity, maxCapacity);
    } else {
        buf = new InstrumentedUnpooledDirectByteBuf(this, initialCapacity, maxCapacity);
    }
    return disableLeakDetector ? buf : toLeakAwareBuffer(buf);
}

UnpooledUnsafeDirectByteBuf

@Override
final void setByteBuffer(ByteBuffer buffer, boolean tryFree) {
    super.setByteBuffer(buffer, tryFree);
    memoryAddress = PlatformDependent.directBufferAddress(buffer);
}
    // 空歌白石：省略部分代码

PooledByteBufAllocator

首先拿到线程局部缓存PoolThreadCache
在线程局部缓存的Area上进行内存分配

PooledByteBuf<T> allocate(PoolThreadCache cache, int reqCapacity, int maxCapacity) {
    PooledByteBuf<T> buf = newByteBuf(maxCapacity);
    allocate(cache, buf, reqCapacity);
    return buf;
}

final int defaultMinNumArena = NettyRuntime.availableProcessors() * 2;

public PooledByteBufAllocator(boolean preferDirect, int nHeapArena, int nDirectArena, int pageSize, int maxOrder,
                                int smallCacheSize, int normalCacheSize,
                                boolean useCacheForAllThreads, int directMemoryCacheAlignment) {
    super(preferDirect);
    threadCache = new PoolThreadLocalCache(useCacheForAllThreads);
    this.smallCacheSize = smallCacheSize;
    this.normalCacheSize = normalCacheSize;

    if (directMemoryCacheAlignment != 0) {
        if (!PlatformDependent.hasAlignDirectByteBuffer()) {
            throw new UnsupportedOperationException("Buffer alignment is not supported. " +
                    "Either Unsafe or ByteBuffer.alignSlice() must be available.");
        }
        pageSize = (int) PlatformDependent.align(pageSize, directMemoryCacheAlignment);
    }
   
    // 空歌白石：省略部分代码
}

PoolThreadLocalCache

private final class PoolThreadLocalCache extends FastThreadLocal<PoolThreadCache> {
    private final boolean useCacheForAllThreads;

    PoolThreadLocalCache(boolean useCacheForAllThreads) {
        this.useCacheForAllThreads = useCacheForAllThreads;
    }

    @Override
    protected synchronized PoolThreadCache initialValue() {   
    // 空歌白石：省略部分代码
            final PoolThreadCache cache = new PoolThreadCache(
                    heapArena, directArena, smallCacheSize, normalCacheSize,
                    DEFAULT_MAX_CACHED_BUFFER_CAPACITY, DEFAULT_CACHE_TRIM_INTERVAL);

            if (DEFAULT_CACHE_TRIM_INTERVAL_MILLIS > 0) {
                if (executor != null) {
                    executor.scheduleAtFixedRate(trimTask, DEFAULT_CACHE_TRIM_INTERVAL_MILLIS,
                            DEFAULT_CACHE_TRIM_INTERVAL_MILLIS, TimeUnit.MILLISECONDS);
                }
            }
   
    // 空歌白石：省略部分代码
    }

    @Override
    protected void onRemoval(PoolThreadCache threadCache) {
        threadCache.free(false);
    }
   
    // 空歌白石：省略部分代码
}

PoolThreadCache


    PoolThreadCache(PoolArena<byte[]> heapArena, PoolArena<ByteBuffer> directArena,
                    int smallCacheSize, int normalCacheSize, int maxCachedBufferCapacity,
                    int freeSweepAllocationThreshold) {
    // 空歌白石：省略部分代码
        if (directArena != null) {
            smallSubPageDirectCaches = createSubPageCaches(
                    smallCacheSize, directArena.numSmallSubpagePools);

            normalDirectCaches = createNormalCaches(
                    normalCacheSize, maxCachedBufferCapacity, directArena);

            directArena.numThreadCaches.getAndIncrement();
        }
   
    // 空歌白石：省略部分代码
    }

PoolArena

DirectArea分配Direct内存的流程

从对象池里面拿到PooledByteBuf进行复用。
从缓存上进行内存分配
从内存堆里面进行内存分配

@Override
protected PooledByteBuf<ByteBuffer> newByteBuf(int maxCapacity) {
    if (HAS_UNSAFE) {
        return PooledUnsafeDirectByteBuf.newInstance(maxCapacity);
    } else {
        return PooledDirectByteBuf.newInstance(maxCapacity);
    }
}

static PooledUnsafeDirectByteBuf newInstance(int maxCapacity) {
    PooledUnsafeDirectByteBuf buf = RECYCLER.get();
    // 空歌白石：内存复用
    buf.reuse(maxCapacity);
    return buf;
}

private static final ObjectPool<PooledUnsafeDirectByteBuf> RECYCLER = ObjectPool.newPool(
        new ObjectCreator<PooledUnsafeDirectByteBuf>() {
    @Override
    public PooledUnsafeDirectByteBuf newObject(Handle<PooledUnsafeDirectByteBuf> handle) {
        return new PooledUnsafeDirectByteBuf(handle, 0);
    }
});
   
    // 空歌白石：省略部分代码

public interface Handle<T> {
    void recycle(T self);
}

Netty内存分配的核心步骤

private void allocate(PoolThreadCache cache, PooledByteBuf<T> buf, final int reqCapacity) {
    final int sizeIdx = size2SizeIdx(reqCapacity);

    if (sizeIdx <= smallMaxSizeIdx) {
        tcacheAllocateSmall(cache, buf, reqCapacity, sizeIdx);
    } else if (sizeIdx < nSizes) {
        tcacheAllocateNormal(cache, buf, reqCapacity, sizeIdx);
    } else {
        int normCapacity = directMemoryCacheAlignment > 0
                ? normalizeSize(reqCapacity) : reqCapacity;
        // Huge allocations are never served via the cache so just call allocateHuge
        allocateHuge(buf, normCapacity);
    }
    // 空歌白石：省略部分代码

NettyRuntime


package io.netty.util;

import io.netty.util.internal.ObjectUtil;
import io.netty.util.internal.SystemPropertyUtil;

import java.util.Locale;

public final class NettyRuntime {

    static class AvailableProcessorsHolder {

        private int availableProcessors;
        synchronized void setAvailableProcessors(final int availableProcessors) {
            ObjectUtil.checkPositive(availableProcessors, "availableProcessors");
            if (this.availableProcessors != 0) {
                final String message = String.format(
                        Locale.ROOT,
                        "availableProcessors is already set to [%d], rejecting [%d]",
                        this.availableProcessors,
                        availableProcessors);
                throw new IllegalStateException(message);
            }
            this.availableProcessors = availableProcessors;
        }
   
    // 空歌白石：省略部分代码
    }
}

Netty内存规格

规格：

tiny： 0-512B
small: 512B-8K
normal: 8K-16M
huge：>16M

名称：

16MB -> chunk
8K -> page
0-8K -> subpage

为什么会是16M？

因为各种内存分配器的chunk都是按照16MB为单位，包括ptmalloc、tcmalloc、jemalloc等

MemoryRegionCache

MemoryRegionCache负责Netty缓存命中的分配逻辑。

MemoryRegionCache分类

queue
- chunk handler
sizeClass
- tiny：0-512B
- small：512B-8K
- normal：8K-16M
size
- tiny：N*16B
- small：512B、1K、2K、4K
- normal：8K、16K、32K

不同类型的节点数量

tiny[32]
small[4]
normal[3]

**注意:**最新源码已经没有tiny类型，只包含small和normal类型。

核心流程：

找到对应的size的MemoryRegionCache
从queue中弹出一个entry给ByteBuf初始化
将弹出的entry扔到对象池进行复用

private abstract static class MemoryRegionCache<T> {
    private final int size;
    private final Queue<Entry<T>> queue;
    private final SizeClass sizeClass;
    private int allocations;

    MemoryRegionCache(int size, SizeClass sizeClass) {
        this.size = MathUtil.safeFindNextPositivePowerOfTwo(size);
        queue = PlatformDependent.newFixedMpscQueue(this.size);
        this.sizeClass = sizeClass;
    }

    /**
    * Init the {@link PooledByteBuf} using the provided chunk and handle with the capacity restrictions.
    */
    protected abstract void initBuf(PoolChunk<T> chunk, ByteBuffer nioBuffer, long handle,
                                    PooledByteBuf<T> buf, int reqCapacity, PoolThreadCache threadCache);

    // 空歌白石：省略部分代码

    /**
        * Clear out this cache and free up all previous cached {@link PoolChunk}s and {@code handle}s.
        */
    public final int free(boolean finalizer) {
        return free(Integer.MAX_VALUE, finalizer);
    }
   
    // 空歌白石：省略部分代码
}

// Hold the caches for the different size classes, which are small and normal.
private final MemoryRegionCache<byte[]>[] smallSubPageHeapCaches;
private final MemoryRegionCache<ByteBuffer>[] smallSubPageDirectCaches;
private final MemoryRegionCache<byte[]>[] normalHeapCaches;
private final MemoryRegionCache<ByteBuffer>[] normalDirectCaches;

private static <T> MemoryRegionCache<T>[] createSubPageCaches(
        int cacheSize, int numCaches) {
   
    // 空歌白石：省略部分代码
        @SuppressWarnings("unchecked")
        MemoryRegionCache<T>[] cache = new MemoryRegionCache[numCaches];
        for (int i = 0; i < cache.length; i++) {
            // TODO: maybe use cacheSize / cache.length
            cache[i] = new SubPageMemoryRegionCache<T>(cacheSize);
        }
   
    // 空歌白石：省略部分代码
}

void release(DefaultHandle<T> handle) {
    MessagePassingQueue<DefaultHandle<T>> handles = pooledHandles;
    handle.toAvailable();
    if (handles != null) {
        handles.relaxedOffer(handle);
    }
}

关键概念

arena

通过Arena可以从系统开辟一块内存供netty使用。

final PoolArena<byte[]> heapArena;
final PoolArena<ByteBuffer> directArena;

enum SizeClass {
    Small,
    Normal
}

final PooledByteBufAllocator parent;

final int numSmallSubpagePools;
final int directMemoryCacheAlignment;
private final PoolSubpage<T>[] smallSubpagePools;

private final PoolChunkList<T> q050;
private final PoolChunkList<T> q025;
private final PoolChunkList<T> q000;
private final PoolChunkList<T> qInit;
private final PoolChunkList<T> q075;
private final PoolChunkList<T> q100;

PoolChunkList使用双向链表关联chunk

final class PoolChunkList<T> implements PoolChunkListMetric {
    private static final Iterator<PoolChunkMetric> EMPTY_METRICS = Collections.<PoolChunkMetric>emptyList().iterator();
    private final PoolArena<T> arena;
    private final PoolChunkList<T> nextList;
    private final int minUsage;
    private final int maxUsage;
    private final int maxCapacity;
    private PoolChunk<T> head;
    private final int freeMinThreshold;
    private final int freeMaxThreshold;

    // This is only update once when create the linked like list of PoolChunkList in PoolArena constructor.
    private PoolChunkList<T> prevList;

protected PoolArena(PooledByteBufAllocator parent, int pageSize,
        int pageShifts, int chunkSize, int cacheAlignment) {
    super(pageSize, pageShifts, chunkSize, cacheAlignment);
    this.parent = parent;
    directMemoryCacheAlignment = cacheAlignment;

    numSmallSubpagePools = nSubpages;
    smallSubpagePools = newSubpagePoolArray(numSmallSubpagePools);
    for (int i = 0; i < smallSubpagePools.length; i ++) {
        smallSubpagePools[i] = newSubpagePoolHead();
    }

    q100 = new PoolChunkList<T>(this, null, 100, Integer.MAX_VALUE, chunkSize);
    q075 = new PoolChunkList<T>(this, q100, 75, 100, chunkSize);
    q050 = new PoolChunkList<T>(this, q075, 50, 100, chunkSize);
    q025 = new PoolChunkList<T>(this, q050, 25, 75, chunkSize);
    q000 = new PoolChunkList<T>(this, q025, 1, 50, chunkSize);
    qInit = new PoolChunkList<T>(this, q000, Integer.MIN_VALUE, 25, chunkSize);
   
    // 空歌白石：省略部分代码
}

chunk

final class PoolChunk<T> implements PoolChunkMetric {
    private static final int SIZE_BIT_LENGTH = 15;
    private static final int INUSED_BIT_LENGTH = 1;
    private static final int SUBPAGE_BIT_LENGTH = 1;
    private static final int BITMAP_IDX_BIT_LENGTH = 32;

    static final int IS_SUBPAGE_SHIFT = BITMAP_IDX_BIT_LENGTH;
    static final int IS_USED_SHIFT = SUBPAGE_BIT_LENGTH + IS_SUBPAGE_SHIFT;
    static final int SIZE_SHIFT = INUSED_BIT_LENGTH + IS_USED_SHIFT;
    static final int RUN_OFFSET_SHIFT = SIZE_BIT_LENGTH + SIZE_SHIFT;

    final PoolArena<T> arena;
    final Object base;
    final T memory;
    final boolean unpooled;

page & subPage


final class PoolSubpage<T> implements PoolSubpageMetric {

    final PoolChunk<T> chunk;
    final int elemSize;
    private final int pageShifts;
    private final int runOffset;
    private final int runSize;
    private final long[] bitmap;

    PoolSubpage<T> prev;
    PoolSubpage<T> next;

    boolean doNotDestroy;
    private int maxNumElems;
    private int bitmapLength;
    private int nextAvail;
    private int numAvail;


PoolSubpage(PoolSubpage<T> head, PoolChunk<T> chunk, int pageShifts, int runOffset, int runSize, int elemSize) {
    this.chunk = chunk;
    this.pageShifts = pageShifts;
    this.runOffset = runOffset;
    this.runSize = runSize;
    this.elemSize = elemSize;
    bitmap = new long[runSize >>> 6 + LOG2_QUANTUM]; // runSize / 64 / QUANTUM

    doNotDestroy = true;
    if (elemSize != 0) {
        maxNumElems = numAvail = runSize / elemSize;
        nextAvail = 0;
        bitmapLength = maxNumElems >>> 6;
        if ((maxNumElems & 63) != 0) {
            bitmapLength ++;
        }

        for (int i = 0; i < bitmapLength; i ++) {
            bitmap[i] = 0;
        }
    }
    addToPool(head);
}

Page级别的allocateNormal

Page级别的内存分配，要么都重新分配，要么都使用缓存。

尝试在现有的chunk上分配
创建一个chunk进行内存分配
初始化PooledByteBuf

private void allocateNormal(PooledByteBuf<T> buf, int reqCapacity, int sizeIdx, PoolThreadCache threadCache) {
    assert lock.isHeldByCurrentThread();
    if (q050.allocate(buf, reqCapacity, sizeIdx, threadCache) ||
        q025.allocate(buf, reqCapacity, sizeIdx, threadCache) ||
        q000.allocate(buf, reqCapacity, sizeIdx, threadCache) ||
        qInit.allocate(buf, reqCapacity, sizeIdx, threadCache) ||
        q075.allocate(buf, reqCapacity, sizeIdx, threadCache)) {
        return;
    }

    // Add a new chunk.
    PoolChunk<T> c = newChunk(pageSize, nPSizes, pageShifts, chunkSize);
    boolean success = c.allocate(buf, reqCapacity, sizeIdx, threadCache);
    assert success;
    qInit.add(c);
}

原文描述：

 * Notation: The following terms are important to understand the code
 * > page  - a page is the smallest unit of memory chunk that can be allocated
 * > run   - a run is a collection of pages
 * > chunk - a chunk is a collection of runs
 * > in this code chunkSize = maxPages * pageSize

 *  A chunk has the following layout:
 *
 *     /-----------------\
 *     | run             |
 *     |                 |
 *     |                 |
 *     |-----------------|
 *     | run             |
 *     |                 |
 *     |-----------------|
 *     | unalloctated    |
 *     | (freed)         |
 *     |                 |
 *     |-----------------|
 *     | subpage         |
 *     |-----------------|
 *     | unallocated     |
 *     | (freed)         |
 *     | ...             |
 *     | ...             |
 *     | ...             |
 *     |                 |
 *     |                 |
 *     |                 |
 *     \-----------------/
 *
 *
 * handle:
 * -------
 * a handle is a long number, the bit layout of a run looks like:
 *
 * oooooooo ooooooos ssssssss ssssssue bbbbbbbb bbbbbbbb bbbbbbbb bbbbbbbb
 *
 * o: runOffset (page offset in the chunk), 15bit
 * s: size (number of pages) of this run, 15bit
 * u: isUsed?, 1bit
 * e: isSubpage?, 1bit
 * b: bitmapIdx of subpage, zero if it's not subpage, 32bit

private long allocateRun(int runSize) {
    int pages = runSize >> pageShifts;
    int pageIdx = arena.pages2pageIdx(pages);

    runsAvailLock.lock();
    try {
        //find first queue which has at least one big enough run
        int queueIdx = runFirstBestFit(pageIdx);
        if (queueIdx == -1) {
            return -1;
        }

        //get run with min offset in this queue
        LongPriorityQueue queue = runsAvail[queueIdx];
   
    // 空歌白石：省略部分代码
}

/**
* Create / initialize a new PoolSubpage of normCapacity. Any PoolSubpage created / initialized here is added to
* subpage pool in the PoolArena that owns this PoolChunk
*
* @param sizeIdx sizeIdx of normalized size
*
* @return index in memoryMap
*/
private long allocateSubpage(int sizeIdx) {
    // Obtain the head of the PoolSubPage pool that is owned by the PoolArena and synchronize on it.
    // This is need as we may add it back and so alter the linked-list structure.
    PoolSubpage<T> head = arena.findSubpagePoolHead(sizeIdx);
    head.lock();
    try {
        //allocate a new run
        int runSize = calculateRunSize(sizeIdx);
        //runSize must be multiples of pageSize
        long runHandle = allocateRun(runSize);
   
    // 空歌白石：省略部分代码
}

subPage级别的allocatesmall

private void tcacheAllocateSmall(PoolThreadCache cache, PooledByteBuf<T> buf, final int reqCapacity,
                                    final int sizeIdx) {

    if (cache.allocateSmall(this, buf, reqCapacity, sizeIdx)) {
        // was able to allocate out of the cache so move on
        return;
    }

    /*
    * Synchronize on the head. This is needed as {@link PoolChunk#allocateSubpage(int)} and
    * {@link PoolChunk#free(long)} may modify the doubly linked list as well.
    */
    final PoolSubpage<T> head = smallSubpagePools[sizeIdx];
    final boolean needsNormalAllocation;
    head.lock();
   
    // 空歌白石：省略部分代码
            allocateNormal(buf, reqCapacity, sizeIdx, cache);
   
    // 空歌白石：省略部分代码
}

/**
* Try to allocate a small buffer out of the cache. Returns {@code true} if successful {@code false} otherwise
*/
boolean allocateSmall(PoolArena<?> area, PooledByteBuf<?> buf, int reqCapacity, int sizeIdx) {
    return allocate(cacheForSmall(area, sizeIdx), buf, reqCapacity);
}

private MemoryRegionCache<?> cacheForSmall(PoolArena<?> area, int sizeIdx) {
    if (area.isDirect()) {
        return cache(smallSubPageDirectCaches, sizeIdx);
    }
    return cache(smallSubPageHeapCaches, sizeIdx);
}

@SuppressWarnings({ "unchecked", "rawtypes" })
private boolean allocate(MemoryRegionCache<?> cache, PooledByteBuf buf, int reqCapacity) {
    if (cache == null) {
        // no cache found so just return false here
        return false;
    }
    boolean allocated = cache.allocate(buf, reqCapacity, this);
    if (++ allocations >= freeSweepAllocationThreshold) {
        allocations = 0;
        trim();
    }
    return allocated;
}

/**
* Create / initialize a new PoolSubpage of normCapacity. Any PoolSubpage created / initialized here is added to
* subpage pool in the PoolArena that owns this PoolChunk
*
* @param sizeIdx sizeIdx of normalized size
*
* @return index in memoryMap
*/
private long allocateSubpage(int sizeIdx) {
    // Obtain the head of the PoolSubPage pool that is owned by the PoolArena and synchronize on it.
    // This is need as we may add it back and so alter the linked-list structure.
    PoolSubpage<T> head = arena.findSubpagePoolHead(sizeIdx);
    head.lock();
    try {
   
    // 空歌白石：省略部分代码
        //runSize must be multiples of pageSize
        long runHandle = allocateRun(runSize);
   
    // 空歌白石：省略部分代码
}

创建PoolChunk时，也会同步初始化PoolSubpage的array。

@SuppressWarnings("unchecked")
PoolChunk(PoolArena<T> arena, Object base, T memory, int pageSize, int pageShifts, int chunkSize, int maxPageIdx) {
    unpooled = false;
    this.arena = arena;
    this.base = base;
    this.memory = memory;
    this.pageSize = pageSize;
    this.pageShifts = pageShifts;
    this.chunkSize = chunkSize;
    freeBytes = chunkSize;

    runsAvail = newRunsAvailqueueArray(maxPageIdx);
    runsAvailLock = new ReentrantLock();
    runsAvailMap = new LongLongHashMap(-1);
    subpages = new PoolSubpage[chunkSize >> pageShifts];

    //insert initial run, offset = 0, pages = chunkSize / pageSize
    int pages = chunkSize >> pageShifts;
    long initHandle = (long) pages << SIZE_SHIFT;
    insertAvailRun(0, pages, initHandle);

    cachedNioBuffers = new ArrayDeque<ByteBuffer>(8);
}

从bitmap中找到未被使用的subpage，可用状态为0，则从pool中移除。

/**
* Returns the bitmap index of the subpage allocation.
*/
long allocate() {
    if (numAvail == 0 || !doNotDestroy) {
        return -1;
    }

    final int bitmapIdx = getNextAvail();
    int q = bitmapIdx >>> 6;
    int r = bitmapIdx & 63;
    assert (bitmap[q] >>> r & 1) == 0;
    bitmap[q] |= 1L << r;

    if (-- numAvail == 0) {
        removeFromPool();
    }

    return toHandle(bitmapIdx);
}

bitmapIdx与memoryMapIdx 拼接。

private long toHandle(int bitmapIdx) {
    int pages = runSize >> pageShifts;
    return (long) runOffset << RUN_OFFSET_SHIFT
            | (long) pages << SIZE_SHIFT
            | 1L << IS_USED_SHIFT
            | 1L << IS_SUBPAGE_SHIFT
            | bitmapIdx;
}

void initBufWithSubpage(PooledByteBuf<T> buf, ByteBuffer nioBuffer, long handle, int reqCapacity,
                        PoolThreadCache threadCache) {
    int runOffset = runOffset(handle);
    int bitmapIdx = bitmapIdx(handle);

    PoolSubpage<T> s = subpages[runOffset];
   
    // 空歌白石：省略部分代码
}

void init(PoolChunk<T> chunk, ByteBuffer nioBuffer,
            long handle, int offset, int length, int maxLength, PoolThreadCache cache) {
    init0(chunk, nioBuffer, handle, offset, length, maxLength, cache);
}

void initUnpooled(PoolChunk<T> chunk, int length) {
    init0(chunk, null, 0, 0, length, length, null);
}
   
    // 空歌白石：省略部分代码
}

内存的释放

核心过程：

连续的内存区段加到缓存
标记联系的内存区段为未使用的内存段
ByteBuf加到对象池

void decrementPinnedMemory(int delta) {
    assert delta > 0;
    pinnedBytes.add(-delta);
}

void free(PoolChunk<T> chunk, ByteBuffer nioBuffer, long handle, int normCapacity, PoolThreadCache cache) {
    if (chunk.unpooled) {
        int size = chunk.chunkSize();
        destroyChunk(chunk);
        activeBytesHuge.add(-size);
        deallocationsHuge.increment();
   
    // 空歌白石：省略部分代码
}

private MemoryRegionCache<?> cache(PoolArena<?> area, int sizeIdx, SizeClass sizeClass) {
    switch (sizeClass) {
    case Normal:
        return cacheForNormal(area, sizeIdx);
    case Small:
        return cacheForSmall(area, sizeIdx);
    default:
        throw new Error();
    }
}
    // 空歌白石：省略部分代码

void freeChunk(PoolChunk<T> chunk, long handle, int normCapacity, SizeClass sizeClass, ByteBuffer nioBuffer,
                boolean finalizer) {
    // 空歌白石：省略部分代码
        destroyChunk = !chunk.parent.free(chunk, handle, normCapacity, nioBuffer);
    // 空歌白石：省略部分代码
}

private void recycle() {
    recyclerHandle.recycle(this);
}
@Override
public void recycle(Object object) {
    if (object != value) {
        throw new IllegalArgumentException("object does not belong to handle");
    }
    localPool.release(this);
}

ObjectPool

package io.netty.util.internal;

import io.netty.util.Recycler;

/**
 * Light-weight object pool.
 *
 * @param <T> the type of the pooled object
 */
public abstract class ObjectPool<T> {
   
    // 空歌白石：省略部分代码

    private static final class RecyclerObjectPool<T> extends ObjectPool<T> {
        private final Recycler<T> recycler;

        RecyclerObjectPool(final ObjectCreator<T> creator) {
             recycler = new Recycler<T>() {
                @Override
                protected T newObject(Handle<T> handle) {
                    return creator.newObject(handle);
                }
            };
        }

        @Override
        public T get() {
            return recycler.get();
        }
    }
}

总结

ByteBuf的api和分类
分配Pooled内存的总步骤
不同规格的Pooled内存分配和释放