Cleaner

public class Cleaner
    extends PhantomReference<Object>
{
    ...
}

在 DirectByteBuffer 的内存回收中，有一个非常关键的对象：cleaner，Cleaner 类继承了 PhantomReference 类，表示它是一个 Java 虚引用类型

cleaner 对象是在创建 DirectByteBuffer 对象时初始化的，其源码如下：

// DirectByteBuffer 创建时初始化 cleaner 对象
cleaner = Cleaner.create(this, new Deallocator(base, size, cap));

// create方法在Cleaner类中
public static Cleaner create(Object ob, Runnable thunk) {
    if (thunk == null)
        return null;
    return add(new Cleaner(ob, thunk));
}

// Cleaner类的构造方法
private Cleaner(Object referent, Runnable thunk) {
    super(referent, dummyQueue);
    this.thunk = thunk;
}

从创建 cleaner 对象的参数可以得知 cleaner 对象持有当前 DirectByteBuffer 对象的虚引用，因为 Java 对象的虚引用的关系，当 DirectByteBuffer 被 gc 回收掉之后，会将 cleaner 对象放入 dummyQueue 队列

同时将 new Deallocator(base, size, cap) 复制给 thunk 属性

创建时回收

分析 DirectByteBuffer 创建时回收需要查看 OpenJDK 源码

DirectByteBuffer(int cap) {                   // package-private

    super(-1, 0, cap, cap);
    boolean pa = VM.isDirectMemoryPageAligned();
    int ps = Bits.pageSize();
    long size = Math.max(1L, (long)cap + (pa ? ps : 0));
    //检查物理内存是否足够，如果不足则进行内存回收
    Bits.reserveMemory(size, cap);

    long base = 0;
    try {
        base = UNSAFE.allocateMemory(size);
    } catch (OutOfMemoryError x) {
        Bits.unreserveMemory(size, cap);
        throw x;
    }
    UNSAFE.setMemory(base, size, (byte) 0);
    if (pa && (base % ps != 0)) {
        // Round up to page boundary
        address = base + ps - (base & (ps - 1));
    } else {
        address = base;
    }
    // 内存回收器
    cleaner = Cleaner.create(this, new Deallocator(base, size, cap));
    att = null;
}

从源码可知 Bits.reserveMemory(size, cap) 正是在回收内存：

static void reserveMemory(long size, int cap) {

    ...
    
    // 剩余的直接内存是否足够本次分配，如果足够则返回，不够则进行回收
    if (tryReserveMemory(size, cap)) {
        return;
    }

    final JavaLangRefAccess jlra = SharedSecrets.getJavaLangRefAccess();
    boolean interrupted = false;
    try {
        boolean refprocActive;
        do {
            try {
                // 等待 ReferenceHandler 线程回收 DirectByteBuffer 内存
                refprocActive = jlra.waitForReferenceProcessing();
            } catch (InterruptedException e) {
                // Defer interrupts and keep trying.
                interrupted = true;
                refprocActive = true;
            }
            if (tryReserveMemory(size, cap)) {
                return;
            }
        } while (refprocActive);

        // 触发一次 full gc
        System.gc();

        long sleepTime = 1;
        int sleeps = 0;
        while (true) {
            if (tryReserveMemory(size, cap)) {
                return;
            }
            // 判断是否达到最大睡眠次数，如果达到了就抛出 OutOfMemoryError
            if (sleeps >= MAX_SLEEPS) {
                break;
            }
            try {
                // 等待 ReferenceHandler 线程回收 DirectByteBuffer 内存
                if (!jlra.waitForReferenceProcessing()) {
                    // 每次睡眠时间都会加长 2 倍
                    Thread.sleep(sleepTime);
                    sleepTime <<= 1;
                    // 增加睡眠次数
                    sleeps++;
                }
            } catch (InterruptedException e) {
                interrupted = true;
            }
        }

        // no luck
        throw new OutOfMemoryError("Direct buffer memory");

    } finally {
        if (interrupted) {
            // don't swallow interrupts
            Thread.currentThread().interrupt();
        }
    }
}

private static boolean tryReserveMemory(long size, int cap) {

    long totalCap;
    // 判断可用的直接内存是否足够本次内存分配
    while (cap <= MAX_MEMORY - (totalCap = TOTAL_CAPACITY.get())) {
        // 使用 CAS 保证已使用内存增加的并发安全性
        if (TOTAL_CAPACITY.compareAndSet(totalCap, totalCap + cap)) {
            RESERVED_MEMORY.addAndGet(size);
            COUNT.incrementAndGet();
            return true;
        }
    }
    return false;
}

从以上源码分析，reserveMemory 释放内存的过程如下：

剩余可用内存是否足够，如果足够则直接返回
jlra.waitForReferenceProcessing() 方法等待 ReferenceHandler 线程回收 DirectByteBuffer 内存
jlra.waitForReferenceProcessing() 被 ReferenceHandler 唤醒后再次判断剩余可用内存是否是否足够，如果不足则继续等待 ReferenceHandler 线程回收 DirectByteBuffer 内存
如果 jlra.waitForReferenceProcessing() 方法返回 false，即 ReferenceHandler 没有在执行DirectByteBuffer 内存回收并且 PendingList 中没有数据（如果 DirectByteBuffer 对象被 gc 回收，因为Cleaner 对象虚引用的关系，会将 Cleaner 对象放入 PendingList，没有数据则表示没有 DirectByteBuffer 被 gc 回收）
触发 full gc
full gc 之后继续等待 ReferenceHandler 线程回收 DirectByteBuffer 内存
等待一定次数之后还是内存不足就会抛出 OutOfMemoryError

ReferenceHandler 线程回收

ReferenceHandler 类继承了 Thread 类，所以直接查看其 run 方法：

public void run() {
    while (true) {
        processPendingReferences();
    }
}

在 run 方法中会循环调用 processPendingReferences 方法：

private static void processPendingReferences() {
    // Only the singleton reference processing thread calls
    // waitForReferencePendingList() and getAndClearReferencePendingList().
    // These are separate operations to avoid a race with other threads
    // that are calling waitForReferenceProcessing().
    waitForReferencePendingList();
    Reference<Object> pendingList;
    synchronized (processPendingLock) {
        pendingList = getAndClearReferencePendingList();
        processPendingActive = true;
    }
    while (pendingList != null) {
        Reference<Object> ref = pendingList;
        pendingList = ref.discovered;
        ref.discovered = null;
        // 如果是 Cleaner 对象，则调用 clean 方法
        if (ref instanceof Cleaner) {
            ((Cleaner)ref).clean();
            // Notify any waiters that progress has been made.
            // This improves latency for nio.Bits waiters, which
            // are the only important ones.
            synchronized (processPendingLock) {
                processPendingLock.notifyAll();
            }
        } else {
            ReferenceQueue<? super Object> q = ref.queue;
            if (q != ReferenceQueue.NULL) q.enqueue(ref);
        }
    }
    // Notify any waiters of completion of current round.
    synchronized (processPendingLock) {
        processPendingActive = false;
        processPendingLock.notifyAll();
    }
}

在 processPendingReferences 方法中会判断 pendingList 中的元素是否是 Cleaner 对象，如果是则调用 Cleaner 对象的 clean 方法：

public void clean() {
    if (!remove(this))
        return;
    try {
        thunk.run();
    } catch (final Throwable x) {
        AccessController.doPrivileged(new PrivilegedAction<>() {
                public Void run() {
                    if (System.err != null)
                        new Error("Cleaner terminated abnormally", x)
                            .printStackTrace();
                    System.exit(1);
                    return null;
                }});
    }
}

在 clean 方法中又调用了 thunk.run() 方法：

public void run() {
    if (address == 0) {
        // Paranoia
        return;
    }
    // 释放分配的内存
    UNSAFE.freeMemory(address);
    address = 0;
    // 减少记录的已使用容量
    Bits.unreserveMemory(size, capacity);
}

可以看到，正是在 thunk.run() 方法中完成了最后的内存释放

总结

DirectByteBuffer 的内存回收是在 DirectByteBuffer 对象被 gc 回收之后通过虚引用的特性将 cleaner 对象放入 pendingList，ReferenceHandler 线程会一直获取 pendingList 中的元素进行处理。而创建时回收其实就是内存不足时等待 ReferenceHandler 回收内存，等待一段时间之后内存还是不足则抛出 OutOfMemoryError

Java DirectByteBuffer内存回收

Cleaner

创建时回收

ReferenceHandler 线程回收

总结