JVM 线上排查实战(三):grep BLOCKED 找不到的死锁,和 jstack 根本不报的死锁连载第三篇:线程卡住

这个系列

「JVM 线上排查实战」共五篇,每篇都在真机上跑、JDK 8 和 JDK 17 两个版本都贴输出:

(一)先把 JVM 看清楚:进程、参数、默认值
(二)CPU 飙高:找到那个线程
(三)线程卡住:死锁、BLOCKED、线程池打满 —— 本篇
(四)内存:OOM 了先干什么
(五)GC 日志:从 JDK 8 升 17,老启动参数会让进程直接起不来

实测环境同前两篇:CentOS 7.9 · JDK 1.8.0_381 装在 /opt/jdk8 · JDK 17.0.8 装在 /usr/java。所有 demo 用 JDK 8 编译,同一份 class 在两个版本上各跑一遍,启动 3 秒后抓 jstack。

先说结论

上一篇是 CPU 高;这一篇是反过来的情况:接口不返回,CPU 却不高。线程没在干活,是在等。等什么,全写在线程栈里,但有四个地方容易看错:

ReentrantLock 死锁的线程状态是 WAITING (parking),不是 BLOCKED。只 grep BLOCKED 找死锁,会一个都找不到
不加 -l,看不到 ReentrantLock 在谁手里。synchronized 的持有者栈里有 - locked <…>,ReentrantLock 的持有者什么都不显示
线程池里「闲着」和「卡住」的线程,状态都是 WAITING (parking)。要看栈里第一行业务代码,不是看状态
线程池自己等自己(父任务在池里 get() 丢回同一个池的子任务),进程卡死,但 jstack 不报死锁

好消息是:ReentrantLock 互锁、synchronized 和 ReentrantLock 混着互锁,jstack 末尾的死锁检测都报出来了,8 和 17 一样。

1. 最标准的:synchronized 死锁 ✅

两个线程,各拿一把锁,再去拿对方那把:

public class DeadSync {
    static final Object ORDER = new Object();
    static final Object STOCK = new Object();

    public static void main(String[] args) {
        new Thread(() -> {
            synchronized (ORDER) {
                sleep(100);
                synchronized (STOCK) { System.out.println("never"); }
            }
        }, "order-thread").start();
        new Thread(() -> {
            synchronized (STOCK) {
                sleep(100);
                synchronized (ORDER) { System.out.println("never"); }
            }
        }, "stock-thread").start();
    }

    static void sleep(long ms) { try { Thread.sleep(ms); } catch (InterruptedException e) { } }
}

jstack <pid> 的最末尾,JDK 8u381:

Found one Java-level deadlock:
=============================
"stock-thread":
  waiting to lock monitor 0x00007fa3d40062c8 (object 0x00000000ec5d8ca8, a java.lang.Object),
  which is held by "order-thread"
"order-thread":
  waiting to lock monitor 0x00007fa3d4004e28 (object 0x00000000ec5d8cb8, a java.lang.Object),
  which is held by "stock-thread"

Java stack information for the threads listed above:
===================================================
"stock-thread":
	at DeadSync.lambda$main$1(DeadSync.java:15)
	- waiting to lock <0x00000000ec5d8ca8> (a java.lang.Object)
	- locked <0x00000000ec5d8cb8> (a java.lang.Object)
	at DeadSync$$Lambda$2/303563356.run(Unknown Source)
	at java.lang.Thread.run(Thread.java:750)
"order-thread":
	at DeadSync.lambda$main$0(DeadSync.java:9)
	- waiting to lock <0x00000000ec5d8cb8> (a java.lang.Object)
	- locked <0x00000000ec5d8ca8> (a java.lang.Object)
	at DeadSync$$Lambda$1/471910020.run(Unknown Source)
	at java.lang.Thread.run(Thread.java:750)

Found 1 deadlock.

JDK 17.0.8 内容一样,只是两段线程之间多了空行,JDK 自带类的栈帧带 java.base@17.0.8/ 前缀。

两个线程在线程列表里的状态都是:

"order-thread" #13 prio=5 os_prio=0 cpu=0.47ms elapsed=3.14s tid=0x00007fcb1010ca10 nid=0x131ff waiting for monitor entry  [0x00007fcaf4383000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at DeadSync.lambda$main$0(DeadSync.java:9)
	- waiting to lock <0x00000000c8b19860> (a java.lang.Object)
	- locked <0x00000000c8b19850> (a java.lang.Object)

(这段是 JDK 17 的,cpu=0.47ms 说明它几乎没用过 CPU —— 死锁不烧 CPU,这也是「CPU 不高但接口不返回」的典型样子。)

读法:- locked 是我拿着的,- waiting to lock 是我在等的。两个线程的这两行地址正好交叉,就是死锁。

2. 坑一:ReentrantLock 死锁里没有 BLOCKED ✅

同样的逻辑,换成 ReentrantLock:

import java.util.concurrent.locks.ReentrantLock;

public class DeadLock {
    static final ReentrantLock ACCOUNT_A = new ReentrantLock();
    static final ReentrantLock ACCOUNT_B = new ReentrantLock();

    public static void main(String[] args) {
        new Thread(() -> transfer(ACCOUNT_A, ACCOUNT_B), "transfer-a2b").start();
        new Thread(() -> transfer(ACCOUNT_B, ACCOUNT_A), "transfer-b2a").start();
    }

    static void transfer(ReentrantLock from, ReentrantLock to) {
        from.lock();
        try {
            try { Thread.sleep(100); } catch (InterruptedException e) { }
            to.lock();
            try { System.out.println("never"); } finally { to.unlock(); }
        } finally {
            from.unlock();
        }
    }
}

jstack 末尾照样报了(JDK 8u381):

Found one Java-level deadlock:
=============================
"transfer-b2a":
  waiting for ownable synchronizer 0x00000000ec5da028, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),
  which is held by "transfer-a2b"
"transfer-a2b":
  waiting for ownable synchronizer 0x00000000ec5da058, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),
  which is held by "transfer-b2a"

但线程本身的状态是:

"transfer-a2b" #9 prio=5 os_prio=0 tid=0x00007fc00c1c4000 nid=0x13066 waiting on condition [0x00007fbffa470000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000ec5da058> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
	at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
	at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
	at DeadLock.transfer(DeadLock.java:16)

WAITING (parking),不是 BLOCKED。 BLOCKED 只用于等 synchronized 的监视器锁;ReentrantLock 底层走的是 LockSupport.park,所以显示成 parking。

再加一个两种锁混着用的(一个线程拿着 synchronized 等 ReentrantLock,另一个反过来):

new Thread(() -> {
    synchronized (CACHE) {
        sleep(100);
        DB.lock();
    }
}, "cache-refresher").start();
new Thread(() -> {
    DB.lock();
    sleep(100);
    synchronized (CACHE) { System.out.println("never"); }
}, "db-writer").start();

jstack 也报了死锁(JDK 8u381):

Found one Java-level deadlock:
=============================
"db-writer":
  waiting to lock monitor 0x00007fcdac004e28 (object 0x00000000ec5d8c30, a java.lang.Object),
  which is held by "cache-refresher"
"cache-refresher":
  waiting for ownable synchronizer 0x00000000ec5d9f68, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),
  which is held by "db-writer"

三个死锁 demo,每个版本数一下 grep -c 'State: BLOCKED':

demo	死锁构成	`BLOCKED` 线程数(8 / 17)	末尾报死锁(8 / 17)
DeadSync	synchronized × synchronized	2 / 2	✅ / ✅
DeadLock	ReentrantLock × ReentrantLock	0 / 0	✅ / ✅
Mixed	synchronized × ReentrantLock	1 / 1	✅ / ✅

结论:找死锁看 jstack 末尾的 Found one Java-level deadlock,别靠 grep BLOCKED。 后者在 ReentrantLock 场景下一个都找不到,混合场景只找到一半。

3. 坑二:不加 `-l`,看不到 ReentrantLock 在谁手里 ✅

死锁 jstack 会替你分析好。但更常见的是没死锁、只是有个线程拿着锁不放,别的线程都在排队。这时要自己找「谁拿着锁」。

实验程序,一个进程里放各种状态的线程:

import java.util.concurrent.locks.LockSupport;
import java.util.concurrent.locks.ReentrantLock;

public class States {
    static final Object MONITOR = new Object();
    static final Object SIGNAL = new Object();
    static final ReentrantLock LOCK = new ReentrantLock();
    static volatile long counter;

    public static void main(String[] args) throws Exception {
        start("busy", () -> { while (true) counter++; });
        start("sync-holder", () -> { synchronized (MONITOR) { sleep(3_600_000); } });
        Thread.sleep(200);
        start("sync-waiter", () -> { synchronized (MONITOR) { counter++; } });
        start("obj-wait", () -> { synchronized (SIGNAL) { try { SIGNAL.wait(); } catch (InterruptedException e) { } } });
        start("parked", LockSupport::park);
        start("sleeper", () -> sleep(3_600_000));
        start("lock-holder", () -> { LOCK.lock(); sleep(3_600_000); });
        Thread.sleep(200);
        start("lock-waiter", () -> { LOCK.lock(); counter++; });
        Thread.sleep(3_600_000);
    }

    static void start(String name, Runnable r) { new Thread(r, name).start(); }
    static void sleep(long ms) { try { Thread.sleep(ms); } catch (InterruptedException e) { } }
}

synchronized 的持有者,普通 jstack 就能看出来(JDK 8u381):

"sync-holder" #10 prio=5 os_prio=0 tid=0x00007fad0c196000 nid=0x130f4 waiting on condition [0x00007facf4f6b000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep(Native Method)
	at States.sleep(States.java:25)
	at States.lambda$main$1(States.java:12)
	- locked <0x00000000ec5d9200> (a java.lang.Object)

- locked <0x00000000ec5d9200>,和 sync-waiter 那行 - waiting to lock <0x00000000ec5d9200> 对得上。

ReentrantLock 的持有者,普通 jstack(JDK 8u381):

"lock-holder" #15 prio=5 os_prio=0 tid=0x00007fad0c1a0000 nid=0x130f9 waiting on condition [0x00007facf4a66000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep(Native Method)
	at States.sleep(States.java:25)
	at States.lambda$main$5(States.java:18)
	at States$$Lambda$7/250421012.run(Unknown Source)
	at java.lang.Thread.run(Thread.java:750)

什么都看不出来。 它明明拿着锁,栈里只是一个在 sleep 的普通线程。

加上 -l(jstack -l <pid>):

"lock-holder" #15 prio=5 os_prio=0 tid=0x00007fad0c1a0000 nid=0x130f9 waiting on condition [0x00007facf4a66000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep(Native Method)
	at States.sleep(States.java:25)
	at States.lambda$main$5(States.java:18)
	at States$$Lambda$7/250421012.run(Unknown Source)
	at java.lang.Thread.run(Thread.java:750)

   Locked ownable synchronizers:
	- <0x00000000ec5da548> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

而排队的 lock-waiter 栈里写着 - parking to wait for <0x00000000ec5da548>。拿等待方的地址,去 -l 的输出里搜 Locked ownable synchronizers 下面的同一个地址,就找到了持有者。 JDK 17.0.8 同样如此(地址是 0x00000000c8b19e98)。

jcmd 也一样,默认不带这段,要显式加 -l。拿 ReentrantLock 死锁那个进程数 Locked ownable synchronizers 出现的次数:

命令	JDK 8u381	JDK 17.0.8
`jstack <pid>`	0	0
`jstack -l <pid>`	11	14
`jcmd <pid> Thread.print`	0	0
`jcmd <pid> Thread.print -l`	11	14

-l 会给每个线程都加这一段,大多数写的是 - None,所以次数比线程池、锁的数量多得多。

-l 还有一个容易看错的地方:线程池里正在跑任务的线程,都会显示拿着一个 ThreadPoolExecutor$Worker:

   Locked ownable synchronizers:
	- <0x00000000ec77bd50> (a java.util.concurrent.ThreadPoolExecutor$Worker)

这是线程池自己的 Worker 在执行任务期间锁着,不是你的业务锁,找持有者时跳过它。

4. 线程状态怎么读:每种状态对应的代码 ✅

上面 States 程序里每个线程的状态和栈顶,两个版本一致:

线程	代码	状态	栈里的关键行
busy	`while (true) counter++`	`RUNNABLE`	直接是业务代码
sync-holder	`synchronized` 里 `sleep`	`TIMED_WAITING (sleeping)`	`- locked <…>`
sync-waiter	进同一个 `synchronized`	`BLOCKED (on object monitor)`	`- waiting to lock <…>`
obj-wait	`SIGNAL.wait()`	`WAITING (on object monitor)`	`- waiting on <…>`
parked	`LockSupport.park()`	`WAITING (parking)`	`LockSupport.park`
sleeper	`Thread.sleep`	`TIMED_WAITING (sleeping)`	`Thread.sleep`
lock-holder	`LOCK.lock()` 后 `sleep`	`TIMED_WAITING (sleeping)`	不加 `-l` 看不出持锁
lock-waiter	`LOCK.lock()`	`WAITING (parking)`	`- parking to wait for <…> (ReentrantLock$NonfairSync)`

几点:

持锁线程没有专门的状态:sync-holder、lock-holder 都是 TIMED_WAITING (sleeping),和没拿任何锁的 sleeper 一模一样 —— 状态只反映它拿着锁在干什么(这里在 sleep)
本文所有 demo 里,BLOCKED 只出现在等 synchronized 的线程上;ReentrantLock 抢锁、队列 take()、Future.get() 全是 WAITING (parking)
WAITING (parking) 本身不说明问题,要看 parking to wait for 后面是什么类型的对象:ReentrantLock$NonfairSync 是在抢锁,FutureTask 是在等结果,AbstractQueuedSynchronizer$ConditionObject 在本文里是在等阻塞队列的 take()(下一节)

5. 线程池打满:连接池拿不到连接 ✅

线上更常见的「卡住」不是死锁,是下游慢,把线程池拖满。模拟一下:业务线程池 4 个线程,连接池只有 2 个连接,每条 SQL 跑 10 分钟;再放一个什么活都没有的空闲池作对照。

import java.util.concurrent.*;
import java.util.concurrent.atomic.AtomicInteger;

public class PoolFull {
    // 模拟一个只有 2 个连接的连接池
    static final BlockingQueue<Object> CONNS = new ArrayBlockingQueue<>(2);

    public static void main(String[] args) throws Exception {
        CONNS.put(new Object()); CONNS.put(new Object());
        ThreadPoolExecutor biz = new ThreadPoolExecutor(4, 4, 0, TimeUnit.SECONDS,
                new LinkedBlockingQueue<>(), named("biz-"));
        ThreadPoolExecutor idle = new ThreadPoolExecutor(2, 2, 0, TimeUnit.SECONDS,
                new LinkedBlockingQueue<>(), named("idle-"));
        idle.prestartAllCoreThreads();
        for (int i = 0; i < 20; i++) biz.execute(PoolFull::handleRequest);
        while (true) {
            System.out.printf("active=%d queue=%d completed=%d%n",
                    biz.getActiveCount(), biz.getQueue().size(), biz.getCompletedTaskCount());
            Thread.sleep(5000);
        }
    }

    static void handleRequest() {
        try {
            Object conn = CONNS.take();          // 拿不到连接就一直等
            try { queryDb(); } finally { CONNS.put(conn); }
        } catch (InterruptedException e) { Thread.currentThread().interrupt(); }
    }

    static void queryDb() throws InterruptedException { Thread.sleep(600_000); } // 慢 SQL

    static ThreadFactory named(String prefix) {
        AtomicInteger n = new AtomicInteger();
        return r -> new Thread(r, prefix + n.incrementAndGet());
    }
}

程序自己打印的线程池状态(两个版本相同):

active=4 queue=16 completed=0

4 个线程全忙,16 个请求在排队,一个都没完成。栈里分成三种(JDK 8u381):

拿到连接、在跑慢 SQL 的(biz-1、biz-2):

"biz-1" #11 prio=5 os_prio=0 tid=0x00007f646419d000 nid=0x1317a waiting on condition [0x00007f644baf9000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep(Native Method)
	at PoolFull.queryDb(PoolFull.java:30)
	at PoolFull.handleRequest(PoolFull.java:26)

拿不到连接、在等的(biz-3、biz-4):

"biz-3" #13 prio=5 os_prio=0 tid=0x00007f64641a0800 nid=0x1317c waiting on condition [0x00007f644b8f7000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000ec5dbad0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
	at PoolFull.handleRequest(PoolFull.java:25)

空闲池里什么都没干的(idle-1):

"idle-1" #9 prio=5 os_prio=0 tid=0x00007f6464198800 nid=0x13178 waiting on condition [0x00007f644bcfb000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000ec77d6d0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

坑三:闲着和卡住,状态一模一样

biz-3(卡住、请求在超时)和 idle-1(闲着、完全健康)状态都是 WAITING (parking),等的对象类型也都是 ConditionObject。区别只在往下看几行:

卡住的:下面有你的业务代码 PoolFull.handleRequest,在它的调用里等 ArrayBlockingQueue.take
闲着的:下面是 ThreadPoolExecutor.getTask,没有业务代码,就是在等新任务

所以「数一下有多少个 WAITING 线程」得不出任何结论。按状态统计的结果是这样的(JDK 8u381):

$ grep 'java.lang.Thread.State' jstack.txt | sort | uniq -c | sort -nr
      6    java.lang.Thread.State: RUNNABLE
      4    java.lang.Thread.State: WAITING (parking)
      3    java.lang.Thread.State: TIMED_WAITING (sleeping)
      2    java.lang.Thread.State: WAITING (on object monitor)

4 个 WAITING (parking) 里,2 个是卡住的,2 个是闲着的,混在一起分不开。

更有用的是按「线程池名 + 状态 + 第一行业务代码」分组。下面这个脚本把线程名末尾的编号去掉(biz-1~biz-4 归成 biz),再找栈里第一个不是 java./sun./jdk. 开头的帧:

#!/bin/bash
# 按「线程名去掉尾部数字 + 状态 + 第一帧业务代码」分组计数
awk 'BEGIN{RS="";FS="\n"} /^"/{
  name=$1; sub(/^"/,"",name); sub(/".*/,"",name); gsub(/[-_ ]?[0-9]+$/,"",name)
  st="-"; fr="(无业务帧)"
  for(i=2;i<=NF;i++){
    if($i ~ /java.lang.Thread.State:/){st=$i; sub(/.*State: /,"",st)}
    else if($i ~ /^\tat / && $i !~ /^\tat (java|javax|sun|jdk|com\.sun)\./){fr=$i; sub(/^\tat /,"",fr); sub(/\(.*/,"",fr); break}
  }
  print name " | " st " | " fr
}' "$1" | sort | uniq -c | sort -nr

同一份 jstack 的结果(JDK 17.0.8,截前几行):

      2 idle | WAITING (parking) | (无业务帧)
      2 biz | WAITING (parking) | PoolFull.handleRequest
      2 biz | TIMED_WAITING (sleeping) | PoolFull.queryDb

一眼就能读出来:biz 池 2 个在跑慢 SQL(queryDb),2 个卡在 handleRequest 里等东西(往上一帧就是 ArrayBlockingQueue.take,即等连接);idle 池闲着。瓶颈是连接池,不是线程池 —— 线程池调大只会让更多线程排队等那 2 个连接。

注意:这个脚本假设线程池给线程起了名字。没起名字的线程叫 pool-1-thread-1,去掉尾部数字后是 pool-1-thread,不同池也能分开;但你就不知道 pool-3 是哪个业务的池了。线程池起名字,排查时省一半事。

顺带一个坑:JDK 17 的栈里冒出了 ForkJoinPool

同样的 biz-3,JDK 17.0.8 的栈:

"biz-3" #17 prio=5 os_prio=0 cpu=0.15ms elapsed=3.15s tid=0x00007f05ec0fc660 nid=0x133d6 waiting on condition  [0x00007f05c51ef000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@17.0.8/Native Method)
	- parking to wait for  <0x00000000c8b1b250> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(java.base@17.0.8/LockSupport.java:341)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@17.0.8/AbstractQueuedSynchronizer.java:506)
	at java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@17.0.8/ForkJoinPool.java:3465)
	at java.util.concurrent.ForkJoinPool.managedBlock(java.base@17.0.8/ForkJoinPool.java:3436)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@17.0.8/AbstractQueuedSynchronizer.java:1623)
	at java.util.concurrent.ArrayBlockingQueue.take(java.base@17.0.8/ArrayBlockingQueue.java:420)
	at PoolFull.handleRequest(PoolFull.java:25)

程序里根本没用 ForkJoinPool,这是 17.0.8 里 ConditionObject.await 自己的实现路径(8u381 同一处直接是 ConditionObject.await → LockSupport.park,没有这两帧)。在 JDK 17 的栈里看到 ForkJoinPool.managedBlock,别以为代码里用了 ForkJoinPool 或并行流,先看它下面挂的是什么。idle-1 在 17 上也一样带着这两帧。

6. 坑四:线程池自己等自己,jstack 不报死锁 ✅

这是最隐蔽的一种。线程池 2 个线程,提交 2 个父任务;每个父任务在池里再提交一个子任务,然后 get() 等它:

import java.util.concurrent.*;

public class Starve {
    public static void main(String[] args) throws Exception {
        ExecutorService pool = Executors.newFixedThreadPool(2, new ThreadFactory() {
            int n;
            public Thread newThread(Runnable r) { return new Thread(r, "order-pool-" + (++n)); }
        });
        for (int i = 0; i < 2; i++) {
            pool.submit(() -> {
                // 父任务占着池里的线程,又把子任务丢回同一个池,然后等它
                Future<String> child = pool.submit(() -> "price");
                return child.get();
            });
        }
    }
}

2 个线程都被父任务占着,子任务排在队列里永远轮不到;父任务永远等不到子任务。这是一个实打实的循环等待,进程永远卡住。

jstack -l 看两个线程(JDK 8u381,order-pool-2 相同):

"order-pool-1" #9 prio=5 os_prio=0 tid=0x00007f81b4190800 nid=0x964 waiting on condition [0x00007f819ce6a000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000ec7a0c58> (a java.util.concurrent.FutureTask)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429)
	at java.util.concurrent.FutureTask.get(FutureTask.java:191)
	at Starve.lambda$main$1(Starve.java:13)
	at Starve$$Lambda$1/1418481495.call(Unknown Source)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)

   Locked ownable synchronizers:
	- <0x00000000ec77bd50> (a java.util.concurrent.ThreadPoolExecutor$Worker)

而输出的末尾:

"VM Periodic Task Thread" os_prio=0 tid=0x00007f81b40d8000 nid=0x963 waiting on condition 

JNI global references: 310

没有 Found one Java-level deadlock。 JDK 17.0.8 也一样,末尾是 JNI global refs: 5, weak refs: 0,没有死锁报告。

为什么不报,从输出能看出端倪(以下是按输出的推断):前三个 demo 的死锁报告里,每一行都是「等某把锁 → which is held by 某个线程」;而这里线程等的是一个 FutureTask,它还排在队列里,没有哪个线程「持有」它,这条「被谁持有」的边接不上。

四种卡法的汇总:

demo	卡法	`BLOCKED` 数(8 / 17)	末尾报死锁(8 / 17)
DeadSync	synchronized 互锁	2 / 2	✅ / ✅
DeadLock	ReentrantLock 互锁	0 / 0	✅ / ✅
Mixed	两种锁混合互锁	1 / 1	✅ / ✅
Starve	线程池自己等自己	0 / 0	❌ / ❌

jstack 没报死锁,不代表没有卡死。 看到一个池的所有线程都停在 FutureTask.get,而且下面的业务代码是同一个池提交的任务,就是这一种。

改法:子任务丢到另一个池。把上面的 pool.submit(() -> "price") 换成一个独立的 pricePool.submit(...),两个版本都实跑通过:

order-pool terminated: true

(这行是程序在 shutdown() 后 awaitTermination(5, TimeUnit.SECONDS) 的返回值,true 表示父任务全部跑完、线程池正常关闭。)

7. 是「慢」还是「卡死」:隔几秒抓三次 ✅

一次 jstack 只是一张快照。一个线程停在 queryDb,可能是这条 SQL 永远不返回,也可能只是刚好在跑一条 200 毫秒的 SQL。隔 5 秒抓 3 次,还停在同一行,才说明它是真卡住。

for i in 1 2 3; do jstack <pid> > jstack_$i.txt; sleep 5; done

PoolFull 的 biz-1 连抓三次,JDK 8u381:

"biz-1" #11 prio=5 os_prio=0 tid=0x00007f8ed41a5000 nid=0x7dd waiting on condition [0x00007f8ebcc68000]
 java.lang.Thread.State: TIMED_WAITING (sleeping)

三次完全相同:同一个 nid、同一个状态,去掉首行后整段栈逐字节一致(三次 md5sum 相同),一直停在 queryDb。但 8 的输出里看不出过了多久。

JDK 17.0.8 多两个字段:

snap1: "biz-1" #15 prio=5 os_prio=0 cpu=0.17ms elapsed=3.13s  ...
snap2: "biz-1" #15 prio=5 os_prio=0 cpu=0.17ms elapsed=8.21s  ...
snap3: "biz-1" #15 prio=5 os_prio=0 cpu=0.17ms elapsed=13.31s ...

cpu=0.17ms 三次一动不动 —— 10 秒里这个线程一点 CPU 都没用,不是在「慢慢算」,是在干等
elapsed= 每次涨约 5 秒,正好是抓取间隔

elapsed= 是线程从启动到现在活了多久,不是「在当前这一行停了多久」。States 程序能看出来:lock-waiter 比 sleeper 晚 200 毫秒启动(中间有一个 Thread.sleep(200)),同一次 jstack 里 sleeper 是 elapsed=2.93s、lock-waiter 是 elapsed=2.73s,正好差 0.2 秒。线程池的线程是复用的,elapsed 很大只说明池建得早,说明不了请求卡了多久。 判断卡没卡住,还是靠「多次快照栈不变 + cpu= 不涨」。

8. 本篇速查

想知道	命令 / 看哪里	注意
有没有死锁	`jstack <pid>` 最末尾的 `Found one Java-level deadlock`	别 `grep BLOCKED`:ReentrantLock 死锁里是 `WAITING (parking)`
synchronized 在谁手里	普通 `jstack`,找 `- locked <地址>`	对上等待方 `- waiting to lock <同一地址>`
ReentrantLock 在谁手里	`jstack -l <pid>` 或 `jcmd <pid> Thread.print -l`,找 `Locked ownable synchronizers` 下的地址	不加 `-l` 看不到;`ThreadPoolExecutor$Worker` 是线程池自己的,跳过
线程池是闲还是卡	按「池名 + 状态 + 第一行业务代码」分组(上文脚本)	只按状态 `uniq -c` 分不开:闲着和卡住都是 `WAITING (parking)`
没报死锁但全池不动	看是不是所有线程都停在 `FutureTask.get`,等的又是同一个池的任务	线程池自己等自己,`jstack` 不报
慢还是卡死	隔 5 秒 `jstack` 抓 3 次	17 上看 `cpu=` 涨不涨;`elapsed=` 是线程寿命,不是卡了多久
JDK 17 栈里有 `ForkJoinPool.managedBlock`	往下看它挂在什么调用下面	17.0.8 里 `ConditionObject.await` 自己会走到这,不代表你用了 ForkJoinPool

(四)内存:OOM 了先干什么 —— 三种 OOM(堆、Metaspace、直接内存)分别构造出来,HeapDumpOnOutOfMemoryError、jmap -histo、JDK 17 上 jmap -heap 换成 jhsdb 的实测。

JVM 线上排查实战(三):grep BLOCKED 找不到的死锁,和 jstack 根本不报的死锁