09-Java工程师的Python第九课-并发编程

线程、进程、异步：Java并发 vs Python并发模型

摘要：Java和Python的并发模型差异巨大。Java有强大的Thread/ExecutorService，Python有GIL的限制和asyncio。

---

写在前面

对Java工程师来说，并发编程是核心技能之一。Thread、ExecutorService、synchronized、volatile、AtomicInteger……这些概念你已经烂熟于心。

但尝试在Python中使用多线程时，会发现一个令人困惑的问题：为什么Python多线程跑不满CPU？

这就是GIL（Global Interpreter Lock）在作怪。

---

一、并发模型对比总览

维度	Java	Python
多线程	原生支持	受GIL限制
多进程	ProcessBuilder	multiprocessing
异步	CompletableFuture	asyncio
并发安全	synchronized/volatile	threading（受GIL限制）
并行计算	ForkJoinPool	multiprocessing/ray
设计哲学	线程优先	进程+异步优先

---

二、GIL：Python的阿喀琉斯之踵

2.1 什么是GIL

GIL（Global Interpreter Lock）是Python解释器（CPython）的一个特性，它确保同一时刻只有一个线程执行Python字节码。

# Python多线程受GIL限制
import threading
import time

def cpu_bound_task(n):
    """消耗CPU的任务"""
    result = 0
    for i in range(n):
        result += i ** 2
    return result

# 测试：2个线程 vs 1个线程
start = time.time()
t1 = threading.Thread(target=cpu_bound_task, args=(10_000_000,))
t2 = threading.Thread(target=cpu_bound_task, args=(10_000_000,))
t1.start(); t2.start()
t1.join(); t2.join()
print(f"2线程耗时: {time.time() - start:.2f}s")

# 结果：耗时和单线程几乎一样！

// Java多线程不受GIL限制
public class CpuBoundTask implements Runnable {
    @Override
    public void run() {
        long result = 0;
        for (int i = 0; i < 10_000_000; i++) {
            result += i * i;
        }
    }

    public static void main(String[] args) throws InterruptedException {
        long start = System.currentTimeMillis();
        Thread t1 = new Thread(new CpuBoundTask());
        Thread t2 = new Thread(new CpuBoundTask());
        t1.start(); t2.start();
        t1.join(); t2.join();
        System.out.println("耗时: " + (System.currentTimeMillis() - start) + "ms");
        // 结果：约为单线程的一半（双核CPU）
    }
}

2.2 GIL的影响

场景	Java多线程	Python多线程
CPU密集型	真正并行	受GIL限制，无法并行
IO密集型	并行	可用（IO时释放GIL）
混合型	复杂，需调优	建议用asyncio

2.3 绕过GIL的方法

# 方法1：multiprocessing（推荐CPU密集型）
from multiprocessing import Pool

def cpu_bound_task(n):
    result = 0
    for i in range(n):
        result += i ** 2
    return result

if __name__ == "__main__":
    with Pool(4) as p:  # 4个进程
        results = p.map(cpu_bound_task, [10_000_000] * 8)
    # 真正并行！

# 方法2：C扩展（不推荐）
# 使用Cython、Numba等编译为C代码

# 方法3：subprocess（轻量级）
import subprocess
subprocess.run(["python", "worker.py"])

---

三、线程对比

3.1 基本使用

// Java - Thread
Thread t = new Thread(() -> {
    System.out.println("Running in thread");
});
t.start();
t.join();

// Java - Runnable
Runnable task = () -> {
    System.out.println("Task executing");
};
new Thread(task).start();

// Java - Callable + Future
ExecutorService executor = Executors.newFixedThreadPool(4);
Future<String> future = executor.submit(() -> {
    Thread.sleep(1000);
    return "Result";
});
String result = future.get();  // 阻塞等待

# Python - threading（受GIL限制）
import threading

def task():
    print("Task executing")

t = threading.Thread(target=task)
t.start()
t.join()

# Python - daemon线程
t = threading.Thread(target=background_task, daemon=True)
t.start()

# Python - Timer
from threading import Timer
def delayed_task():
    print("Delayed")

t = Timer(5.0, delayed_task)  # 5秒后执行
t.start()

3.2 线程同步对比

// Java - synchronized
public class Counter {
    private int count = 0;

    public synchronized void increment() {
        count++;
    }

    public synchronized int getCount() {
        return count;
    }
}

// Java - ReentrantLock
private final ReentrantLock lock = new ReentrantLock();

public void increment() {
    lock.lock();
    try {
        count++;
    } finally {
        lock.unlock();
    }
}

# Python - threading的Lock
import threading

counter = 0
lock = threading.Lock()

def increment():
    global counter
    with lock:
        counter += 1

# Python - RLock（可重入锁）
rlock = threading.RLock()

# Python - Semaphore
semaphore = threading.Semaphore(3)  # 最多3个并发

def limited_access():
    with semaphore:
        # 同时最多3个线程执行这里
        pass

3.3 线程安全数据结构

// Java - ConcurrentHashMap
ConcurrentHashMap<String, Integer> map = new ConcurrentHashMap<>();
map.put("a", 1);
map.computeIfAbsent("b", k -> 2);
map.getOrDefault("c", 0);

// Java - AtomicInteger
AtomicInteger count = new AtomicInteger(0);
count.incrementAndGet();
count.addAndGet(5);

# Python - queue是线程安全的
from queue import Queue
q = Queue()
q.put(item)
item = q.get()

# Python - threading local数据
local_data = threading.local()

def process_request(request_id):
    local_data.request_id = request_id
    # 每个线程有自己的request_id

# Python - list/dict不是线程安全的！
# 多个线程同时修改list会有问题

---

四、进程对比

4.1 基本使用

// Java - ProcessBuilder
ProcessBuilder pb = new ProcessBuilder("python", "script.py");
pb.directory(new File("/path/to/dir"));
Process process = pb.start();
int exitCode = process.waitFor();

// Java - ProcessHandle
ProcessHandle.current()
    .children()
    .forEach(handle -> System.out.println(handle.pid()));

# Python - multiprocessing（推荐）
from multiprocessing import Process, Pool

def worker(name):
    print(f"Worker {name}")

if __name__ == "__main__":
    p = Process(target=worker, args=("Alice",))
    p.start()
    p.join()

    # Pool（推荐）
    with Pool(4) as pool:
        results = pool.map(cpu_bound_task, [10_000_000] * 8)

4.2 进程间通信

// Java - Pipe
PipedOutputStream out = new PipedOutputStream();
PipedInputStream in = new PipedInputStream(out);

// Java - Socket
ServerSocket server = new ServerSocket(8080);
Socket client = server.accept();

// Java - File/Memory-Mapped File

# Python - multiprocessing.Queue
from multiprocessing import Process, Queue

def producer(q):
    q.put("message")

def consumer(q):
    msg = q.get()

if __name__ == "__main__":
    q = Queue()
    p1 = Process(target=producer, args=(q,))
    p2 = Process(target=consumer, args=(q,))
    p1.start(); p2.start()
    p1.join(); p2.join()

# Python - Pipe
from multiprocessing import Pipe
parent_conn, child_conn = Pipe()

# Python - Manager（共享状态）
from multiprocessing import Manager
manager = Manager()
shared_dict = manager.dict()
shared_list = manager.list()

4.3 进程池对比

// Java - ExecutorService
ExecutorService executor = Executors.newFixedThreadPool(4);

List<Future<String>> futures = executor.invokeAll(Arrays.asList(
    () -> callService("A"),
    () -> callService("B"),
    () -> callService("C")
));

executor.shutdown();

// Java - ForkJoinPool（适合分治任务）
ForkJoinPool fjPool = new ForkJoinPool();
Integer result = fjPool.invoke(new SumTask(0, 1000000));

# Python - Pool
from multiprocessing import Pool

def cpu_bound_task(n):
    return sum(i**2 for i in range(n))

with Pool(4) as pool:
    # map - 顺序保持
    results = pool.map(cpu_bound_task, [10**6] * 10)

    # apply_async - 异步
    async_result = pool.apply_async(cpu_bound_task, (10**6,))
    result = async_result.get()

    # starmap - 多参数
    results = pool.starmap(add, [(1, 2), (3, 4), (5, 6)])

    # 回调
    def callback(result):
        print(f"Done: {result}")
    pool.apply_async(cpu_bound_task, (10**6,), callback=callback)

---

五、异步编程对比

5.1 Java CompletableFuture

// Java - CompletableFuture
CompletableFuture<String> future = CompletableFuture
    .supplyAsync(() -> callApi("user/1"))  // 异步执行
    .thenApply(result -> parseJson(result))  // 转换
    .thenCompose(user -> getUserOrders(user.getId()))  // 扁平化
    .exceptionally(ex -> {
        log.error("Error", ex);
        return "default";
    });

String result = future.join();  // 阻塞等待

// 组合多个Future
CompletableFuture.allOf(future1, future2, future3).join();

// 任意一个完成
CompletableFuture.anyOf(future1, future2).join();

5.2 Python asyncio

# Python - asyncio（核心）
import asyncio

async def fetch_user(user_id):
    await asyncio.sleep(1)  # 模拟IO
    return {"id": user_id, "name": "Alice"}

async def main():
    # 创建任务
    task1 = asyncio.create_task(fetch_user(1))
    task2 = asyncio.create_task(fetch_user(2))

    # 等待所有完成
    results = await asyncio.gather(task1, task2)
    print(results)

    # 等待任意一个
    done, pending = await asyncio.wait([task1, task2])
    for task in done:
        print(task.result())

asyncio.run(main())

# 异步上下文
async with asyncio.Lock():
    # 临界区
    pass

5.3 async/await对比

// Java - 虚拟线程（Java 21+）
try (var executor = Executors.newVirtualThreadPerTaskExecutor()) {
    Future<String> future = executor.submit(() -> {
        return callApi("user/1");
    });
    String result = future.get();
}

# Python - async/await
async def get_user(user_id):
    async with aiohttp.ClientSession() as session:
        async with session.get(f"/user/{user_id}") as response:
            return await response.json()

# Python - aiohttp（异步HTTP）
import aiohttp

async def fetch_all_users(user_ids):
    async with aiohttp.ClientSession() as session:
        tasks = [get_user(session, uid) for uid in user_ids]
        return await asyncio.gather(*tasks)

5.4 异步对比表

操作	Java	Python
创建异步任务	CompletableFuture.supplyAsync()	asyncio.create_task()
等待多个	allOf() / join()	asyncio.gather()
链式调用	thenApply()	await
超时	orTimeout()	asyncio.wait_for()
取消	future.cancel()	task.cancel()

---

六、并发工具对比

6.1 CountDownLatch

// Java
CountDownLatch latch = new CountDownLatch(3);

executor.submit(() -> {
    processA();
    latch.countDown();
});
executor.submit(() -> {
    processB();
    latch.countDown();
});
executor.submit(() -> {
    processC();
    latch.countDown();
});

latch.await();  // 等待所有任务完成
System.out.println("All done");

# Python - threading版本
from threading import Thread, CountDownLatch

latch = CountDownLatch(3)

def process_a():
    print("A done")
    latch.count_down()

# 不推荐用threading，推荐用asyncio

6.2 CyclicBarrier

// Java
CyclicBarrier barrier = new CyclicBarrier(3);

executor.submit(() -> {
    // 等待其他两个线程都到达屏障
    barrier.await();
    System.out.println("All at barrier");
});

# Python - 没有直接等价
# 用threading.Barrier
from threading import Barrier

barrier = Barrier(3)

def process():
    barrier.wait()
    print("All at barrier")

---

七、实战：并发下载文件

7.1 Java实现

// Java - 使用WebClient（WebFlux）或RestTemplate + ExecutorService
ExecutorService executor = Executors.newFixedThreadPool(10);
List<Future<Path>> futures = urls.stream()
    .map(url -> executor.submit(() -> downloadFile(url)))
    .collect(Collectors.toList());

List<Path> paths = futures.stream()
    .map(Future::get)
    .collect(Collectors.toList());

executor.shutdown();

7.2 Python实现

# Python - aiohttp + asyncio（推荐IO密集型）
import aiohttp
import asyncio
import aiofiles

async def download_file(session, url, path):
    async with session.get(url) as response:
        content = await response.read()
        async with aiofiles.open(path, "wb") as f:
            await f.write(content)

async def download_all(urls):
    async with aiohttp.ClientSession() as session:
        tasks = [
            download_file(session, url, f"files/{i}")
            for i, url in enumerate(urls)
        ]
        await asyncio.gather(*tasks)

# Python - multiprocessing（CPU密集型）
from multiprocessing import Pool

def download_file(url):
    # CPU密集型处理
    return process(download(url))

with Pool(10) as pool:
    results = pool.map(download_file, urls)

---

八、性能对比与选择建议

8.1 场景选择

场景	Java	Python
CPU密集型	ThreadPoolExecutor	multiprocessing
IO密集型	CompletableFuture	asyncio + aiohttp
混合型	ForkJoinPool	multiprocessing + asyncio
并行计算	ForkJoinPool	ray/numba/dask
简单并发	Thread	asyncio

8.2 GIL的正确理解

GIL不是Python的bug，而是设计选择：

• GIL简化了Python的内存管理（引用计数）
• 对于IO密集型任务，GIL影响不大
• 多进程可以绕过GIL，但增加了进程间通信开销

8.3 现代Python并发推荐

# 推荐的并发方案

# 1. asyncio（IO密集型，Web开发）
# Flask + asyncio / FastAPI / aiohttp

# 2. multiprocessing（CPU密集型，数据处理）
# from concurrent.futures import ProcessPoolExecutor
# with ProcessPoolExecutor() as executor:

# 3. concurrent.futures（统一接口）
from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor

# IO密集型用线程
with ThreadPoolExecutor() as executor:
    futures = [executor.submit(download, url) for url in urls]

# CPU密集型用进程
with ProcessPoolExecutor() as executor:
    futures = [executor.submit(process, data) for data in datasets]

---

九、总结

维度	Java	Python
CPU密集型并发	ThreadPoolExecutor	multiprocessing
IO密集型并发	CompletableFuture	asyncio
全局解释器锁	无	有（GIL）
线程安全集合	ConcurrentHashMap等	queue.Queue（推荐）
进程创建	Runtime.exec/ProcessBuilder	multiprocessing
异步生态	Spring WebFlux	asyncio/aiohttp

Python的并发模型比Java更复杂，因为GIL的存在。但通过asyncio（IO密集型）和multiprocessing（CPU密集型）的组合，Python也能实现高效的并发。关键是理解任务特性，选择正确的并发方案。