Kudu TCMalloc Memory 泄漏

112 阅读1分钟

TCMalloc是谷歌开发的内存分配器。Kudu版本为 1.10.0,有 10GB 的 block cache,一直存在 top 命令看到的 kudu-tserver 进程占用内存虚高的情况。在 kudu-tserver:8050/memz 中查看内存使用详情:

------------------------------------------------
MALLOC:    13606721656 (12976.4 MiB) Bytes in use by application
MALLOC: +   4908195840 ( 4680.8 MiB) Bytes in page heap freelist
MALLOC: +    444714736 (  424.1 MiB) Bytes in central cache freelist
MALLOC: +       286720 (    0.3 MiB) Bytes in transfer cache freelist
MALLOC: +     28481688 (   27.2 MiB) Bytes in thread cache freelists
MALLOC: +     52559872 (   50.1 MiB) Bytes in malloc metadata
MALLOC:   ------------
MALLOC: =  19040960512 (18158.9 MiB) Actual memory used (physical + swap)
MALLOC: +    919805952 (  877.2 MiB) Bytes released to OS (aka unmapped)
MALLOC:   ------------
MALLOC: =  19960766464 (19036.1 MiB) Virtual address space used
MALLOC:
MALLOC:         308671              Spans in use
MALLOC:            178              Thread heaps in use
MALLOC:           8192              Tcmalloc page size
------------------------------------------------
Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()).
Bytes released to the OS take up virtual address space but no physical memory.

--tcmalloc_max_free_bytes_percentage默认为10,12976 * 10% = 1297MB,小于4680MB,正常情况下应该是会调用GcTcmalloc(),释放 4680MB - 1297MB 给操作系统。但是 scan / compactions / wal 不使用memtrackers,也就无法触发MemTracker::Release,从而无法调用GcTcmalloc()。如果 scan 比较多可能会出现这种情况,1.11 通过增加了一个后台定时GC的线程解决了这个问题。

参考:issues.apache.org/jira/browse…