Analyzing Linux Slab Memory Usage
Analyzing Linux Slab Memory Usage
In Linux memory management, Slab is an important mechanism used by the kernel for efficient allocation of small memory blocks. By analyzing Slab memory usage, you can effectively detect kernel memory leaks and excessive cache usage issues.
Common Slab Memory Issue Troubleshooting
Abnormal Kernel Cache Growth, Insufficient System Memory
The symptoms may appear as follows:
# free -h shows very little available memory
total used free shared buff/cache available
Mem: 15Gi 2.1Gi 256Mi 128Mi 13Gi 12Gi
# But memory recovery is limited after drop cache
echo 3 > /proc/sys/vm/drop_cachesWhy does free show little available memory but can't be dropped??? Because used memory includes Slab memory calculations. If Slab occupies too much space, it will appear as very little free available value!!
Using Tools for Detection
Check kernel cache statistics through /proc/slabinfo:
# slabinfo - version: 2.1
# name <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <limit> <batchcount> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail>
dentry 350000 351000 192 42 2 : tunables 0 0 0 : slabdata 8357 8357 0
inode_cache 180000 180900 608 13 2 : tunables 0 0 0 : slabdata 13915 13915 0
kmalloc-1024 45000 45000 1024 16 4 : tunables 0 0 0 : slabdata 2812 2812 0You can see abnormal growth in certain cache items:
- dentry cache: 350,000 objects, occupying about 67MB
- inode_cache: 180,000 objects, occupying about 109MB
- kmalloc-1024: 45,000 objects, occupying about 45MB
These caches cannot be released through drop_caches, indicating kernel object leaks.
But there's a troublesome aspect:
- /proc/slabinfo output is extensive and difficult to review!!
- /proc/slabinfo typically shows gradual data changes when problems occur, such as slow growth that's hard for human eyes to detect!!
Therefore, Ctbots.com has created a simple version to solve this problem.
https://ctbots.com/en/linux/performance/slabinfo.html
The tool is very simple to use, mainly divided into three steps:
Follow instructions to generate shell commands for execution, which will record log information.
Download the specified output log folder content to local storage.
Upload the log folder content to Ctbots.com for analysis, directly view online tables, compare and search for differences.
Directly use the analysis function to simply analyze whether leaks exist.
Slab Memory Issue Analysis
Why does this situation occur?
- Applications frequently create files but don't close them properly, leading to dentry/inode cache accumulation
- Kernel modules have memory leaks, continuously allocating kmalloc objects without releasing them
- File system anomalies preventing normal cache object recovery
Kernel Slab memory management contains multiple cache pools:
- dentry cache: Directory entry cache for accelerating file path resolution
- inode cache: Index node cache storing file system metadata
- kmalloc cache: General memory allocator classified by size
- Specialized caches: Such as skbuff_head_cache network cache, etc.
What to Do When Slab Issues Are Found?
Based on ctbots tool analysis recommendations:
Excessive dentry/inode cache
# Check number of open files
lsof | wc -l
# View process file handle usage
lsof -p <pid> | wc -l
# Adjust kernel parameters to limit cache size
echo 100000 > /proc/sys/fs/dentry-statekmalloc leak investigation
# Check kernel module memory usage
cat /proc/modules | grep -v '0$'
# View kernel memory allocation statistics
cat /proc/meminfo | grep -i slab
# Use kmemleak to detect kernel memory leaks (requires kernel support)
echo scan > /sys/kernel/debug/kmemleak
cat /sys/kernel/debug/kmemleakNetwork cache anomalies
# Check network connection status
ss -tuln | wc -l
# View network cache statistics
cat /proc/net/sockstatIf you suspect network connection leaks, you can use Ctbots.com to detect TCP or UDP connection leaks.
https://ctbots.com/en/linux/findBug/performanceIndex.html
In the menu, you can see TCP and UDP connection leak troubleshooting tools.