Provided by: bpfcc-tools_0.8.0-4_all
stackcount - Count function calls and their stack traces. Uses Linux eBPF/bcc.
stackcount [-h] [-p PID] [-i INTERVAL] [-D DURATION] [-T] [-r] [-s] [-P] [-K] [-U] [-v] [-d] [-f] pattern
stackcount traces functions and frequency counts them with their entire stack trace, kernel stack and user stack, summarized in-kernel for efficiency. This allows higher frequency events to be studied. The output consists of unique stack traces, and their occurrence counts. In addition to kernel and user functions, kernel tracepoints and USDT tracepoint are also supported. The pattern is a string with optional '*' wildcards, similar to file globbing. If you'd prefer to use regular expressions, use the -r option. This tool only works on Linux 4.6+. Stack traces are obtained using the new `BPF_STACK_TRACE` APIs. For kernels older than 4.6, see the version under tools/old.
CONFIG_BPF and bcc.
-h Print usage message. -r Allow regular expressions for the search pattern. The default allows "*" wildcards only. -s Show address offsets. -T Include a timestamp with interval output. -v Show raw addresses. -d Print a delimiter ("--") in-between the kernel and user stacks. --debug Print the source of the BPF program when loading it (for debugging purposes). -i interval Summary interval, in seconds. -D duration Total duration of trace, in seconds. -f Folded output format. -p PID Trace this process ID only (filtered in-kernel). pattern A function name, or a search pattern. Can include wildcards ("*"). If the -r option is used, can include regular expressions.
Count kernel and user stack traces for submit_bio(): # stackcount submit_bio Count stacks with a delimiter for submit_bio(): # stackcount -d submit_bio Count kernel stack trace only for submit_bio(): # stackcount -K submit_bio Count user stack trace only for submit_bio(): # stackcount -U submit_bio Count stack traces for ip_output(): # stackcount ip_output Show symbol offsets: # stackcount -s ip_output Show offsets and raw addresses (verbose): # stackcount -sv ip_output Count stacks for kernel functions matching tcp_send*: # stackcount 'tcp_send*' Same as previous, but using regular expressions: # stackcount -r '^tcp_send.*' Output every 5 seconds, with timestamps: # stackcount -Ti 5 ip_output Only count stacks when PID 185 is on-CPU: # stackcount -p 185 ip_output Count user stacks for dynamic heap allocations with malloc in PID 185: # stackcount -p 185 c:malloc Count user stacks for thread creation (USDT tracepoint) in PID 185: # stackcount -p 185 u:pthread:pthread_create Count stacks for context switch events using a kernel tracepoint: # stackcount t:sched:sched_switch
This summarizes unique stack traces in-kernel for efficiency, allowing it to trace a higher rate of function calls than methods that post-process in user space. The stack trace data is only copied to user space when the output is printed, which usually only happens once. The stack walking also happens in an optimized code path in the kernel thanks to the new BPF_STACK_TRACE table APIs, which should be more efficient than the manual walker in the eBPF tracer which older versions of this script used. With this in mind, call rates of < 10,000/sec would incur negligible overhead. Test before production use. You can also use funccount to get a handle on function call rates first.
This is from bcc. https://github.com/iovisor/bcc Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.
Unstable - in development.
Brendan Gregg, Sasha Goldshtein