NAME

       offwaketime - Summarize blocked time by off-CPU stack + waker stack. Uses Linux eBPF/bcc.

SYNOPSIS

       offwaketime  [-h]  [-p  PID  |  -t  TID  |  -u  | -k] [-U | -K] [-f] [--stack-storage-size
       STACK_STORAGE_SIZE] [-m MIN_BLOCK_TIME] [-M MAX_BLOCK_TIME] [duration]

DESCRIPTION

       This program shows kernel stack traces and task names that  were  blocked  and  "off-CPU",
       along  with  the stack traces and task names for the threads that woke them, and the total
       elapsed time from when they blocked to  when  they  were  woken  up.   This  combines  the
       summaries  from  both  the  offcputime and wakeuptime tools.  The time measurement will be
       very similar to off-CPU time, however, off-CPU time may include a little extra time  spent
       waiting  on  a run queue to be scheduled.  The combined stacks, task names, and total time
       is summarized in kernel context for efficiency, using an eBPF map.

       The output summary will further help you identify reasons why threads were  blocking,  and
       quantify  the  time  from  when  they  were  blocked to woken up.  This spans all types of
       blocking activity:  disk  I/O,  network  I/O,  locks,  page  faults,  swapping,  sleeping,
       involuntary context switches, etc.

       This is complementary to CPU profiling (e.g., CPU flame graphs) which shows the time spent
       on-CPU. This shows the time spent blocked off-CPU,  and  the  output,  especially  the  -f
       format, can be used to generate an "off-wake time flame graph".

       See http://www.brendangregg.com/FlameGraphs/offcpuflamegraphs.html

REQUIREMENTS

       CONFIG_BPF and bcc.

OPTIONS

       -h     Print usage message.

       -f     Print output in folded stack format.

       -p PID Trace this process ID only (filtered in-kernel).

       -t TID Trace this thread ID only (filtered in-kernel).

       -u     Only trace user threads (no kernel threads).

       -k     Only trace kernel threads (no user threads).

       -U     Show stacks from user space only (no kernel space stacks).

       -K     Show stacks from kernel space only (no user space stacks).

       --stack-storage-size STACK_STORAGE_SIZE
              Change the number of unique stack traces that can be stored and displayed.

       duration
              Duration to trace, in seconds.

       -m MIN_BLOCK_TIME
              The amount of time in microseconds over which we store traces (default 1)

       -M MAX_BLOCK_TIME
              The amount of time in microseconds under which we store traces (default U64_MAX)

EXAMPLES

       Trace  all  thread  blocking  events, and summarize (in-kernel) by user and kernel off-CPU
       stack trace, waker stack traces, task names, and total blocked time:
              # offwaketime

       Trace for 5 seconds only:
              # offwaketime 5

       Trace for 5 seconds, and emit output in folded stack format (suitable for  flame  graphs),
       user-mode threads only:
              # offwaketime -fu 5

       Trace PID 185 only:
              # offwaketime -p 185

OVERHEAD

       This  summarizes unique stack trace pairs in-kernel for efficiency, allowing it to trace a
       higher rate of events than methods that post-process in user space.  The stack  trace  and
       time  data  is  only  copied  to  user space once, when the output is printed. While these
       techniques greatly lower overhead, scheduler events are still a high frequency  event,  as
       they  can  exceed  1  million events per second, and so caution should still be used. Test
       before production use.

       If the overhead is still a problem, take a look at the min block option.  If your  aim  is
       to  chase  down  longer  blocking  events,  then this could be increased to filter shorter
       blocking events, further lowering overhead.

SOURCE

       This is from bcc.

              https://github.com/iovisor/bcc

       Also look in the bcc distribution for a companion _examples.txt  file  containing  example
       usage, output, and commentary for this tool.

OS

       Linux

STABILITY

       Unstable - in development.

AUTHOR

       Brendan Gregg

SEE ALSO

       offcputime(8), wakeuptime(8)