NAME

       runqlat - Run queue (scheduler) latency as a histogram.

SYNOPSIS

       runqlat [-h] [-T] [-m] [-P] [--pidnss] [-L] [-p PID] [interval] [count]

DESCRIPTION

       This  measures the time a task spends waiting on a run queue (or equivalent scheduler data
       structure) for a turn on-CPU, and shows this time as a  histogram.  This  time  should  be
       small,  but a task may need to wait its turn due to CPU load. The higher the CPU load, the
       longer a task will generally need to wait its turn.

       This tool measures two types of run queue latency:

       1. The time from a task being enqueued on a run queue to its context switch and execution.
       This  traces  enqueue_task_*()  ->  finish_task_switch(),  and  instruments  the run queue
       latency after a voluntary context switch.

       2. The time from when a task was involuntary context switched and still  in  the  runnable
       state, to when it next executed. This is instrumented from finish_task_switch() alone.

       This  tool  uses  in-kernel  eBPF  maps  for  storing  timestamps  and  the histogram, for
       efficiency. Despite this, the overhead of  this  tool  may  become  significant  for  some
       workloads: see the OVERHEAD section.

       This  works  by tracing various kernel scheduler functions using dynamic tracing, and will
       need updating to match any changes to these functions.

       Since this uses BPF, only the root user can use this tool.

REQUIREMENTS

       CONFIG_BPF and bcc.

OPTIONS

       -h     Print usage message.

       -T     Include timestamps on output.

       -m     Output histogram in milliseconds.

       -P     Print a histogram for each PID.

       --pidnss
              Print a histogram for each PID namespace (short for PID namespaces). For  container
              analysis.

       -L     Print a histogram for each thread ID.

       -p PID Only show this PID (filtered in kernel for efficiency).

       interval
              Output interval, in seconds.

       count  Number of outputs.

EXAMPLES

       Summarize run queue latency as a histogram:
              # runqlat

       Print 1 second summaries, 10 times:
              # runqlat 1 10

       Print  1  second  summaries,  using  milliseconds  as units for the histogram, and include
       timestamps on output:
              # runqlat -mT 1

       Trace PID 186 only, 1 second summaries:
              # runqlat -P 185 1

FIELDS

       usecs  Microsecond range

       msecs  Millisecond range

       count  How many times a task event fell into this range

       distribution
              An ASCII bar chart to visualize the distribution (count column)

OVERHEAD

       This traces scheduler functions, which can become very frequent. While eBPF has  very  low
       overhead,  and  this  tool  uses in-kernel maps for efficiency, the frequency of scheduler
       events for some workloads may be high enough  that  the  overhead  of  this  tool  becomes
       significant. Measure in a lab environment to quantify the overhead before use.

SOURCE

       This is from bcc.

              https://github.com/iovisor/bcc

       Also  look  in  the bcc distribution for a companion _examples.txt file containing example
       usage, output, and commentary for this tool.

OS

       Linux

STABILITY

       Unstable - in development.

AUTHOR

       Brendan Gregg

SEE ALSO

       runqlen(8), pidstat(1)