NAME

       cpuunclaimed  -  Sample  CPU  run  queues  and  calculate  unclaimed  idle CPU. Uses Linux
       eBPF/bcc.

SYNOPSIS

       cpuunclaimed [-T] [-j] [-J] [interval [count]]

DESCRIPTION

       This tool samples the length of the run queues and determine when there are idle CPUs, yet
       queued threads waiting their turn. It reports the amount of idle (yet unclaimed by waiting
       threads) CPU as a system-wide percentage.

       This situation can happen for a number of reasons:

       -      An application has been bound to some, but not all, CPUs, and has runnable  threads
              that cannot migrate to other CPUs due to this configuration.

       -      CPU  affinity: an optimization that leaves threads on CPUs where the CPU caches are
              warm, even if this means short periods of waiting while other CPUs  are  idle.  The
              wait period is tunale (see sysctl, kernel.sched*).

       -      Scheduler bugs.

       An  unclaimed  idle  of  <  1%  is  likely to be CPU affinity, and not usually a cause for
       concern. By leaving the CPU idle, overall throughput of the system may be  improved.  This
       tool  is  best  for identifying larger issues, > 2%, due to the coarseness of its 99 Hertz
       samples.

       This is an experimental tool that currently works by use of  sampling  to  keep  overheads
       low. Tool assumptions:

       -      CPU samples consistently fire around the same offset. There will sometimes be a lag
              as a sample is delayed  by  higher-priority  interrupts,  but  it  is  assumed  the
              subsequent  samples will catch up to the expected offsets (as is seen in practice).
              You can use -J to inspect sample offsets. Some systems can  power  down  CPUs  when
              idle,  and  when  they wake up again they may begin firing at a skewed offset: this
              tool will detect the skew, print an error, and exit.

       -      All CPUs are online (see ncpu).

       If this identifies unclaimed CPU, you can double check it by dumping raw samples (-j),  as
       well  as  using  other  tracing tools to instrument scheduler events (although this latter
       approach has much higher overhead).

       Since this uses BPF, only the root user can use this tool.

REQUIREMENTS

       CONFIG_BPF and bcc.

EXAMPLES

       Sample and calculate unclaimed idle CPUs, output every 1 second (default:
              # cpuunclaimed

       Print 5 second summaries, 10 times:
              # cpuunclaimed 5 10

       Print 1 second summaries with timestamps:
              # cpuunclaimed -T 1

       Raw dump of all samples (verbose), as comma-separated values:
              # cpuunclaimed -j

FIELDS

       %CPU   CPU utilization as a system-wide percentage.

       unclaimed idle
              Percentage of CPU resources that were idle when work was queued on other CPUs, as a
              system-wide percentage.

       TIME   Time (HH:MM:SS)

       TIMESTAMP_ns
              Timestamp, nanoseconds.

       CPU#   CPU ID.

       OFFSET_ns_CPU#
              Time offset that a sample fired within a sample group for this CPU.

OVERHEAD

       The  overhead  is expected to be low/negligible as this tool uses sampling at 99 Hertz (on
       all CPUs), which has a fixed and low cost, rather than sampling every scheduler  event  as
       many other approaches use (which can involve instrumenting millions of events per second).
       Sampled CPUs, run queue lengths, and timestamps are  written  to  ring  buffers  that  are
       periodically read by user space for reporting. Measure overhead in a test environment.

SOURCE

       This is from bcc.

              https://github.com/iovisor/bcc

       Also  look  in  the bcc distribution for a companion _examples.txt file containing example
       usage, output, and commentary for this tool.

OS

       Linux

STABILITY

       Unstable - in development.

AUTHOR

       Brendan Gregg

SEE ALSO

       runqlen(8)