Provided by: bpfcc-tools_0.8.0-4_all bug


       trace  - Trace a function and print its arguments or return value, optionally evaluating a
       filter. Uses Linux eBPF/bcc.


       trace [-h] [-b BUFFER_PAGES] [-p PID] [-L TID] [-v] [-Z STRING_SIZE] [-S]
                [-M MAX_EVENTS] [-t] [-T] [-C] [-K] [-U] [-a] [-I header]
                probe [probe ...]


       trace probes functions you specify and displays trace messages if a  particular  condition
       is  met.  You  can  control  the  message  format to display function arguments and return

       Since this uses BPF, only the root user can use this tool.


       CONFIG_BPF and bcc.


       -h     Print usage message.

       -p PID Trace only functions in the process PID.

       -L TID Trace only functions in the thread TID.

       -v     Display the generated BPF program, for debugging purposes.

       -z STRING_SIZE
              When collecting string  arguments  (of  type  char*),  collect  up  to  STRING_SIZE
              characters. Longer strings will be truncated.

       -S     If  set,  trace messages from trace's own process. By default, this is off to avoid
              tracing storms -- for example, if you trace the write  system  call,  and  consider
              that trace is writing to the standard output.

       -M MAX_EVENTS
              Print up to MAX_EVENTS trace messages and then exit.

       -t     Print times relative to the beginning of the trace (offsets), in seconds.

       -T     Print the time column.

       -C     Print CPU id.

       -B     Treat argument of STRCMP helper as a binary value

       -K     Print the kernel stack for each event.

       -U     Print  the  user stack for each event.  -a Print virtual address in kernel and user

       -I header
              Additional header files to include in the BPF  program.  This  is  needed  if  your
              filter  or print expressions use types or data structures that are not available in
              the standard headers. For example: 'linux/mm.h'

       probe [probe ...]
              One or  more  probes  that  attach  to  functions,  filter  conditions,  and  print
              information. See PROBE SYNTAX below.


       The general probe syntax is as follows:

       [{p,r}]:[library]:function[(signature)] [(predicate)] ["format string"[, arguments]]

       {t:category:event,u:library:probe} [(predicate)] ["format string"[, arguments]]

              Probe  type  -  "p"  for  function  entry,  "r" for function return, "t" for kernel
              tracepoint, "u" for USDT probe. The default probe type is "p".

              Library containing the probe.  Specify the full path to the .so or executable  file
              where  the  function  to probe resides. Alternatively, you can specify just the lib
              name: for example, "c" refers to libc. If no library name is specified, the  kernel
              is assumed. Also, you can specify an executable name (without a full path) if it is
              in the PATH.  For example, "bash".

              The tracepoint category. For example, "sched" or "irq".

              The function to probe.

              The optional signature of the function to probe. This can make it easier to  access
              the  function's  arguments,  instead  of  using  the  "arg1", "arg2" etc.  argument
              specifiers. For example, "(struct timespec *ts)" in the signature position lets you
              use "ts" in the filter or print expressions.

       event  The tracepoint event. For example, "block_rq_complete".

       probe  The USDT probe name. For example, "pthread_create".

              The  filter applied to the captured data. Only if the filter evaluates as true, the
              trace message will be printed. The filter can  use  any  valid  C  expression  that
              refers to the argument values: arg1, arg2, etc., or to the return value retval in a
              return probe. If necessary, use C cast operators to coerce  the  arguments  to  the
              desired  type. For example, if arg1 is of type int, use the expression ((int)arg1 <
              0) to trace only invocations where arg1 is negative.  Note that only arg1-arg6  are
              supported,  and  only if the function is using the standard x86_64 convention where
              the first six arguments are in the RDI, RSI, RDX, RCX,  R8,  R9  registers.  If  no
              predicate is specified, all function invocations are traced.

              The  predicate  expression  may  also  use  the STRCMP pseudo-function to compare a
              predefined string to a string argument. For  example:  STRCMP("test",  arg1).   The
              order  of  arguments  is  important:  the  first  argument MUST be a quoted literal
              string, and the second  argument  can  be  a  runtime  string,  most  typically  an

       ["format string"[, arguments]]
              A  printf-style  format string that will be used for the trace message. You can use
              the following format specifiers: %s, %d, %u, %lld, %llu, %hd, %hu, %c, %x, %llx  --
              with  the  same  semantics  as  printf's.  Make  sure  to  pass the exact number of
              arguments as there are placeholders in the  format  string.  The  format  specifier
              replacements  may  be any C expressions, and may refer to the same special keywords
              as in the predicate (arg1, arg2, etc.).

              In addition to the above format specifiers, you can also use %K  and  %U  when  the
              expression  is  an  address  that  potentially  points  to executable code (i.e., a
              symbol). trace will resolve %K specifiers to a kernel symbol,  such  as  vfs__read,
              and  will  resolve  %U  specifiers  to a user-space symbol in that process, such as

              In tracepoints, both the predicate and the arguments may refer  to  the  tracepoint
              format  structure, which is stored in the special "args" variable. For example, the
              block:block_rq_complete tracepoint can  print  or  filter  by  args->nr_sector.  To
              discover the format of your tracepoint, use the tplist tool.

              In  USDT  probes, the arg1, ..., argN variables refer to the probe's arguments.  To
              determine which arguments your probe has, use the tplist tool.

              The predicate expression and the format specifier  replacements  for  printing  may
              also  use  the  following  special  keywords:  $pid,  $tgid to refer to the current
              process' pid and tgid; $uid, $gid to refer to the current user's uid and gid;  $cpu
              to refer to the current processor number.


       Trace all invocations of the open system call with the name of the file being opened:
              # trace '::do_sys_open "%s", arg2'

       Trace  all  invocations  of  the  read  system call where the number of bytes requested is
       greater than 20,000:
              # trace '::sys_read (arg3 > 20000) "read %d bytes", arg3'

       Trace all malloc calls and print the size of the requested allocation:
              # trace ':c:malloc "size = %d", arg1'

       Trace returns from the readline function in bash and print the return value as a string:
              # trace 'r:bash:readline "%s", retval'

       Trace the block:block_rq_complete tracepoint and print the number of sectors completed:
              # trace 't:block:block_rq_complete "%d sectors", args->nr_sector'

       Trace the pthread_create USDT probe from the pthread library and print the address of  the
       thread's start function:
              # trace 'u:pthread:pthread_create "start addr = %llx", arg3'

       Trace the nanosleep system call and print the sleep duration in nanoseconds:
              # trace 'p::SyS_nanosleep(struct timespec *ts) sleep for %lld ns , ts->tv_nsec'


       This is from bcc.


       Also  look  in  the bcc distribution for a companion _examples.txt file containing example
       usage, output, and commentary for this tool.




       Unstable - in development.


       Sasha Goldshtein