Provided by: uftrace_0.9.0-1_amd64 bug


       uftrace-record - Run a command and record its trace data


       uftrace record [options] COMMAND [command-options]


       This command runs COMMAND and gathers function trace data from it, and saves it into files
       under the uftrace data directory - without displaying anything.

       This data can then be inspected later on, using uftrace replay or uftrace report.


       -b SIZE, --buffer=SIZE
              Size of internal buffer in which trace data will be saved.  Default size is 128k.

       -F FUNC, --filter=FUNC
              Set filter to trace selected functions only.  This option can  be  used  more  than
              once.  See FILTERS.

       -N FUNC, --notrace=FUNC
              Set  filter  not  to  trace  selected functions (or the functions called underneath
              them).  This option can be used more than once.  See FILTERS.

       -T TRG, --trigger=TRG
              Set trigger on selected functions.  This option can be used more  than  once.   See

       -t TIME, --time-filter=TIME
              Do  not  show  functions  which  run  under  the time threshold.  If some functions
              explicitly have the `trace' trigger applied, those are always traced regardless  of
              execution time.

              Allow  running  uftrace  even if some problems occur.  When uftrace record finds no
              mcount symbol (which is generated by compiler) in the executable, it quits with  an
              error  message  since  uftrace  can not trace the program.  However, it is possible
              that the user is only interested in functions within a dynamically-linked  library,
              in  which  case  this  option  can  be  used  to  cause  uftrace to run the program
              regardless.  Also, the -A/--argument and -R/--retval options work only for binaries
              built  with  -pg, so uftrace will normally exit when it tries to run binaries built
              without that option.  This option ignores the warning and goes on  tracing  without
              the argument and/or return value.

       -L PATH, --library-path=PATH
              Load necessary internal libraries from this path.  This is for testing purposes.

              Do  not  record library function invocations.  Library calls are normally traced by
              hooking the dynamic linker's resolve function in the PLT.  One can disable it  with
              this option.

              Do  not  bind dynamic symbol address.  This option uses the LD_BIND_NOT environment
              variable to trace library function calls which might be missing due  to  concurrent
              (first)  accesses.   It  is not meaningful to use this option with the --no-libcall

              Trace function calls between libraries.  By default, uftrace  only  record  library
              call from the main executable.  Implies --force.

       -D DEPTH, --depth=DEPTH
              Set global trace limit in nesting level.

              Set the max function stack depth for tracing.  Default is 1024.

       --nop  Do  not  record any functions.  This is a no-op and only meaningful for performance

       --time Print running time of children in time(1)-style.

       -k, --kernel
              Trace kernel functions as well as user functions.  Only kernel entry/exit functions
              will be traced by default.  Use the --kernel-depth option to override this.

       -H HOST, --host=HOST
              Send  trace  data  to  given  host  via  the  network,  not  writing to files.  The
              uftrace recv command should be run on the destination host to receive the data.

              When sending data to the network (with -H), use  the  given  port  instead  of  the
              default (8090).

              Start  uftrace  with  tracing  disabled.   This is only meaningful when used with a
              trace_on trigger.

       -A SPEC, --argument=SPEC
              Record function arguments.  This option can be used more than once.  See ARGUMENTS.

       -R SPEC, --retval=SPEC
              Record function return values.  This option  can  be  used  more  than  once.   See

       -a, --auto-args
              Automatically  record  arguments  and  return values of known functions.  These are
              usually functions in standard (C language or system) libraries but if debug info is
              available it includes functions in the user program.

              Use NUM threads to record trace data.  Default is 1/4 of online CPUs (but when full
              kernel tracing is enabled, it will use the full number of CPUs).

              Use single thread version of libmcount for faster recording.  This  is  ignored  if
              the target program calls pthread_create().

              Boost  priority  of  recording  threads  to real-time (FIFO) with priority of PRIO.
              This is particularly useful for high-volume data such as full kernel tracing.

       -K DEPTH, --kernel-depth=DEPTH
              Set kernel max function depth separately.  Implies --kernel.

              Set kernel tracing buffer size.  The default value (in the kernel) is 1408k.

       -P FUNC, --patch=FUNC
              Patch FUNC dynamically.  This  is  only  applicable  binaries  built  by  gcc  with
              -pg -mfentry -mnop-mcount or clang with -fxray-instrument.  This option can be used
              more than once.  See DYNAMIC TRACING.

       -E EVENT, --event=EVENT
              Enable event tracing.  The event should be available on the system.

              Disable event recording which is used by default.  Note that explicit event tracing
              by --event option is not affected by this.

              Retain  same pid for traced program.  For some daemon processes, it is important to
              have same pid when forked.  Running under uftrace normally changes pid as it  calls
              fork() again internally.

       -S SCRIPT_PATH, --script=SCRIPT_PATH
              Add  a script to do additional work at the entry and exit of function.  The type of
              script is detected by the postfix such as `.py' for python.

              Use pattern match using TYPE.  Possible types  are  regex  and  glob.   Default  is

              Disable ASLR (Address Space Layout Randomization).  It makes the target process fix
              its address space layout.


       The uftrace tool supports filtering out  uninteresting  functions.   Filtering  is  highly
       recommended  since  it helps users focus on the interesting functions and reduces the data
       size.  When uftrace is called it receives two types of function filter; an  opt-in  filter
       with  -F/--filter  and  an opt-out filter with -N/--notrace.  These filters can be applied
       either at record time or replay time.

       The first one is an opt-in filter.  By default, it doesn't trace anything.  But  when  one
       of  the  specified  functions is executed, tracing is started.  When the function returns,
       tracing is stopped again.

       For example, consider a simple program which calls a(), b() and c() in turn.

              $ cat abc.c
              void c(void) {
                  /* do nothing */

              void b(void) {

              void a(void) {

              int main(void) {
                  return 0;

              $ gcc -pg -o abc abc.c

       Normally uftrace will trace all the functions from main() to c().

              $ uftrace ./abc
              # DURATION    TID     FUNCTION
               138.494 us [ 1234] | __cxa_atexit();
                          [ 1234] | main() {
                          [ 1234] |   a() {
                          [ 1234] |     b() {
                 3.880 us [ 1234] |       c();
                 5.475 us [ 1234] |     } /* b */
                 6.448 us [ 1234] |   } /* a */
                 8.631 us [ 1234] | } /* main */

       But when the -F b filter option is used, it will not trace main() or a() but only b()  and

              $ uftrace record -F b ./abc
              $ uftrace replay
              # DURATION    TID     FUNCTION
                          [ 1234] | b() {
                 3.880 us [ 1234] |   c();
                 5.475 us [ 1234] | } /* b */

       The  second  type of filter is opt-out.  By default, everything is traced, but when one of
       the specified functions is executed, tracing stops.  When the excluded  function  returns,
       tracing is started again.

       In  the  above  example,  you can omit the function b() and all calls it makes with the -N

              $ uftrace record -N b ./abc
              $ uftrace replay
              # DURATION    TID     FUNCTION
               138.494 us [ 1234] | __cxa_atexit();
                          [ 1234] | main() {
                 6.448 us [ 1234] |   a();
                 8.631 us [ 1234] | } /* main */

       In addition, you can limit the print nesting level with the -D option.

              $ uftrace record -D 3 ./abc
              $ uftrace replay
              # DURATION    TID     FUNCTION
               138.494 us [ 1234] | __cxa_atexit();
                          [ 1234] | main() {
                          [ 1234] |   a() {
                 5.475 us [ 1234] |     b();
                 6.448 us [ 1234] |   } /* a */
                 8.631 us [ 1234] | } /* main */

       In the above example, uftrace only prints functions up to a depth of 3, so  leaf  function
       c() was omitted.  Note that the -D option works with -F.

       Sometimes  it's useful to see long-running functions only.  This is good because there are
       usually many tiny  functions  that  are  not  interesting.   The  -t/--time-filter  option
       implements  the  time-based  filter  that only records functions which run longer than the
       given threshold.  In the above example, the user might want to see functions running  more
       than 5 microseconds like below:

              $ uftrace record -t 5us ./abc
              $ uftrace replay
              # DURATION    TID     FUNCTION
               138.494 us [ 1234] | __cxa_atexit();
                          [ 1234] | main() {
                          [ 1234] |   a() {
                 5.475 us [ 1234] |     b();
                 6.448 us [ 1234] |   } /* a */
                 8.631 us [ 1234] | } /* main */

       The  -t/--time-filter  option  works  for user-level functions only.  It does not work for
       recording kernel functions, but they can be hidden  in  replay,  report,  dump  and  graph
       commands with -t/--time-filter option.

       You can also set triggers on filtered functions.  See TRIGGERS section below for details.

       When  kernel function tracing is enabled, you can also set the filters on kernel functions
       by marking the symbol with the @kernel modifier.  The following example will show all user
       functions and the (kernel) page fault handler.

              $ sudo uftrace -k -F '.*page_fault@kernel' ./abc
              # DURATION    TID     FUNCTION
                         [14721] | main() {
                7.713 us [14721] |   __do_page_fault();
                6.600 us [14721] |   __do_page_fault();
                6.544 us [14721] |   __do_page_fault();
                         [14721] |   a() {
                         [14721] |     b() {
                         [14721] |       c() {
                0.860 us [14721] |         getpid();
                2.346 us [14721] |       } /* c */
                2.956 us [14721] |     } /* b */
                3.340 us [14721] |   } /* a */
               79.086 us [14721] | } /* main */


       The  uftrace  tool  supports triggering actions on selected function calls with or without
       filters.   Currently  supported  triggers  are  listed  below.   The   BNF   for   trigger
       specification is:

              <trigger>    :=  <symbol> "@" <actions>
              <actions>    :=  <action>  | <action> "," <actions>
              <action>     :=  "depth="<num> | "trace" | "trace_on" | "trace_off" |
                               "time="<time_spec> | "read="<read_spec> | "finish" |
                               "filter" | "notrace" | "recover"
              <time_spec>  :=  <num> [ <time_unit> ]
              <time_unit>  :=  "ns" | "us" | "ms" | "s"
              <read_spec>  :=  "proc/statm" | "page-fault" | "pmu-cycle" | "pmu-cache" |

       The  depth  trigger is to change filter depth during execution of the function.  It can be
       used to apply different filter depths for different functions.

       The following example shows how triggers work.  The global filter maximum depth is 5,  but
       when function b() is called, it is changed to 1, so functions below b() will not shown.

              $ uftrace record -D 5 -T 'b@depth=1' ./abc
              $ uftrace replay
              # DURATION    TID     FUNCTION
               138.494 us [ 1234] | __cxa_atexit();
                          [ 1234] | main() {
                          [ 1234] |   a() {
                 5.475 us [ 1234] |     b();
                 6.448 us [ 1234] |   } /* a */
                 8.631 us [ 1234] | } /* main */

       The backtrace trigger is only meaningful in the replay command.

       The  traceon  and  traceoff  actions  (the  _  can be omitted from trace_on and trace_off)
       control whether uftrace records the specified functions or not.

       The `recover' trigger is for some corner cases in which the process accesses the callstack
       directly.   During  tracing  of  the  v8  javascript  engine, for example, it kept getting
       segfaults in the garbage collection stage.  It was  because  v8  incorporates  the  return
       address  into  compiled code objects(?).  The recover trigger restores the original return
       address at the function entry point and resets to the uftrace return hook address again at
       function  exit.   I was managed to work around the segfault by setting the recover trigger
       on the related function (specifically ExitFrame::Iterate).

       The `time' trigger is to change time filter setting during execution of the function.   It
       can be used to apply differernt time filter for different functions.

       The  read  trigger is to read some information at runtime.  The result will be recorded as
       (builtin) events at the beginning and the end of a given function.  As of  now,  following
       events are supported:

       · “proc/statm”: process memory stat from /proc filesystem

       · “page-fault”: number of page faults using getrusage(2)

       · “pmu-cycle”: cpu cycles and instructions using Linux perf-event syscall

       · “pmu-cache”: (cpu) cache-references and misses using Linux perf-event syscall

       · “pmu-branch”: branch instructions and misses using Linux perf-event syscall

       The results are printed in comments like below.

              $ uftrace record -T a@read=proc/statm ./abc
              $ uftrace replay
              # DURATION    TID     FUNCTION
                          [ 1234] | main() {
                          [ 1234] |   a() {
                          [ 1234] |     /* read:proc/statm (size=6808KB, rss=776KB, shared=712KB) */
                          [ 1234] |     b() {
                          [ 1234] |       c() {
                 1.448 us [ 1234] |         getpid();
                10.270 us [ 1234] |       } /* c */
                11.250 us [ 1234] |     } /* b */
                          [ 1234] |     /* read2:proc/statm (size=+4KB, rss=+0KB, shared=+0KB) */
                18.380 us [ 1234] |   } /* a */
                19.537 us [ 1234] | } /* main */

       The  `finish'  trigger  is  to  end  recording.  The process still can run and this can be
       useful to trace unterminated processes like daemon.

       The `filter' and `notrace' triggers have  same  effect  as  -F/--filter  and  -N/--notrace
       options respectively.

       Triggers only work for user-level functions for now.


       The  uftrace  tool  supports  recording  function arguments and/or return values using the
       -A/--argument and -R/--retval options respectively.  The syntax is very similar to that of

              <argument>    :=  <symbol> "@" <specs>
              <specs>       :=  <spec> | <spec> "," <spec>
              <spec>        :=  ( <int_spec> | <float_spec> | <ret_spec> )
              <int_spec>    :=  "arg" N [ "/" <format> [ <size> ] ] [ "%" ( <reg> | <stack> ) ]
              <float_spec>  :=  "fparg" N [ "/" ( <size> | "80" ) ] [ "%" ( <reg> | <stack> ) ]
              <ret_spec>    :=  "retval" [ "/" <format> [ <size> ] ]
              <format>      :=  "d" | "i" | "u" | "x" | "s" | "c" | "f" | "S" | "p"
              <size>        :=  "8" | "16" | "32" | "64"
              <reg>         :=  <arch-specific register name>  # "rdi", "xmm0", "r0", ...
              <stack>       :=  "stack" [ "+" ] <offset>

       The  -A/--argument  option  takes  argN  where  N is an index of the arguments.  The index
       starts from 1 and corresponds to the argument passing order of the calling  convention  on
       the  system.   Note  that  the indexes of arguments are separately counted for integer (or
       pointer) and floating-point  type,  and  they  can  interfere  depending  on  the  calling
       convention.  The argN is for integer arguments and fpargN is for floating-point arguments.

       Users  can  optionally  specify  a format and size for the arguments and/or return values.
       The “d” format or without format field,  uftrace  treats  them  as  `long  int'  type  for
       integers  and `double' for floating-point numbers.  The “i” format makes it signed integer
       type and “u” format is for unsigned type.  Both are printed as decimal  while  “x”  format
       makes  it  printed  as hexadecimal.  The “s” format is for null-terminated string type and
       “c” format is for character type.  The “f”  format  is  for  floating-point  type  and  is
       meaningful  only  for  return value (generally).  Note that fpargN doesn't take the format
       field since it's always floating-point.  The “S” format is for std::string,  but  it  only
       supports  libstdc++  library  as of yet.  Finally, the “p” format is for function pointer.
       Once the target address is recorded, it will be displayed as function name.

       Please beware when using string type arguments since it  can  crash  the  program  if  the
       (pointer) value is invalid.

       It  is also possible to specify a certain register name or stack offset for arguments (but
       not for return value).  The following register names can be used for argument:

       · x86: rdi, rsi, rdx, rcx, r8, r9 (for integer), xmm[0-7] (for floating-point)

       · arm: r[0-3] (for integer), s[0-15] or d[0-7] (for floating-point)

       Examples are below:

              $ uftrace record -A main@arg1/x -R main@retval/i32 ./abc
              $ uftrace replay
              # DURATION    TID     FUNCTION
               138.494 us [ 1234] | __cxa_atexit();
                          [ 1234] | main(0x1) {
                          [ 1234] |   a() {
                          [ 1234] |     b() {
                 3.880 us [ 1234] |       c();
                 5.475 us [ 1234] |     } /* b */
                 6.448 us [ 1234] |   } /* a */
                 8.631 us [ 1234] | } = 0; /* main */

              $ uftrace record -A puts@arg1/s -R puts@retval ./hello
              Hello world

              $ uftrace replay
              # DURATION    TID     FUNCTION
                 1.457 us [21534] | __monstartup();
                 0.997 us [21534] | __cxa_atexit();
                          [21534] | main() {
                 7.226 us [21534] |   puts("Hello world") = 12;
                 8.708 us [21534] | } /* main */

       Note that these arguments and return value are recorded only if the executable  was  built
       with  the -pg option.  Executables built with -finstrument-functions will ignore it except
       for library calls.  Recording of arguments and return values only  works  with  user-level
       functions for now.

       If  the target program is built with debug info like DWARF, uftrace can identify number of
       arguments and their types automatically (when  built  with  libdw).   Also  arguments  and
       return  value  of some well-known library functions are provided even if the debug info is
       not available.  In these cases user don't need to specify  format  of  the  arguments  and
       return  value  manually  -  just  function  name  (or pattern) is enough.  In fact, manual
       argspec will suppress the automatic argspec.

       For example, the above example can be written like below:

              $ uftrace record -A . -R main ./hello
              Hello world

              $ uftrace replay -F main
              # DURATION     TID     FUNCTION
                          [ 18948] | main(1, 0x7ffeeb7590b8) {
                 7.183 us [ 18948] |   puts("Hello world");
                 9.832 us [ 18948] | } = 0; /* main */

       Note that argument pattern (“.”) matches to any character so it recorded  all  (supported)
       functions.  It shows two arguments for “main” and a single string argument for “puts”.  If
       you simply want to see all arguments and return values of every functions (if  supported),
       use -a/--auto-args option.


       The  uftrace  tool  supports  dynamic  function  tracing  which  can be enabled at runtime
       (load-time, to be precise) on x86_64.  Before recording functions, normally  you  need  to
       build  the  target  program  with  -pg  (or  -finstrument-functions),  then  it  has  some
       performance impact because all functions call mcount().

       With dynamic tracing, you can trace  specific  functions  only  given  by  the  -P/--patch
       option.  However you need to add some more compiler (gcc) options when building the target
       program.  The gcc 5.1 or more recent versions provide -mfentry  and  -mnop-mcount  options
       which add instrumentation code (i.e. calling mcount() function) at the very beginning of a
       function and convert the instruction to a  NOP.   Then  it  has  almost  zero  performance
       overhead  when  running  in  a  normal condition.  The uftrace can convert it back to call
       mcount() if users want to (using -P option).

       The following example shows a  error  message  when  normally  running  uftrace  with  the
       executable  built  with  -pg -mfentry -mnop-mcount.   Because  the binary doesn't call any
       instrumentation code (i.e.  `mcount').

              $ gcc -o abc -pg -mfentry -mnop-mcount tests/s-abc.c
              $ uftrace abc
              uftrace: /home/namhyung/project/uftrace/cmd-record.c:1305:check_binary
                ERROR: Can't find 'mcount' symbol in the 'abc'.
                       It seems not to be compiled with -pg or -finstrument-functions flag
                       which generates traceable code.  Please check your binary file.

       But when the -P a patch option is used, and then only it can dynamically trace a().

              $ uftrace record --no-libcall -P a abc
              $ uftrace replay
              # DURATION    TID     FUNCTION
                 0.923 us [19379] | a();

       In addition, you can enable all functions at load time  using  `.'  that  matches  to  any
       character in a regex pattern with P option.

              $ uftrace record --no-libcall -P . abc
              $ uftrace replay
              # DURATION    TID     FUNCTION
                          [19387] | main() {
                          [19387] |   a() {
                          [19387] |     b() {
                 0.940 us [19387] |       c();
                 2.030 us [19387] |     } /* b */
                 2.451 us [19387] |   } /* a */
                 3.289 us [19387] | } /* main */

       Clang/LLVM    4.0    provides   a   dynamic   instrumentation   technique   called   X-ray
       (     It's     similar     to     a     combination     of
       gcc -mfentry -mnop-mcount  and  -finstrument-functions.  The uftrace also supports dynamic
       tracing on the executables built with the X-ray.

       For example, you can build the target program by clang with the below option  and  equally
       use -P option for dynamic tracing like below:

              $ clang -fxray-instrument -fxray-instruction-threshold=1 -o abc-xray  tests/s-abc.c
              $ uftrace record -P main abc-xray
              $ uftrace replay
              # DURATION    TID     FUNCTION
                          [11093] | main() {
                 1.659 us [11093] |   getpid();
                 5.963 us [11093] | } /* main */

              $ uftrace record -P . abc-xray
              $ uftrace replay
              # DURATION    TID     FUNCTION
                          [11098] | main() {
                          [11098] |   a() {
                          [11098] |     b() {
                          [11098] |       c() {
                 0.753 us [11098] |         getpid();
                 1.430 us [11098] |       } /* c */
                 1.915 us [11098] |     } /* b */
                 2.405 us [11098] |   } /* a */
                 3.005 us [11098] | } /* main */


       The  uftrace  tool  supports  script  execution  for  each  function  entry and exit.  The
       supported script is only Python 2.7 as of now.

       The user can write  four  functions.   `uftrace_entry'  and  `uftrace_exit'  are  executed
       whenever  each  function  is  executed at the entry and exit.  However `uftrace_begin' and
       `uftrace_end' are only executed once when the target program begins and ends.

              $ cat scripts/
              def uftrace_begin(ctx):
                  print("program begins...")

              def uftrace_entry(ctx):
                  func = ctx["name"]
                  print("entry : " + func + "()")

              def uftrace_exit(ctx):
                  func = ctx["name"]
                  print("exit  : " + func + "()")

              def uftrace_end():
                  print("program is finished")

       The above script can be executed in record time as follows:

              $ uftrace record -S scripts/ -F main tests/t-abc
              program begins...
              entry : main()
              entry : a()
              entry : b()
              entry : c()
              entry : getpid()
              exit  : getpid()
              exit  : c()
              exit  : b()
              exit  : a()
              exit  : main()
              program is finished

       The `ctx' variable is a dictionary type that contains the below information.

              /* context information passed to script */
              script_context = {
                  int       tid;
                  int       depth;
                  long      timestamp;
                  long      duration;    # exit only
                  long      address;
                  string    name;
                  list      args;        # entry only (if available)
                  value     retval;      # exit  only (if available)

       Each  field  in  `script_context'  can  be   read   inside   the   script.    Please   see
       uftrace-script(1) for details about scripting.


       uftrace(1),   uftrace-replay(1),  uftrace-report(1),  uftrace-recv(1),  uftrace-script(1),


       Namhyung Kim <>.