Provided by: uftrace_0.8.2-1_amd64 
NAME
uftrace-record - Run a command and record its trace data
SYNOPSIS
uftrace record [options] COMMAND [command-options]
DESCRIPTION
This command runs COMMAND and gathers function trace data from it, and saves it into files
under the uftrace data directory - without displaying anything.
This data can then be inspected later on, using uftrace replay or uftrace report.
OPTIONS
-b SIZE, --buffer=SIZE
Size of internal buffer in which trace data will be saved. Default size is 128k.
-F FUNC, --filter=FUNC
Set filter to trace selected functions only. This option can be used more than
once. See FILTERS.
-N FUNC, --notrace=FUNC
Set filter not to trace selected functions (or the functions called underneath
them). This option can be used more than once. See FILTERS.
-T TRG, --trigger=TRG
Set trigger on selected functions. This option can be used more than once. See
TRIGGERS.
-t TIME, --time-filter=TIME
Do not show functions which run under the time threshold. If some functions
explicitly have the 'trace' trigger applied, those are always traced regardless of
execution time.
--force
Allow running uftrace even if some problems occur. When uftrace record finds no
mcount symbol (which is generated by compiler) in the executable, it quits with an
error message since uftrace can not trace the program. However, it is possible
that the user is only interested in functions within a dynamically-linked library,
in which case this option can be used to cause uftrace to run the program
regardless. Also, the -A/--argument and -R/--retval options work only for binaries
built with -pg, so uftrace will normally exit when it tries to run binaries built
without that option. This option ignores the warning and goes on tracing without
the argument and/or return value.
-L PATH, --library-path=PATH
Load necessary internal libraries from this path. This is for testing purposes.
--no-libcall
Do not record library function invocations. Library calls are normally traced by
hooking the dynamic linker's resolve function in the PLT. One can disable it with
this option.
--no-pltbind
Do not bind dynamic symbol address. This option uses the LD_BIND_NOT environment
variable to trace library function calls which might be missing due to concurrent
(first) accesses. It is not meaningful to use this option with the --no-libcall
option.
--nest-libcall
Trace function calls between libraries. By default, uftrace only record library
call from the main executable. Implies --force.
-D DEPTH, --depth=DEPTH
Set global trace limit in nesting level.
--max-stack=DEPTH
Set the max function stack depth for tracing. Default is 1024.
--nop Do not record any functions. This is a no-op and only meaningful for performance
comparisons.
--time Print running time of children in time(1)-style.
-k, --kernel
Trace kernel functions as well as user functions. Only kernel entry/exit functions
will be traced by default. Use the --kernel-depth option to override this.
-H HOST, --host=HOST
Send trace data to given host via the network, not writing to files. The
uftrace recv command should be run on the destination host to receive the data.
--port=PORT
When sending data to the network (with -H), use the given port instead of the
default (8090).
--disable
Start uftrace with tracing disabled. This is only meaningful when used with a
trace_on trigger.
-A SPEC, --argument=SPEC
Record function arguments. This option can be used more than once. See ARGUMENTS.
-R SPEC, --retval=SPEC
Record function return values. This option can be used more than once. See
ARGUMENTS.
--auto-args
Automatically record arguments and return values of well-known library functions.
Recommend to use it with --nest-libcall.
--num-thread=NUM
Use NUM threads to record trace data. Default is 1/4 of online CPUs (but when full
kernel tracing is enabled, it will use the full number of CPUs).
--libmcount-single
Use single thread version of libmcount for faster recording. This is ignored if
the target program calls pthread_create().
--rt-prio=PRIO
Boost priority of recording threads to real-time (FIFO) with priority of PRIO.
This is particularly useful for high-volume data such as full kernel tracing.
-K DEPTH, --kernel-depth=DEPTH
Set kernel max function depth separately. Implies --kernel.
--kernel-buffer=SIZE
Set kernel tracing buffer size. The default value (in the kernel) is 1408k.
-P FUNC, --patch=FUNC
Patch FUNC dynamically. This is only applicable binaries built with
-pg -mfentry -mnop-mcount on x86_64. This option can be used more than once. See
DYNAMIC TRACING.
-E EVENT, --event=EVENT
Enable event tracing. The event should be available on the system.
--keep-pid
Retain same pid for traced program. For some daemon processes, it is important to
have same pid when forked. Running under uftrace normally changes pid as it calls
fork() again internally.
-S SCRIPT_PATH, --script=SCRIPT_PATH
Add a script to do additional work at the entry and exit of function. The type of
script is detected by the postfix such as '.py' for python.
FILTERS
The uftrace tool supports filtering out uninteresting functions. Filtering is highly
recommended since it helps users focus on the interesting functions and reduces the data
size. When uftrace is called it receives two types of function filter; an opt-in filter
with -F/--filter and an opt-out filter with -N/--notrace. These filters can be applied
either at record time or replay time.
The first one is an opt-in filter. By default, it doesn't trace anything. But when one
of the specified functions is executed, tracing is started. When the function returns,
tracing is stopped again.
For example, consider a simple program which calls a(), b() and c() in turn.
$ cat abc.c
void c(void) {
/* do nothing */
}
void b(void) {
c();
}
void a(void) {
b();
}
int main(void) {
a();
return 0;
}
$ gcc -pg -o abc abc.c
Normally uftrace will trace all the functions from main() to c().
$ uftrace ./abc
# DURATION TID FUNCTION
138.494 us [ 1234] | __cxa_atexit();
[ 1234] | main() {
[ 1234] | a() {
[ 1234] | b() {
3.880 us [ 1234] | c();
5.475 us [ 1234] | } /* b */
6.448 us [ 1234] | } /* a */
8.631 us [ 1234] | } /* main */
But when the -F b filter option is used, it will not trace main() or a() but only b() and
c().
$ uftrace record -F b ./abc
$ uftrace replay
# DURATION TID FUNCTION
[ 1234] | b() {
3.880 us [ 1234] | c();
5.475 us [ 1234] | } /* b */
The second type of filter is opt-out. By default, everything is traced, but when one of
the specified functions is executed, tracing stops. When the excluded function returns,
tracing is started again.
In the above example, you can omit the function b() and all calls it makes with the -N
option.
$ uftrace record -N b ./abc
$ uftrace replay
# DURATION TID FUNCTION
138.494 us [ 1234] | __cxa_atexit();
[ 1234] | main() {
6.448 us [ 1234] | a();
8.631 us [ 1234] | } /* main */
In addition, you can limit the print nesting level with the -D option.
$ uftrace record -D 3 ./abc
$ uftrace replay
# DURATION TID FUNCTION
138.494 us [ 1234] | __cxa_atexit();
[ 1234] | main() {
[ 1234] | a() {
5.475 us [ 1234] | b();
6.448 us [ 1234] | } /* a */
8.631 us [ 1234] | } /* main */
In the above example, uftrace only prints functions up to a depth of 3, so leaf function
c() was omitted. Note that the -D option works with -F.
Sometimes it's useful to see long-running functions only. This is good because there are
usually many tiny functions that are not interesting. The -t/--time-filter option
implements the time-based filter that only records functions which run longer than the
given threshold. In the above example, the user might want to see functions running more
than 5 microseconds like below:
$ uftrace record -t 5us ./abc
$ uftrace replay
# DURATION TID FUNCTION
138.494 us [ 1234] | __cxa_atexit();
[ 1234] | main() {
[ 1234] | a() {
5.475 us [ 1234] | b();
6.448 us [ 1234] | } /* a */
8.631 us [ 1234] | } /* main */
The -t/--time-filter option works for user-level functions only. It does not work for
recording kernel functions, but they can be hidden in replay, report, dump and graph
commands with -t/--time-filter option.
You can also set triggers on filtered functions. See TRIGGERS section below for details.
When kernel function tracing is enabled, you can also set the filters on kernel functions
by marking the symbol with the @kernel modifier. The following example will show all user
functions and the (kernel) page fault handler.
$ sudo uftrace -k -F '*page_fault@kernel' ./abc
# DURATION TID FUNCTION
[14721] | main() {
7.713 us [14721] | __do_page_fault();
6.600 us [14721] | __do_page_fault();
6.544 us [14721] | __do_page_fault();
[14721] | a() {
[14721] | b() {
[14721] | c() {
0.860 us [14721] | getpid();
2.346 us [14721] | } /* c */
2.956 us [14721] | } /* b */
3.340 us [14721] | } /* a */
79.086 us [14721] | } /* main */
TRIGGERS
The uftrace tool supports triggering actions on selected function calls with or without
filters. Currently supported triggers are listed below. The BNF for trigger
specification is:
<trigger> := <symbol> "@" <actions>
<actions> := <action> | <action> "," <actions>
<action> := "depth="<num> | "trace" | "trace_on" | "trace_off" | "recover" |
"time="<time_spec> | "read="<read_spec> | "finish" |
"filter" | "notrace"
<time_spec> := <num> [ <time_unit> ]
<time_unit> := "ns" | "us" | "ms" | "s"
<read_spec> := "proc/statm" | "page-fault"
The depth trigger is to change filter depth during execution of the function. It can be
used to apply different filter depths for different functions.
The following example shows how triggers work. The global filter maximum depth is 5, but
when function b() is called, it is changed to 1, so functions below b() will not shown.
$ uftrace record -D 5 -T 'b@depth=1' ./abc
$ uftrace replay
# DURATION TID FUNCTION
138.494 us [ 1234] | __cxa_atexit();
[ 1234] | main() {
[ 1234] | a() {
5.475 us [ 1234] | b();
6.448 us [ 1234] | } /* a */
8.631 us [ 1234] | } /* main */
The backtrace trigger is only meaningful in the replay command.
The traceon and traceoff actions (the _ can be omitted from trace_on and trace_off)
control whether uftrace records the specified functions or not.
The 'recover' trigger is for some corner cases in which the process accesses the callstack
directly. During tracing of the v8 javascript engine, for example, it kept getting
segfaults in the garbage collection stage. It was because v8 incorporates the return
address into compiled code objects(?). The recover trigger restores the original return
address at the function entry point and resets to the uftrace return hook address again at
function exit. I was managed to work around the segfault by setting the recover trigger
on the related function (specifically ExitFrame::Iterate).
The 'time' trigger is to change time filter setting during execution of the function. It
can be used to apply differernt time filter for different functions.
The read trigger is to read some information at runtime. As of now, reading process
memory stat ("proc/statm") from /proc filesystem and number of page faults ("page-fault")
using getrusage(2) are supported only. The results are printed in comments like below.
$ uftrace record -T b@read=proc/statm ./abc
$ uftrace replay
# DURATION TID FUNCTION
[ 1234] | main() {
[ 1234] | a() {
[ 1234] | /* read:proc/statm (size=6808KB, rss=777KB, shared=713KB) */
[ 1234] | b() {
[ 1234] | c() {
1.448 us [ 1234] | getpid();
10.270 us [ 1234] | } /* c */
11.250 us [ 1234] | } /* b */
18.380 us [ 1234] | } /* a */
19.537 us [ 1234] | } /* main */
The 'finish' trigger is to end recording. The process still can run and this can be
useful to trace unterminated processes like daemon.
The 'filter' and 'notrace' triggers have same effect as -F/--filter and -N/--notrace
options respectively.
Triggers only work for user-level functions for now.
ARGUMENTS
The uftrace tool supports recording function arguments and/or return values using the
-A/--argument and -R/--retval options respectively. The syntax is very similar to that of
triggers:
<argument> := <symbol> "@" <specs>
<specs> := <spec> | <spec> "," <spec>
<spec> := ( <int_spec> | <float_spec> | <ret_spec> )
<int_spec> := "arg" N [ "/" <format> [ <size> ] ] [ "%" ( <reg> | <stack> ) ]
<float_spec> := "fparg" N [ "/" ( <size> | "80" ) ] [ "%" ( <reg> | <stack> ) ]
<ret_spec> := "retval" [ "/" <format> [ <size> ] ]
<format> := "i" | "u" | "x" | "s" | "c" | "f" | "S" | "p"
<size> := "8" | "16" | "32" | "64"
<reg> := <arch-specific register name> # "rdi", "xmm0", "r0", ...
<stack> := "stack" [ "+" ] <offset>
The -A/--argument option takes argN where N is an index of the arguments. The index
starts from 1 and corresponds to the argument passing order of the calling convention on
the system. Note that the indexes of arguments are separately counted for integer (or
pointer) and floating-point type, and they can interfere depending on the calling
convention. The argN is for integer arguments and fpargN is for floating-point arguments.
Users can optionally specify a format and size for the arguments and/or return values.
Without this, uftrace treats them as 'long int' type for integers and 'double' for
floating-point numbers. The "i" format makes it signed integer type and "u" format is for
unsigned type. Both are printed as decimal while "x" format makes it printed as
hexadecimal. The "s" format is for null-terminated string type and "c" format is for
character type. The "f" format is for floating-point type and is meaningful only for
return value (generally). Note that fpargN doesn't take the format field since it's
always floating-point. The "S" format is for std::string, but it only supports libstdc++
library as of yet. Finally, the "p" format is for function pointer. Once the target
address is recorded, it will be displayed as function name.
Please beware when using string type arguments since it can crash the program if the
(pointer) value is invalid.
It is also possible to specify a certain register name or stack offset for arguments (but
not for return value). The following register names can be used for argument:
• x86: rdi, rsi, rdx, rcx, r8, r9 (for integer), xmm[0-7] (for floating-point)
• arm: r[0-3] (for integer), s[0-15] or d[0-7] (for floating-point)
Examples are below:
$ uftrace record -A main@arg1/x -R main@retval/i32 ./abc
$ uftrace replay
# DURATION TID FUNCTION
138.494 us [ 1234] | __cxa_atexit();
[ 1234] | main(0x1) {
[ 1234] | a() {
[ 1234] | b() {
3.880 us [ 1234] | c();
5.475 us [ 1234] | } /* b */
6.448 us [ 1234] | } /* a */
8.631 us [ 1234] | } = 0; /* main */
$ uftrace record -A puts@arg1/s -R puts@retval ./hello
Hello world
$ uftrace replay
# DURATION TID FUNCTION
1.457 us [21534] | __monstartup();
0.997 us [21534] | __cxa_atexit();
[21534] | main() {
7.226 us [21534] | puts("Hello world") = 12;
8.708 us [21534] | } /* main */
Note that these arguments and return value are recorded only if the executable was built
with the -pg option. Executables built with -finstrument-functions will cause uftrace to
exit with an error message. Recording of arguments and return values only works with
user-level functions for now.
DYNAMIC TRACING
The uftrace tool supports dynamic function tracing which can be enabled at runtime
(load-time, to be precise) on x86_64. Before recording functions, normally you need to
build the target program with -pg (or -finstrument-functions), then it has some
performance impact because all functions call mcount().
With dynamic tracing, you can trace specific functions only given by the -P/--patch
option. However you need to add some more compiler (gcc) options when building the target
program. The gcc 5.1 or more recent versions provide -mfentry and -mnop-mcount options
which add instrumentation code (i.e. calling mcount() function) at the very beginning of
a function and convert the instruction to a NOP. Then it has almost zero performance
overhead when running in a normal condition. The uftrace can convert it back to call
mcount() if users want to (using -P option).
The following example shows a error message when normally running uftrace with the
executable built with -pg -mfentry -mnop-mcount. Because the binary doesn't call any
instrumentation code (i.e. 'mcount').
$ gcc -o abc -pg -mfentry -mnop-mcount tests/s-abc.c
$ uftrace abc
uftrace: /home/namhyung/project/uftrace/cmd-record.c:1305:check_binary
ERROR: Can't find 'mcount' symbol in the 'abc'.
It seems not to be compiled with -pg or -finstrument-functions flag
which generates traceable code. Please check your binary file.
But when the -P a patch option is used, and then only it can dynamically trace a().
$ uftrace record --no-libcall -P a abc
$ uftrace replay
# DURATION TID FUNCTION
0.923 us [19379] | a();
In addition, you can enable all functions at load time using '.' that matches to any
character in a regex pattern with P option.
$ uftrace record --no-libcall -P . abc
$ uftrace replay
# DURATION TID FUNCTION
[19387] | main() {
[19387] | a() {
[19387] | b() {
0.940 us [19387] | c();
2.030 us [19387] | } /* b */
2.451 us [19387] | } /* a */
3.289 us [19387] | } /* main */
Clang/LLVM 4.0 provides a dynamic instrumentation technique called X-ray
(http://llvm.org/docs/XRay.html). It's similar to a combination of
gcc -mfentry -mnop-mcount and -finstrument-functions. The uftrace also supports dynamic
tracing on the executables built with the X-ray.
For example, you can build the target program by clang with the below option and equally
use -P option for dynamic tracing like below:
$ clang -fxray-instrument -fxray-instruction-threshold=1 -o abc-xray tests/s-abc.c
$ uftrace record -P main abc-xray
$ uftrace replay
# DURATION TID FUNCTION
[11093] | main() {
1.659 us [11093] | getpid();
5.963 us [11093] | } /* main */
$ uftrace record -P . abc-xray
$ uftrace replay
# DURATION TID FUNCTION
[11098] | main() {
[11098] | a() {
[11098] | b() {
[11098] | c() {
0.753 us [11098] | getpid();
1.430 us [11098] | } /* c */
1.915 us [11098] | } /* b */
2.405 us [11098] | } /* a */
3.005 us [11098] | } /* main */
SCRIPT EXECUTION
The uftrace tool supports script execution for each function entry and exit. The
supported script is only Python 2.7 as of now.
The user can write four functions. 'uftrace_entry' and 'uftrace_exit' are executed
whenever each function is executed at the entry and exit. However 'uftrace_begin' and
'uftrace_end' are only executed once when the target program begins and ends.
$ cat scripts/simple.py
def uftrace_begin():
print("program begins...")
def uftrace_entry(ctx):
func = ctx["name"]
print("entry : " + func + "()")
def uftrace_exit(ctx):
func = ctx["name"]
print("exit : " + func + "()")
def uftrace_end():
print("program is finished")
The above script can be executed in record time as follows:
$ uftrace record -S scripts/simple.py -F main tests/t-abc
program begins...
entry : main()
entry : a()
entry : b()
entry : c()
entry : getpid()
exit : getpid()
exit : c()
exit : b()
exit : a()
exit : main()
program is finished
The 'ctx' variable is a dictionary type that contains the below information.
/* context information passed to script */
script_context = {
int tid;
int depth;
long timestamp;
long duration; # exit only
long address;
string name;
list args; # entry only (if available)
value retval; # exit only (if available)
};
Each field in 'script_context' can be read inside the script. Please see
uftrace-script(1) for details about scripting.
SEE ALSO
uftrace(1), uftrace-replay(1), uftrace-report(1), uftrace-recv(1), uftrace-script(1)
AUTHORS
Namhyung Kim <namhyung@gmail.com>.