Provided by: uftrace_0.8.2-1_amd64 
NAME
uftrace-live - Trace functions in a command during live execution
SYNOPSIS
uftrace [live] [options] COMMAND [command-options]
DESCRIPTION
This command runs COMMAND and prints its functions with time and thread info. This is basically the same
as running the uftrace record and uftrace replay commands in turn, but it does not save a data file.
This command accepts most options that are accepted by the record or replay commands.
OPTIONS
-b SIZE, --buffer=SIZE
Size of internal buffer in which trace data will be saved. Default size is 128k.
-F FUNC, --filter=FUNC
Set filter to trace selected functions only. This option can be used more than once. See FIL‐
TERS.
-N FUNC, --notrace=FUNC
Set filter not to trace selected functions (or the functions called underneath them). This option
can be used more than once. See FILTERS.
-T TRG, --trigger=TRG
Set trigger on selected functions. This option can be used more than once. See TRIGGERS.
-D DEPTH, --depth=DEPTH
Set global trace limit in nesting level.
-t TIME, --time-filter=TIME
Do not show small functions under the time threshold. If some functions explicitly have 'trace'
trigger, those are always traced regardless of execution time.
-A SPEC, --argument=SPEC
Record function arguments. This option can be used more than once. See ARGUMENTS.
-R SPEC, --retval=SPEC
Record function return value. This option can be used more than once. See ARGUMENTS.
--auto-args
Automatically record arguments and return values of well-known library functions. Recommend to
use it with --nest-libcall.
-f FIELD, --output-fields=FIELD
Customize field in the output. Possible values are: duration, tid, time, delta, elapsed and addr.
Multiple fields can be set by using comma. Special field of 'none' can be used (solely) to hide
all fields. Default is 'duration,tid'. See FIELDS.
-r RANGE, --time-range=RANGE
Only show functions executed within the time RANGE. The RANGE can be <start>~<stop> (separated by
"~") and one of <start> and <stop> can be omitted. The <start> and <stop> are timestamp or
elapsed time if they have <time_unit> postfix, for example '100us'. However, it is highly recom‐
mended to use only elapsed time because there is no way to know the timestamp before actually run‐
ning the program. The timestamp or elapsed time can be shown with -f time or -f elapsed option
respectively.
--force
Allow running uftrace even if some problems occur. When uftrace record finds no mcount symbol
(which is generated by compiler) in the executable, it quits with an error message since uftrace
can not trace the executable. However, it is possible that the user is only interested in func‐
tions within a dynamically-linked library, in which case this option can be used to cause uftrace
to run the program regardless. Also, the -A/--argument and -R/--retval options work only for bi‐
naries built with -pg, so uftrace will normally exit when it tries to run binaries built without
that option. This option ignores the warning and goes on tracing without the argument and/or re‐
turn value.
--flat Print flat format rather than C-like format. This is usually for debugging and testing purpose.
-L PATH, --library-path=PATH
Load necessary internal libraries from this path. This is for testing.
-k, --kernel
Trace kernel functions as well as user functions. This is simply a shortcut to --kernel-depth=1.
--no-libcall
Do not record library function invocations. Library calls are normally traced by hooking the dy‐
namic linker's resolve function in the PLT. One can disable it with this option.
--no-pltbind
Do not bind dynamic symbol address. This option uses the LD_BIND_NOT environment variable to
trace library function calls which might be missing due to concurrent (first) accesses. It is not
meaningful to use this option with the --no-libcall option.
--nest-libcall
Trace function calls between libraries. By default, uftrace only record library call from the
main executable. Implies --force.
--disable
Start uftrace with tracing disabled. This is only meaningful when used with a trace_on trigger.
--demangle=TYPE
Demangle C++ symbol names. Possible values are "full", "simple" and "no". Default is "simple"
which ignores function arguments and template parameters.
--report
Show live-report before replay.
--column-view
Show each task in separate column. This makes easy to distinguish functions in different tasks.
--column-offset=DEPTH
When --column-view option is used, this option specifies the amount of offset between each task.
Default is 8.
--task-newline
Interleave a new line when task is changed. This makes easy to distinguish functions in different
tasks.
--num-thread=NUM
Use NUM threads to record trace data. Default is 1/4 of online CPUs (but when full kernel tracing
is enabled, it will use the full number of CPUs).
--no-comment
Do not show comments of returned functions.
--libmcount-single
Use single thread version of libmcount for faster recording. This is ignored if the target pro‐
gram calls pthread_create().
--rt-prio=PRIO
Boost priority of recording threads to real-time (FIFO) with priority of PRIO. This is particu‐
larly useful for high-volume data such as full kernel tracing.
-K DEPTH, --kernel-depth=DEPTH
Set kernel max function depth separately. Implies --kernel.
--kernel-buffer=SIZE
Set kernel tracing buffer size. The default value (in the kernel) is 1408k.
--kernel-skip-out
Do not show kernel functions called outside of user functions. This option is deprecated and set
to true by default.
--kernel-full
Show all kernel functions called outside of user functions. This option is the inverse of --ker‐
nel-skip-out.
--kernel-only
Show kernel functions only without user functions.
-P FUNC, --patch=FUNC
Patch FUNC dynamically. This is only applicable binaries built with -pg -mfentry -mnop-mcount on
x86_64. This option can be used more than once. See DYNAMIC TRACING.
-E EVENT, --event=EVENT
Enable event tracing. The event should be available on the system.
--list-event
Show available events in the process.
--keep-pid
Retain same pid for traced program. For some daemon processes, it is important to have same pid
when forked. Running under uftrace normally changes pid as it calls fork() again internally.
Note that it might corrupt terminal setting so it'd be better using it with --no-pager option.
-S SCRIPT_PATH, --script=SCRIPT_PATH
Add a script to do additional work at the entry and exit of function. The type of script is de‐
tected by the postfix such as '.py' for python.
--event-full
Show all (user) events outside of user functions.
FILTERS
The uftrace tool supports filtering out uninteresting functions. When uftrace is called it receives two
types of function filter; an opt-in filter with -F/--filter and an opt-out filter with -N/--notrace.
These filters can be applied either at record time or replay time.
The first one is an opt-in filter. By default, it doesn't trace anything. But when one of the specified
functions is executed, tracing is started. When the function returns, tracing is stopped again.
For example, consider a simple program which calls a(), b() and c() in turn.
$ cat abc.c
void c(void) {
/* do nothing */
}
void b(void) {
c();
}
void a(void) {
b();
}
int main(void) {
a();
return 0;
}
$ gcc -pg -o abc abc.c
Normally uftrace will trace all the functions from main() to c().
$ uftrace ./abc
# DURATION TID FUNCTION
138.494 us [ 1234] | __cxa_atexit();
[ 1234] | main() {
[ 1234] | a() {
[ 1234] | b() {
3.880 us [ 1234] | c();
5.475 us [ 1234] | } /* b */
6.448 us [ 1234] | } /* a */
8.631 us [ 1234] | } /* main */
But when the -F b filter option is used, it will not trace main() or a() but only b() and c().
$ uftrace -F b ./abc
# DURATION TID FUNCTION
[ 1234] | b() {
3.880 us [ 1234] | c();
5.475 us [ 1234] | } /* b */
The second type of filter is opt-out. By default, everything is traced, but when one of the specified
functions is executed, tracing stops. When the excluded function returns, tracing is started again.
In the above example, you can omit the function b() and all calls it makes with the -N option.
$ uftrace live -N b ./abc
# DURATION TID FUNCTION
138.494 us [ 1234] | __cxa_atexit();
[ 1234] | main() {
6.448 us [ 1234] | a();
8.631 us [ 1234] | } /* main */
In addition, you can limit the print nesting level with the -D option.
$ uftrace -D 3 ./abc
# DURATION TID FUNCTION
138.494 us [ 1234] | __cxa_atexit();
[ 1234] | main() {
[ 1234] | a() {
5.475 us [ 1234] | b();
6.448 us [ 1234] | } /* a */
8.631 us [ 1234] | } /* main */
In the above example, uftrace only prints functions up to a depth of 3, so leaf function c() was omitted.
Note that the -D option works with -F.
Sometimes it's useful to see long-running functions only. This is good because there are usually many
tiny functions that are not interesting. The -t/--time-filter option implements the time-based filter
that only records functions which run longer than the given threshold. In the above example, the user
might want to see functions running more than 5 microseconds like below:
$ uftrace live -t 5us ./abc
# DURATION TID FUNCTION
138.494 us [ 1234] | __cxa_atexit();
[ 1234] | main() {
[ 1234] | a() {
5.475 us [ 1234] | b();
6.448 us [ 1234] | } /* a */
8.631 us [ 1234] | } /* main */
You can also set triggers on filtered functions. See TRIGGERS section below for details.
TRIGGERS
The uftrace tool supports triggering actions on selected function calls with or without filters. Cur‐
rently supported triggers are depth (for record and replay) and backtrace (for replay only). The BNF for
trigger specifications is like below:
<trigger> := <symbol> "@" <actions>
<actions> := <action> | <action> "," <actions>
<action> := "depth="<num> | "backtrace" | "trace" | "trace_on" | "trace_off" |
"recover" | "color="<color> | "time="<time_spec> | "read="<read_spec> |
"finish" | "filter" | "notrace"
<time_spec> := <num> [ <time_unit> ]
<time_unit> := "ns" | "nsec" | "us" | "usec" | "ms" | "msec" | "s" | "sec" | "m" | "min"
<read_spec> := "proc/statm" | "page-fault"
The depth trigger is to change filter depth during execution of the function. It can be used to apply
different filter depths for different functions. And the backtrace trigger is used to print a stack
backtrace at replay time.
The color trigger is to change the color of the function in replay output. The supported colors are red,
green, blue, yellow, magenta, cyan, bold, and gray.
The following example shows how triggers work. The global filter maximum depth is 5, but when function
b() is called, it is changed to 1, so functions below b() will not shown.
$ uftrace live -D 5 -T 'b@depth=1' ./abc
# DURATION TID FUNCTION
138.494 us [ 1234] | __cxa_atexit();
[ 1234] | main() {
[ 1234] | a() {
5.475 us [ 1234] | b();
6.448 us [ 1234] | } /* a */
8.631 us [ 1234] | } /* main */
The backtrace trigger is only meaningful in the replay command.
The traceon and traceoff actions (the _ can be omitted from trace_on and trace_off) control whether uf‐
trace records the specified functions or not.
The 'recover' trigger is for some corner cases in which the process accesses the callstack directly.
During tracing of the v8 javascript engine, for example, it kept getting segfaults in the garbage collec‐
tion stage. It was because v8 incorporates the return address into compiled code objects(?). The recov‐
er trigger restores the original return address at the function entry point and resets to the uftrace re‐
turn hook address again at function exit. I was managed to work around the segfault by setting the re‐
cover trigger on the related function (specifically ExitFrame::Iterate).
The 'time' trigger is to change time filter setting during execution of the function. It can be used to
apply different time filter for different functions.
The read trigger is to read some information at runtime. As of now, reading process memory stat
("proc/statm") from the /proc filesystem and number of page faults ("page-fault") using getrusage(2) are
supported only. The results are printed in comments like below.
$ uftrace -T b@read=proc/statm ./abc
# DURATION TID FUNCTION
[ 1234] | main() {
[ 1234] | a() {
[ 1234] | /* read:proc/statm (size=6808KB, rss=777KB, shared=713KB) */
[ 1234] | b() {
[ 1234] | c() {
1.448 us [ 1234] | getpid();
10.270 us [ 1234] | } /* c */
11.250 us [ 1234] | } /* b */
18.380 us [ 1234] | } /* a */
19.537 us [ 1234] | } /* main */
The 'finish' trigger is to end recording. The process still can run and this can be useful to trace un‐
terminated processes like daemon.
The 'filter' and 'notrace' triggers have same effect as -F/--filter and -N/--notrace options respective‐
ly.
Triggers only work for user-level functions for now.
ARGUMENTS
The uftrace tool supports recording function arguments and/or return values using the -A/--argument and
-R/--retval options respectively. The syntax is very similar to that of triggers:
<argument> := <symbol> "@" <specs>
<specs> := <spec> | <spec> "," <spec>
<spec> := ( <int_spec> | <float_spec> | <ret_spec> )
<int_spec> := "arg" N [ "/" <format> [ <size> ] ] [ "%" ( <reg> | <stack> ) ]
<float_spec> := "fparg" N [ "/" ( <size> | "80" ) ] [ "%" ( <reg> | <stack> ) ]
<ret_spec> := "retval" [ "/" <format> [ <size> ] ]
<format> := "i" | "u" | "x" | "s" | "c" | "f" | "S" | "p"
<size> := "8" | "16" | "32" | "64"
<reg> := <arch-specific register name> # "rdi", "xmm0", "r0", ...
<stack> := "stack" [ "+" ] <offset>
The -A/--argument option takes argN where N is an index of the arguments. The index starts from 1 and
corresponds to the argument passing order of the calling convention on the system. Note that the indexes
of arguments are separately counted for integer (or pointer) and floating-point type, and they can inter‐
fere depending on the calling convention. The argN is for integer arguments and fpargN is for float‐
ing-point arguments.
Users can optionally specify a format and size for the arguments and/or return values. Without this, uf‐
trace treats them as 'long int' type for integers and 'double' for floating-point numbers. The "i" for‐
mat makes it signed integer type and "u" format is for unsigned type. Both are printed as decimal while
"x" format makes it printed as hexadecimal. The "s" format is for null-terminated string type and "c"
format is for character type. The "f" format is for floating-point type and is meaningful only for re‐
turn value (generally). Note that fpargN doesn't take the format field since it's always floating-point.
The "S" format is for std::string, but it only supports libstdc++ library as of yet. Finally, the "p"
format is for function pointer. Once the target address is recorded, it will be displayed as function
name.
Please beware when using string type arguments since it can crash the program if the (pointer) value is
invalid.
It is also possible to specify a certain register name or stack offset for arguments (but not for return
value). The following register names can be used for argument:
• x86: rdi, rsi, rdx, rcx, r8, r9 (for integer), xmm[0-7] (for floating-point)
• arm: r[0-3] (for integer), s[0-15] or d[0-7] (for floating-point)
Examples are below:
$ uftrace -A main@arg1/x -R main@retval/i32 ./abc
# DURATION TID FUNCTION
138.494 us [ 1234] | __cxa_atexit();
[ 1234] | main(0x1) {
[ 1234] | a() {
[ 1234] | b() {
3.880 us [ 1234] | c();
5.475 us [ 1234] | } /* b */
6.448 us [ 1234] | } /* a */
8.631 us [ 1234] | } = 0; /* main */
$ uftrace -A puts@arg1/s -R puts@retval ./hello
Hello world
# DURATION TID FUNCTION
1.457 us [21534] | __monstartup();
0.997 us [21534] | __cxa_atexit();
[21534] | main() {
7.226 us [21534] | puts("Hello world") = 12;
8.708 us [21534] | } /* main */
Note that these arguments and return value are recorded only if the executable was built with the -pg op‐
tion. Executables built with -finstrument-functions will cause uftrace to exit with an error message.
Recording of arguments and return values only works with user-level functions for now.
FIELDS
The uftrace allows for user to customize the replay output with a couple of fields. Here the field means
info on the left side of the pipe (|) character. By default it uses duration and tid fields, but you can
use other fields in any order like:
$ uftrace -f time,delta,duration,tid,addr ./abc
# TIMESTAMP TIMEDELTA DURATION TID ADDRESS FUNCTION
75059.205379813 1.374 us [27804] 4004d0 | __monstartup();
75059.205384184 4.371 us 0.737 us [27804] 4004f0 | __cxa_atexit();
75059.205386655 2.471 us [27804] 4006b1 | main() {
75059.205386838 0.183 us [27804] 400656 | a() {
75059.205386961 0.123 us [27804] 400669 | b() {
75059.205387078 0.117 us [27804] 40067c | c() {
75059.205387264 0.186 us 0.643 us [27804] 4004b0 | getpid();
75059.205388501 1.237 us 1.423 us [27804] 40067c | } /* c */
75059.205388724 0.223 us 1.763 us [27804] 400669 | } /* b */
75059.205388878 0.154 us 2.040 us [27804] 400656 | } /* a */
75059.205389030 0.152 us 2.375 us [27804] 4006b1 | } /* main */
Each field has following meaning:
• tid: task id (obtained by gettid(2))
• duration: function execution time
• time: timestamp at the execution
• delta: difference between two timestamp in a task
• elapsed: elapsed time from the first timestamp
• addr: address of the function
The default value is 'duration,tid'. If given field name starts with "+", then it'll be appended to the
default fields. So "-f +time" is as same as "-f duration,tid,time". And it also accepts a special field
name of 'none' which disables the field display and shows function output only.
DYNAMIC TRACING
The uftrace tool supports dynamic function tracing which can be enabled at runtime (load-time, to be pre‐
cise) on x86_64. Before recording functions, normally you need to build the target program with -pg (or
-finstrument-functions), then it has some performance impact because all functions call mcount().
With dynamic tracing, you can trace specific functions only given by the -P/--patch option. However you
need to add some more compiler (gcc) options when building the target program. The gcc 5.1 or more re‐
cent versions provide -mfentry and -mnop-mcount options which add instrumentation code (i.e. calling
mcount() function) at the very beginning of a function and convert the instruction to a NOP. Then it has
almost zero performance overhead when running in a normal condition. The uftrace can convert it back to
call mcount() if users want to (using -P option).
The following example shows a error message when normally running uftrace with the executable built with
-pg -mfentry -mnop-mcount. Because the binary doesn't call any instrumentation code (i.e. 'mcount').
$ gcc -o abc -pg -mfentry -mnop-mcount tests/s-abc.c
$ uftrace abc
uftrace: /home/namhyung/project/uftrace/cmd-record.c:1305:check_binary
ERROR: Can't find 'mcount' symbol in the 'abc'.
It seems not to be compiled with -pg or -finstrument-functions flag
which generates traceable code. Please check your binary file.
But when the -P a patch option is used, and then only it can dynamically trace a().
$ uftrace --no-libcall -P a abc
# DURATION TID FUNCTION
0.923 us [19379] | a();
In addition, you can enable all functions at load time using '.' that matches to any character in a regex
pattern with P option.
$ uftrace --no-libcall -P . abc
# DURATION TID FUNCTION
[19387] | main() {
[19387] | a() {
[19387] | b() {
0.940 us [19387] | c();
2.030 us [19387] | } /* b */
2.451 us [19387] | } /* a */
3.289 us [19387] | } /* main */
Clang/LLVM 4.0 provides a dynamic instrumentation technique called X-ray
(http://llvm.org/docs/XRay.html). It's similar to a combination of gcc -mfentry -mnop-mcount and -fin‐
strument-functions. The uftrace also supports dynamic tracing on the executables built with the X-ray.
For example, you can build the target program by clang with the below option and equally use -P option
for dynamic tracing like below:
$ clang -fxray-instrument -fxray-instruction-threshold=1 -o abc-xray tests/s-abc.c
$ uftrace -P main abc-xray
# DURATION TID FUNCTION
[11093] | main() {
1.659 us [11093] | getpid();
5.963 us [11093] | } /* main */
$ uftrace -P . abc-xray
# DURATION TID FUNCTION
[11098] | main() {
[11098] | a() {
[11098] | b() {
[11098] | c() {
0.753 us [11098] | getpid();
1.430 us [11098] | } /* c */
1.915 us [11098] | } /* b */
2.405 us [11098] | } /* a */
3.005 us [11098] | } /* main */
SCRIPT EXECUTION
The uftrace tool supports script execution for each function entry and exit. The supported script is on‐
ly Python 2.7 as of now.
The user can write four functions. 'uftrace_entry' and 'uftrace_exit' are executed whenever each func‐
tion is executed at the entry and exit. However 'uftrace_begin' and 'uftrace_end' are only executed once
when the target program begins and ends.
$ cat scripts/simple.py
def uftrace_begin():
print("program begins...")
def uftrace_entry(ctx):
func = ctx["name"]
print("entry : " + func + "()")
def uftrace_exit(ctx):
func = ctx["name"]
print("exit : " + func + "()")
def uftrace_end():
print("program is finished")
The above script can be executed in record time as follows:
$ uftrace -S scripts/simple.py -F main tests/t-abc
program begins...
entry : main()
entry : a()
entry : b()
entry : c()
entry : getpid()
exit : getpid()
exit : c()
exit : b()
exit : a()
exit : main()
program is finished
# DURATION TID FUNCTION
[10929] | main() {
[10929] | a() {
[10929] | b() {
[10929] | c() {
4.293 us [10929] | getpid();
19.017 us [10929] | } /* c */
27.710 us [10929] | } /* b */
37.007 us [10929] | } /* a */
55.260 us [10929] | } /* main */
The 'ctx' variable is a dictionary type that contains the below information.
/* context information passed to script */
script_context = {
int tid;
int depth;
long timestamp;
long duration; # exit only
long address;
string name;
list args; # entry only (if available)
value retval; # exit only (if available)
};
Each field in 'script_context' can be read inside the script. Please see uftrace-script(1) for details
about scripting.
SEE ALSO
uftrace-record(1), uftrace-replay(1), uftrace-report(1), uftrace-script(1)
AUTHORS
Namhyung Kim <namhyung@gmail.com>.
Uftrace User Manuals May, 2016 UFTRACE-LIVE(1)