argdist(8) System Manager's Manual argdist(8)
argdist - Trace a function and display a histogram or frequency count of its parameter
values. Uses Linux eBPF/bcc.
argdist [-h] [-p PID] [-z STRING_SIZE] [-i INTERVAL] [-n COUNT] [-v] [-T TOP] [-H speci‐
fier] [-C specifier] [-I header]
argdist attaches to function entry and exit points, collects specified parameter values,
and stores them in a histogram or a frequency collection that counts the number of times a
parameter value occurred. It can also filter parameter values and instrument multiple
entry points at once.
Since this uses BPF, only the root user can use this tool.
CONFIG_BPF and bcc.
-h Print usage message.
-p PID Trace only functions in the process PID.
When collecting string arguments (of type char*), collect up to STRING_SIZE charac‐
ters. Longer strings will be truncated.
Print the collected data every INTERVAL seconds. The default is 1 second.
Print the collected data COUNT times and then exit.
-v Display the generated BPF program, for debugging purposes.
-T TOP When collecting frequency counts, display only the top TOP entries.
-H specifiers, -C specifiers
One or more probe specifications that instruct argdist which functions to probe,
which parameters to collect, how to aggregate them, and whether to perform any fil‐
tering. See SPECIFIER SYNTAX below.
One or more header files that should be included in the BPF program. This enables
the use of structure definitions, enumerations, and constants that are available in
these headers. You should provide the same path you would include in the BPF pro‐
gram, e.g. 'linux/blkdev.h' or 'linux/time.h'. Note: in many cases, argdist will
deduce the necessary header files automatically.
The general specifier syntax is as follows:
Probe type - "p" for function entry, "r" for function return, "t" for kernel trace‐
point, "u" for USDT probe; -H for histogram collection, -C for frequency count.
Indicates where to place the probe and whether the probe should collect frequency
count information, or aggregate the collected values into a histogram. Counting
probes will collect the number of times every parameter value was observed, whereas
histogram probes will collect the parameter values into a histogram. Only integral
types can be used with histogram probes; there is no such limitation for counting
Library containing the probe. Specify the full path to the .so or executable file
where the function to probe resides. Alternatively, you can specify just the lib
name: for example, "c" refers to libc. If no library name is specified, the kernel
The category of the kernel tracepoint. For example: net, sched, block.
The function to probe, and its signature. The function name must match exactly for
the probe to be placed. The signature, on the other hand, is only required if you
plan to collect parameter values based on that signature. For example, if you only
want to collect the first parameter, you don't have to specify the rest of the
parameters in the signature. When capturing kernel tracepoints, this should be the
name of the event, e.g. net_dev_start_xmit. The signature for kernel tracepoints
should be empty. When capturing USDT probes, this should be the name of the probe,
e.g. reloc_complete. The signature for USDT probes should be empty.
The type(s) of the expression(s) to capture. This is the type of the keys in the
histogram or raw event collection that are collected by the probes.
The expression(s) to capture. These are the values that are assigned to the his‐
togram or raw event collection. You may use the parameters directly, or valid C
expressions that involve the parameters, such as "size % 10". Tracepoints may
access a special structure called "args" that is formatted according to the trace‐
point format (which you can obtain using tplist). For example, the
block:block_rq_complete tracepoint can access args->nr_sector. USDT probes may
access the arguments defined by the tracing program in the special arg1, arg2, ...
variables. To obtain their types, use the tplist tool. Return probes can use the
argument values received by the function when it was entered, through the
$entry(paramname) special variable. Return probes can also access the function's
return value in $retval, and the function's execution time in nanoseconds in
$latency. Note that adding the $latency or $entry(paramname) variables to the
expression will introduce an additional probe at the function's entry to collect
this data, and therefore introduce additional overhead.
The filter applied to the captured data. Only parameter values that pass the fil‐
ter will be collected. This is any valid C expression that refers to the parameter
values, such as "fd == 1 && length > 16". The $entry, $retval, and $latency vari‐
ables can be used here as well, in return probes. The filter expression may also
use the STRCMP pseudo-function to compare a predefined string to a string argument.
For example: STRCMP("test.txt", file). The order of arguments is important: the
first argument MUST be a quoted literal string, and the second argument can be a
The label that will be displayed when printing the probed values. By default, this
is the probe specifier.
Print a histogram of allocation sizes passed to kmalloc:
# argdist -H 'p::__kmalloc(u64 size):u64:size'
Print a count of how many times process 1005 called malloc with an allocation size of 16
# argdist -p 1005 -C 'p:c:malloc(size_t size):size_t:size:size==16'
Snoop on all strings returned by gets():
# argdist -C 'r:c:gets():char*:$retval'
Print a histogram of read sizes that were longer than 1ms:
# argdist -H 'r::__vfs_read(void *file, void *buf, size_t
count):size_t:$entry(count):$latency > 1000000'
Print frequency counts of how many times writes were issued to a particular file descrip‐
tor number, in process 1005:
# argdist -p 1005 -C 'p:c:write(int fd):int:fd'
Print a histogram of error codes returned by read() in process 1005:
# argdist -p 1005 -H 'r:c:read()'
Print a histogram of buffer sizes passed to write() across all processes, where the file
descriptor was 1 (STDOUT):
# argdist -H 'p:c:write(int fd, const void *buf, size_t count):size_t:count:fd==1'
Count fork() calls in libc across all processes, grouped by pid:
# argdist -C 'p:c:fork():int:$PID;fork per process'
Print histogram of number of sectors in completing block I/O requests:
# argdist -H 't:block:block_rq_complete():u32:nr_sector'
Aggregate interrupts by interrupt request (IRQ):
# argdist -C 't:irq:irq_handler_entry():int:irq'
Print the functions used as thread entry points and how common they are:
# argdist -C 'u:pthread:pthread_start():u64:arg2' -p 1337
Print histograms of sleep() and nanosleep() parameter values:
# argdist -H 'p:c:sleep(u32 seconds):u32:seconds' -H 'p:c:nanosleep(struct timespec
Spy on writes to STDOUT performed by process 2780, up to a string size of 120 characters:
# argdist -p 2780 -z 120 -C 'p:c:write(int fd, char* buf, size_t
Group files being read from and the read sizes from __vfs_read:
# argdist -C 'p::__vfs_read(struct file *file, void *buf, size_t
This is from bcc.
Also look in the bcc distribution for a companion _examples.txt file containing example
usage, output, and commentary for this tool.
Unstable - in development.
USER COMMANDS 2016-02-11 argdist(8)