bcc-inject

title: bcc-inject(8)

text:
# bcc-inject(8)

## NAME
inject - injects appropriate error into function if input call chain and predicates are satisfied. Uses Linux eBPF/bcc.

## SYNOPSIS
inject -h [-I header] [-P probability] [-v] [-c count] <mode> <spec>

## DESCRIPTION
inject injects errors into specified kernel functionality when a given call chain and associated predicates are satisfied.

WARNING: This tool injects failures into key kernel functions and may crash the kernel. You should know what you're doing if you're using this tool.

This makes use of a Linux 4.16 feature (bpf_override_return())

Since this uses BPF, only the root user can use this tool.

## OPTIONS
-h

Print usage message.

-v

Display the generated BPF program, for debugging or modification.

-I header

Necessary headers to be included.

-P probability

Optional probability of failure, default 1.

-c count

Number of errors to inject before stopping, default never stops.

## MODE
kmalloc

Make the following function indicate failure

int should_failslab(struct kmem_cache *s, gfp_t gfpflags)

bio

Make the following function indicate failure

int should_fail_bio(struct bio *bio)

alloc_page

Make the following function indicate failure

bool should_fail_alloc_page(gfp_t gfp_mask, unsigned int order)

## SPEC
FUNCTION([ARGS])[(TEST)] [=> ...]

A list of predicates separated by "=>". A predicate is a function signature (name and arguments) in a call stack and a test on the function's arguments.

Missing predicates are implicitly true. Missing tests are implicitly true. Specifying the function arguments is optional if the test does not use them. If the error injection function is not listed as the first predicate, it is implicitly added.

Functions are listed in the reverse order that they are called, ie. if a() calls b(), the spec would be "b() => a()".

## REQUIREMENTS
CONFIG_BPF, CONFIG_BPF_KPROBE_OVERRIDE, bcc

## EXAMPLES

```
inject kmalloc -v 'SyS_mount()'
```

```
inject kmalloc -v 'mount_subtree() => btrfs_mount()'
```

```
inject -P 0.5 -c 100 alloc_page "should_fail_alloc_page(gfp_t gfp_mask, unsigned int order) (order == 1) => qlge_refill_bq()"
```

Please see the output of '-h' and tools/inject_example.txt for more examples.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Howard McLauchlan


bcc-javacalls

package: bcc-tools

title: bcc-javacalls(8)

summary: Summarize method calls

text:
# bcc-javacalls(8)

## NAME
ucalls, javacalls, perlcalls, phpcalls, pythoncalls, rubycalls, tclcalls - Summarize method calls from high-level languages and Linux syscalls.

## SYNOPSIS
javacalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

perlcalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

phpcalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

pythoncalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

rubycalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

tclcalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

ucalls [-l {java,perl,php,python,ruby}] [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

## DESCRIPTION
This tool summarizes method calls from high-level languages such as Java, Perl, PHP, Python, Ruby, and Tcl. It can also trace Linux system calls. Whenever a method is invoked, ucalls records the call count and optionally the method's execution time (latency) and displays a summary.

This uses in-kernel eBPF maps to store per process summaries for efficiency.

This tool relies on USDT probes embedded in many high-level languages, such as Java, Perl, PHP, Python, Ruby, and Tcl. It requires a runtime instrumented with these probes, which in some cases requires building from source with a USDT-specific flag, such as "--enable-dtrace" or "--with-dtrace". For Java, method probes are not enabled by default, and can be turned on by running the Java process with the "-XX:+ExtendedDTraceProbes" flag. For PHP processes, the environment variable USE_ZEND_DTRACE must be set to 1.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-l {java,perl,php,python,ruby,tcl}

The language to trace. If not provided, only syscalls are traced (when the -S option is used).

-T TOP

Print only the top methods by frequency or latency.

-L

Collect method invocation latency (duration).

-S

Collect Linux syscalls frequency and timing.

-v

Print the resulting BPF program, for debugging purposes.

-m

Print times in milliseconds (the default is microseconds).

pid

The process id to trace.

interval

Print summary after this number of seconds and then exit. By default, wait for Ctrl+C to terminate.

## EXAMPLES
Trace the top 10 Ruby method calls:

\# ucalls -T 10 -l ruby 1344

Trace Python method calls and Linux syscalls including latency in milliseconds:

\# ucalls -l python -mL 2020

Trace only syscalls and print a summary after 10 seconds:

\# ucalls -S 788 10

## OVERHEAD
Tracing individual method calls will produce a considerable overhead in all high-level languages. For languages with just-in-time compilation, such as Java, the overhead can be more considerable than for interpreted languages. On the other hand, syscall tracing will typically be tolerable for most processes, unless they have a very unusual rate of system calls.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _example.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein

## SEE ALSO
ustat(8), argdist(8)

branch: sisyphus


bcc-javaflow

title: bcc-javaflow(8)

summary: Print a flow graph of method

text:
# bcc-javaflow(8)

## NAME
uflow, javaflow, perlflow, phpflow, pythonflow, rubyflow, tclflow - Print a flow graph of method calls in high-level languages.

## SYNOPSIS
javaflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

perlflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

phpflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

pythonflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

rubyflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

tclflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

uflow [-h] [-M METHOD] [-C CLAZZ] [-v] [-l {java,perl,php,python,ruby,tcl}] pid

## DESCRIPTION
uflow traces method calls and prints them in a flow graph that can facilitate debugging and diagnostics by following the program's execution (method flow).

This tool relies on USDT probes embedded in many high-level languages, such as Java, Perl, PHP, Python, Ruby, and Tcl. It requires a runtime instrumented with these probes, which in some cases requires building from source with a USDT-specific flag, such as "--enable-dtrace" or "--with-dtrace". For Java processes, the startup flag "-XX:+ExtendedDTraceProbes" is required. For PHP processes, the environment variable USE_ZEND_DTRACE must be set to 1.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-M METHOD

Print only method calls where the method name begins with this string.

-C CLAZZ

Print only method calls where the class name begins with this string. The class name interpretation strongly depends on the language. For example, in Java use "package/subpackage/ClassName" to refer to classes.

-v

Print the resulting BPF program, for debugging purposes.

{java,perl,php,python,ruby,tcl}

The language to trace.

pid

The process id to trace.

## EXAMPLES
Follow method flow in a Ruby process:

\# uflow ruby 148

Follow method flow in a Java process where the class name is java.lang.Thread:

\# uflow -C java/lang/Thread java 1802

## FIELDS
CPU

The CPU number on which the method was invoked. This is useful to easily see where the output skips to a different CPU.

PID

The process id.

TID

The thread id.

TIME

The duration of the method call.

METHOD

The method name.

## OVERHEAD
This tool has extremely high overhead because it prints every method call. For some scenarios, you might see lost samples in the output as the tool is unable to keep up with the rate of data coming from the kernel. Filtering by class or method prefix can help reduce the amount of data printed, but there is still a very high overhead in the collection mechanism. Do not use for performance- sensitive production scenarios, and always test first.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _example.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein

## SEE ALSO
trace(8), ustat(8)


bcc-javagc

title: bcc-javagc(8)

summary: Trace garbage collection events in

text:
# bcc-javagc(8)

## NAME
ugc, javagc, nodegc, pythongc, rubygc - Trace garbage collection events in high-level languages.

## SYNOPSIS
javagc [-h] [-v] [-m] [-M MINIMUM] [-F FILTER] pid

nodegc [-h] [-v] [-m] [-M MINIMUM] [-F FILTER] pid

pythongc [-h] [-v] [-m] [-M MINIMUM] [-F FILTER] pid

rubygc [-h] [-v] [-m] [-M MINIMUM] [-F FILTER] pid

ugc [-h] [-v] [-m] [-M MINIMUM] [-F FILTER] [-l {java,node,python,ruby}] pid

## DESCRIPTION
This traces garbage collection events as they occur, including their duration and any additional information (such as generation collected or type of GC) provided by the respective language's runtime.

This tool relies on USDT probes embedded in many high-level languages, such as Java, Node, Python, and Ruby. It requires a runtime instrumented with these probes, which in some cases requires building from source with a USDT-specific flag, such as "--enable-dtrace" or "--with-dtrace".

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-v

Print the resulting BPF program, for debugging purposes.

-m

Print times in milliseconds. The default is microseconds.

-M MINIMUM

Display only collections that are longer than this threshold. The value is given in milliseconds. The default is to display all collections.

-F FILTER

Display only collections whose textual description matches (contains) this string. The default is to display all collections. Note that the filtering here is performed in user-space, and not as part of the BPF program. This means that if you have thousands of collection events, specifying this filter will not reduce the amount of data that has to be transferred from the BPF program to the user-space script.

{java,node,python,ruby}

The language to trace.

pid

The process id to trace.

## EXAMPLES
Trace garbage collections in a specific Node process:

\# ugc node 148

Trace garbage collections in a specific Java process, and print GC times in

milliseconds: # ugc -m java 6004

Trace garbage collections in a specific Java process, and display them only if

they are longer than 10ms and have the string "Tenured" in their detailed description: # ugc -M 10 -F Tenured java 6004

## FIELDS
START

The start time of the GC, in seconds from the beginning of the trace.

TIME

The duration of the garbage collection event.

DESCRIPTION

The runtime-provided description of this garbage collection event.

## OVERHEAD
Garbage collection events, even if frequent, should not produce a considerable overhead when traced because they are still not very common. Even hundreds of GCs per second (which is a very high rate) will still produce a fairly negligible overhead.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _example.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein

## SEE ALSO
trace(8), ustat(8), uobjnew(8)


bcc-javaobjnew

title: bcc-javaobjnew(8)

summary: Summarize object allocations in

text:
# bcc-javaobjnew(8)

## NAME
uobjnew, cobjnew, javaobjnew, rubyobjnew, tclobjnew - Summarize object allocations in high-level languages.

## SYNOPSIS
cobjnew [-h] [-C TOP_COUNT] [-S TOP_SIZE] [-v] pid [interval]

javaobjnew [-h] [-C TOP_COUNT] [-S TOP_SIZE] [-v] pid [interval]

rubyobjnew [-h] [-C TOP_COUNT] [-S TOP_SIZE] [-v] pid [interval]

tclobjnew [-h] [-C TOP_COUNT] [-S TOP_SIZE] [-v] pid [interval]

uobjnew [-h] [-C TOP_COUNT] [-S TOP_SIZE] [-v] [-l {c,java,ruby,tcl}] pid [interval]

## DESCRIPTION
uobjnew traces object allocations in high-level languages (including "malloc") and prints summaries of the most frequently allocated types by number of objects or number of bytes.

This tool relies on USDT probes embedded in many high-level languages, such as C, Java, Ruby, and Tcl. It requires a runtime instrumented with these probes, which in some cases requires building from source with a USDT-specific flag, such as "--enable-dtrace" or "--with-dtrace". For Java, the Java process must be started with the "-XX:+ExtendedDTraceProbes" flag.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-C TOP_COUNT

Print the top object types sorted by number of instances.

-S TOP_SIZE

Print the top object types sorted by size.

-v

Print the resulting BPF program, for debugging purposes.

{c,java,ruby,tcl}

The language to trace.

pid

The process id to trace.

interval

Wait this many seconds and then print the summary and exit. By default, wait for Ctrl+C to exit.

## EXAMPLES
Trace object allocations in a Ruby process:

\# uobjnew ruby 148

Trace object allocations from "malloc" and print the top 10 by total size:

\# uobjnew -S 10 c 1788

## FIELDS
TYPE

The object type being allocated. For C (malloc), this is the block size.

ALLOCS

The number of objects allocated.

BYTES

The number of bytes allocated.

## OVERHEAD
Object allocation events are quite frequent, and therefore the overhead from running this tool can be considerable. Use with caution and make sure to test before using in a production environment. Nonetheless, even thousands of allocations per second will likely produce a reasonable overhead when investigating a problem.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _example.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein

## SEE ALSO
ustat(8), ugc(8), memleak(8)


bcc-javastat

title: bcc-javastat(8)

summary: Activity stats from

text:
# bcc-javastat(8)

## NAME
ustat, javastat, nodestat, perlstat, phpstat, pythonstat, rubystat, tclstat - Activity stats from high-level languages.

## SYNOPSIS
javastat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

nodestat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

perlstat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

phpstat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

pythonstat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

rubystat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

tclstat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

ustat [-l {java,node,perl,php,python,ruby,tcl}] [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

## DESCRIPTION
This is "top" for high-level language events, such as garbage collections, exceptions, thread creations, object allocations, method calls, and more. The events are aggregated for each process and printed in a top-like table, which can be sorted by various fields. Not all language runtimes provide the same set of details.

This uses in-kernel eBPF maps to store per process summaries for efficiency.

This tool relies on USDT probes embedded in many high-level languages, such as Java, Node, Perl, PHP, Python, Ruby, and Tcl. It requires a runtime instrumented with these probes, which in some cases requires building from source with a USDT-specific flag, such as "--enable-dtrace" or "--with-dtrace". For Java, some probes are not enabled by default, and can be turned on by running the Java process with the "-XX:+ExtendedDTraceProbes" flag. For PHP processes, the environment variable USE_ZEND_DTRACE must be set to 1.

Newly-created processes will only be traced at the next interval. If you run this tool with a short interval (say, 1-5 seconds), this should be virtually unnoticeable. For longer intervals, you might miss processes that were started and terminated during the interval window.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-l {java,node,perl,php,python,ruby,tcl}

The language to trace. By default, all languages are traced.

-C

Do not clear the screen between updates.

-S {cload,excp,gc,method,objnew,thread}

Sort the output by the specified field.

-r MAXROWS

Do not print more than this number of rows.

-d

Print the resulting BPF program, for debugging purposes.

interval

Interval between updates, seconds.

count

Number of interval summaries.

## EXAMPLES
Summarize activity in high-level languages, 1 second refresh:

\# ustat

Don't clear the screen, and top 8 rows only:

\# ustat -Cr 8

5 second summaries, 10 times only:

\# ustat 5 10

## FIELDS
loadavg

The contents of /proc/loadavg

PID

Process ID.

CMDLINE

Process command line (often the second and following arguments will give you a hint as to which application is being run.

METHOD/s

Count of method invocations during interval.

GC/s

Count of garbage collections during interval.

OBJNEW/s

Count of objects allocated during interval.

CLOAD/s

Count of classes loaded during interval.

EXC/s

Count of exceptions thrown during interval.

THR/s

Count of threads created during interval.

## OVERHEAD
When using this tool with high-frequency events, such as method calls, a very significant slow-down can be expected. However, many of the high-level languages covered by this tool already have a fairly high per-method invocation cost, especially when running in interpreted mode. For the lower-frequency events, such as garbage collections or thread creations, the overhead should not be significant. Specifically, when probing Java processes and not using the "-XX:+ExtendedDTraceProbes" flag, the most expensive probes are not emitted, and the overhead should be acceptable.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _example.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein

## SEE ALSO
trace(8), argdist(8), tplist(8)


bcc-javathreads

title: bcc-javathreads(8)

text:
# bcc-javathreads(8)

## NAME
uthreads, cthreads, javathreads - Trace thread creation events in Java or pthreads.

## SYNOPSIS
cthreads [-h] [-v] pid javathreads [-h] [-v] pid uthreads [-h] [-l {c,java,none}] [-v] pid

## DESCRIPTION
This traces thread creation events in Java processes, or pthread creation events in any process. When a thread is created, its name or start address is printed.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-l {c,java,none}

The language to trace. C and none select tracing pthreads only, regardless of the runtime being traced.

-v

Print the resulting BPF program, for debugging purposes.

pid

The process id to trace.

## EXAMPLES
Trace Java thread creations:

\# uthreads -l java 148

Trace pthread creations:

\# uthreads 1802

## FIELDS
TIME

The event's time in seconds from the beginning of the trace.

ID

The thread's ID. The information in this column depends on the runtime.

TYPE

Event type -- thread start, stop, or pthread event.

DESCRIPTION

The thread's name or start address function name.

## OVERHEAD
Thread start and stop events are usually not very frequent, which makes this tool's overhead negligible.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _example.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein

## SEE ALSO
ustat(8), trace(8)


bcc-killsnoop

title: bcc-killsnoop(8)

text:
# bcc-killsnoop(8)

## NAME
killsnoop - Trace signals issued by the kill() syscall. Uses Linux eBPF/bcc.

## SYNOPSIS
killsnoop [-h] [-x] [-p PID] [-T PID] [-s SIGNAL]

## DESCRIPTION
killsnoop traces the kill() syscall, to show signals sent via this method. This may be useful to troubleshoot failing applications, where an unknown mechanism is sending signals.

This works by tracing the kernel sys_kill() function using dynamic tracing, and will need updating to match any changes to this function.

This makes use of a Linux 4.4 feature (bpf_perf_event_output()); for kernels older than 4.4, see the version under tools/old, which uses an older mechanism.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print usage message.

-x

Only print failed kill() syscalls.

-p PID

Trace this process ID only which is the sender of signal (filtered in-kernel).

-T PID

Trace this target process ID only which is the receiver of signal (filtered in-kernel).

-s SIGNAL

Trace this signal only (filtered in-kernel).

## EXAMPLES
Trace all kill() syscalls:

\# killsnoop

Trace only kill() syscalls that failed:

\# killsnoop -x

Trace PID 181 only:

\# killsnoop -p 181

Trace target PID 189 only:

\# killsnoop -T 189

Trace signal 9 only:

\# killsnoop -s 9

## FIELDS
TIME

Time of the kill call.

PID

Source process ID

COMM

Source process name

SIG

Signal number. See signal(7).

TPID

Target process ID

RES

Result. 0 == success, a negative value (of the error code) for failure.

## OVERHEAD
This traces the kernel kill function and prints output for each event. As the rate of this is generally expected to be low (< 100/s), the overhead is also expected to be negligible. If you have an application that is calling a very high rate of kill()s for some reason, then test and understand overhead before use.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
opensnoop(8), funccount(8)


bcc-klockstat

title: bcc-klockstat(8)

text:
# bcc-klockstat(8)

## NAME
klockstat - Traces kernel mutex lock events and display locks statistics. Uses Linux eBPF/bcc.

## SYNOPSIS
klockstat [-h] [-i] [-n] [-s] [-c] [-S FIELDS] [-p] [-t] [-d DURATION]

## DESCRIPTION
klockstat traces kernel mutex lock events and display locks statistics and displays following data:

Caller Avg Spin Count Max spin Total spin

psi_avgs_work+0x2e 3675 5 5468 18379

flush_to_ldisc+0x22 2833 2 4210 5667

n_tty_write+0x30c 3914 1 3914 3914

isig+0x5d 2390 1 2390 2390

tty_buffer_flush+0x2a 1604 1 1604 1604

commit_echoes+0x22 1400 1 1400 1400

n_tty_receive_buf_common+0x3b9 1399 1 1399 1399

Caller Avg Hold Count Max hold Total hold

flush_to_ldisc+0x22 42558 2 76135 85116

psi_avgs_work+0x2e 14821 5 20446 74106

n_tty_receive_buf_common+0x3b9 12300 1 12300 12300

n_tty_write+0x30c 10712 1 10712 10712

isig+0x5d 3362 1 3362 3362

tty_buffer_flush+0x2a 3078 1 3078 3078

commit_echoes+0x22 3017 1 3017 3017

Every caller to using kernel's mutex is displayed on every line.

First portion of lines show the lock acquiring data, showing the amount of time it took to acquired given lock.

'Caller' - symbol acquiring the mutex

'Average Spin' - average time to acquire the mutex

'Count' - number of times mutex was acquired

'Max spin' - maximum time to acquire the mutex

'Total spin' - total time spent in acquiring the mutex

Second portion of lines show the lock holding data, showing the amount of time it took to hold given lock.

'Caller' - symbol holding the mutex

'Average Hold' - average time mutex was held

'Count' - number of times mutex was held

'Max hold' - maximum time mutex was held

'Total hold' - total time spent in holding the mutex

This works by tracing mutex_lock/unlock kprobes, updating the lock stats in maps and processing them in the python part.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print usage message.

-i SEC

print summary at this interval (seconds)

-c CALLER

print locks taken by given caller filter (e.g., pipe_)

-S FIELDS

sort data on specific columns defined by FIELDS string (by default the data is sorted by Max columns) FIELDS string contains 1 or 2 fields describing columns to sort on for both acquiring and holding data. Following fields are available:

acq_max for 'Max spin' column

acq_total for 'Total spin' column

acq_count for 'Count' column

hld_max for 'Max hold' column

hld_total for 'Total hold' column

hld_count for 'Count' column

See EXAMPLES.

-n COUNT

print COUNT number of locks

-s COUNT

print COUNT number of stack entries

-p PID

Trace this process ID only (filtered in-kernel).

-t TID

Trace this thread ID only (filtered in-kernel).

-d DURATION

Total duration of trace in seconds.

--stack-storage-size STACK_STORAGE_SIZE

Change the number of unique stack traces that can be stored and displayed.

## EXAMPLES
Sort lock acquired results on acquired count:

\# klockstat -S acq_count

Sort lock held results on total held time:

\# klockstat -S hld_total

Combination of above:

\# klockstat -S acq_count,hld_total

Trace system wide:

\# klockstat

Trace for 5 seconds only:

\# klockstat -d 5

Display stats every 5 seconds

\# klockstat -i 5

Trace locks for PID 123:

\# klockstat -p 123

Trace locks for TID 321:

\# klockstat -t 321

Display stats only for lock callers with 'pipe_' substring

\# klockstat -c pipe_

Display 3 locks:

\# klockstat -n 3

Display 10 levels of stack for the most expensive lock:

\# klockstat -n 1 -s 10 Tracing lock events... Hit Ctrl-C to end. ^C

Caller Avg Spin Count Max spin Total spin

pipe_wait+0xa9 670 397691 17273 266775437

pipe_wait+0xa9

pipe_read+0x206

new_sync_read+0x12a

vfs_read+0x9d

ksys_read+0x5f

do_syscall_64+0x5b

entry_SYSCALL_64_after_hwframe+0x44

Caller Avg Hold Count Max hold Total hold

flush_to_ldisc+0x22 28381 3 65484 85144

flush_to_ldisc+0x22

process_one_work+0x1b0

worker_thread+0x50

kthread+0xfb

ret_from_fork+0x35

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## CREDITS
This tool is based on work of David Valin <dvalin@redhat.com> and his script.

## AUTHOR
Jiri Olsa


bcc-ksnoop

title: bcc-ksnoop(8)

text:
# bcc-ksnoop(8)

## NAME
KSNOOP - tool for tracing kernel function entry/return showing arguments/return values

## SYNOPSIS
ksnoop [OPTIONS] { COMMAND FUNC | help } OPTIONS := { { -V | --version } | { -h | --help } | { [-P | --pages] nr_pages} | { [-p | --pid] pid} | [{ -s | --stack }] | [{ -d | --debug }] }

COMMAND := { trace | info }

FUNC := { name | name(arg[,**arg]) }

## DESCRIPTION
ksnoop allows for inspection of arguments and return values associated with function entry/return. ksnoop info FUNC

Show function description, arguments and return value types.

ksnoop trace FUNC [FUNC]

Trace function entry and return, showing arguments and return values. A function name can simply be specified, or a function name along with named arguments, return values. return is used to specify the return value.

ksnoop requires the kernel to provide BTF for itself, and if tracing of module data is required, module BTF must be present also. Check /sys/kernel/btf to see if BTF is present.

ksnoop requires CAP_BPF and CAP_TRACING capabilities.

## OPTIONS
-h, --help

Show help information

-V, --version

Show version.

-d, --debug

Show debug output.

-p, --pid

Filter events by pid.

-P, --pages

Specify number of pages used per-CPU for perf event collection. Default is 8.

-s, --stack

Specified set of functions are traced if and only if they are encountered in the order specified.

## EXAMPLES
\# ksnoop info ip_send_skb

```
int  ip_send_skb(struct net  * net, struct sk_buff  * skb);
```

Show function description.

\# ksnoop trace ip_send_skb

```
          TIME  CPU      PID FUNCTION/ARGS
78101668506811    1     2813 ip_send_skb(

                                 net = *(0xffffffffb5959840)

                                  (struct net){

                                   .passive = (refcount_t){

                                    .refs = (atomic_t){

                                     .counter = (int)0x2,

                                    },

                                   },

                                   .dev_base_seq = (unsigned int)0x18,

                                   .ifindex = (int)0xf,

                                   .list = (struct list_head){

                                    .next = (struct list_head *)0xffff9895440dc120,

                                    .prev = (struct list_head *)0xffffffffb595a8d0,

                                   },

                                 ...
79561322965250    1     2813 ip_send_skb(

                                 return =

                                  (int)0x0

                             );
```

Show entry/return for ip_send_skb() with arguments, return values.

\# ksnoop trace "ip_send_skb(skb)"

```
         TIME  CPU      PID FUNCTION/ARGS
78142420834537    1     2813 ip_send_skb(

                                 skb = *(0xffff989750797c00)

                                  (struct sk_buff){

                                   (union){

                                    .sk = (struct sock *)0xffff98966ce19200,

                                    .ip_defrag_offset = (int)0x6ce19200,

                                   },

                                   (union){

                                    (struct){

                                     ._skb_refdst = (long unsigned int)0xffff98981dde2d80,

                                     .destructor = (void (*)(struct sk_buff *))0xffffffffb3e1beb0,

                                    },

                                ...
```

Show entry argument skb.

\# ksnoop trace "ip_send_skb(return)"

```
         TIME  CPU      PID FUNCTION/ARGS
78178228354796    1     2813 ip_send_skb(

                                 return =

                                  (int)0x0

                             );
```

Show return value from ip_send_skb().

\# ksnoop trace "ip_send_skb(skb->sk)"

```
          TIME  CPU      PID FUNCTION/ARGS
78207649138829    2     2813 ip_send_skb(

                                 skb->sk = *(0xffff98966ce19200)

                                  (struct sock){

                                   .__sk_common = (struct sock_common){

                                    (union){

                                     .skc_addrpair = (__addrpair)0x1701a8c017d38f8d,

                                     (struct){

                                      .skc_daddr = (__be32)0x17d38f8d,

                                      .skc_rcv_saddr = (__be32)0x1701a8c0,

                                     },

                                    },

                                  ...
```

Trace member information associated with argument. Only one level of membership is supported.

\# ksnoop -p 2813 "ip_rcv(dev)"

```
          TIME  CPU      PID FUNCTION/ARGS
78254803164920    1     2813 ip_rcv(

                                 dev = *(0xffff9895414cb000)

                                  (struct net_device){

                                   .name = (char[16])[

                                    'l',

                                    'o',

                                   ],

                                   .name_node = (struct netdev_name_node *)0xffff989541515ec0,

                                   .state = (long unsigned int)0x3,

                                 ...
```

Trace dev argument of ip_rcv(). Specify process id 2813 for events for that process only.

\# ksnoop -s tcp_sendmsg __tcp_transmit_skb ip_output

```
         TIME  CPU      PID FUNCTION/ARGS
71827770952903    1     4777 __tcp_transmit_skb(

                                 sk = *(0xffff9852460a2300)

                                  (struct sock){

                                   .__sk_common = (struct sock_common){

                                    (union){

                                     .skc_addrpair = (__addrpair)0x61b2af0a35cbfe0a,
```

Trace entry/return of tcp_sendmsg, __tcp_transmit_skb and ip_output when tcp_sendmsg leads to a call to __tcp_transmit_skb and that in turn leads to a call to ip_output; i.e. with a call graph matching the order specified. The order does not have to be direct calls, i.e. function A can call another function that calls function B.

\# ksnoop "ip_send_skb(skb->len > 100, skb)"

```
          TIME  CPU      PID FUNCTION/ARGS
39267395709745    1     2955 ip_send_skb(

                                 skb->len =

                                  (unsigned int)0x89,

                                 skb = *(0xffff89c8be81e500)

                                  (struct sk_buff){

                                   (union){

                                    .sk = (struct sock *)0xffff89c6c59e5580,

                                    .ip_defrag_offset = (int)0xc59e5580,

                                   },
```

Trace ip_send_skb() skbs which have len > 100.

## SEE ALSO
bpf(2),


bcc-kvmexit

title: bcc-kvmexit(8)

text:
# bcc-kvmexit(8)

## NAME
kvmexit - Display the exit_reason and its statistics of each vm exit.

## SYNOPSIS
kvmexit [-h] [-p PID [-v VCPU | -a] ] [-t TID | -T 'TID1,TID2'] [duration]

## DESCRIPTION
Considering virtual machines' frequent exits can cause performance problems, this tool aims to locate the frequent exited reasons and then find solutions to reduce or even avoid the exit, by displaying the detail exit reasons and the counts of each vm exit for all vms running on one physical machine.

This tool uses a PERCPU_ARRAY: pcpuArrayA and a percpu_hash: hashA to collaboratively store each kvm exit reason and its count. The reason is there exists a rule when one vcpu exits and re-enters, it tends to continue to run on the same physical cpu as the last cycle, which is also called 'cache hit'. Thus we turn to use a PERCPU_ARRAY to record the 'cache hit' situation to speed things up; and for other cases, then use a percpu_hash.

As RAW_TRACEPOINT_PROBE(kvm_exit) consumes less cpu cycles, when this tool is used, it firstly tries to employ raw tracepoints in modules, and if failes, then fall back to regular tracepoint.

Limitation: In view of the hardware-assisted virtualization technology of different architectures, currently we only adapt on vmx in intel.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

This also requires Linux 4.7+ (BPF_PROG_TYPE_TRACEPOINT support).

## OPTIONS
-h

Print usage message.

-p PID

Display process with this PID only, collpase all tids with exit reasons sorted in descending order.

-v VCPU

Display this VCPU only for this PID.

-a ALLTIDS

Display all TIDS for this PID.

-t TID

Display thread with this TID only with exit reasons sorted in descending order.

-T 'TID1,TID2'

Display threads for a union like {395490, 395491}.

duration

Duration of display, after sleeping several seconds.

## EXAMPLES
Display kvm exit reasons and statistics for all threads... Hit Ctrl-C to end:

\# kvmexit

Display kvm exit reasons and statistics for all threads after sleeping 6 secs:

\# kvmexit 6

Display kvm exit reasons and statistics for PID 1273795 after sleeping 5 secs:

\# kvmexit -p 1273795 5

Display kvm exit reasons and statistics for PID 1273795 and its all threads after sleeping 5 secs:

\# kvmexit -p 1273795 5 -a

Display kvm exit reasons and statistics for PID 1273795 VCPU 0... Hit Ctrl-C to end:

\# kvmexit -p 1273795 -v 0

Display kvm exit reasons and statistics for PID 1273795 VCPU 0 after sleeping 4 secs:

\# kvmexit -p 1273795 -v 0 4

Display kvm exit reasons and statistics for TID 1273819 after sleeping 10 secs:

\# kvmexit -t 1273819 10

Display kvm exit reasons and statistics for TIDS ['1273820', '1273819']... Hit Ctrl-C to end:

\# kvmexit -T '1273820,1273819'

## OVERHEAD
This traces the "kvm_exit" kernel function, records the exit reason and calculates its counts. Contrast with filling more vm-exit reason debug entries, this tool is more easily and flexibly: the bcc python logic could provide nice kernel aggregation and custom output, the bpf in-kernel percpu_array and percpu_cache further improves performance.

The impact of using this tool on the host should be negligible. While this tool is very efficient, it does affect the guest virtual machine itself, the average test results on guest vm are as follows:

| cpu cycles

no TP | 1127

regular TP | 1277 (13% downgrade)

RAW TP | 1187 (5% downgrade)

Host: echo 1 > /proc/sys/net/core/bpf_jit_enable

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Fei Li <lifei.shirley@bytedance.com>


bcc-llcstat

title: bcc-llcstat(8)

text:
# bcc-llcstat(8)

## NAME
llcstat - Summarize CPU cache references and misses by process. Uses Linux eBPF/bcc.

## SYNOPSIS
llcstat [-h] [-c SAMPLE_PERIOD] [duration]

## DESCRIPTION
llcstat instruments CPU cache references and cache misses system-side, and summarizes them by PID and CPU. These events have different meanings on different architecture. For x86-64, they mean misses and references to LLC. This can be useful to locate and debug performance issues caused by cache hit rate.

This works by sampling corresponding events defined in uapi/linux/perf_event.h, namely PERF_COUNT_HW_CACHE_REFERENCES and PERF_COUNT_HW_CACHE_MISSES, using BPF perf event tracing. Upon each sampled event, the attached BPF program records the PID and CPU ID on which the event happened, and stores it in table.

This makes use of a Linux 4.9 feature (BPF_PROG_TYPE_PERF_EVENT).

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print usage message.

-c SAMPLE_PERIOD

Sample one in this many cache reference and cache miss events.

-t

Summarize cache references and misses by PID/TID

duration

Duration to trace, in seconds.

## EXAMPLES
Sample one in 100 events, trace for 20 seconds:

\# llcstat -c 100 20

## FIELDS
PID

Process ID

NAME

Process name

CPU

CPU ID

REFERENCE

Number of cache reference events

MISS

Number of cache miss events

HIT%

Cache hit ratio

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Teng Qin

## SEE ALSO
Perf can be used as a generic event counter tool. An example for LLC:

\# perf top -e cache-misses -e cache-references -a -ns pid,cpu,comm


bcc-mdflush

title: bcc-mdflush(8)

text:
# bcc-mdflush(8)

## NAME
mdflush - Trace md flush events. Uses Linux eBPF/bcc.

## SYNOPSIS
mdflush

## DESCRIPTION
This tool traces flush events by md, the Linux multiple device driver (software RAID). The timestamp and md device for the flush are printed. Knowing when these flushes happen can be useful for correlation with unexplained spikes in disk latency.

This works by tracing the kernel md_flush_request() function using dynamic tracing, and will need updating to match any changes to this function.

Note that the flushes themselves are likely to originate from higher in the I/O stack, such as from the file systems.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## EXAMPLES
Trace md flush events:

\# mdflush

## FIELDS
TIME

Time of the flush event (HH:MM:SS).

PID

The process ID that was on-CPU when the event was issued. This may identify the cause of the flush (eg, the "sync" command), but will often identify a kernel worker thread that was managing I/O.

COMM

The command name for the PID.

DEVICE

The md device name.

## OVERHEAD
Expected to be negligible.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
biosnoop(8)


bcc-memleak

title: bcc-memleak(8)

text:
# bcc-memleak(8)

## NAME
memleak - Print a summary of outstanding allocations and their call stacks to detect memory leaks. Uses Linux eBPF/bcc.

## SYNOPSIS
memleak [-h] [-p PID] [-t] [-a] [-o OLDER] [-c COMMAND] [--combined-only] [--wa-missing-free] [-s SAMPLE_RATE] [-T TOP] [-z MIN_SIZE] [-Z MAX_SIZE] [-O OBJ] [INTERVAL] [COUNT]

## DESCRIPTION
memleak traces and matches memory allocation and deallocation requests, and collects call stacks for each allocation. memleak can then print a summary of which call stacks performed allocations that weren't subsequently freed.

When tracing a specific process, memleak instruments a list of allocation functions from libc, specifically: malloc, calloc, realloc, posix_memalign, valloc, memalign, pvalloc, aligned_alloc, and free. When tracing all processes, memleak instruments kmalloc/kfree, kmem_cache_alloc/kmem_cache_free, and also page allocations made by get_free_pages/free_pages.

memleak may introduce significant overhead when tracing processes that allocate and free many blocks very quickly. See the OVERHEAD section below.

This tool only works on Linux 4.6+. Stack traces are obtained using the new BPF_STACK_TRACE` APIs. For kernels older than 4.6, see the version under tools/old. Kernel memory allocations are intercepted through tracepoints, which are available on Linux 4.7+.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print usage message.

-p PID

Trace this process ID only (filtered in-kernel). This traces libc allocator.

-t

Print a trace of all allocation and free requests and results.

-a

Print a list of allocations that weren't freed (and their sizes) in addition to their call stacks.

-o OLDER

Print only allocations older than OLDER milliseconds. Useful to remove false positives. The default value is 500 milliseconds.

-c COMMAND

Run the specified command and trace its allocations only. This traces libc allocator.

--combined-only

Use statistics precalculated in kernel space. Amount of data to be pulled from kernel significantly decreases, at the cost of losing capabilities of time-based false positives filtering (-o).

--wa-missing-free

Make up the action of free to alleviate misjudgments when free is missing.

-s SAMPLE_RATE

Record roughly every SAMPLE_RATE-th allocation to reduce overhead.

-t TOP

Print only the top TOP stacks (sorted by size). The default value is 10.

-z MIN_SIZE

Capture only allocations that are larger than or equal to MIN_SIZE bytes.

-Z MAX_SIZE

Capture only allocations that are smaller than or equal to MAX_SIZE bytes.

-O OBJ

Attach to allocation functions in specified object instead of resolving libc. Ignored when kernel allocations are profiled.

INTERVAL

Print a summary of outstanding allocations and their call stacks every INTERVAL seconds. The default interval is 5 seconds.

COUNT

Print the outstanding allocations summary COUNT times and then exit.

## EXAMPLES
Print outstanding kernel allocation stacks every 3 seconds:

\# memleak 3

Print user outstanding allocation stacks and allocation details for the process 1005:

\# memleak -p 1005 -a

Sample roughly every 5th allocation (~20%) of the call stacks and print the top 5

stacks 10 times before quitting. # memleak -s 5 --top=5 10

Run ./allocs and print outstanding allocation stacks for that process:

\# memleak -c ./allocs

Capture only allocations between 16 and 32 bytes in size:

\# memleak -z 16 -Z 32

## OVERHEAD
memleak can have significant overhead if the target process or kernel performs allocations at a very high rate. Pathological cases may exhibit up to 100x degradation in running time. Most of the time, however, memleak shouldn't cause a significant slowdown. You can use the -s switch to reduce the overhead further by capturing only every N-th allocation. The -z and -Z switches can also reduce overhead by capturing only allocations of specific sizes.

Additionally, option --combined-only saves processing time by reusing already calculated allocation statistics from kernel. It's faster, but lacks information about particular allocations.

Also, option --wa-missing-free makes memleak more accuracy in the complicated environment.

To determine the rate at which your application is calling malloc/free, or the rate at which your kernel is calling kmalloc/kfree, place a probe with perf and collect statistics. For example, to determine how many calls to __kmalloc are placed in a typical period of 10 seconds:

\# perf probe '__kmalloc'

\# perf stat -a -e 'probe:__kmalloc' -- sleep 10

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein


bcc-mountsnoop

title: bcc-mountsnoop(8)

text:
# bcc-mountsnoop(8)

## NAME
mountsnoop - Trace mount() and umount() syscalls. Uses Linux eBPF/bcc.

## SYNOPSIS
mountsnoop

## DESCRIPTION
mountsnoop traces the mount() and umount() syscalls, showing which processes are mounting and unmounting filesystems in what mount namespaces. This can be useful for troubleshooting system and container setup.

This works by tracing the kernel sys_mount() and sys_umount() functions using dynamic tracing, and will need updating to match any changes to this function.

This makes use of a Linux 4.8 feature (bpf_get_current_task()).

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## FIELDS
COMM

Process name

PID

Process ID

TID

Thread ID

MNT_NS

Mount namespace inode number

CALL

System call, arguments, and return value

## OVERHEAD
This traces the kernel mount and umount functions and prints output for each event. As the rate of these calls is generally expected to be very low, the overhead is also expected to be negligible. If your system calls mount() and umount() at a high rate, then test and understand overhead before use.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Omar Sandoval

## SEE ALSO
mount(2) umount(2)


title: bcc-mysqld_qslower(8)

text:
# bcc-mysqld_qslower(8)

## NAME
mysqld_qslower - Trace MySQL server queries slower than a threshold.

## SYNOPSIS
mysqld_qslower PID [min_ms]

## DESCRIPTION
This traces queries served by a MySQL server, and prints those that exceed a custom latency (query duration) threshold. By default, a minimum threshold of 1 millisecond is used. If a threshold of 0 is used, all queries are printed.

This uses User Statically-Defined Tracing (USDT) probes, a feature added to MySQL for DTrace support, but which may not be enabled on a given MySQL installation. See requirements.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF, bcc, and MySQL server with USDT probe support (when configuring the build: -DENABLE_DTRACE=1).

## OPTIONS
PID Trace this mysqld PID.

min_ms

Minimum query latency (duration) to trace, in milliseconds. Default is 1 ms.

## EXAMPLES
Trace MySQL server queries slower than 1 ms for PID 1981:

\# mysqld_qslower 1981

Trace slower than 10 ms for PID 1981:

\# mysqld_qslower 1981 10

## FIELDS
TIME(s)

Time of query start, in seconds.

PID

Process ID of the traced server.

MS

Milliseconds for the query, from start to end.

QUERY

Query string, truncated to 128 characters.

## OVERHEAD
This adds low-overhead instrumentation to MySQL queries, and only emits output data from kernel to user-level if they query exceeds the threshold. If the server query rate is less than 1,000/sec, the overhead is expected to be negligible. If the query rate is higher, test to gauge overhead.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
biosnoop(8)


bcc-netqtop

title: bcc-netqtop(8)

text:
# bcc-netqtop(8)

## NAME
netqtop - Summarize PPS, BPS, average size of packets and packet counts ordered by packet sizes on each queue of a network interface.

## SYNOPSIS
netqtop [-n nic] [-i interval] [-t throughput]

## DESCRIPTION
netqtop accounts statistics of both transmitted and received packets on each queue of a specified network interface to help developers check if its traffic load is balanced. The result is displayed as a table with columns of PPS, BPS, average size and packet counts in range [0,64), [64, 5120), [512, 2048), [2048, 16K), [16K, 64K). This is printed every given interval (default 1) in seconds.

The tool uses the net:net_dev_start_xmit and net:netif_receive_skb kernel tracepoints. Since it uses tracepoint, the tool only works on Linux 4.7+.

netqtop introduces significant overhead while network traffic is large. See OVERHEAD section below.

## REQUIREMENTS
CONFIG_bpf and bcc

## OPTIONS
-n NIC

Specify the network interface card

-i INTERVAL

Print results every INTERVAL seconds. The default value is 1.

-t THROUGHPUT

Print BPS and PPS of each queue.

## EXAMPLES
Account statistics of eth0 and output every 2 seconds:

\# netqtop -n eth0 -i 1

## OVERHEAD
In performance test, netqtop introduces a overhead up to 30% PPS drop while printing interval is set to 1 second. So be mindful of potential packet drop when using this tool.

It also increases ping-pong latency by about 1 usec.

## SOURCE
This is from bcc

https://github.com/iovisor/bcc

Also look in the bcc distribution for a netqtop_example.txt file containing example usage, output and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development

## AUTHOR
Yolandajn


bcc-nfsdist

title: bcc-nfsdist(8)

text:
# bcc-nfsdist(8)

## NAME
nfsdist - Summarize NFS operation latency. Uses Linux eBPF/bcc.

## SYNOPSIS
nfsdist [-h] [-T] [-m] [-p PID] [interval] [count]

## DESCRIPTION
This tool summarizes time (latency) spent in common NFS file operations: reads, writes, opens, and getattrs, and presents it as a power-of-2 histogram. It uses an in-kernel eBPF map to store the histogram for efficiency.

Since this works by tracing the nfs_file_operations interface functions, it will need updating to match any changes to these functions.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print usage message.

-T

Don't include timestamps on interval output.

-m

Output in milliseconds.

-p PID

Trace this PID only.

## EXAMPLES
Trace NFS operation time, and print a summary on Ctrl-C:

\# nfsdist

Trace PID 181 only:

\# nfsdist -p 181

Print 1 second summaries, 10 times:

\# nfsdist 1 10

1 second summaries, printed in milliseconds

\# nfsdist -m 1

## FIELDS
msecs

Range of milliseconds for this bucket.

usecs

Range of microseconds for this bucket.

count

Number of operations in this time range.

distribution

ASCII representation of the distribution (the count column).

## OVERHEAD
This adds low-overhead instrumentation to these NFS operations, including reads and writes from the file system cache. Such reads and writes can be very frequent (depending on the workload; eg, 1M/sec), at which point the overhead of this tool may become noticeable. Measure and quantify before use.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Samuel Nair

## SEE ALSO
nfsslower(8)


bcc-nfsslower

title: bcc-nfsslower(8)

text:
# bcc-nfsslower(8)

## NAME
nfsslower - Trace slow NFS file operations, with per-event details.

## SYNOPSIS
nfsslower [-h] [-j] [-p PID] [min_ms]

## DESCRIPTION
This tool traces common NFSv3 & NFSv4 file operations: reads, writes, opens, and getattrs. It measures the time spent in these operations, and prints details for each that exceeded a threshold.

WARNING: See the OVERHEAD section.

By default, a minimum millisecond threshold of 10 is used. If a threshold of 0 is used, all events are printed (warning: verbose).

Since this works by tracing the nfs_file_operations interface functions, it will need updating to match any changes to these functions.

This tool uses kprobes to instrument the kernel for entry and exit information, in the future a preferred way would be to use tracepoints. Currently there aren't any tracepoints available for nfs_read_file, nfs_write_file and nfs_open_file, nfs_getattr does have entry and exit tracepoints but we chose to use kprobes for consistency

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-p PID Trace this PID only.

-j

Trace output in CSV format.

min_ms

Minimum I/O latency (duration) to trace, in milliseconds. Default is 10 ms.

## EXAMPLES
Trace synchronous file reads and writes slower than 10 ms:

\# nfsslower

Trace slower than 1 ms:

\# nfsslower 1

Trace slower than 1 ms, and output just the fields in parsable format (CSV):

\# nfsslower -j 1

Trace all file reads and writes (warning: the output will be verbose):

\# nfsslower 0

Trace slower than 1 ms, for PID 181 only:

\# nfsslower -p 181 1

## FIELDS
TIME(s)

Time of I/O completion since the first I/O seen, in seconds.

COMM

Process name.

PID

Process ID.

T

Type of operation. R == read, W == write, O == open, G == getattr.

OFF_KB

File offset for the I/O, in Kbytes.

BYTES

Size of I/O, in bytes.

LAT(ms)

Latency (duration) of I/O, measured from when it was issued by VFS to the filesystem, to when it completed. This time is inclusive of RPC latency, network latency, cache lookup, remote fileserver processing latency, etc. Its a more accurate measure of the latency suffered by applications performing NFS read/write calls to a fileserver.

FILENAME

A cached kernel file name (comes from dentry->d_name.name).

ENDTIME_us

Completion timestamp, microseconds (-j only).

OFFSET_b

File offset, bytes (-j only).

LATENCY_us

Latency (duration) of the I/O, in microseconds (-j only).

## OVERHEAD
This adds low-overhead instrumentation to NFS operations, including reads and writes from the file system cache. Such read, writes and particularly getattrs can be very frequent (depending on the workload; eg, 1M/sec), at which point the overhead of this tool (even if it prints no "slower" events) can begin to become significant. Measure and quantify before use. If this continues to be a problem, consider switching to a tool that prints in-kernel summaries only. This tool has been tested with NFSv3 & NVSv4, but it might work with NFSv{1,2}, since it is tracing the generic functions from nfs_file_operations.

Note that the overhead of this tool should be less than fileslower(8), as this tool targets NFS functions only, and not all file read/write paths.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion nfsslower_examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Samuel Nair

## SEE ALSO
biosnoop(8), funccount(8), fileslower(8)


bcc-nodegc

title: bcc-nodegc(8)

summary: Trace garbage collection events in

text:
# bcc-nodegc(8)

## NAME
ugc, javagc, nodegc, pythongc, rubygc - Trace garbage collection events in high-level languages.

## SYNOPSIS
javagc [-h] [-v] [-m] [-M MINIMUM] [-F FILTER] pid

nodegc [-h] [-v] [-m] [-M MINIMUM] [-F FILTER] pid

pythongc [-h] [-v] [-m] [-M MINIMUM] [-F FILTER] pid

rubygc [-h] [-v] [-m] [-M MINIMUM] [-F FILTER] pid

ugc [-h] [-v] [-m] [-M MINIMUM] [-F FILTER] [-l {java,node,python,ruby}] pid

## DESCRIPTION
This traces garbage collection events as they occur, including their duration and any additional information (such as generation collected or type of GC) provided by the respective language's runtime.

This tool relies on USDT probes embedded in many high-level languages, such as Java, Node, Python, and Ruby. It requires a runtime instrumented with these probes, which in some cases requires building from source with a USDT-specific flag, such as "--enable-dtrace" or "--with-dtrace".

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-v

Print the resulting BPF program, for debugging purposes.

-m

Print times in milliseconds. The default is microseconds.

-M MINIMUM

Display only collections that are longer than this threshold. The value is given in milliseconds. The default is to display all collections.

-F FILTER

Display only collections whose textual description matches (contains) this string. The default is to display all collections. Note that the filtering here is performed in user-space, and not as part of the BPF program. This means that if you have thousands of collection events, specifying this filter will not reduce the amount of data that has to be transferred from the BPF program to the user-space script.

{java,node,python,ruby}

The language to trace.

pid

The process id to trace.

## EXAMPLES
Trace garbage collections in a specific Node process:

\# ugc node 148

Trace garbage collections in a specific Java process, and print GC times in

milliseconds: # ugc -m java 6004

Trace garbage collections in a specific Java process, and display them only if

they are longer than 10ms and have the string "Tenured" in their detailed description: # ugc -M 10 -F Tenured java 6004

## FIELDS
START

The start time of the GC, in seconds from the beginning of the trace.

TIME

The duration of the garbage collection event.

DESCRIPTION

The runtime-provided description of this garbage collection event.

## OVERHEAD
Garbage collection events, even if frequent, should not produce a considerable overhead when traced because they are still not very common. Even hundreds of GCs per second (which is a very high rate) will still produce a fairly negligible overhead.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _example.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein

## SEE ALSO
trace(8), ustat(8), uobjnew(8)


bcc-nodestat

title: bcc-nodestat(8)

summary: Activity stats from

text:
# bcc-nodestat(8)

## NAME
ustat, javastat, nodestat, perlstat, phpstat, pythonstat, rubystat, tclstat - Activity stats from high-level languages.

## SYNOPSIS
javastat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

nodestat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

perlstat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

phpstat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

pythonstat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

rubystat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

tclstat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

ustat [-l {java,node,perl,php,python,ruby,tcl}] [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

## DESCRIPTION
This is "top" for high-level language events, such as garbage collections, exceptions, thread creations, object allocations, method calls, and more. The events are aggregated for each process and printed in a top-like table, which can be sorted by various fields. Not all language runtimes provide the same set of details.

This uses in-kernel eBPF maps to store per process summaries for efficiency.

This tool relies on USDT probes embedded in many high-level languages, such as Java, Node, Perl, PHP, Python, Ruby, and Tcl. It requires a runtime instrumented with these probes, which in some cases requires building from source with a USDT-specific flag, such as "--enable-dtrace" or "--with-dtrace". For Java, some probes are not enabled by default, and can be turned on by running the Java process with the "-XX:+ExtendedDTraceProbes" flag. For PHP processes, the environment variable USE_ZEND_DTRACE must be set to 1.

Newly-created processes will only be traced at the next interval. If you run this tool with a short interval (say, 1-5 seconds), this should be virtually unnoticeable. For longer intervals, you might miss processes that were started and terminated during the interval window.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-l {java,node,perl,php,python,ruby,tcl}

The language to trace. By default, all languages are traced.

-C

Do not clear the screen between updates.

-S {cload,excp,gc,method,objnew,thread}

Sort the output by the specified field.

-r MAXROWS

Do not print more than this number of rows.

-d

Print the resulting BPF program, for debugging purposes.

interval

Interval between updates, seconds.

count

Number of interval summaries.

## EXAMPLES
Summarize activity in high-level languages, 1 second refresh:

\# ustat

Don't clear the screen, and top 8 rows only:

\# ustat -Cr 8

5 second summaries, 10 times only:

\# ustat 5 10

## FIELDS
loadavg

The contents of /proc/loadavg

PID

Process ID.

CMDLINE

Process command line (often the second and following arguments will give you a hint as to which application is being run.

METHOD/s

Count of method invocations during interval.

GC/s

Count of garbage collections during interval.

OBJNEW/s

Count of objects allocated during interval.

CLOAD/s

Count of classes loaded during interval.

EXC/s

Count of exceptions thrown during interval.

THR/s

Count of threads created during interval.

## OVERHEAD
When using this tool with high-frequency events, such as method calls, a very significant slow-down can be expected. However, many of the high-level languages covered by this tool already have a fairly high per-method invocation cost, especially when running in interpreted mode. For the lower-frequency events, such as garbage collections or thread creations, the overhead should not be significant. Specifically, when probing Java processes and not using the "-XX:+ExtendedDTraceProbes" flag, the most expensive probes are not emitted, and the overhead should be acceptable.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _example.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein

## SEE ALSO
trace(8), argdist(8), tplist(8)


bcc-numasched

title: bcc-numasched(8)

text:
# bcc-numasched(8)

## NAME
numasched - Tracing task switch NUMA. Uses bcc/eBPF.

## SYNOPSIS
numasched

## DESCRIPTION
numasched tracked task switch of NUMA.

This program is also a basic example of bcc and tracepoint.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-p, --pid PID

Trace this PID only.

-t, --tid TID

Trace this TID only.

-c, --comm COMM

Trace this COMM only.

## EXAMPLES
Tracing task switch NUMA:

\# numasched

Trace PID 181 only:

\# numasched -p 181

## FIELDS
TIME

A timestamp on the output, in "HH:MM:SS" format.

PID

The process ID.

TID

The thread ID.

SRC_NID

Source NUMA ID.

DST_NID

Target NUMA ID.

COMM

The process COMM.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Rong Tao <rongtao@cestc.cn>

## SEE ALSO
opensnoop(8)


bcc-offcputime

title: bcc-offcputime(8)

text:
# bcc-offcputime(8)

## NAME
offcputime - Summarize off-CPU time by kernel stack trace. Uses Linux eBPF/bcc.

## SYNOPSIS
offcputime [-h] [-p PID | -t TID | -u | -k] [-U | -K] [-d] [-f] [--stack-storage-size STACK_STORAGE_SIZE] [-m MIN_BLOCK_TIME] [-M MAX_BLOCK_TIME] [--state STATE] [duration]

## DESCRIPTION
This program shows stack traces and task names that were blocked and "off-CPU", and the total duration they were not running: their "off-CPU time". It works by tracing when threads block and when they return to CPU, measuring both the time they were off-CPU and the blocked stack trace and the task name. This data is summarized in the kernel using an eBPF map, and by summing the off-CPU time by unique stack trace and task name.

The output summary will help you identify reasons why threads were blocking, and quantify the time they were off-CPU. This spans all types of blocking activity: disk I/O, network I/O, locks, page faults, involuntary context switches, etc.

This is complementary to CPU profiling (e.g., CPU flame graphs) which shows the time spent on-CPU. This shows the time spent off-CPU, and the output, especially the -f format, can be used to generate an "off-CPU time flame graph".

See http://www.brendangregg.com/FlameGraphs/offcpuflamegraphs.html

This tool only works on Linux 4.6+. It uses the new `BPF_STACK_TRACE` table APIs to generate the in-kernel stack traces. For kernels older than 4.6, see the version under tools/old.

Note: this tool only traces off-CPU times that began and ended while tracing.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print usage message.

-p PID

Trace this process ID only (filtered in-kernel).

-t TID

Trace this thread ID only (filtered in-kernel).

-u

Only trace user threads (no kernel threads).

-k

Only trace kernel threads (no user threads).

-U

Show stacks from user space only (no kernel space stacks).

-K

Show stacks from kernel space only (no user space stacks).

-d

Insert delimiter between kernel/user stacks.

-f

Print output in folded stack format.

--stack-storage-size STACK_STORAGE_SIZE

Change the number of unique stack traces that can be stored and displayed.

-m MIN_BLOCK_TIME

The minimum time in microseconds over which we store traces (default 1)

-M MAX_BLOCK_TIME

The maximum time in microseconds under which we store traces (default U64_MAX)

--state

Filter on this thread state bitmask (eg, 2 == TASK_UNINTERRUPTIBLE). See include/linux/sched.h for states.

duration

Duration to trace, in seconds.

## EXAMPLES
Trace all thread blocking events, and summarize (in-kernel) by kernel stack trace and total off-CPU time:

\# offcputime

Trace for 5 seconds only:

\# offcputime 5

Trace for 5 seconds, and emit output in folded stack format (suitable for flame graphs):

\# offcputime -f 5

Trace PID 185 only:

\# offcputime -p 185

## OVERHEAD
This summarizes unique stack traces in-kernel for efficiency, allowing it to trace a higher rate of events than methods that post-process in user space. The stack trace and time data is only copied to user space once, when the output is printed. While these techniques greatly lower overhead, scheduler events are still a high frequency event, as they can exceed 1 million events per second, and so caution should still be used. Test before production use.

If the overhead is still a problem, take a look at the MINBLOCK_US tunable in the code. If your aim is to chase down longer blocking events, then this could be increased to filter shorter blocking events, further lowering overhead.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
stackcount(8)


bcc-offwaketime

title: bcc-offwaketime(8)

text:
# bcc-offwaketime(8)

## NAME
offwaketime - Summarize blocked time by off-CPU stack + waker stack. Uses Linux eBPF/bcc.

## SYNOPSIS
offwaketime [-h] [-p PID | -t TID | -u | -k] [-U | -K] [-d] [-f] [--stack-storage-size STACK_STORAGE_SIZE] [-m MIN_BLOCK_TIME] [-M MAX_BLOCK_TIME] [--state STATE] [duration]

## DESCRIPTION
This program shows kernel stack traces and task names that were blocked and "off-CPU", along with the stack traces and task names for the threads that woke them, and the total elapsed time from when they blocked to when they were woken up. This combines the summaries from both the offcputime and wakeuptime tools. The time measurement will be very similar to off-CPU time, however, off-CPU time may include a little extra time spent waiting on a run queue to be scheduled. The combined stacks, task names, and total time is summarized in kernel context for efficiency, using an eBPF map.

The output summary will further help you identify reasons why threads were blocking, and quantify the time from when they were blocked to woken up. This spans all types of blocking activity: disk I/O, network I/O, locks, page faults, swapping, sleeping, involuntary context switches, etc.

This is complementary to CPU profiling (e.g., CPU flame graphs) which shows the time spent on-CPU. This shows the time spent blocked off-CPU, and the output, especially the -f format, can be used to generate an "off-wake time flame graph".

See http://www.brendangregg.com/FlameGraphs/offcpuflamegraphs.html

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print usage message.

-f

Print output in folded stack format.

-p PID

Trace this process ID only (filtered in-kernel). Can be a comma separated list of PIDS.

-t TID

Trace this thread ID only (filtered in-kernel). Can be a comma separated list of TIDS.

-u

Only trace user threads (no kernel threads).

-k

Only trace kernel threads (no user threads).

-U

Show stacks from user space only (no kernel space stacks).

-K

Show stacks from kernel space only (no user space stacks).

-d, --delimited

insert delimiter between kernel/user stacks

--stack-storage-size STACK_STORAGE_SIZE

Change the number of unique stack traces that can be stored and displayed.

duration

Duration to trace, in seconds.

-m MIN_BLOCK_TIME

The amount of time in microseconds over which we store traces (default 1)

-M MAX_BLOCK_TIME

The amount of time in microseconds under which we store traces (default U64_MAX)

--state

Filter on this thread state bitmask (eg, 2 == TASK_UNINTERRUPTIBLE). See include/linux/sched.h for states.

## EXAMPLES
Trace all thread blocking events, and summarize (in-kernel) by user and kernel off-CPU stack trace, waker stack traces, task names, and total blocked time:

\# offwaketime

Trace for 5 seconds only:

\# offwaketime 5

Trace for 5 seconds, and emit output in folded stack format (suitable for flame graphs), user-mode threads only:

\# offwaketime -fu 5

Trace PID 185 only:

\# offwaketime -p 185

## OVERHEAD
This summarizes unique stack trace pairs in-kernel for efficiency, allowing it to trace a higher rate of events than methods that post-process in user space. The stack trace and time data is only copied to user space once, when the output is printed. While these techniques greatly lower overhead, scheduler events are still a high frequency event, as they can exceed 1 million events per second, and so caution should still be used. Test before production use.

If the overhead is still a problem, take a look at the min block option. If your aim is to chase down longer blocking events, then this could be increased to filter shorter blocking events, further lowering overhead.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
offcputime(8), wakeuptime(8)


bcc-oomkill

title: bcc-oomkill(8)

text:
# bcc-oomkill(8)

## NAME
oomkill - Trace oom_kill_process(). Uses Linux eBPF/bcc.

## SYNOPSIS
oomkill

## DESCRIPTION
This traces the kernel out-of-memory killer, and prints basic details, including the system load averages at the time of the OOM kill. This can provide more context on the system state at the time: was it getting busier or steady, based on the load averages? This tool may also be useful to customize for investigations; for example, by adding other task_struct details at the time of OOM.

This program is also a basic example of eBPF/bcc.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## EXAMPLES
Trace OOM kill events:

\# oomkill

## FIELDS
Triggered by ...

The process ID and process name of the task that was running when another task was OOM killed.

OOM kill of ...

The process ID and name of the target process that was OOM killed.

loadavg

Contents of /proc/loadavg. The first three numbers are 1, 5, and 15 minute load averages (where the average is an exponentially damped moving sum, and those numbers are constants in the equation); then there is the number of running tasks, a slash, and the total number of tasks; and then the last number is the last PID to be created.

## OVERHEAD
Negligible.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
memleak(8)


bcc-opensnoop

title: bcc-opensnoop(8)

text:
# bcc-opensnoop(8)

## NAME
opensnoop - Trace open() syscalls. Uses Linux eBPF/bcc.

## SYNOPSIS
opensnoop [-h] [-T] [-U] [-x] [-p PID] [-t TID] [-u UID]

[-d DURATION] [-n NAME] [-e] [-f FLAG_FILTER] [-F]

[--cgroupmap MAPPATH] [--mntnsmap MAPPATH]

## DESCRIPTION
opensnoop traces the open() syscall, showing which processes are attempting to open which files. This can be useful for determining the location of config and log files, or for troubleshooting applications that are failing, specially on startup.

This works by tracing the kernel sys_open() function using dynamic tracing, and will need updating to match any changes to this function.

This makes use of a Linux 4.4 feature (bpf_perf_event_output()); for kernels older than 4.4, see the version under tools/old, which uses an older mechanism.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print usage message.

-T

Include a timestamp column.

-U

Show UID.

-x

Only print failed opens.

-p PID

Trace this process ID only (filtered in-kernel).

-t TID

Trace this thread ID only (filtered in-kernel).

-u UID

Trace this UID only (filtered in-kernel).

-d DURATION

Total duration of trace in seconds.

-n name

Only print processes where its name partially matches 'name'

-e

Show extended fields.

-f FLAG

Filter on open() flags, e.g., O_WRONLY.

-F

Show full path for an open file with relative path.

--cgroupmap MAPPATH

Trace cgroups in this BPF map only (filtered in-kernel).

--mntnsmap MAPPATH

Trace mount namespaces in this BPF map only (filtered in-kernel).

## EXAMPLES
Trace all open() syscalls:

\# opensnoop

Trace all open() syscalls, for 10 seconds only:

\# opensnoop -d 10

Trace all open() syscalls, and include timestamps:

\# opensnoop -T

Show UID:

\# opensnoop -U

Trace only open() syscalls that failed:

\# opensnoop -x

Trace PID 181 only:

\# opensnoop -p 181

Trace UID 1000 only:

\# opensnoop -u 1000

Trace all open() syscalls from processes where its name partially matches 'ed':

\# opensnoop -n ed

Show extended fields:

\# opensnoop -e

Only print calls for writing:

\# opensnoop -f O_WRONLY -f O_RDWR

Trace a set of cgroups only (see special_filtering.md from bcc sources for more details):

\# opensnoop --cgroupmap /sys/fs/bpf/test01

## FIELDS
TIME(s)

Time of the call, in seconds.

UID

User ID

PID

Process ID

TID

Thread ID

COMM

Process name

FD

File descriptor (if success), or -1 (if failed)

ERR

Error number (see the system's errno.h)

FLAGS

Flags passed to open(2), in octal

PATH

Open path

## OVERHEAD
This traces the kernel open function and prints output for each event. As the rate of this is generally expected to be low (< 1000/s), the overhead is also expected to be negligible. If you have an application that is calling a high rate of open()s, then test and understand overhead before use.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg, Rocky Xing

## SEE ALSO
execsnoop(8), funccount(1)


bcc-perlcalls

title: bcc-perlcalls(8)

summary: Summarize method calls

text:
# bcc-perlcalls(8)

## NAME
ucalls, javacalls, perlcalls, phpcalls, pythoncalls, rubycalls, tclcalls - Summarize method calls from high-level languages and Linux syscalls.

## SYNOPSIS
javacalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

perlcalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

phpcalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

pythoncalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

rubycalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

tclcalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

ucalls [-l {java,perl,php,python,ruby}] [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

## DESCRIPTION
This tool summarizes method calls from high-level languages such as Java, Perl, PHP, Python, Ruby, and Tcl. It can also trace Linux system calls. Whenever a method is invoked, ucalls records the call count and optionally the method's execution time (latency) and displays a summary.

This uses in-kernel eBPF maps to store per process summaries for efficiency.

This tool relies on USDT probes embedded in many high-level languages, such as Java, Perl, PHP, Python, Ruby, and Tcl. It requires a runtime instrumented with these probes, which in some cases requires building from source with a USDT-specific flag, such as "--enable-dtrace" or "--with-dtrace". For Java, method probes are not enabled by default, and can be turned on by running the Java process with the "-XX:+ExtendedDTraceProbes" flag. For PHP processes, the environment variable USE_ZEND_DTRACE must be set to 1.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-l {java,perl,php,python,ruby,tcl}

The language to trace. If not provided, only syscalls are traced (when the -S option is used).

-T TOP

Print only the top methods by frequency or latency.

-L

Collect method invocation latency (duration).

-S

Collect Linux syscalls frequency and timing.

-v

Print the resulting BPF program, for debugging purposes.

-m

Print times in milliseconds (the default is microseconds).

pid

The process id to trace.

interval

Print summary after this number of seconds and then exit. By default, wait for Ctrl+C to terminate.

## EXAMPLES
Trace the top 10 Ruby method calls:

\# ucalls -T 10 -l ruby 1344

Trace Python method calls and Linux syscalls including latency in milliseconds:

\# ucalls -l python -mL 2020

Trace only syscalls and print a summary after 10 seconds:

\# ucalls -S 788 10

## OVERHEAD
Tracing individual method calls will produce a considerable overhead in all high-level languages. For languages with just-in-time compilation, such as Java, the overhead can be more considerable than for interpreted languages. On the other hand, syscall tracing will typically be tolerable for most processes, unless they have a very unusual rate of system calls.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _example.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein

## SEE ALSO
ustat(8), argdist(8)


bcc-perlflow

title: bcc-perlflow(8)

summary: Print a flow graph of method

text:
# bcc-perlflow(8)

## NAME
uflow, javaflow, perlflow, phpflow, pythonflow, rubyflow, tclflow - Print a flow graph of method calls in high-level languages.

## SYNOPSIS
javaflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

perlflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

phpflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

pythonflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

rubyflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

tclflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

uflow [-h] [-M METHOD] [-C CLAZZ] [-v] [-l {java,perl,php,python,ruby,tcl}] pid

## DESCRIPTION
uflow traces method calls and prints them in a flow graph that can facilitate debugging and diagnostics by following the program's execution (method flow).

This tool relies on USDT probes embedded in many high-level languages, such as Java, Perl, PHP, Python, Ruby, and Tcl. It requires a runtime instrumented with these probes, which in some cases requires building from source with a USDT-specific flag, such as "--enable-dtrace" or "--with-dtrace". For Java processes, the startup flag "-XX:+ExtendedDTraceProbes" is required. For PHP processes, the environment variable USE_ZEND_DTRACE must be set to 1.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-M METHOD

Print only method calls where the method name begins with this string.

-C CLAZZ

Print only method calls where the class name begins with this string. The class name interpretation strongly depends on the language. For example, in Java use "package/subpackage/ClassName" to refer to classes.

-v

Print the resulting BPF program, for debugging purposes.

{java,perl,php,python,ruby,tcl}

The language to trace.

pid

The process id to trace.

## EXAMPLES
Follow method flow in a Ruby process:

\# uflow ruby 148

Follow method flow in a Java process where the class name is java.lang.Thread:

\# uflow -C java/lang/Thread java 1802

## FIELDS
CPU

The CPU number on which the method was invoked. This is useful to easily see where the output skips to a different CPU.

PID

The process id.

TID

The thread id.

TIME

The duration of the method call.

METHOD

The method name.

## OVERHEAD
This tool has extremely high overhead because it prints every method call. For some scenarios, you might see lost samples in the output as the tool is unable to keep up with the rate of data coming from the kernel. Filtering by class or method prefix can help reduce the amount of data printed, but there is still a very high overhead in the collection mechanism. Do not use for performance- sensitive production scenarios, and always test first.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _example.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein

## SEE ALSO
trace(8), ustat(8)


bcc-perlstat

title: bcc-perlstat(8)

summary: Activity stats from

text:
# bcc-perlstat(8)

## NAME
ustat, javastat, nodestat, perlstat, phpstat, pythonstat, rubystat, tclstat - Activity stats from high-level languages.

## SYNOPSIS
javastat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

nodestat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

perlstat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

phpstat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

pythonstat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

rubystat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

tclstat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

ustat [-l {java,node,perl,php,python,ruby,tcl}] [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

## DESCRIPTION
This is "top" for high-level language events, such as garbage collections, exceptions, thread creations, object allocations, method calls, and more. The events are aggregated for each process and printed in a top-like table, which can be sorted by various fields. Not all language runtimes provide the same set of details.

This uses in-kernel eBPF maps to store per process summaries for efficiency.

This tool relies on USDT probes embedded in many high-level languages, such as Java, Node, Perl, PHP, Python, Ruby, and Tcl. It requires a runtime instrumented with these probes, which in some cases requires building from source with a USDT-specific flag, such as "--enable-dtrace" or "--with-dtrace". For Java, some probes are not enabled by default, and can be turned on by running the Java process with the "-XX:+ExtendedDTraceProbes" flag. For PHP processes, the environment variable USE_ZEND_DTRACE must be set to 1.

Newly-created processes will only be traced at the next interval. If you run this tool with a short interval (say, 1-5 seconds), this should be virtually unnoticeable. For longer intervals, you might miss processes that were started and terminated during the interval window.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-l {java,node,perl,php,python,ruby,tcl}

The language to trace. By default, all languages are traced.

-C

Do not clear the screen between updates.

-S {cload,excp,gc,method,objnew,thread}

Sort the output by the specified field.

-r MAXROWS

Do not print more than this number of rows.

-d

Print the resulting BPF program, for debugging purposes.

interval

Interval between updates, seconds.

count

Number of interval summaries.

## EXAMPLES
Summarize activity in high-level languages, 1 second refresh:

\# ustat

Don't clear the screen, and top 8 rows only:

\# ustat -Cr 8

5 second summaries, 10 times only:

\# ustat 5 10

## FIELDS
loadavg

The contents of /proc/loadavg

PID

Process ID.

CMDLINE

Process command line (often the second and following arguments will give you a hint as to which application is being run.

METHOD/s

Count of method invocations during interval.

GC/s

Count of garbage collections during interval.

OBJNEW/s

Count of objects allocated during interval.

CLOAD/s

Count of classes loaded during interval.

EXC/s

Count of exceptions thrown during interval.

THR/s

Count of threads created during interval.

## OVERHEAD
When using this tool with high-frequency events, such as method calls, a very significant slow-down can be expected. However, many of the high-level languages covered by this tool already have a fairly high per-method invocation cost, especially when running in interpreted mode. For the lower-frequency events, such as garbage collections or thread creations, the overhead should not be significant. Specifically, when probing Java processes and not using the "-XX:+ExtendedDTraceProbes" flag, the most expensive probes are not emitted, and the overhead should be acceptable.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _example.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein

## SEE ALSO
trace(8), argdist(8), tplist(8)


bcc-phpcalls

title: bcc-phpcalls(8)

summary: Summarize method calls

text:
# bcc-phpcalls(8)

## NAME
ucalls, javacalls, perlcalls, phpcalls, pythoncalls, rubycalls, tclcalls - Summarize method calls from high-level languages and Linux syscalls.

## SYNOPSIS
javacalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

perlcalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

phpcalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

pythoncalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

rubycalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

tclcalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

ucalls [-l {java,perl,php,python,ruby}] [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

## DESCRIPTION
This tool summarizes method calls from high-level languages such as Java, Perl, PHP, Python, Ruby, and Tcl. It can also trace Linux system calls. Whenever a method is invoked, ucalls records the call count and optionally the method's execution time (latency) and displays a summary.

This uses in-kernel eBPF maps to store per process summaries for efficiency.

This tool relies on USDT probes embedded in many high-level languages, such as Java, Perl, PHP, Python, Ruby, and Tcl. It requires a runtime instrumented with these probes, which in some cases requires building from source with a USDT-specific flag, such as "--enable-dtrace" or "--with-dtrace". For Java, method probes are not enabled by default, and can be turned on by running the Java process with the "-XX:+ExtendedDTraceProbes" flag. For PHP processes, the environment variable USE_ZEND_DTRACE must be set to 1.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-l {java,perl,php,python,ruby,tcl}

The language to trace. If not provided, only syscalls are traced (when the -S option is used).

-T TOP

Print only the top methods by frequency or latency.

-L

Collect method invocation latency (duration).

-S

Collect Linux syscalls frequency and timing.

-v

Print the resulting BPF program, for debugging purposes.

-m

Print times in milliseconds (the default is microseconds).

pid

The process id to trace.

interval

Print summary after this number of seconds and then exit. By default, wait for Ctrl+C to terminate.

## EXAMPLES
Trace the top 10 Ruby method calls:

\# ucalls -T 10 -l ruby 1344

Trace Python method calls and Linux syscalls including latency in milliseconds:

\# ucalls -l python -mL 2020

Trace only syscalls and print a summary after 10 seconds:

\# ucalls -S 788 10

## OVERHEAD
Tracing individual method calls will produce a considerable overhead in all high-level languages. For languages with just-in-time compilation, such as Java, the overhead can be more considerable than for interpreted languages. On the other hand, syscall tracing will typically be tolerable for most processes, unless they have a very unusual rate of system calls.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _example.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein

## SEE ALSO
ustat(8), argdist(8)


bcc-phpflow

title: bcc-phpflow(8)

summary: Print a flow graph of method

text:
# bcc-phpflow(8)

## NAME
uflow, javaflow, perlflow, phpflow, pythonflow, rubyflow, tclflow - Print a flow graph of method calls in high-level languages.

## SYNOPSIS
javaflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

perlflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

phpflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

pythonflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

rubyflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

tclflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

uflow [-h] [-M METHOD] [-C CLAZZ] [-v] [-l {java,perl,php,python,ruby,tcl}] pid

## DESCRIPTION
uflow traces method calls and prints them in a flow graph that can facilitate debugging and diagnostics by following the program's execution (method flow).

This tool relies on USDT probes embedded in many high-level languages, such as Java, Perl, PHP, Python, Ruby, and Tcl. It requires a runtime instrumented with these probes, which in some cases requires building from source with a USDT-specific flag, such as "--enable-dtrace" or "--with-dtrace". For Java processes, the startup flag "-XX:+ExtendedDTraceProbes" is required. For PHP processes, the environment variable USE_ZEND_DTRACE must be set to 1.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-M METHOD

Print only method calls where the method name begins with this string.

-C CLAZZ

Print only method calls where the class name begins with this string. The class name interpretation strongly depends on the language. For example, in Java use "package/subpackage/ClassName" to refer to classes.

-v

Print the resulting BPF program, for debugging purposes.

{java,perl,php,python,ruby,tcl}

The language to trace.

pid

The process id to trace.

## EXAMPLES
Follow method flow in a Ruby process:

\# uflow ruby 148

Follow method flow in a Java process where the class name is java.lang.Thread:

\# uflow -C java/lang/Thread java 1802

## FIELDS
CPU

The CPU number on which the method was invoked. This is useful to easily see where the output skips to a different CPU.

PID

The process id.

TID

The thread id.

TIME

The duration of the method call.

METHOD

The method name.

## OVERHEAD
This tool has extremely high overhead because it prints every method call. For some scenarios, you might see lost samples in the output as the tool is unable to keep up with the rate of data coming from the kernel. Filtering by class or method prefix can help reduce the amount of data printed, but there is still a very high overhead in the collection mechanism. Do not use for performance- sensitive production scenarios, and always test first.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _example.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein

## SEE ALSO
trace(8), ustat(8)


bcc-phpstat

title: bcc-phpstat(8)

summary: Activity stats from

text:
# bcc-phpstat(8)

## NAME
ustat, javastat, nodestat, perlstat, phpstat, pythonstat, rubystat, tclstat - Activity stats from high-level languages.

## SYNOPSIS
javastat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

nodestat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

perlstat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

phpstat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

pythonstat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

rubystat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

tclstat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

ustat [-l {java,node,perl,php,python,ruby,tcl}] [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

## DESCRIPTION
This is "top" for high-level language events, such as garbage collections, exceptions, thread creations, object allocations, method calls, and more. The events are aggregated for each process and printed in a top-like table, which can be sorted by various fields. Not all language runtimes provide the same set of details.

This uses in-kernel eBPF maps to store per process summaries for efficiency.

This tool relies on USDT probes embedded in many high-level languages, such as Java, Node, Perl, PHP, Python, Ruby, and Tcl. It requires a runtime instrumented with these probes, which in some cases requires building from source with a USDT-specific flag, such as "--enable-dtrace" or "--with-dtrace". For Java, some probes are not enabled by default, and can be turned on by running the Java process with the "-XX:+ExtendedDTraceProbes" flag. For PHP processes, the environment variable USE_ZEND_DTRACE must be set to 1.

Newly-created processes will only be traced at the next interval. If you run this tool with a short interval (say, 1-5 seconds), this should be virtually unnoticeable. For longer intervals, you might miss processes that were started and terminated during the interval window.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-l {java,node,perl,php,python,ruby,tcl}

The language to trace. By default, all languages are traced.

-C

Do not clear the screen between updates.

-S {cload,excp,gc,method,objnew,thread}

Sort the output by the specified field.

-r MAXROWS

Do not print more than this number of rows.

-d

Print the resulting BPF program, for debugging purposes.

interval

Interval between updates, seconds.

count

Number of interval summaries.

## EXAMPLES
Summarize activity in high-level languages, 1 second refresh:

\# ustat

Don't clear the screen, and top 8 rows only:

\# ustat -Cr 8

5 second summaries, 10 times only:

\# ustat 5 10

## FIELDS
loadavg

The contents of /proc/loadavg

PID

Process ID.

CMDLINE

Process command line (often the second and following arguments will give you a hint as to which application is being run.

METHOD/s

Count of method invocations during interval.

GC/s

Count of garbage collections during interval.

OBJNEW/s

Count of objects allocated during interval.

CLOAD/s

Count of classes loaded during interval.

EXC/s

Count of exceptions thrown during interval.

THR/s

Count of threads created during interval.

## OVERHEAD
When using this tool with high-frequency events, such as method calls, a very significant slow-down can be expected. However, many of the high-level languages covered by this tool already have a fairly high per-method invocation cost, especially when running in interpreted mode. For the lower-frequency events, such as garbage collections or thread creations, the overhead should not be significant. Specifically, when probing Java processes and not using the "-XX:+ExtendedDTraceProbes" flag, the most expensive probes are not emitted, and the overhead should be acceptable.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _example.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein

## SEE ALSO
trace(8), argdist(8), tplist(8)