bcc-pidpersec

title: bcc-pidpersec(8)

text:
# bcc-pidpersec(8)

## NAME
pidpersec - Count new processes (via fork()). Uses Linux eBPF/bcc.

## SYNOPSIS
pidpersec

## DESCRIPTION
pidpersec shows how many new processes were created each second. There can be performance issues caused by many short-lived processes, which may not be visible in sampling tools like top(1). pidpersec provides one way to investigate this behavior.

This works by tracing the kernel sched_fork() function using dynamic tracing, and will need updating to match any changes to this function.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## EXAMPLES
Count new processes created each second:

\# pidpersec

## OVERHEAD
This traces the kernel fork function, and maintains an in-kernel count which is read asynchronously from user-space. As the rate of this is generally expected to be low (<< 1000/s), the overhead is also expected to be negligible.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
top(1)


bcc-ppchcalls

package: bcc-tools

title: bcc-ppchcalls(8)

text:
# bcc-ppchcalls(8)

## NAME
ppchcalls - Summarize ppc hcall counts and latencies.

## SYNOPSIS
ppchcalls [-h] [-p PID] [-t TID] [-i INTERVAL] [-d DURATION] [-T TOP] [-x] [-e ERRNO] [-L] [-m] [-P] [-l] [--hcall HCALL]

## DESCRIPTION
This tool traces hcall entry and exit raw tracepoints and summarizes either the number of hcalls of each type, or the number of hcalls per process. It can also collect min, max and average latency for each hcall or each process.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc. Linux 4.17+ is required to attach a BPF program to the raw_hcalls:hcall_{enter,exit} tracepoints, used by this tool.

## OPTIONS
-h

Print usage message.

-p PID

Trace only this process.

-t TID

Trace only this thread.

-i INTERVAL

Print the summary at the specified interval (in seconds).

-d DURATION

Total duration of trace (in seconds).

-T TOP

Print only this many entries. Default: 10.

-x

Trace only failed hcalls (i.e., the return value from the hcall was < 0).

-e ERRNO

Trace only hcalls that failed with that error (e.g. -e EPERM or -e 1).

-m

Display times in milliseconds. Default: microseconds.

-P

Summarize by process and not by hcall.

-l

List the hcalls recognized by the tool (hard-coded list). Hcalls beyond this list will still be displayed, as "[unknown: nnn]" where nnn is the hcall number.

--hcall HCALL

Trace this hcall only (use option -l to get all recognized hcalls).

## EXAMPLES
Summarize all hcalls by hcall:

\# ppchcalls

Summarize all hcalls by process:

\# ppchcalls -P

Summarize only failed hcalls:

\# ppchcalls -x

Summarize only hcalls that failed with EPERM:

\# ppchcalls -e EPERM

Trace PID 181 only:

\# ppchcalls -p 181

Summarize hcalls counts and latencies:

\# ppchcalls -L

## FIELDS
PID

Process ID

COMM

Process name

HCALL

Hcall name, or "[unknown: nnn]" for hcalls that aren't recognized

COUNT

The number of events

MIN

The minimum elapsed time (in us or ms)

MAX

The maximum elapsed time (in us or ms)

AVG

The average elapsed time (in us or ms)

## OVERHEAD
For most applications, the overhead should be manageable if they perform 1000's or even 10,000's of hcalls per second. For higher rates, the overhead may become considerable.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Harsh Prateek Bora

## SEE ALSO
syscount(8)

branch: sisyphus


bcc-profile

title: bcc-profile(8)

text:
# bcc-profile(8)

## NAME
profile - Profile CPU usage by sampling stack traces. Uses Linux eBPF/bcc.

## SYNOPSIS
profile [-adfh] [-p PID | -L TID] [-U | -K] [-F FREQUENCY | -c COUNT] [--stack-storage-size COUNT] [-C CPU] [--cgroupmap CGROUPMAP] [--mntnsmap MAPPATH] [duration]

## DESCRIPTION
This is a CPU profiler. It works by taking samples of stack traces at timed intervals. It will help you understand and quantify CPU usage: which code is executing, and by how much, including both user-level and kernel code.

By default this samples at 49 Hertz (samples per second), across all CPUs. This frequency can be tuned using a command line option. The reason for 49, and not 50, is to avoid lock-step sampling.

This is also an efficient profiler, as stack traces are frequency counted in kernel context, rather than passing each stack to user space for frequency counting there. Only the unique stacks and counts are passed to user space at the end of the profile, greatly reducing the kernel<->user transfer.

## REQUIREMENTS
CONFIG_BPF and bcc.

This also requires Linux 4.9+ (BPF_PROG_TYPE_PERF_EVENT support). See tools/old for an older version that may work on Linux 4.6 - 4.8.

## OPTIONS
-h

Print usage message.

-p PID

Trace process with one or more comma separated PIDs only (filtered in-kernel).

-L TID

Trace thread with one or more comma separated TIDs only (filtered in-kernel).

-F frequency

Frequency to sample stacks.

-c count

Sample stacks every one in this many events.

-f

Print output in folded stack format.

-d

Include an output delimiter between kernel and user stacks (either "--", or, in folded mode, "-").

-U

Show stacks from user space only (no kernel space stacks).

-K

Show stacks from kernel space only (no user space stacks).

-I

Include CPU idle stacks (by default these are excluded).

--stack-storage-size COUNT

The maximum number of unique stack traces that the kernel will count (default 16384). If the sampled count exceeds this, a warning will be printed.

-C cpu

Collect stacks only from specified cpu.

--cgroupmap MAPPATH

Profile cgroups in this BPF map only (filtered in-kernel).

duration

Duration to trace, in seconds.

## EXAMPLES
Profile (sample) stack traces system-wide at 49 Hertz (samples per second) until Ctrl-C:

\# profile

Profile for 5 seconds only:

\# profile 5

Profile at 99 Hertz for 5 seconds only:

\# profile -F 99 5

Profile 1 in a million events for 5 seconds only:

\# profile -c 1000000 5

Profile process with PID 181 only:

\# profile -p 181

Profile thread with TID 181 only:

\# profile -L 181

Profile for 5 seconds and output in folded stack format (suitable as input for flame graphs), including a delimiter between kernel and user stacks:

\# profile -df 5

Profile kernel stacks only:

\# profile -K

Profile a set of cgroups only (see special_filtering.md from bcc sources for more details):

\# profile --cgroupmap /sys/fs/bpf/test01

## DEBUGGING
See "[unknown]" frames with bogus addresses? This can happen for different reasons. Your best approach is to get Linux perf to work first, and then to try this tool. Eg, "perf record -F 49 -a -g -- sleep 1; perf script", and to check for unknown frames there.

The most common reason for "[unknown]" frames is that the target software has not been compiled with frame pointers, and so we can't use that simple method for walking the stack. The fix in that case is to use software that does have frame pointers, eg, gcc -fno-omit-frame-pointer, or Java's -XX:+PreserveFramePointer.

Another reason for "[unknown]" frames is JIT compilers, which don't use a traditional symbol table. The fix in that case is to populate a /tmp/perf-PID.map file with the symbols, which this tool should read. How you do this depends on the runtime (Java, Node.js).

If you seem to have unrelated samples in the output, check for other sampling or tracing tools that may be running. The current version of this tool can include their events if profiling happened concurrently. Those samples may be filtered in a future version.

## OVERHEAD
This is an efficient profiler, as stack traces are frequency counted in kernel context, and only the unique stacks and their counts are passed to user space. Contrast this with the current "perf record -F 99 -a" method of profiling, which writes each sample to user space (via a ring buffer), and then to the file system (perf.data), which must be post-processed.

This uses perf_event_open to setup a timer which is instrumented by BPF, and for efficiency it does not initialize the perf ring buffer, so the redundant perf samples are not collected.

It's expected that the overhead while sampling at 49 Hertz (the default), across all CPUs, should be negligible. If you increase the sample rate, the overhead might begin to be measurable.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
offcputime(8)


bcc-pythoncalls

title: bcc-pythoncalls(8)

summary: Summarize method calls

text:
# bcc-pythoncalls(8)

## NAME
ucalls, javacalls, perlcalls, phpcalls, pythoncalls, rubycalls, tclcalls - Summarize method calls from high-level languages and Linux syscalls.

## SYNOPSIS
javacalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

perlcalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

phpcalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

pythoncalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

rubycalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

tclcalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

ucalls [-l {java,perl,php,python,ruby}] [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

## DESCRIPTION
This tool summarizes method calls from high-level languages such as Java, Perl, PHP, Python, Ruby, and Tcl. It can also trace Linux system calls. Whenever a method is invoked, ucalls records the call count and optionally the method's execution time (latency) and displays a summary.

This uses in-kernel eBPF maps to store per process summaries for efficiency.

This tool relies on USDT probes embedded in many high-level languages, such as Java, Perl, PHP, Python, Ruby, and Tcl. It requires a runtime instrumented with these probes, which in some cases requires building from source with a USDT-specific flag, such as "--enable-dtrace" or "--with-dtrace". For Java, method probes are not enabled by default, and can be turned on by running the Java process with the "-XX:+ExtendedDTraceProbes" flag. For PHP processes, the environment variable USE_ZEND_DTRACE must be set to 1.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-l {java,perl,php,python,ruby,tcl}

The language to trace. If not provided, only syscalls are traced (when the -S option is used).

-T TOP

Print only the top methods by frequency or latency.

-L

Collect method invocation latency (duration).

-S

Collect Linux syscalls frequency and timing.

-v

Print the resulting BPF program, for debugging purposes.

-m

Print times in milliseconds (the default is microseconds).

pid

The process id to trace.

interval

Print summary after this number of seconds and then exit. By default, wait for Ctrl+C to terminate.

## EXAMPLES
Trace the top 10 Ruby method calls:

\# ucalls -T 10 -l ruby 1344

Trace Python method calls and Linux syscalls including latency in milliseconds:

\# ucalls -l python -mL 2020

Trace only syscalls and print a summary after 10 seconds:

\# ucalls -S 788 10

## OVERHEAD
Tracing individual method calls will produce a considerable overhead in all high-level languages. For languages with just-in-time compilation, such as Java, the overhead can be more considerable than for interpreted languages. On the other hand, syscall tracing will typically be tolerable for most processes, unless they have a very unusual rate of system calls.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _example.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein

## SEE ALSO
ustat(8), argdist(8)


bcc-pythonflow

title: bcc-pythonflow(8)

summary: Print a flow graph of method

text:
# bcc-pythonflow(8)

## NAME
uflow, javaflow, perlflow, phpflow, pythonflow, rubyflow, tclflow - Print a flow graph of method calls in high-level languages.

## SYNOPSIS
javaflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

perlflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

phpflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

pythonflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

rubyflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

tclflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

uflow [-h] [-M METHOD] [-C CLAZZ] [-v] [-l {java,perl,php,python,ruby,tcl}] pid

## DESCRIPTION
uflow traces method calls and prints them in a flow graph that can facilitate debugging and diagnostics by following the program's execution (method flow).

This tool relies on USDT probes embedded in many high-level languages, such as Java, Perl, PHP, Python, Ruby, and Tcl. It requires a runtime instrumented with these probes, which in some cases requires building from source with a USDT-specific flag, such as "--enable-dtrace" or "--with-dtrace". For Java processes, the startup flag "-XX:+ExtendedDTraceProbes" is required. For PHP processes, the environment variable USE_ZEND_DTRACE must be set to 1.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-M METHOD

Print only method calls where the method name begins with this string.

-C CLAZZ

Print only method calls where the class name begins with this string. The class name interpretation strongly depends on the language. For example, in Java use "package/subpackage/ClassName" to refer to classes.

-v

Print the resulting BPF program, for debugging purposes.

{java,perl,php,python,ruby,tcl}

The language to trace.

pid

The process id to trace.

## EXAMPLES
Follow method flow in a Ruby process:

\# uflow ruby 148

Follow method flow in a Java process where the class name is java.lang.Thread:

\# uflow -C java/lang/Thread java 1802

## FIELDS
CPU

The CPU number on which the method was invoked. This is useful to easily see where the output skips to a different CPU.

PID

The process id.

TID

The thread id.

TIME

The duration of the method call.

METHOD

The method name.

## OVERHEAD
This tool has extremely high overhead because it prints every method call. For some scenarios, you might see lost samples in the output as the tool is unable to keep up with the rate of data coming from the kernel. Filtering by class or method prefix can help reduce the amount of data printed, but there is still a very high overhead in the collection mechanism. Do not use for performance- sensitive production scenarios, and always test first.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _example.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein

## SEE ALSO
trace(8), ustat(8)


bcc-pythongc

title: bcc-pythongc(8)

summary: Trace garbage collection events in

text:
# bcc-pythongc(8)

## NAME
ugc, javagc, nodegc, pythongc, rubygc - Trace garbage collection events in high-level languages.

## SYNOPSIS
javagc [-h] [-v] [-m] [-M MINIMUM] [-F FILTER] pid

nodegc [-h] [-v] [-m] [-M MINIMUM] [-F FILTER] pid

pythongc [-h] [-v] [-m] [-M MINIMUM] [-F FILTER] pid

rubygc [-h] [-v] [-m] [-M MINIMUM] [-F FILTER] pid

ugc [-h] [-v] [-m] [-M MINIMUM] [-F FILTER] [-l {java,node,python,ruby}] pid

## DESCRIPTION
This traces garbage collection events as they occur, including their duration and any additional information (such as generation collected or type of GC) provided by the respective language's runtime.

This tool relies on USDT probes embedded in many high-level languages, such as Java, Node, Python, and Ruby. It requires a runtime instrumented with these probes, which in some cases requires building from source with a USDT-specific flag, such as "--enable-dtrace" or "--with-dtrace".

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-v

Print the resulting BPF program, for debugging purposes.

-m

Print times in milliseconds. The default is microseconds.

-M MINIMUM

Display only collections that are longer than this threshold. The value is given in milliseconds. The default is to display all collections.

-F FILTER

Display only collections whose textual description matches (contains) this string. The default is to display all collections. Note that the filtering here is performed in user-space, and not as part of the BPF program. This means that if you have thousands of collection events, specifying this filter will not reduce the amount of data that has to be transferred from the BPF program to the user-space script.

{java,node,python,ruby}

The language to trace.

pid

The process id to trace.

## EXAMPLES
Trace garbage collections in a specific Node process:

\# ugc node 148

Trace garbage collections in a specific Java process, and print GC times in

milliseconds: # ugc -m java 6004

Trace garbage collections in a specific Java process, and display them only if

they are longer than 10ms and have the string "Tenured" in their detailed description: # ugc -M 10 -F Tenured java 6004

## FIELDS
START

The start time of the GC, in seconds from the beginning of the trace.

TIME

The duration of the garbage collection event.

DESCRIPTION

The runtime-provided description of this garbage collection event.

## OVERHEAD
Garbage collection events, even if frequent, should not produce a considerable overhead when traced because they are still not very common. Even hundreds of GCs per second (which is a very high rate) will still produce a fairly negligible overhead.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _example.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein

## SEE ALSO
trace(8), ustat(8), uobjnew(8)


bcc-pythonstat

title: bcc-pythonstat(8)

summary: Activity stats from

text:
# bcc-pythonstat(8)

## NAME
ustat, javastat, nodestat, perlstat, phpstat, pythonstat, rubystat, tclstat - Activity stats from high-level languages.

## SYNOPSIS
javastat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

nodestat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

perlstat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

phpstat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

pythonstat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

rubystat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

tclstat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

ustat [-l {java,node,perl,php,python,ruby,tcl}] [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

## DESCRIPTION
This is "top" for high-level language events, such as garbage collections, exceptions, thread creations, object allocations, method calls, and more. The events are aggregated for each process and printed in a top-like table, which can be sorted by various fields. Not all language runtimes provide the same set of details.

This uses in-kernel eBPF maps to store per process summaries for efficiency.

This tool relies on USDT probes embedded in many high-level languages, such as Java, Node, Perl, PHP, Python, Ruby, and Tcl. It requires a runtime instrumented with these probes, which in some cases requires building from source with a USDT-specific flag, such as "--enable-dtrace" or "--with-dtrace". For Java, some probes are not enabled by default, and can be turned on by running the Java process with the "-XX:+ExtendedDTraceProbes" flag. For PHP processes, the environment variable USE_ZEND_DTRACE must be set to 1.

Newly-created processes will only be traced at the next interval. If you run this tool with a short interval (say, 1-5 seconds), this should be virtually unnoticeable. For longer intervals, you might miss processes that were started and terminated during the interval window.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-l {java,node,perl,php,python,ruby,tcl}

The language to trace. By default, all languages are traced.

-C

Do not clear the screen between updates.

-S {cload,excp,gc,method,objnew,thread}

Sort the output by the specified field.

-r MAXROWS

Do not print more than this number of rows.

-d

Print the resulting BPF program, for debugging purposes.

interval

Interval between updates, seconds.

count

Number of interval summaries.

## EXAMPLES
Summarize activity in high-level languages, 1 second refresh:

\# ustat

Don't clear the screen, and top 8 rows only:

\# ustat -Cr 8

5 second summaries, 10 times only:

\# ustat 5 10

## FIELDS
loadavg

The contents of /proc/loadavg

PID

Process ID.

CMDLINE

Process command line (often the second and following arguments will give you a hint as to which application is being run.

METHOD/s

Count of method invocations during interval.

GC/s

Count of garbage collections during interval.

OBJNEW/s

Count of objects allocated during interval.

CLOAD/s

Count of classes loaded during interval.

EXC/s

Count of exceptions thrown during interval.

THR/s

Count of threads created during interval.

## OVERHEAD
When using this tool with high-frequency events, such as method calls, a very significant slow-down can be expected. However, many of the high-level languages covered by this tool already have a fairly high per-method invocation cost, especially when running in interpreted mode. For the lower-frequency events, such as garbage collections or thread creations, the overhead should not be significant. Specifically, when probing Java processes and not using the "-XX:+ExtendedDTraceProbes" flag, the most expensive probes are not emitted, and the overhead should be acceptable.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _example.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein

## SEE ALSO
trace(8), argdist(8), tplist(8)


bcc-rdmaucma

title: bcc-rdmaucma(8)

text:
# bcc-rdmaucma(8)

## NAME
rdmaucma - Trace RDMA Userspace Connection Manager Access Event. For Linux, uses BCC, eBPF.

## SYNOPSIS
rdmaucma [-h] [-D]

## DESCRIPTION
This program traces RDMA UCMA(Userspace Connection Manager Access) events, This can be useful to analyze issues on RDMA CM.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print usage message.

-D

Show debug infomation of bpf text.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
zhenwei pi

## SEE ALSO
rdma(8)


bcc-readahead

title: bcc-readahead(8)

summary: Show performance of read-ahead cache

text:
# bcc-readahead(8)

## NAME
readahead - Show performance of read-ahead cache

## SYNOPSIS
readahead [-d DURATION]

## DESCRIPTION
The tool shows the performance of read-ahead caching on the system under a given load to investigate any caching issues. It shows a count of unused pages in the cache and also prints a histogram showing how long they have remained there.

This tool traces the __do_page_cache_readahead() kernel function to track entry and exit in the readahead mechanism in the kernel and then uses __page_cache_alloc() and mark_page_accessed() functions to calculate the age of the page in the cache as well as see how many are left unaccessed.

Since this uses BPF, only the root user can use this tool.

### NOTE ON KPROBES USAGE
Since the tool uses Kprobes, depending on your linux kernel's compilation, these functions may be inlined and hence not available for Kprobes. To see whether you have the functions available, check vmlinux source and binary to confirm whether inlining is happening or not. You can also check /proc/kallsyms on the host and verify if the target functions are present there before using this.

## REQUIREMENTS
CONFIG_BPF, bcc

## OPTIONS
-h Print usage message

-d DURATION

Trace the read-ahead caching system for DURATION seconds

## EXAMPLES
Trace for 30 seconds and show histogram of page age (ms) in read-ahead cache along with unused page count:

\# readahead -d 30

## OVERHEAD
The kernel functions instrumented by this program could be high-frequency depending on the profile of the application (for example sequential IO). We advise the users to measure and monitor the overhead before leaving this turned on in production environments.

## SOURCE
This originated as a bpftrace tool from the book "BPF Performance Tools", published by Addison Wesley (2019):

http://www.brendangregg.com/bpf-performance-tools-book.html

See the book for more documentation on this tool.

This version is in the BCC repository:

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Suchakra Sharma

## SEE ALSO
readahead(2), madvise(2)


bcc-reset-trace

title: bcc-reset-trace(8)

summary: reset the state of tracing.

text:
# bcc-reset-trace(8)

## NAME
reset-trace - reset the state of tracing.

## SYNOPSIS
reset-trace [-F] [-h] [-q] [-v]

## DESCRIPTION
You will probably never need this tool. If you kill -9 a bcc tool (plus other signals, like SIGTERM), or if a bcc tool crashes, then kernel tracing can be left in a semi-enabled state. It's not as bad as it sounds: there may just be overhead for writing to ring buffers that are never read. This tool can be used to clean up the tracing state, and reset and disable active tracing.

Make sure no other tracing sessions are active. This tool might stop them from functioning (perhaps ungracefully).

This specifically clears the state in at least the following files in /sys/kernel/debug/tracing: kprobe_events, uprobe_events, trace_pipe. Other tracing facilities (ftrace) are checked, and if not in an expected state, a note is printed. All tracing files can be reset with -F for force, but this will interfere with any other running tracing sessions (eg, ftrace).

## REQUIREMENTS
/sys/kernel/debug mounted as debugfs

## OPTIONS
-F

Force. Will reset all tracing facilities, including those not used by bcc (ftrace). You shouldn't need to use this.

-h

USAGE message.

-q

Quiet. No output while working.

-v

Verbose: print what it is doing.

## EXAMPLES
Reset the state of tracing:

\# reset-trace

Verbose:

\# reset-trace -v

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg


bcc-rubycalls

title: bcc-rubycalls(8)

summary: Summarize method calls

text:
# bcc-rubycalls(8)

## NAME
ucalls, javacalls, perlcalls, phpcalls, pythoncalls, rubycalls, tclcalls - Summarize method calls from high-level languages and Linux syscalls.

## SYNOPSIS
javacalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

perlcalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

phpcalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

pythoncalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

rubycalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

tclcalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

ucalls [-l {java,perl,php,python,ruby}] [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

## DESCRIPTION
This tool summarizes method calls from high-level languages such as Java, Perl, PHP, Python, Ruby, and Tcl. It can also trace Linux system calls. Whenever a method is invoked, ucalls records the call count and optionally the method's execution time (latency) and displays a summary.

This uses in-kernel eBPF maps to store per process summaries for efficiency.

This tool relies on USDT probes embedded in many high-level languages, such as Java, Perl, PHP, Python, Ruby, and Tcl. It requires a runtime instrumented with these probes, which in some cases requires building from source with a USDT-specific flag, such as "--enable-dtrace" or "--with-dtrace". For Java, method probes are not enabled by default, and can be turned on by running the Java process with the "-XX:+ExtendedDTraceProbes" flag. For PHP processes, the environment variable USE_ZEND_DTRACE must be set to 1.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-l {java,perl,php,python,ruby,tcl}

The language to trace. If not provided, only syscalls are traced (when the -S option is used).

-T TOP

Print only the top methods by frequency or latency.

-L

Collect method invocation latency (duration).

-S

Collect Linux syscalls frequency and timing.

-v

Print the resulting BPF program, for debugging purposes.

-m

Print times in milliseconds (the default is microseconds).

pid

The process id to trace.

interval

Print summary after this number of seconds and then exit. By default, wait for Ctrl+C to terminate.

## EXAMPLES
Trace the top 10 Ruby method calls:

\# ucalls -T 10 -l ruby 1344

Trace Python method calls and Linux syscalls including latency in milliseconds:

\# ucalls -l python -mL 2020

Trace only syscalls and print a summary after 10 seconds:

\# ucalls -S 788 10

## OVERHEAD
Tracing individual method calls will produce a considerable overhead in all high-level languages. For languages with just-in-time compilation, such as Java, the overhead can be more considerable than for interpreted languages. On the other hand, syscall tracing will typically be tolerable for most processes, unless they have a very unusual rate of system calls.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _example.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein

## SEE ALSO
ustat(8), argdist(8)


bcc-rubyflow

title: bcc-rubyflow(8)

summary: Print a flow graph of method

text:
# bcc-rubyflow(8)

## NAME
uflow, javaflow, perlflow, phpflow, pythonflow, rubyflow, tclflow - Print a flow graph of method calls in high-level languages.

## SYNOPSIS
javaflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

perlflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

phpflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

pythonflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

rubyflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

tclflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

uflow [-h] [-M METHOD] [-C CLAZZ] [-v] [-l {java,perl,php,python,ruby,tcl}] pid

## DESCRIPTION
uflow traces method calls and prints them in a flow graph that can facilitate debugging and diagnostics by following the program's execution (method flow).

This tool relies on USDT probes embedded in many high-level languages, such as Java, Perl, PHP, Python, Ruby, and Tcl. It requires a runtime instrumented with these probes, which in some cases requires building from source with a USDT-specific flag, such as "--enable-dtrace" or "--with-dtrace". For Java processes, the startup flag "-XX:+ExtendedDTraceProbes" is required. For PHP processes, the environment variable USE_ZEND_DTRACE must be set to 1.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-M METHOD

Print only method calls where the method name begins with this string.

-C CLAZZ

Print only method calls where the class name begins with this string. The class name interpretation strongly depends on the language. For example, in Java use "package/subpackage/ClassName" to refer to classes.

-v

Print the resulting BPF program, for debugging purposes.

{java,perl,php,python,ruby,tcl}

The language to trace.

pid

The process id to trace.

## EXAMPLES
Follow method flow in a Ruby process:

\# uflow ruby 148

Follow method flow in a Java process where the class name is java.lang.Thread:

\# uflow -C java/lang/Thread java 1802

## FIELDS
CPU

The CPU number on which the method was invoked. This is useful to easily see where the output skips to a different CPU.

PID

The process id.

TID

The thread id.

TIME

The duration of the method call.

METHOD

The method name.

## OVERHEAD
This tool has extremely high overhead because it prints every method call. For some scenarios, you might see lost samples in the output as the tool is unable to keep up with the rate of data coming from the kernel. Filtering by class or method prefix can help reduce the amount of data printed, but there is still a very high overhead in the collection mechanism. Do not use for performance- sensitive production scenarios, and always test first.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _example.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein

## SEE ALSO
trace(8), ustat(8)


bcc-rubygc

title: bcc-rubygc(8)

summary: Trace garbage collection events in

text:
# bcc-rubygc(8)

## NAME
ugc, javagc, nodegc, pythongc, rubygc - Trace garbage collection events in high-level languages.

## SYNOPSIS
javagc [-h] [-v] [-m] [-M MINIMUM] [-F FILTER] pid

nodegc [-h] [-v] [-m] [-M MINIMUM] [-F FILTER] pid

pythongc [-h] [-v] [-m] [-M MINIMUM] [-F FILTER] pid

rubygc [-h] [-v] [-m] [-M MINIMUM] [-F FILTER] pid

ugc [-h] [-v] [-m] [-M MINIMUM] [-F FILTER] [-l {java,node,python,ruby}] pid

## DESCRIPTION
This traces garbage collection events as they occur, including their duration and any additional information (such as generation collected or type of GC) provided by the respective language's runtime.

This tool relies on USDT probes embedded in many high-level languages, such as Java, Node, Python, and Ruby. It requires a runtime instrumented with these probes, which in some cases requires building from source with a USDT-specific flag, such as "--enable-dtrace" or "--with-dtrace".

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-v

Print the resulting BPF program, for debugging purposes.

-m

Print times in milliseconds. The default is microseconds.

-M MINIMUM

Display only collections that are longer than this threshold. The value is given in milliseconds. The default is to display all collections.

-F FILTER

Display only collections whose textual description matches (contains) this string. The default is to display all collections. Note that the filtering here is performed in user-space, and not as part of the BPF program. This means that if you have thousands of collection events, specifying this filter will not reduce the amount of data that has to be transferred from the BPF program to the user-space script.

{java,node,python,ruby}

The language to trace.

pid

The process id to trace.

## EXAMPLES
Trace garbage collections in a specific Node process:

\# ugc node 148

Trace garbage collections in a specific Java process, and print GC times in

milliseconds: # ugc -m java 6004

Trace garbage collections in a specific Java process, and display them only if

they are longer than 10ms and have the string "Tenured" in their detailed description: # ugc -M 10 -F Tenured java 6004

## FIELDS
START

The start time of the GC, in seconds from the beginning of the trace.

TIME

The duration of the garbage collection event.

DESCRIPTION

The runtime-provided description of this garbage collection event.

## OVERHEAD
Garbage collection events, even if frequent, should not produce a considerable overhead when traced because they are still not very common. Even hundreds of GCs per second (which is a very high rate) will still produce a fairly negligible overhead.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _example.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein

## SEE ALSO
trace(8), ustat(8), uobjnew(8)


bcc-rubyobjnew

title: bcc-rubyobjnew(8)

summary: Summarize object allocations in

text:
# bcc-rubyobjnew(8)

## NAME
uobjnew, cobjnew, javaobjnew, rubyobjnew, tclobjnew - Summarize object allocations in high-level languages.

## SYNOPSIS
cobjnew [-h] [-C TOP_COUNT] [-S TOP_SIZE] [-v] pid [interval]

javaobjnew [-h] [-C TOP_COUNT] [-S TOP_SIZE] [-v] pid [interval]

rubyobjnew [-h] [-C TOP_COUNT] [-S TOP_SIZE] [-v] pid [interval]

tclobjnew [-h] [-C TOP_COUNT] [-S TOP_SIZE] [-v] pid [interval]

uobjnew [-h] [-C TOP_COUNT] [-S TOP_SIZE] [-v] [-l {c,java,ruby,tcl}] pid [interval]

## DESCRIPTION
uobjnew traces object allocations in high-level languages (including "malloc") and prints summaries of the most frequently allocated types by number of objects or number of bytes.

This tool relies on USDT probes embedded in many high-level languages, such as C, Java, Ruby, and Tcl. It requires a runtime instrumented with these probes, which in some cases requires building from source with a USDT-specific flag, such as "--enable-dtrace" or "--with-dtrace". For Java, the Java process must be started with the "-XX:+ExtendedDTraceProbes" flag.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-C TOP_COUNT

Print the top object types sorted by number of instances.

-S TOP_SIZE

Print the top object types sorted by size.

-v

Print the resulting BPF program, for debugging purposes.

{c,java,ruby,tcl}

The language to trace.

pid

The process id to trace.

interval

Wait this many seconds and then print the summary and exit. By default, wait for Ctrl+C to exit.

## EXAMPLES
Trace object allocations in a Ruby process:

\# uobjnew ruby 148

Trace object allocations from "malloc" and print the top 10 by total size:

\# uobjnew -S 10 c 1788

## FIELDS
TYPE

The object type being allocated. For C (malloc), this is the block size.

ALLOCS

The number of objects allocated.

BYTES

The number of bytes allocated.

## OVERHEAD
Object allocation events are quite frequent, and therefore the overhead from running this tool can be considerable. Use with caution and make sure to test before using in a production environment. Nonetheless, even thousands of allocations per second will likely produce a reasonable overhead when investigating a problem.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _example.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein

## SEE ALSO
ustat(8), ugc(8), memleak(8)


bcc-rubystat

title: bcc-rubystat(8)

summary: Activity stats from

text:
# bcc-rubystat(8)

## NAME
ustat, javastat, nodestat, perlstat, phpstat, pythonstat, rubystat, tclstat - Activity stats from high-level languages.

## SYNOPSIS
javastat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

nodestat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

perlstat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

phpstat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

pythonstat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

rubystat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

tclstat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

ustat [-l {java,node,perl,php,python,ruby,tcl}] [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

## DESCRIPTION
This is "top" for high-level language events, such as garbage collections, exceptions, thread creations, object allocations, method calls, and more. The events are aggregated for each process and printed in a top-like table, which can be sorted by various fields. Not all language runtimes provide the same set of details.

This uses in-kernel eBPF maps to store per process summaries for efficiency.

This tool relies on USDT probes embedded in many high-level languages, such as Java, Node, Perl, PHP, Python, Ruby, and Tcl. It requires a runtime instrumented with these probes, which in some cases requires building from source with a USDT-specific flag, such as "--enable-dtrace" or "--with-dtrace". For Java, some probes are not enabled by default, and can be turned on by running the Java process with the "-XX:+ExtendedDTraceProbes" flag. For PHP processes, the environment variable USE_ZEND_DTRACE must be set to 1.

Newly-created processes will only be traced at the next interval. If you run this tool with a short interval (say, 1-5 seconds), this should be virtually unnoticeable. For longer intervals, you might miss processes that were started and terminated during the interval window.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-l {java,node,perl,php,python,ruby,tcl}

The language to trace. By default, all languages are traced.

-C

Do not clear the screen between updates.

-S {cload,excp,gc,method,objnew,thread}

Sort the output by the specified field.

-r MAXROWS

Do not print more than this number of rows.

-d

Print the resulting BPF program, for debugging purposes.

interval

Interval between updates, seconds.

count

Number of interval summaries.

## EXAMPLES
Summarize activity in high-level languages, 1 second refresh:

\# ustat

Don't clear the screen, and top 8 rows only:

\# ustat -Cr 8

5 second summaries, 10 times only:

\# ustat 5 10

## FIELDS
loadavg

The contents of /proc/loadavg

PID

Process ID.

CMDLINE

Process command line (often the second and following arguments will give you a hint as to which application is being run.

METHOD/s

Count of method invocations during interval.

GC/s

Count of garbage collections during interval.

OBJNEW/s

Count of objects allocated during interval.

CLOAD/s

Count of classes loaded during interval.

EXC/s

Count of exceptions thrown during interval.

THR/s

Count of threads created during interval.

## OVERHEAD
When using this tool with high-frequency events, such as method calls, a very significant slow-down can be expected. However, many of the high-level languages covered by this tool already have a fairly high per-method invocation cost, especially when running in interpreted mode. For the lower-frequency events, such as garbage collections or thread creations, the overhead should not be significant. Specifically, when probing Java processes and not using the "-XX:+ExtendedDTraceProbes" flag, the most expensive probes are not emitted, and the overhead should be acceptable.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _example.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein

## SEE ALSO
trace(8), argdist(8), tplist(8)


bcc-runqlat

title: bcc-runqlat(8)

text:
# bcc-runqlat(8)

## NAME
runqlat - Run queue (scheduler) latency as a histogram.

## SYNOPSIS
runqlat [-h] [-T] [-m] [-P] [--pidnss] [-L] [-p PID] [interval] [count]

## DESCRIPTION
This measures the time a task spends waiting on a run queue (or equivalent scheduler data structure) for a turn on-CPU, and shows this time as a histogram. This time should be small, but a task may need to wait its turn due to CPU load. The higher the CPU load, the longer a task will generally need to wait its turn.

This tool measures two types of run queue latency:

1. The time from a task being enqueued on a run queue to its context switch and execution. This traces ttwu_do_wakeup(), wake_up_new_task() -> finish_task_switch() with either raw tracepoints (if supported) or kprobes and instruments the run queue latency after a voluntary context switch.

2. The time from when a task was involuntary context switched and still in the runnable state, to when it next executed. This is instrumented from finish_task_switch() alone.

This tool uses in-kernel eBPF maps for storing timestamps and the histogram, for efficiency. Despite this, the overhead of this tool may become significant for some workloads: see the OVERHEAD section.

This works by tracing various kernel scheduler functions using dynamic tracing, and will need updating to match any changes to these functions.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print usage message.

-T

Include timestamps on output.

-m

Output histogram in milliseconds.

-P

Print a histogram for each PID.

--pidnss

Print a histogram for each PID namespace (short for PID namespaces). For container analysis.

-L

Print a histogram for each thread ID.

-p PID

Only show this PID (filtered in kernel for efficiency).

interval

Output interval, in seconds.

count

Number of outputs.

## EXAMPLES
Summarize run queue latency as a histogram:

\# runqlat

Print 1 second summaries, 10 times:

\# runqlat 1 10

Print 1 second summaries, using milliseconds as units for the histogram, and include timestamps on output:

\# runqlat -mT 1

Trace PID 186 only, 1 second summaries:

\# runqlat -P 185 1

## FIELDS
usecs

Microsecond range

msecs

Millisecond range

count

How many times a task event fell into this range

distribution

An ASCII bar chart to visualize the distribution (count column)

## OVERHEAD
This traces scheduler functions, which can become very frequent. While eBPF has very low overhead, and this tool uses in-kernel maps for efficiency, the frequency of scheduler events for some workloads may be high enough that the overhead of this tool becomes significant. Measure in a lab environment to quantify the overhead before use.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
runqlen(8), runqslower(8), pidstat(1)


bcc-runqlen

title: bcc-runqlen(8)

text:
# bcc-runqlen(8)

## NAME
runqlen - Scheduler run queue length as a histogram.

## SYNOPSIS
runqlen [-h] [-T] [-O] [-C] [interval] [count]

## DESCRIPTION
This program summarizes scheduler queue length as a histogram, and can also show run queue occupancy. It works by sampling the run queue length on all CPUs at 99 Hertz.

This tool can be used to identify imbalances, eg, when processes are bound to CPUs causing queueing, or interrupt mappings causing the same.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print usage message.

-T

Include timestamps on output.

-O

Report run queue occupancy.

-C

Report for each CPU.

interval

Output interval, in seconds.

count

Number of outputs.

## EXAMPLES
Summarize run queue length as a histogram:

\# runqlen

Print 1 second summaries, 10 times:

\# runqlen 1 10

Print output every second, with timestamps, and show each CPU separately:

\# runqlen -CT 1

Print run queue occupancy every second:

\# runqlen -O 1

Print run queue occupancy, with timestamps, for each CPU:

\# runqlen -COT 1

## FIELDS
runqlen

Scheduler run queue length: the number of threads (tasks) waiting to run, (excluding including the currently running task).

count

Number of samples at this queue length.

distribution

An ASCII bar chart to visualize the distribution (count column)

## OVERHEAD
This uses sampling at 99 Hertz (on all CPUs), and in-kernel summaries, which should make overhead negligible. This does not trace scheduler events, like runqlen does, which comes at a much higher overhead cost.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
runqlat(8), runqslower(8), pidstat(1)


bcc-runqslower

title: bcc-runqslower(8)

summary: Trace long process scheduling delays.

text:
# bcc-runqslower(8)

## NAME
runqslower - Trace long process scheduling delays.

## SYNOPSIS
runqslower [-p PID] [-t TID] [-P] [min_us]

## DESCRIPTION
This measures the time a task spends waiting on a run queue (or equivalent scheduler data structure) for a turn on-CPU, and shows occurrences of time exceeding passed threshold. This time should be small, but a task may need to wait its turn due to CPU load. The higher the CPU load, the longer a task will generally need to wait its turn.

This tool measures two types of run queue latency:

1. The time from a task being enqueued on a run queue to its context switch and execution. This traces ttwu_do_wakeup(), wake_up_new_task() -> finish_task_switch() with either raw tracepoints (if supported) or kprobes and instruments the run queue latency after a voluntary context switch.

2. The time from when a task was involuntary context switched and still in the runnable state, to when it next executed. This is instrumented from finish_task_switch() alone.

The overhead of this tool may become significant for some workloads: see the OVERHEAD section.

This works by tracing various kernel scheduler functions using dynamic tracing, and will need updating to match any changes to these functions.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print usage message.

-p PID

Only show this PID (filtered in kernel for efficiency).

-t TID

Only show this TID (filtered in kernel for efficiency).

min_us

Minimum scheduling delay in microseconds to output.

-P

Also show previous task comm and TID.

## EXAMPLES
Show scheduling delays longer than 10ms:

\# runqslower

Show scheduling delays longer than 1ms for process with PID 123:

\# runqslower -p 123 1000

## FIELDS
TIME

Time of when scheduling event occurred.

COMM

Process name.

PID

Process ID.

LAT(us)

Scheduling latency from time when task was ready to run to the time it was assigned to a CPU to run.

## OVERHEAD
This traces scheduler functions, which can become very frequent. While eBPF has very low overhead, and this tool uses in-kernel maps for efficiency, the frequency of scheduler events for some workloads may be high enough that the overhead of this tool becomes significant. Measure in a lab environment to quantify the overhead before use.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Ivan Babrou, original BCC Python version Andrii Nakryiko, CO-RE version

## SEE ALSO
runqlen(8), runqlat(8), pidstat(1)


bcc-shmsnoop

title: bcc-shmsnoop(8)

text:
# bcc-shmsnoop(8)

## NAME
shmsnoop - Trace System V shared memory syscalls. Uses Linux eBPF/bcc.

## SYNOPSIS
shmsnoop [-h] [-T] [-p] [-t] [-d DURATION] [-n NAME]

## DESCRIPTION
shmsnoop traces System V shared memory syscalls: shmget, shmat, shmdt, shmctl

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print usage message.

-T

Include a timestamp column.

-p PID

Trace this process ID only (filtered in-kernel).

-t TID

Trace this thread ID only (filtered in-kernel).

-d DURATION

Total duration of trace in seconds.

-n NAME

Only print command lines matching this command name (regex)

## EXAMPLES
Trace all shm* syscalls:

\# shmsnoop

Trace all shm* syscalls, and include timestamps:

\# shmsnoop -T

Only trace shm* syscalls where the process contains "server":

\# shmsnoop -n server

## FIELDS
TIME(s)

Time of shm syscall return, in seconds.

PID

Process ID

COMM

Process/command name.

RET

Return value of shm syscall.

ARGS

"arg: value" couples that represent given syscall arguments as described in their manpage

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Jiri Olsa

## SEE ALSO
opensnoop(1)


bcc-slabratetop

title: bcc-slabratetop(8)

text:
# bcc-slabratetop(8)

## NAME
slabratetop - Kernel SLAB/SLUB memory cache allocation rate top. Uses Linux BPF/bcc.

## SYNOPSIS
slabratetop [-h] [-C] [-r MAXROWS] [interval] [count]

## DESCRIPTION
This is top for the the rate of kernel SLAB/SLUB memory allocations. It works by tracing kmem_cache_alloc() calls, a commonly used interface for kernel memory allocation (SLAB or SLUB). It summarizes the rate and total bytes allocated of these calls per interval: the activity. Compare this to slabtop(1), which shows the current static volume of the caches.

This tool uses kernel dynamic tracing of the kmem_cache_alloc() function.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-C

Don't clear the screen.

-r MAXROWS

Maximum number of rows to print. Default is 20.

interval

Interval between updates, seconds.

count

Number of interval summaries.

## EXAMPLES
Summarize active kernel SLAB/SLUB calls (kmem_cache_alloc()), showing the top 20 caches every second:

\# slabratetop

Don't clear the screen, and top 8 rows only:

\# slabratetop -Cr 8

5 second summaries, 10 times only:

\# slabratetop 5 10

## FIELDS
loadavg:

The contents of /proc/loadavg

CACHE

Kernel cache name.

ALLOCS

Allocations (number of calls).

BYTES

Total bytes allocated.

## OVERHEAD
If kmem_cache_alloc() is called at a high rate (eg, >100k/second) the overhead of this tool might begin to be measurable. The rate can be seen in the ALLOCS column of the output.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
slabtop(1)


bcc-sofdsnoop

title: bcc-sofdsnoop(8)

summary: traces FDs passed by sockets

text:
# bcc-sofdsnoop(8)

## NAME
sofdsnoop - traces FDs passed by sockets

## SYNOPSIS
usage: sofdsnoop [-h] [-T] [-p PID] [-t TID] [-n NAME] [-d DURATION]

## DESCRIPTION
Trace file descriptors passed via socket

### optional arguments:
-h, --help

show this help message and exit

-T, --timestamp

include timestamp on output

-p PID, --pid PID

trace this PID only

-t TID, --tid TID

trace this TID only

-n NAME, --name NAME

only print process names containing this name

-d DURATION, --duration DURATION

total duration of trace in seconds

## EXAMPLES
Trace passed file descriptors

\# sofdsnoop

Include timestamps

\# sofdsnoop -T

Only trace PID 181

\# sofdsnoop -p 181

Only trace TID 123

\# sofdsnoop -t 123

Trace for 10 seconds only

\# sofdsnoop -d 10

Only print process names containing "main"

\# sofdsnoop -n main

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.


bcc-softirqs

title: bcc-softirqs(8)

text:
# bcc-softirqs(8)

## NAME
softirqs - Measure soft IRQ (soft interrupt) event time. Uses Linux eBPF/bcc.

## SYNOPSIS
softirqs [-h] [-T] [-N] [-C] [-d] [-c CPU] [interval] [count]

## DESCRIPTION
This summarizes the time spent servicing soft IRQs (soft interrupts), and can show this time as either totals or histogram distributions. A system-wide summary of this time is shown by the %soft column of mpstat(1), and soft IRQ event counts (but not times) are available in /proc/softirqs.

This tool uses the irq:softirq_enter and irq:softirq_exit kernel tracepoints, which is a stable tracing mechanism. BPF programs can attach to tracepoints from Linux 4.7 only. An older version of this tool is available in tools/old, and uses kprobes instead of tracepoints.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print usage message.

-T

Include timestamps on output.

-N

Output in nanoseconds.

-C

Show the number of soft irq events.

-d

Show IRQ time distribution as histograms.

-c CPU

Trace on this CPU only.

## EXAMPLES
Sum soft IRQ event time until Ctrl-C:

\# softirqs

Show the number of soft irq events:

\# softirqs -C

Show soft IRQ event time as histograms:

\# softirqs -d

Print 1 second summaries, 10 times:

\# softirqs 1 10

1 second summaries, printed in nanoseconds, with timestamps:

\# softirqs -NT 1

Sum soft IRQ event time on CPU 1 until Ctrl-C:

\# softirqs -c 1

## FIELDS
SOFTIRQ

The kernel function name that performs the soft IRQ action.

TOTAL_usecs

Total time spent in this soft IRQ function in microseconds.

TOTAL_nsecs

Total time spent in this soft IRQ function in nanoseconds.

usecs

Range of microseconds for this bucket.

nsecs

Range of nanoseconds for this bucket.

count

Number of soft IRQs in this time range.

distribution

ASCII representation of the distribution (the count column).

## OVERHEAD
This traces kernel functions and maintains in-kernel counts, which are asynchronously copied to user-space. While the rate of interrupts be very high (>1M/sec), this is a relatively efficient way to trace these events, and so the overhead is expected to be small for normal workloads, but could become noticeable for heavy workloads. Measure in a test environment before use.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHORS
Brendan Gregg, Sasha Goldshtein, Rocky Xing

## SEE ALSO
hardirqs(8)


bcc-softirqslower

title: bcc-softirqslower(8)

text:
# bcc-softirqslower(8)

## NAME
softirqslower - Trace softirq handlers with latency exceeding a threshold

## SYNOPSIS
softirqslower.py [min_us]

softirqslower.py [-c CPU] [min_us]

## DESCRIPTION
The softirqslower tool traces softirq events that exceed a specified latency threshold. This tool helps diagnose interrupt handling bottlenecks by measuring two critical latency dimensions:

- irq(hard) to softirq: Time delay between hardware interrupt completion and softirq execution start

- softirq runtime: Actual execution duration of softirq handlers

Developed as part of the BCC (BPF Compiler Collection) tools, it's particularly useful for network stack optimization, scheduler tuning, and real-time systems analysis.

## OPTIONS
min_us

Minimum latency threshold to trace (in microseconds). Default: 10,000 μs (10 ms)

-c, --cpu CPU

Trace only events on the specified CPU core

## OUTPUT FORMAT
The tool outputs events in the following table format:

TIME STAGE SOFTIRQ LAT(us) CPU COMM

With fields defined as:

TIME

Event timestamp (HH:MM:SS format)

STAGE

Latency measurement type:

irq(hard) to softirq

Time from hardware interrupt completion to softirq dispatch

softirq runtime

SoftIRQ handler execution duration

SOFTIRQ

SoftIRQ category (case-sensitive). Common values:

NET_RX

Network reception processing

NET_TX

Network transmission processing

TIMER

Timer callbacks

SCHED

Scheduler operations

RCU

Read-Copy-Update synchronization

TASKLET

Deferred task execution

HRTIMER

High-resolution timers

BLOCK

Block device operations

LAT(us)

Measured latency in microseconds

CPU

CPU core where softirq was handled (0-based numbering)

COMM

Process context handling the softirq:

swapper/N

Idle thread for CPU N

ksoftirqd/N

Softirq daemon for CPU N

<process>

User-space process

## EXAMPLES
Trace softirqs exceeding 10μs latency:

softirqslower 10

Monitor only CPU core 1:

softirqslower -c 1

Trace network-related softirq delays (>50μs):

softirqslower 50 | grep -E 'net_rx|net_tx'

Capture RCU delays longer than 100μs to file:

softirqslower 100 | grep rcu > rcu_latency.log

## SIGNALS
Ctrl+C Stop tracing and exit gracefully

## AUTHOR
Chenyue Zhou

## SEE ALSO
runqslower(8), hardirqs(8), softirqs(8), trace(8), funclatency(8),

## REPORTING BUGS
BCC Tools Issue Tracker: https://github.com/iovisor/bcc/issues


bcc-solisten

title: bcc-solisten(8)

summary: Trace listening socket

text:
# bcc-solisten(8)

## NAME
solisten - Trace listening socket

## SYNOPSIS
usage: solisten [-h] [--show-netns] [-p PID] [-n NETNS]

## DESCRIPTION
All IPv4 and IPv6 listen attempts are traced, even if they ultimately fail or the listening program is not willing to accept().

### optional arguments:
-h, --help

show this help message and exit

--show-netns

show network namespace

-p PID, --pid PID

trace this PID only

-n NETNS, --netns NETNS

trace this Network Namespace only

## EXAMPLES
Stream socket listen:

\# solisten

Stream socket listen for specified PID only

\# solisten -p 1234

Stream socket listen for the specified network namespace ID only

\# solisten --netns 4242

Show network ns ID (useful for containers)

\# solisten --show-netns

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.


bcc-spfdsnoop

title: bcc-spfdsnoop(8)

text:
# bcc-spfdsnoop(8)

## NAME
sofdsnoop - Trace FDs passed through unix sockets. Uses Linux eBPF/bcc.

## SYNOPSIS
sofdsnoop [-h] [-T] [-p PID] [-t TID] [-n NAME] [-d DURATION]

## DESCRIPTION
sofdsnoop traces FDs passed through unix sockets

Every file descriptor that is passed via unix sockets os displayed on separate line together with process info (TID/COMM columns), ACTION details (SEND/RECV), file descriptor number (FD) and its translation to file if available (NAME).

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print usage message.

-T

Include a timestamp column.

-p PID

Trace this process ID only (filtered in-kernel).

-t TID

Trace this thread ID only (filtered in-kernel).

-d DURATION

Total duration of trace in seconds.

-n NAME

Only print command lines matching this command name (regex)

## EXAMPLES
Trace all sockets:

\# sofdsnoop

Trace all sockets, and include timestamps:

\# sofdsnoop -T

Only trace sockets where the process contains "server":

\# sofdsnoop -n server

## FIELDS
TIME(s)

Time of SEDN/RECV actions, in seconds.

ACTION

Operation on the fd SEND/RECV.

TID

Process TID

COMM

Parent process/command name.

SOCKET

The socket carrier.

FD

file descriptor number

NAME

file name for SEND lines

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Jiri Olsa

## SEE ALSO
opensnoop(1)


bcc-sslsniff

title: bcc-sslsniff(8)

text:
# bcc-sslsniff(8)

## NAME
sslsniff - Print data passed to OpenSSL, GnuTLS or NSS. Uses Linux eBPF/bcc.

## SYNOPSIS
sslsniff [-h] [-p PID] [-u UID] [-x] [-c COMM] [-o] [-g] [-n] [-d] [--hexdump] [--max-buffer-size SIZE] [-l] [--handshake] [--extra-lib EXTRA_LIB]

## DESCRIPTION
sslsniff prints data sent to write/send and read/recv functions of OpenSSL, GnuTLS and NSS, allowing us to read plain text content before encryption (when writing) and after decryption (when reading).

This works reading the second parameter of both functions (*buf).

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print usage message.

-p PID

Trace only functions in this process PID.

-u UID

Trace only calls made by this UID.

-x

Show extra fields: UID and TID.

-c COMM

Show only processes that match this COMM exactly.

-o, --no-openssl

Do not trace OpenSSL functions.

-g, --no-gnutls

Do not trace GnuTLS functions.

-n, --no-nss

Do not trace GnuTLS functions.

--hexdump

Show data as hexdump instead of trying to decode it as UTF-8

--max-buffer-size SIZE

Sets maximum buffer size of intercepted data. Longer values would be truncated. Default value is 8 Kib, maximum possible value is a bit less than 32 Kib.

-l, --latency

Show function latency in ms.

--handshake

Show handshake latency, enabled only if latency option is on.

--extra-lib EXTRA_LIB

Consist type of the library and library path separated by colon. Supported library types are: openssl, gnutls, nss. Can be specified multiple times.

## EXAMPLES
Print all calls to SSL write/send and read/recv system-wide:

\# sslsniff

Print only OpenSSL calls issued by user with UID 1000

\# sslsniff -u 1000 --no-nss --no-gnutls

Print SSL handshake event and latency for all traced functions:

\# sslsniff -l --handshake

Print only calls to OpenSSL from /some/path/libssl.so

sslsniff --no-openssl --no-gnutls --no-nss --extra-lib openssl:/some/path/libssl.so

## FIELDS
FUNC

Which function is being called (write/send or read/recv)

TIME

Time of the command, in seconds.

COMM

Entered command.

PID

Process ID calling SSL.

LEN

Bytes written or read by SSL functions.

UID

UID of the process, displayed only if launched with -x.

TID

Thread ID, displayed only if launched with -x.

LAT(ms)

Function latency in ms.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHORS
Adrian Lopez and Mark Drayton

## SEE ALSO
trace(8)


bcc-stackcount

title: bcc-stackcount(8)

text:
# bcc-stackcount(8)

## NAME
stackcount - Count function calls and their stack traces. Uses Linux eBPF/bcc.

## SYNOPSIS
stackcount [-h] [-p PID] [-c CPU] [-i INTERVAL] [-D DURATION] [-T]

[-r] [-s] [-P] [-K] [-U] [-v] [-d] [-f] [--debug] pattern

## DESCRIPTION
stackcount traces functions and frequency counts them with their entire stack trace, kernel stack and user stack, summarized in-kernel for efficiency. This allows higher frequency events to be studied. The output consists of unique stack traces, and their occurrence counts. In addition to kernel and user functions, kernel tracepoints and USDT tracepoint are also supported.

The pattern is a string with optional '*' wildcards, similar to file globbing. If you'd prefer to use regular expressions, use the -r option.

This tool only works on Linux 4.6+. Stack traces are obtained using the new `BPF_STACK_TRACE` APIs. For kernels older than 4.6, see the version under tools/old.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print usage message.

-r

Allow regular expressions for the search pattern. The default allows "*" wildcards only.

-s

Show address offsets.

-P

Display stacks separately for each process.

-K

Show kernel stack only.

-U

Show user stack only.

-T

Include a timestamp with interval output.

-v

Show raw addresses.

-d

Print a delimiter ("--") in-between the kernel and user stacks.

--debug

Print the source of the BPF program when loading it (for debugging purposes).

-i interval

Summary interval, in seconds.

-D duration

Total duration of trace, in seconds. -f Folded output format.

-p PID

Trace this process ID only (filtered in-kernel).

-c CPU

Trace this CPU only (filtered in-kernel).

pattern

A function name, or a search pattern. Can include wildcards ("*"). If the -r option is used, can include regular expressions.

## EXAMPLES
Count kernel and user stack traces for submit_bio():

\# stackcount submit_bio

Count stacks with a delimiter for submit_bio():

\# stackcount -d submit_bio

Count kernel stack trace only for submit_bio():

\# stackcount -K submit_bio

Count user stack trace only for submit_bio():

\# stackcount -U submit_bio

Count stack traces for ip_output():

\# stackcount ip_output

Show symbol offsets:

\# stackcount -s ip_output

Show offsets and raw addresses (verbose):

\# stackcount -sv ip_output

Count stacks for kernel functions matching tcp_send*:

\# stackcount 'tcp_send*'

Same as previous, but using regular expressions:

\# stackcount -r '^tcp_send.*'

Output every 5 seconds, with timestamps:

\# stackcount -Ti 5 ip_output

Only count stacks when PID 185 is on-CPU:

\# stackcount -p 185 ip_output

Only count stacks for CPU 1:

\# stackcount -c 1 put_prev_entity

Count user stacks for dynamic heap allocations with malloc in PID 185:

\# stackcount -p 185 c:malloc

Count user stacks for thread creation (USDT tracepoint) in PID 185:

\# stackcount -p 185 u:pthread:pthread_create

Count stacks for context switch events using a kernel tracepoint:

\# stackcount t:sched:sched_switch

## OVERHEAD
This summarizes unique stack traces in-kernel for efficiency, allowing it to trace a higher rate of function calls than methods that post-process in user space. The stack trace data is only copied to user space when the output is printed, which usually only happens once. The stack walking also happens in an optimized code path in the kernel thanks to the new BPF_STACK_TRACE table APIs, which should be more efficient than the manual walker in the eBPF tracer which older versions of this script used. With this in mind, call rates of < 10,000/sec would incur negligible overhead. Test before production use. You can also use funccount to get a handle on function call rates first.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg, Sasha Goldshtein

## SEE ALSO
stacksnoop(8), funccount(8)


bcc-statsnoop

title: bcc-statsnoop(8)

text:
# bcc-statsnoop(8)

## NAME
statsnoop - Trace stat() syscalls. Uses Linux eBPF/bcc.

## SYNOPSIS
statsnoop [-h] [-t] [-x] [-p PID]

## DESCRIPTION
statsnoop traces the different stat() syscalls, showing which processes are attempting to read information about which files. This can be useful for determining the location of config and log files, or for troubleshooting applications that are failing, especially on startup.

This works by tracing various kernel sys_stat() functions using dynamic tracing, and will need updating to match any changes to these functions.

This makes use of a Linux 4.4 feature (bpf_perf_event_output()); for kernels older than 4.4, see the version under tools/old, which uses an older mechanism.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print usage message.

-t

Include a timestamp column: in seconds since the first event, with decimal places.

-x

Only print failed stats.

-p PID

Trace this process ID only (filtered in-kernel).

## EXAMPLES
Trace all stat() syscalls:

\# statsnoop

Trace all stat() syscalls, and include timestamps:

\# statsnoop -t

Trace only stat() syscalls that failed:

\# statsnoop -x

Trace PID 181 only:

\# statsnoop -p 181

## FIELDS
TIME(s)

Time of the call, in seconds.

PID

Process ID

COMM

Process name

FD

File descriptor (if success), or -1 (if failed)

ERR

Error number (see the system's errno.h)

PATH

Open path

## OVERHEAD
This traces the kernel stat function and prints output for each event. As the rate of this is generally expected to be low (< 1000/s), the overhead is also expected to be negligible. If you have an application that is calling a high rate of stat()s, then test and understand overhead before use.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
opensnoop(1)


bcc-swapin

title: bcc-swapin(8)

text:
# bcc-swapin(8)

## NAME
swapin - Count swapins by process. Uses BCC/eBPF.

## SYNOPSIS
swapin

-h, --help

show this help message and exit

-T, --notime

do not show the timestamp (HH:MM:SS)

## DESCRIPTION
This tool counts swapins by process, to show which process is affected by swapping (if swap devices are in use). This can explain a significant source of application latency, if it has began swapping due to memory pressure on the system.

This works by tracing the swap_readpage() kernel function using dynamic instrumentation. This tool may need maintenance to keep working if that function changes in later kernels.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and BCC.

## EXAMPLES
Count swapins by process, showing per-second summaries.

\# swapin

## FIELDS
1st

The process name.

2nd

The process ID.

3rd

The count of swapins during that interval.

## OVERHEAD
The rate of swapins should be low (bounded by swapin device IOPS), such that the overhead of this tool is expected to be negligible.

## SOURCE
This originated as a bpftrace tool from the book "BPF Performance Tools", published by Addison Wesley (2019):

http://www.brendangregg.com/bpf-performance-tools-book.html

See the book for more documentation on this tool.

This version is in the BCC repository:

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
swapon(8)


bcc-syncsnoop

title: bcc-syncsnoop(8)

text:
# bcc-syncsnoop(8)

## NAME
syncsnoop - Trace sync() syscall. Uses Linux eBPF/bcc.

## SYNOPSIS
syncsnoop

## DESCRIPTION
syncsnoop traces calls to sync(), which flushes file system buffers to storage devices. These calls can cause performance perturbations, and it can be useful to know if they are happening and how frequently.

This works by tracing the kernel sys_sync() function using dynamic tracing, and will need updating to match any changes to this function.

This makes use of a Linux 4.4 feature (bpf_perf_event_output()); for kernels older than 4.4, see the version under tools/old, which uses an older mechanism.

This program is also a basic example of eBPF/bcc.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## EXAMPLES
Trace calls to sync():

\# syncsnoop

## FIELDS
TIME(s)

Time of the call, in seconds.

CALL

Call traced.

## OVERHEAD
This traces the kernel sync function and prints output for each event. As the rate of this is generally expected to be low (<< 100/s), the overhead is also expected to be negligible.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg, original BCC Python version Tiago Ilieve, CO-RE version

## SEE ALSO
iostat(1)


bcc-syscount

title: bcc-syscount(8)

summary: Summarize syscall counts and latencies.

text:
# bcc-syscount(8)

## NAME
syscount - Summarize syscall counts and latencies.

## SYNOPSIS
syscount [-h] [-p PID] [-t TID] [-c PPID] [-i INTERVAL] [-d DURATION] [-T TOP] [-x] [-e ERRNO] [-L] [-m] [-P] [-l] [--syscall SYSCALL]

## DESCRIPTION
This tool traces syscall entry and exit tracepoints and summarizes either the number of syscalls of each type, or the number of syscalls per process. It can also collect latency (invocation time) for each syscall or each process.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc. Linux 4.7+ is required to attach a BPF program to the raw_syscalls:sys_{enter,exit} tracepoints, used by this tool.

## OPTIONS
-h

Print usage message.

-p PID

Trace only this process.

-t TID

Trace only this thread.

-c PPID

Trace only child of this pid.

-i INTERVAL

Print the summary at the specified interval (in seconds).

-d DURATION

Total duration of trace (in seconds).

-T TOP

Print only this many entries. Default: 10.

-x

Trace only failed syscalls (i.e., the return value from the syscall was < 0).

-e ERRNO

Trace only syscalls that failed with that error (e.g. -e EPERM or -e 1).

-m

Display times in milliseconds. Default: microseconds.

-P

Summarize by process and not by syscall.

-l

List the syscalls recognized by the tool (hard-coded list). Syscalls beyond this list will still be displayed, as "[unknown: nnn]" where nnn is the syscall number.

--syscall SYSCALL

Trace this syscall only (use option -l to get all recognized syscalls).

## EXAMPLES
Summarize all syscalls by syscall:

\# syscount

Summarize all syscalls by process:

\# syscount -P

Summarize only failed syscalls:

\# syscount -x

Summarize only syscalls that failed with EPERM:

\# syscount -e EPERM

Trace PID 181 only:

\# syscount -p 181

Summarize syscalls counts and latencies:

\# syscount -L

## FIELDS
PID

Process ID

COMM

Process name

SYSCALL

Syscall name, or "[unknown: nnn]" for syscalls that aren't recognized

COUNT

The number of events

TIME

The total elapsed time (in us or ms)

## OVERHEAD
For most applications, the overhead should be manageable if they perform 1000's or even 10,000's of syscalls per second. For higher rates, the overhead may become considerable. For example, tracing a loop of 4 million calls to geteuid(), slows it down by 1.85x when tracing only syscall counts, and slows it down by more than 5x when tracing syscall counts and latencies. However, this represents a rate of >3.5 million syscalls per second, which should not be typical.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein, Rocky Xing

## SEE ALSO
funccount(8), ucalls(8), argdist(8), trace(8), funclatency(8)


bcc-tclcalls

title: bcc-tclcalls(8)

summary: Summarize method calls

text:
# bcc-tclcalls(8)

## NAME
ucalls, javacalls, perlcalls, phpcalls, pythoncalls, rubycalls, tclcalls - Summarize method calls from high-level languages and Linux syscalls.

## SYNOPSIS
javacalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

perlcalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

phpcalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

pythoncalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

rubycalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

tclcalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

ucalls [-l {java,perl,php,python,ruby}] [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

## DESCRIPTION
This tool summarizes method calls from high-level languages such as Java, Perl, PHP, Python, Ruby, and Tcl. It can also trace Linux system calls. Whenever a method is invoked, ucalls records the call count and optionally the method's execution time (latency) and displays a summary.

This uses in-kernel eBPF maps to store per process summaries for efficiency.

This tool relies on USDT probes embedded in many high-level languages, such as Java, Perl, PHP, Python, Ruby, and Tcl. It requires a runtime instrumented with these probes, which in some cases requires building from source with a USDT-specific flag, such as "--enable-dtrace" or "--with-dtrace". For Java, method probes are not enabled by default, and can be turned on by running the Java process with the "-XX:+ExtendedDTraceProbes" flag. For PHP processes, the environment variable USE_ZEND_DTRACE must be set to 1.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-l {java,perl,php,python,ruby,tcl}

The language to trace. If not provided, only syscalls are traced (when the -S option is used).

-T TOP

Print only the top methods by frequency or latency.

-L

Collect method invocation latency (duration).

-S

Collect Linux syscalls frequency and timing.

-v

Print the resulting BPF program, for debugging purposes.

-m

Print times in milliseconds (the default is microseconds).

pid

The process id to trace.

interval

Print summary after this number of seconds and then exit. By default, wait for Ctrl+C to terminate.

## EXAMPLES
Trace the top 10 Ruby method calls:

\# ucalls -T 10 -l ruby 1344

Trace Python method calls and Linux syscalls including latency in milliseconds:

\# ucalls -l python -mL 2020

Trace only syscalls and print a summary after 10 seconds:

\# ucalls -S 788 10

## OVERHEAD
Tracing individual method calls will produce a considerable overhead in all high-level languages. For languages with just-in-time compilation, such as Java, the overhead can be more considerable than for interpreted languages. On the other hand, syscall tracing will typically be tolerable for most processes, unless they have a very unusual rate of system calls.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _example.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein

## SEE ALSO
ustat(8), argdist(8)


bcc-tclflow

title: bcc-tclflow(8)

summary: Print a flow graph of method

text:
# bcc-tclflow(8)

## NAME
uflow, javaflow, perlflow, phpflow, pythonflow, rubyflow, tclflow - Print a flow graph of method calls in high-level languages.

## SYNOPSIS
javaflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

perlflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

phpflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

pythonflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

rubyflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

tclflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

uflow [-h] [-M METHOD] [-C CLAZZ] [-v] [-l {java,perl,php,python,ruby,tcl}] pid

## DESCRIPTION
uflow traces method calls and prints them in a flow graph that can facilitate debugging and diagnostics by following the program's execution (method flow).

This tool relies on USDT probes embedded in many high-level languages, such as Java, Perl, PHP, Python, Ruby, and Tcl. It requires a runtime instrumented with these probes, which in some cases requires building from source with a USDT-specific flag, such as "--enable-dtrace" or "--with-dtrace". For Java processes, the startup flag "-XX:+ExtendedDTraceProbes" is required. For PHP processes, the environment variable USE_ZEND_DTRACE must be set to 1.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-M METHOD

Print only method calls where the method name begins with this string.

-C CLAZZ

Print only method calls where the class name begins with this string. The class name interpretation strongly depends on the language. For example, in Java use "package/subpackage/ClassName" to refer to classes.

-v

Print the resulting BPF program, for debugging purposes.

{java,perl,php,python,ruby,tcl}

The language to trace.

pid

The process id to trace.

## EXAMPLES
Follow method flow in a Ruby process:

\# uflow ruby 148

Follow method flow in a Java process where the class name is java.lang.Thread:

\# uflow -C java/lang/Thread java 1802

## FIELDS
CPU

The CPU number on which the method was invoked. This is useful to easily see where the output skips to a different CPU.

PID

The process id.

TID

The thread id.

TIME

The duration of the method call.

METHOD

The method name.

## OVERHEAD
This tool has extremely high overhead because it prints every method call. For some scenarios, you might see lost samples in the output as the tool is unable to keep up with the rate of data coming from the kernel. Filtering by class or method prefix can help reduce the amount of data printed, but there is still a very high overhead in the collection mechanism. Do not use for performance- sensitive production scenarios, and always test first.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _example.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein

## SEE ALSO
trace(8), ustat(8)


bcc-tclobjnew

title: bcc-tclobjnew(8)

summary: Summarize object allocations in

text:
# bcc-tclobjnew(8)

## NAME
uobjnew, cobjnew, javaobjnew, rubyobjnew, tclobjnew - Summarize object allocations in high-level languages.

## SYNOPSIS
cobjnew [-h] [-C TOP_COUNT] [-S TOP_SIZE] [-v] pid [interval]

javaobjnew [-h] [-C TOP_COUNT] [-S TOP_SIZE] [-v] pid [interval]

rubyobjnew [-h] [-C TOP_COUNT] [-S TOP_SIZE] [-v] pid [interval]

tclobjnew [-h] [-C TOP_COUNT] [-S TOP_SIZE] [-v] pid [interval]

uobjnew [-h] [-C TOP_COUNT] [-S TOP_SIZE] [-v] [-l {c,java,ruby,tcl}] pid [interval]

## DESCRIPTION
uobjnew traces object allocations in high-level languages (including "malloc") and prints summaries of the most frequently allocated types by number of objects or number of bytes.

This tool relies on USDT probes embedded in many high-level languages, such as C, Java, Ruby, and Tcl. It requires a runtime instrumented with these probes, which in some cases requires building from source with a USDT-specific flag, such as "--enable-dtrace" or "--with-dtrace". For Java, the Java process must be started with the "-XX:+ExtendedDTraceProbes" flag.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-C TOP_COUNT

Print the top object types sorted by number of instances.

-S TOP_SIZE

Print the top object types sorted by size.

-v

Print the resulting BPF program, for debugging purposes.

{c,java,ruby,tcl}

The language to trace.

pid

The process id to trace.

interval

Wait this many seconds and then print the summary and exit. By default, wait for Ctrl+C to exit.

## EXAMPLES
Trace object allocations in a Ruby process:

\# uobjnew ruby 148

Trace object allocations from "malloc" and print the top 10 by total size:

\# uobjnew -S 10 c 1788

## FIELDS
TYPE

The object type being allocated. For C (malloc), this is the block size.

ALLOCS

The number of objects allocated.

BYTES

The number of bytes allocated.

## OVERHEAD
Object allocation events are quite frequent, and therefore the overhead from running this tool can be considerable. Use with caution and make sure to test before using in a production environment. Nonetheless, even thousands of allocations per second will likely produce a reasonable overhead when investigating a problem.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _example.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein

## SEE ALSO
ustat(8), ugc(8), memleak(8)


bcc-tclstat

title: bcc-tclstat(8)

summary: Activity stats from

text:
# bcc-tclstat(8)

## NAME
ustat, javastat, nodestat, perlstat, phpstat, pythonstat, rubystat, tclstat - Activity stats from high-level languages.

## SYNOPSIS
javastat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

nodestat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

perlstat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

phpstat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

pythonstat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

rubystat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

tclstat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

ustat [-l {java,node,perl,php,python,ruby,tcl}] [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

## DESCRIPTION
This is "top" for high-level language events, such as garbage collections, exceptions, thread creations, object allocations, method calls, and more. The events are aggregated for each process and printed in a top-like table, which can be sorted by various fields. Not all language runtimes provide the same set of details.

This uses in-kernel eBPF maps to store per process summaries for efficiency.

This tool relies on USDT probes embedded in many high-level languages, such as Java, Node, Perl, PHP, Python, Ruby, and Tcl. It requires a runtime instrumented with these probes, which in some cases requires building from source with a USDT-specific flag, such as "--enable-dtrace" or "--with-dtrace". For Java, some probes are not enabled by default, and can be turned on by running the Java process with the "-XX:+ExtendedDTraceProbes" flag. For PHP processes, the environment variable USE_ZEND_DTRACE must be set to 1.

Newly-created processes will only be traced at the next interval. If you run this tool with a short interval (say, 1-5 seconds), this should be virtually unnoticeable. For longer intervals, you might miss processes that were started and terminated during the interval window.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-l {java,node,perl,php,python,ruby,tcl}

The language to trace. By default, all languages are traced.

-C

Do not clear the screen between updates.

-S {cload,excp,gc,method,objnew,thread}

Sort the output by the specified field.

-r MAXROWS

Do not print more than this number of rows.

-d

Print the resulting BPF program, for debugging purposes.

interval

Interval between updates, seconds.

count

Number of interval summaries.

## EXAMPLES
Summarize activity in high-level languages, 1 second refresh:

\# ustat

Don't clear the screen, and top 8 rows only:

\# ustat -Cr 8

5 second summaries, 10 times only:

\# ustat 5 10

## FIELDS
loadavg

The contents of /proc/loadavg

PID

Process ID.

CMDLINE

Process command line (often the second and following arguments will give you a hint as to which application is being run.

METHOD/s

Count of method invocations during interval.

GC/s

Count of garbage collections during interval.

OBJNEW/s

Count of objects allocated during interval.

CLOAD/s

Count of classes loaded during interval.

EXC/s

Count of exceptions thrown during interval.

THR/s

Count of threads created during interval.

## OVERHEAD
When using this tool with high-frequency events, such as method calls, a very significant slow-down can be expected. However, many of the high-level languages covered by this tool already have a fairly high per-method invocation cost, especially when running in interpreted mode. For the lower-frequency events, such as garbage collections or thread creations, the overhead should not be significant. Specifically, when probing Java processes and not using the "-XX:+ExtendedDTraceProbes" flag, the most expensive probes are not emitted, and the overhead should be acceptable.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _example.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein

## SEE ALSO
trace(8), argdist(8), tplist(8)


bcc-tcpaccept

title: bcc-tcpaccept(8)

text:
# bcc-tcpaccept(8)

## NAME
tcpaccept - Trace TCP passive connections (accept()). Uses Linux eBPF/bcc.

## SYNOPSIS
tcpaccept [-h] [-T] [-t] [-p PID] [-P PORTS] [-4 | -6] [--cgroupmap MAPPATH] [--mntnsmap MAPPATH]

## DESCRIPTION
This tool traces passive TCP connections (eg, via an accept() syscall; connect() are active connections). This can be useful for general troubleshooting to see what new connections the local server is accepting.

This uses dynamic tracing of the kernel inet_csk_accept() socket function (from tcp_prot.accept), and will need to be modified to match kernel changes.

This tool only traces successful TCP accept()s. Connection attempts to closed ports will not be shown (those can be traced via other functions).

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print usage message.

-T

Include a time column on output (HH:MM:SS).

-t

Include a timestamp column.

-p PID

Trace this process ID only (filtered in-kernel).

-P PORTS

Comma-separated list of local ports to trace (filtered in-kernel).

-4

Trace IPv4 family only.

-6

Trace IPv6 family only.

--cgroupmap MAPPATH

Trace cgroups in this BPF map only (filtered in-kernel).

--mntnsmap MAPPATH

Trace mount namespaces in this BPF map only (filtered in-kernel).

## EXAMPLES
Trace all passive TCP connections (accept()s):

\# tcpaccept

Trace all TCP accepts, and include timestamps:

\# tcpaccept -t

Trace connections to local ports 80 and 81 only:

\# tcpaccept -P 80,81

Trace PID 181 only:

\# tcpaccept -p 181

Trace IPv4 family only:

\# tcpaccept -4

Trace IPv6 family only:

\# tcpaccept -6

Trace a set of cgroups only (see special_filtering.md from bcc sources for more details):

\# tcpaccept --cgroupmap /sys/fs/bpf/test01

## FIELDS
TIME

Time of the event, in HH:MM:SS format.

TIME(s)

Time of the event, in seconds.

PID

Process ID

COMM

Process name

IP

IP address family (4 or 6)

RADDR

Remote IP address.

RPORT

Remote port

LADDR

Local IP address.

LPORT

Local port

## OVERHEAD
This traces the kernel inet_csk_accept function and prints output for each event. The rate of this depends on your server application. If it is a web or proxy server accepting many tens of thousands of connections per second, then the overhead of this tool may be measurable (although, still a lot better than tracing every packet). If it is less than a thousand a second, then the overhead is expected to be negligible. Test and understand this overhead before use.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
tcptracer(8), tcpconnect(8), funccount(8), tcpdump(8)


bcc-tcpcong

title: bcc-tcpcong(8)

text:
# bcc-tcpcong(8)

## NAME
tcpcong - Measure tcp congestion state duration. Uses Linux eBPF/bcc.

## SYNOPSIS
tcpcong [-h] [-T] [-L] [-R] [-u] [-d] [interval] [outputs]

## DESCRIPTION
this tool measures tcp sockets congestion control status duration, and prints a summary of tcp congestion state durations along with the number of total state changes.

It uses dynamic tracing of kernel tcp congestion control status updating functions, and will need to be updated to match kernel changes.

The traced functions are only called when there is congestion state update, and therefore have low overhead. we also use BPF map to store traced data to reduce overhead. See the OVERHEAD section for more details. Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print usage message.

-T

Include a timestamp column.

-L

Specify local tcp port range.

-R

Specify remote tcp port range.

-u

Output in microseconds.

-d

Show congestion status duration distribution as histograms.

## EXAMPLES
Show all tcp sockets congestion status duration until Ctrl-C:

\# tcpcongestdura

Show all tcp sockets congestion status duration every 1 second and 10 times:

\# tcpcong 1 10

Show only local port 3000-3006 congestion status duration every 1 second:

\# tcpcong -L 3000-3006 1

Show only remote port 5000-5005 congestion status duration every 1 second:

\# tcpcong -R 5000-5005 1

Show 1 second summaries, printed in microseconds, with timestamps:

\# tcpcong -uT 1

Show all tcp sockets congestion status duration as histograms:

\# tcpcong -d

## FIELDS
LAddrPort

local ip address and tcp socket port.

RAddrPort

remote ip address and tcp socket port.

Open_us

Total duration in open status for microseconds.

Dod_us

Total duration in disorder status for microseconds.

Rcov_us

Total duration in recovery status for microseconds.

Cwr_us

Total duration in cwr status for microseconds.

Los_us

Total duration in loss status for microseconds.

Open_ms

Total duration in open status for milliseconds.

Dod_ms

Total duration in disorder status for milliseconds.

Rcov_ms

Total duration in recovery status for milliseconds.

Cwr_ms

Total duration in cwr status for milliseconds.

Loss_ms

Total duration in loss status for milliseconds.

Chgs

Total number of status change.

usecs

Range of microseconds for this bucket.

msecs

Range of milliseconds for this bucket.

count

Number of congestion status in this time range.

distribution

ASCII representation of the distribution (the count column).

## OVERHEAD
This traces the kernel tcp congestion status change functions. As called rate per second of these functions per socket is low(<10000), the overhead is also expected to be negligible. If you have an application that will create thousands of tcp connections, then test and understand overhead before use.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
jacky gan

## SEE ALSO
tcpretrans(8), tcpconnect(8), tcptop(8), tcpdrop(8)