bcc-nfsslower

title: bcc-nfsslower(8)

text:
# bcc-nfsslower(8)

## NAME
nfsslower - Trace slow NFS file operations, with per-event details.

## SYNOPSIS
nfsslower [-h] [-j] [-p PID] [min_ms]

## DESCRIPTION
This tool traces common NFSv3 & NFSv4 file operations: reads, writes, opens, and getattrs. It measures the time spent in these operations, and prints details for each that exceeded a threshold.

WARNING: See the OVERHEAD section.

By default, a minimum millisecond threshold of 10 is used. If a threshold of 0 is used, all events are printed (warning: verbose).

Since this works by tracing the nfs_file_operations interface functions, it will need updating to match any changes to these functions.

This tool uses kprobes to instrument the kernel for entry and exit information, in the future a preferred way would be to use tracepoints. Currently there aren't any tracepoints available for nfs_read_file, nfs_write_file and nfs_open_file, nfs_getattr does have entry and exit tracepoints but we chose to use kprobes for consistency

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-p PID Trace this PID only.

-j

Trace output in CSV format.

min_ms

Minimum I/O latency (duration) to trace, in milliseconds. Default is 10 ms.

## EXAMPLES
Trace synchronous file reads and writes slower than 10 ms:

\# nfsslower

Trace slower than 1 ms:

\# nfsslower 1

Trace slower than 1 ms, and output just the fields in parsable format (CSV):

\# nfsslower -j 1

Trace all file reads and writes (warning: the output will be verbose):

\# nfsslower 0

Trace slower than 1 ms, for PID 181 only:

\# nfsslower -p 181 1

## FIELDS
TIME(s)

Time of I/O completion since the first I/O seen, in seconds.

COMM

Process name.

PID

Process ID.

T

Type of operation. R == read, W == write, O == open, G == getattr.

OFF_KB

File offset for the I/O, in Kbytes.

BYTES

Size of I/O, in bytes.

LAT(ms)

Latency (duration) of I/O, measured from when it was issued by VFS to the filesystem, to when it completed. This time is inclusive of RPC latency, network latency, cache lookup, remote fileserver processing latency, etc. Its a more accurate measure of the latency suffered by applications performing NFS read/write calls to a fileserver.

FILENAME

A cached kernel file name (comes from dentry->d_name.name).

ENDTIME_us

Completion timestamp, microseconds (-j only).

OFFSET_b

File offset, bytes (-j only).

LATENCY_us

Latency (duration) of the I/O, in microseconds (-j only).

## OVERHEAD
This adds low-overhead instrumentation to NFS operations, including reads and writes from the file system cache. Such read, writes and particularly getattrs can be very frequent (depending on the workload; eg, 1M/sec), at which point the overhead of this tool (even if it prints no "slower" events) can begin to become significant. Measure and quantify before use. If this continues to be a problem, consider switching to a tool that prints in-kernel summaries only. This tool has been tested with NFSv3 & NVSv4, but it might work with NFSv{1,2}, since it is tracing the generic functions from nfs_file_operations.

Note that the overhead of this tool should be less than fileslower(8), as this tool targets NFS functions only, and not all file read/write paths.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion nfsslower_examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Samuel Nair

## SEE ALSO
biosnoop(8), funccount(8), fileslower(8)


bcc-nodegc

package: bcc-tools

title: bcc-nodegc(8)

summary: Trace garbage collection events in

text:
# bcc-nodegc(8)

## NAME
ugc, javagc, nodegc, pythongc, rubygc - Trace garbage collection events in high-level languages.

## SYNOPSIS
javagc [-h] [-v] [-m] [-M MINIMUM] [-F FILTER] pid

nodegc [-h] [-v] [-m] [-M MINIMUM] [-F FILTER] pid

pythongc [-h] [-v] [-m] [-M MINIMUM] [-F FILTER] pid

rubygc [-h] [-v] [-m] [-M MINIMUM] [-F FILTER] pid

ugc [-h] [-v] [-m] [-M MINIMUM] [-F FILTER] [-l {java,node,python,ruby}] pid

## DESCRIPTION
This traces garbage collection events as they occur, including their duration and any additional information (such as generation collected or type of GC) provided by the respective language's runtime.

This tool relies on USDT probes embedded in many high-level languages, such as Java, Node, Python, and Ruby. It requires a runtime instrumented with these probes, which in some cases requires building from source with a USDT-specific flag, such as "--enable-dtrace" or "--with-dtrace".

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-v

Print the resulting BPF program, for debugging purposes.

-m

Print times in milliseconds. The default is microseconds.

-M MINIMUM

Display only collections that are longer than this threshold. The value is given in milliseconds. The default is to display all collections.

-F FILTER

Display only collections whose textual description matches (contains) this string. The default is to display all collections. Note that the filtering here is performed in user-space, and not as part of the BPF program. This means that if you have thousands of collection events, specifying this filter will not reduce the amount of data that has to be transferred from the BPF program to the user-space script.

{java,node,python,ruby}

The language to trace.

pid

The process id to trace.

## EXAMPLES
Trace garbage collections in a specific Node process:

\# ugc node 148

Trace garbage collections in a specific Java process, and print GC times in

milliseconds: # ugc -m java 6004

Trace garbage collections in a specific Java process, and display them only if

they are longer than 10ms and have the string "Tenured" in their detailed description: # ugc -M 10 -F Tenured java 6004

## FIELDS
START

The start time of the GC, in seconds from the beginning of the trace.

TIME

The duration of the garbage collection event.

DESCRIPTION

The runtime-provided description of this garbage collection event.

## OVERHEAD
Garbage collection events, even if frequent, should not produce a considerable overhead when traced because they are still not very common. Even hundreds of GCs per second (which is a very high rate) will still produce a fairly negligible overhead.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _example.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein

## SEE ALSO
trace(8), ustat(8), uobjnew(8)


bcc-nodestat

title: bcc-nodestat(8)

summary: Activity stats from

text:
# bcc-nodestat(8)

## NAME
ustat, javastat, nodestat, perlstat, phpstat, pythonstat, rubystat, tclstat - Activity stats from high-level languages.

## SYNOPSIS
javastat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

nodestat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

perlstat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

phpstat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

pythonstat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

rubystat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

tclstat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

ustat [-l {java,node,perl,php,python,ruby,tcl}] [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

## DESCRIPTION
This is "top" for high-level language events, such as garbage collections, exceptions, thread creations, object allocations, method calls, and more. The events are aggregated for each process and printed in a top-like table, which can be sorted by various fields. Not all language runtimes provide the same set of details.

This uses in-kernel eBPF maps to store per process summaries for efficiency.

This tool relies on USDT probes embedded in many high-level languages, such as Java, Node, Perl, PHP, Python, Ruby, and Tcl. It requires a runtime instrumented with these probes, which in some cases requires building from source with a USDT-specific flag, such as "--enable-dtrace" or "--with-dtrace". For Java, some probes are not enabled by default, and can be turned on by running the Java process with the "-XX:+ExtendedDTraceProbes" flag. For PHP processes, the environment variable USE_ZEND_DTRACE must be set to 1.

Newly-created processes will only be traced at the next interval. If you run this tool with a short interval (say, 1-5 seconds), this should be virtually unnoticeable. For longer intervals, you might miss processes that were started and terminated during the interval window.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-l {java,node,perl,php,python,ruby,tcl}

The language to trace. By default, all languages are traced.

-C

Do not clear the screen between updates.

-S {cload,excp,gc,method,objnew,thread}

Sort the output by the specified field.

-r MAXROWS

Do not print more than this number of rows.

-d

Print the resulting BPF program, for debugging purposes.

interval

Interval between updates, seconds.

count

Number of interval summaries.

## EXAMPLES
Summarize activity in high-level languages, 1 second refresh:

\# ustat

Don't clear the screen, and top 8 rows only:

\# ustat -Cr 8

5 second summaries, 10 times only:

\# ustat 5 10

## FIELDS
loadavg

The contents of /proc/loadavg

PID

Process ID.

CMDLINE

Process command line (often the second and following arguments will give you a hint as to which application is being run.

METHOD/s

Count of method invocations during interval.

GC/s

Count of garbage collections during interval.

OBJNEW/s

Count of objects allocated during interval.

CLOAD/s

Count of classes loaded during interval.

EXC/s

Count of exceptions thrown during interval.

THR/s

Count of threads created during interval.

## OVERHEAD
When using this tool with high-frequency events, such as method calls, a very significant slow-down can be expected. However, many of the high-level languages covered by this tool already have a fairly high per-method invocation cost, especially when running in interpreted mode. For the lower-frequency events, such as garbage collections or thread creations, the overhead should not be significant. Specifically, when probing Java processes and not using the "-XX:+ExtendedDTraceProbes" flag, the most expensive probes are not emitted, and the overhead should be acceptable.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _example.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein

## SEE ALSO
trace(8), argdist(8), tplist(8)


bcc-offcputime

title: bcc-offcputime(8)

text:
# bcc-offcputime(8)

## NAME
offcputime - Summarize off-CPU time by kernel stack trace. Uses Linux eBPF/bcc.

## SYNOPSIS
offcputime [-h] [-p PID | -t TID | -u | -k] [-U | -K] [-d] [-f] [--stack-storage-size STACK_STORAGE_SIZE] [-m MIN_BLOCK_TIME] [-M MAX_BLOCK_TIME] [--state STATE] [duration]

## DESCRIPTION
This program shows stack traces and task names that were blocked and "off-CPU", and the total duration they were not running: their "off-CPU time". It works by tracing when threads block and when they return to CPU, measuring both the time they were off-CPU and the blocked stack trace and the task name. This data is summarized in the kernel using an eBPF map, and by summing the off-CPU time by unique stack trace and task name.

The output summary will help you identify reasons why threads were blocking, and quantify the time they were off-CPU. This spans all types of blocking activity: disk I/O, network I/O, locks, page faults, involuntary context switches, etc.

This is complementary to CPU profiling (e.g., CPU flame graphs) which shows the time spent on-CPU. This shows the time spent off-CPU, and the output, especially the -f format, can be used to generate an "off-CPU time flame graph".

See http://www.brendangregg.com/FlameGraphs/offcpuflamegraphs.html

This tool only works on Linux 4.6+. It uses the new `BPF_STACK_TRACE` table APIs to generate the in-kernel stack traces. For kernels older than 4.6, see the version under tools/old.

Note: this tool only traces off-CPU times that began and ended while tracing.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print usage message.

-p PID

Trace this process ID only (filtered in-kernel).

-t TID

Trace this thread ID only (filtered in-kernel).

-u

Only trace user threads (no kernel threads).

-k

Only trace kernel threads (no user threads).

-U

Show stacks from user space only (no kernel space stacks).

-K

Show stacks from kernel space only (no user space stacks).

-d

Insert delimiter between kernel/user stacks.

-f

Print output in folded stack format.

--stack-storage-size STACK_STORAGE_SIZE

Change the number of unique stack traces that can be stored and displayed.

-m MIN_BLOCK_TIME

The minimum time in microseconds over which we store traces (default 1)

-M MAX_BLOCK_TIME

The maximum time in microseconds under which we store traces (default U64_MAX)

--state

Filter on this thread state bitmask (eg, 2 == TASK_UNINTERRUPTIBLE). See include/linux/sched.h for states.

duration

Duration to trace, in seconds.

## EXAMPLES
Trace all thread blocking events, and summarize (in-kernel) by kernel stack trace and total off-CPU time:

\# offcputime

Trace for 5 seconds only:

\# offcputime 5

Trace for 5 seconds, and emit output in folded stack format (suitable for flame graphs):

\# offcputime -f 5

Trace PID 185 only:

\# offcputime -p 185

## OVERHEAD
This summarizes unique stack traces in-kernel for efficiency, allowing it to trace a higher rate of events than methods that post-process in user space. The stack trace and time data is only copied to user space once, when the output is printed. While these techniques greatly lower overhead, scheduler events are still a high frequency event, as they can exceed 1 million events per second, and so caution should still be used. Test before production use.

If the overhead is still a problem, take a look at the MINBLOCK_US tunable in the code. If your aim is to chase down longer blocking events, then this could be increased to filter shorter blocking events, further lowering overhead.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
stackcount(8)


bcc-offwaketime

title: bcc-offwaketime(8)

text:
# bcc-offwaketime(8)

## NAME
offwaketime - Summarize blocked time by off-CPU stack + waker stack. Uses Linux eBPF/bcc.

## SYNOPSIS
offwaketime [-h] [-p PID | -t TID | -u | -k] [-U | -K] [-f] [--stack-storage-size STACK_STORAGE_SIZE] [-m MIN_BLOCK_TIME] [-M MAX_BLOCK_TIME] [--state STATE] [duration]

## DESCRIPTION
This program shows kernel stack traces and task names that were blocked and "off-CPU", along with the stack traces and task names for the threads that woke them, and the total elapsed time from when they blocked to when they were woken up. This combines the summaries from both the offcputime and wakeuptime tools. The time measurement will be very similar to off-CPU time, however, off-CPU time may include a little extra time spent waiting on a run queue to be scheduled. The combined stacks, task names, and total time is summarized in kernel context for efficiency, using an eBPF map.

The output summary will further help you identify reasons why threads were blocking, and quantify the time from when they were blocked to woken up. This spans all types of blocking activity: disk I/O, network I/O, locks, page faults, swapping, sleeping, involuntary context switches, etc.

This is complementary to CPU profiling (e.g., CPU flame graphs) which shows the time spent on-CPU. This shows the time spent blocked off-CPU, and the output, especially the -f format, can be used to generate an "off-wake time flame graph".

See http://www.brendangregg.com/FlameGraphs/offcpuflamegraphs.html

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print usage message.

-f

Print output in folded stack format.

-p PID

Trace this process ID only (filtered in-kernel). Can be a comma separated list of PIDS.

-t TID

Trace this thread ID only (filtered in-kernel). Can be a comma separated list of TIDS.

-u

Only trace user threads (no kernel threads).

-k

Only trace kernel threads (no user threads).

-U

Show stacks from user space only (no kernel space stacks).

-K

Show stacks from kernel space only (no user space stacks).

--stack-storage-size STACK_STORAGE_SIZE

Change the number of unique stack traces that can be stored and displayed.

duration

Duration to trace, in seconds.

-m MIN_BLOCK_TIME

The amount of time in microseconds over which we store traces (default 1)

-M MAX_BLOCK_TIME

The amount of time in microseconds under which we store traces (default U64_MAX)

--state

Filter on this thread state bitmask (eg, 2 == TASK_UNINTERRUPTIBLE). See include/linux/sched.h for states.

## EXAMPLES
Trace all thread blocking events, and summarize (in-kernel) by user and kernel off-CPU stack trace, waker stack traces, task names, and total blocked time:

\# offwaketime

Trace for 5 seconds only:

\# offwaketime 5

Trace for 5 seconds, and emit output in folded stack format (suitable for flame graphs), user-mode threads only:

\# offwaketime -fu 5

Trace PID 185 only:

\# offwaketime -p 185

## OVERHEAD
This summarizes unique stack trace pairs in-kernel for efficiency, allowing it to trace a higher rate of events than methods that post-process in user space. The stack trace and time data is only copied to user space once, when the output is printed. While these techniques greatly lower overhead, scheduler events are still a high frequency event, as they can exceed 1 million events per second, and so caution should still be used. Test before production use.

If the overhead is still a problem, take a look at the min block option. If your aim is to chase down longer blocking events, then this could be increased to filter shorter blocking events, further lowering overhead.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
offcputime(8), wakeuptime(8)


bcc-oomkill

title: bcc-oomkill(8)

text:
# bcc-oomkill(8)

## NAME
oomkill - Trace oom_kill_process(). Uses Linux eBPF/bcc.

## SYNOPSIS
oomkill

## DESCRIPTION
This traces the kernel out-of-memory killer, and prints basic details, including the system load averages at the time of the OOM kill. This can provide more context on the system state at the time: was it getting busier or steady, based on the load averages? This tool may also be useful to customize for investigations; for example, by adding other task_struct details at the time of OOM.

This program is also a basic example of eBPF/bcc.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## EXAMPLES
Trace OOM kill events:

\# oomkill

## FIELDS
Triggered by ...

The process ID and process name of the task that was running when another task was OOM killed.

OOM kill of ...

The process ID and name of the target process that was OOM killed.

loadavg

Contents of /proc/loadavg. The first three numbers are 1, 5, and 15 minute load averages (where the average is an exponentially damped moving sum, and those numbers are constants in the equation); then there is the number of running tasks, a slash, and the total number of tasks; and then the last number is the last PID to be created.

## OVERHEAD
Negligible.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
memleak(8)


bcc-opensnoop

title: bcc-opensnoop(8)

text:
# bcc-opensnoop(8)

## NAME
opensnoop - Trace open() syscalls. Uses Linux eBPF/bcc.

## SYNOPSIS
opensnoop.py [-h] [-T] [-U] [-x] [-p PID] [-t TID] [-u UID]

[-d DURATION] [-n NAME] [-e] [-f FLAG_FILTER]

[--cgroupmap MAPPATH] [--mntnsmap MAPPATH]

## DESCRIPTION
opensnoop traces the open() syscall, showing which processes are attempting to open which files. This can be useful for determining the location of config and log files, or for troubleshooting applications that are failing, specially on startup.

This works by tracing the kernel sys_open() function using dynamic tracing, and will need updating to match any changes to this function.

This makes use of a Linux 4.4 feature (bpf_perf_event_output()); for kernels older than 4.4, see the version under tools/old, which uses an older mechanism.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print usage message.

-T

Include a timestamp column.

-U

Show UID.

-x

Only print failed opens.

-p PID

Trace this process ID only (filtered in-kernel).

-t TID

Trace this thread ID only (filtered in-kernel).

-u UID

Trace this UID only (filtered in-kernel).

-d DURATION

Total duration of trace in seconds.

-n name

Only print processes where its name partially matches 'name'

-e

Show extended fields.

-f FLAG

Filter on open() flags, e.g., O_WRONLY.

--cgroupmap MAPPATH

Trace cgroups in this BPF map only (filtered in-kernel).

--mntnsmap MAPPATH

Trace mount namespaces in this BPF map only (filtered in-kernel).

## EXAMPLES
Trace all open() syscalls:

\# opensnoop

Trace all open() syscalls, for 10 seconds only:

\# opensnoop -d 10

Trace all open() syscalls, and include timestamps:

\# opensnoop -T

Show UID:

\# opensnoop -U

Trace only open() syscalls that failed:

\# opensnoop -x

Trace PID 181 only:

\# opensnoop -p 181

Trace UID 1000 only:

\# opensnoop -u 1000

Trace all open() syscalls from processes where its name partially matches 'ed':

\# opensnoop -n ed

Show extended fields:

\# opensnoop -e

Only print calls for writing:

\# opensnoop -f O_WRONLY -f O_RDWR

Trace a set of cgroups only (see special_filtering.md from bcc sources for more details):

\# opensnoop --cgroupmap /sys/fs/bpf/test01

## FIELDS
TIME(s)

Time of the call, in seconds.

UID

User ID

PID

Process ID

TID

Thread ID

COMM

Process name

FD

File descriptor (if success), or -1 (if failed)

ERR

Error number (see the system's errno.h)

FLAGS

Flags passed to open(2), in octal

PATH

Open path

## OVERHEAD
This traces the kernel open function and prints output for each event. As the rate of this is generally expected to be low (< 1000/s), the overhead is also expected to be negligible. If you have an application that is calling a high rate of open()s, then test and understand overhead before use.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
execsnoop(8), funccount(1)


bcc-perlcalls

title: bcc-perlcalls(8)

summary: Summarize method calls

text:
# bcc-perlcalls(8)

## NAME
ucalls, javacalls, perlcalls, phpcalls, pythoncalls, rubycalls, tclcalls - Summarize method calls from high-level languages and Linux syscalls.

## SYNOPSIS
javacalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

perlcalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

phpcalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

pythoncalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

rubycalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

tclcalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

ucalls [-l {java,perl,php,python,ruby}] [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

## DESCRIPTION
This tool summarizes method calls from high-level languages such as Java, Perl, PHP, Python, Ruby, and Tcl. It can also trace Linux system calls. Whenever a method is invoked, ucalls records the call count and optionally the method's execution time (latency) and displays a summary.

This uses in-kernel eBPF maps to store per process summaries for efficiency.

This tool relies on USDT probes embedded in many high-level languages, such as Java, Perl, PHP, Python, Ruby, and Tcl. It requires a runtime instrumented with these probes, which in some cases requires building from source with a USDT-specific flag, such as "--enable-dtrace" or "--with-dtrace". For Java, method probes are not enabled by default, and can be turned on by running the Java process with the "-XX:+ExtendedDTraceProbes" flag. For PHP processes, the environment variable USE_ZEND_DTRACE must be set to 1.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-l {java,perl,php,python,ruby,tcl}

The language to trace. If not provided, only syscalls are traced (when the -S option is used).

-T TOP

Print only the top methods by frequency or latency.

-L

Collect method invocation latency (duration).

-S

Collect Linux syscalls frequency and timing.

-v

Print the resulting BPF program, for debugging purposes.

-m

Print times in milliseconds (the default is microseconds).

pid

The process id to trace.

interval

Print summary after this number of seconds and then exit. By default, wait for Ctrl+C to terminate.

## EXAMPLES
Trace the top 10 Ruby method calls:

\# ucalls -T 10 -l ruby 1344

Trace Python method calls and Linux syscalls including latency in milliseconds:

\# ucalls -l python -mL 2020

Trace only syscalls and print a summary after 10 seconds:

\# ucalls -S 788 10

## OVERHEAD
Tracing individual method calls will produce a considerable overhead in all high-level languages. For languages with just-in-time compilation, such as Java, the overhead can be more considerable than for interpreted languages. On the other hand, syscall tracing will typically be tolerable for most processes, unless they have a very unusual rate of system calls.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _example.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein

## SEE ALSO
ustat(8), argdist(8)


bcc-perlflow

title: bcc-perlflow(8)

summary: Print a flow graph of method

text:
# bcc-perlflow(8)

## NAME
uflow, javaflow, perlflow, phpflow, pythonflow, rubyflow, tclflow - Print a flow graph of method calls in high-level languages.

## SYNOPSIS
javaflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

perlflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

phpflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

pythonflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

rubyflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

tclflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

uflow [-h] [-M METHOD] [-C CLAZZ] [-v] [-l {java,perl,php,python,ruby,tcl}] pid

## DESCRIPTION
uflow traces method calls and prints them in a flow graph that can facilitate debugging and diagnostics by following the program's execution (method flow).

This tool relies on USDT probes embedded in many high-level languages, such as Java, Perl, PHP, Python, Ruby, and Tcl. It requires a runtime instrumented with these probes, which in some cases requires building from source with a USDT-specific flag, such as "--enable-dtrace" or "--with-dtrace". For Java processes, the startup flag "-XX:+ExtendedDTraceProbes" is required. For PHP processes, the environment variable USE_ZEND_DTRACE must be set to 1.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-M METHOD

Print only method calls where the method name begins with this string.

-C CLAZZ

Print only method calls where the class name begins with this string. The class name interpretation strongly depends on the language. For example, in Java use "package/subpackage/ClassName" to refer to classes.

-v

Print the resulting BPF program, for debugging purposes.

{java,perl,php,python,ruby,tcl}

The language to trace.

pid

The process id to trace.

## EXAMPLES
Follow method flow in a Ruby process:

\# uflow ruby 148

Follow method flow in a Java process where the class name is java.lang.Thread:

\# uflow -C java/lang/Thread java 1802

## FIELDS
CPU

The CPU number on which the method was invoked. This is useful to easily see where the output skips to a different CPU.

PID

The process id.

TID

The thread id.

TIME

The duration of the method call.

METHOD

The method name.

## OVERHEAD
This tool has extremely high overhead because it prints every method call. For some scenarios, you might see lost samples in the output as the tool is unable to keep up with the rate of data coming from the kernel. Filtering by class or method prefix can help reduce the amount of data printed, but there is still a very high overhead in the collection mechanism. Do not use for performance- sensitive production scenarios, and always test first.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _example.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein

## SEE ALSO
trace(8), ustat(8)


bcc-perlstat

title: bcc-perlstat(8)

summary: Activity stats from

text:
# bcc-perlstat(8)

## NAME
ustat, javastat, nodestat, perlstat, phpstat, pythonstat, rubystat, tclstat - Activity stats from high-level languages.

## SYNOPSIS
javastat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

nodestat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

perlstat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

phpstat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

pythonstat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

rubystat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

tclstat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

ustat [-l {java,node,perl,php,python,ruby,tcl}] [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

## DESCRIPTION
This is "top" for high-level language events, such as garbage collections, exceptions, thread creations, object allocations, method calls, and more. The events are aggregated for each process and printed in a top-like table, which can be sorted by various fields. Not all language runtimes provide the same set of details.

This uses in-kernel eBPF maps to store per process summaries for efficiency.

This tool relies on USDT probes embedded in many high-level languages, such as Java, Node, Perl, PHP, Python, Ruby, and Tcl. It requires a runtime instrumented with these probes, which in some cases requires building from source with a USDT-specific flag, such as "--enable-dtrace" or "--with-dtrace". For Java, some probes are not enabled by default, and can be turned on by running the Java process with the "-XX:+ExtendedDTraceProbes" flag. For PHP processes, the environment variable USE_ZEND_DTRACE must be set to 1.

Newly-created processes will only be traced at the next interval. If you run this tool with a short interval (say, 1-5 seconds), this should be virtually unnoticeable. For longer intervals, you might miss processes that were started and terminated during the interval window.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-l {java,node,perl,php,python,ruby,tcl}

The language to trace. By default, all languages are traced.

-C

Do not clear the screen between updates.

-S {cload,excp,gc,method,objnew,thread}

Sort the output by the specified field.

-r MAXROWS

Do not print more than this number of rows.

-d

Print the resulting BPF program, for debugging purposes.

interval

Interval between updates, seconds.

count

Number of interval summaries.

## EXAMPLES
Summarize activity in high-level languages, 1 second refresh:

\# ustat

Don't clear the screen, and top 8 rows only:

\# ustat -Cr 8

5 second summaries, 10 times only:

\# ustat 5 10

## FIELDS
loadavg

The contents of /proc/loadavg

PID

Process ID.

CMDLINE

Process command line (often the second and following arguments will give you a hint as to which application is being run.

METHOD/s

Count of method invocations during interval.

GC/s

Count of garbage collections during interval.

OBJNEW/s

Count of objects allocated during interval.

CLOAD/s

Count of classes loaded during interval.

EXC/s

Count of exceptions thrown during interval.

THR/s

Count of threads created during interval.

## OVERHEAD
When using this tool with high-frequency events, such as method calls, a very significant slow-down can be expected. However, many of the high-level languages covered by this tool already have a fairly high per-method invocation cost, especially when running in interpreted mode. For the lower-frequency events, such as garbage collections or thread creations, the overhead should not be significant. Specifically, when probing Java processes and not using the "-XX:+ExtendedDTraceProbes" flag, the most expensive probes are not emitted, and the overhead should be acceptable.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _example.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein

## SEE ALSO
trace(8), argdist(8), tplist(8)


bcc-phpcalls

title: bcc-phpcalls(8)

summary: Summarize method calls

text:
# bcc-phpcalls(8)

## NAME
ucalls, javacalls, perlcalls, phpcalls, pythoncalls, rubycalls, tclcalls - Summarize method calls from high-level languages and Linux syscalls.

## SYNOPSIS
javacalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

perlcalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

phpcalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

pythoncalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

rubycalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

tclcalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

ucalls [-l {java,perl,php,python,ruby}] [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

## DESCRIPTION
This tool summarizes method calls from high-level languages such as Java, Perl, PHP, Python, Ruby, and Tcl. It can also trace Linux system calls. Whenever a method is invoked, ucalls records the call count and optionally the method's execution time (latency) and displays a summary.

This uses in-kernel eBPF maps to store per process summaries for efficiency.

This tool relies on USDT probes embedded in many high-level languages, such as Java, Perl, PHP, Python, Ruby, and Tcl. It requires a runtime instrumented with these probes, which in some cases requires building from source with a USDT-specific flag, such as "--enable-dtrace" or "--with-dtrace". For Java, method probes are not enabled by default, and can be turned on by running the Java process with the "-XX:+ExtendedDTraceProbes" flag. For PHP processes, the environment variable USE_ZEND_DTRACE must be set to 1.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-l {java,perl,php,python,ruby,tcl}

The language to trace. If not provided, only syscalls are traced (when the -S option is used).

-T TOP

Print only the top methods by frequency or latency.

-L

Collect method invocation latency (duration).

-S

Collect Linux syscalls frequency and timing.

-v

Print the resulting BPF program, for debugging purposes.

-m

Print times in milliseconds (the default is microseconds).

pid

The process id to trace.

interval

Print summary after this number of seconds and then exit. By default, wait for Ctrl+C to terminate.

## EXAMPLES
Trace the top 10 Ruby method calls:

\# ucalls -T 10 -l ruby 1344

Trace Python method calls and Linux syscalls including latency in milliseconds:

\# ucalls -l python -mL 2020

Trace only syscalls and print a summary after 10 seconds:

\# ucalls -S 788 10

## OVERHEAD
Tracing individual method calls will produce a considerable overhead in all high-level languages. For languages with just-in-time compilation, such as Java, the overhead can be more considerable than for interpreted languages. On the other hand, syscall tracing will typically be tolerable for most processes, unless they have a very unusual rate of system calls.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _example.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein

## SEE ALSO
ustat(8), argdist(8)


bcc-phpflow

title: bcc-phpflow(8)

summary: Print a flow graph of method

text:
# bcc-phpflow(8)

## NAME
uflow, javaflow, perlflow, phpflow, pythonflow, rubyflow, tclflow - Print a flow graph of method calls in high-level languages.

## SYNOPSIS
javaflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

perlflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

phpflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

pythonflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

rubyflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

tclflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

uflow [-h] [-M METHOD] [-C CLAZZ] [-v] [-l {java,perl,php,python,ruby,tcl}] pid

## DESCRIPTION
uflow traces method calls and prints them in a flow graph that can facilitate debugging and diagnostics by following the program's execution (method flow).

This tool relies on USDT probes embedded in many high-level languages, such as Java, Perl, PHP, Python, Ruby, and Tcl. It requires a runtime instrumented with these probes, which in some cases requires building from source with a USDT-specific flag, such as "--enable-dtrace" or "--with-dtrace". For Java processes, the startup flag "-XX:+ExtendedDTraceProbes" is required. For PHP processes, the environment variable USE_ZEND_DTRACE must be set to 1.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-M METHOD

Print only method calls where the method name begins with this string.

-C CLAZZ

Print only method calls where the class name begins with this string. The class name interpretation strongly depends on the language. For example, in Java use "package/subpackage/ClassName" to refer to classes.

-v

Print the resulting BPF program, for debugging purposes.

{java,perl,php,python,ruby,tcl}

The language to trace.

pid

The process id to trace.

## EXAMPLES
Follow method flow in a Ruby process:

\# uflow ruby 148

Follow method flow in a Java process where the class name is java.lang.Thread:

\# uflow -C java/lang/Thread java 1802

## FIELDS
CPU

The CPU number on which the method was invoked. This is useful to easily see where the output skips to a different CPU.

PID

The process id.

TID

The thread id.

TIME

The duration of the method call.

METHOD

The method name.

## OVERHEAD
This tool has extremely high overhead because it prints every method call. For some scenarios, you might see lost samples in the output as the tool is unable to keep up with the rate of data coming from the kernel. Filtering by class or method prefix can help reduce the amount of data printed, but there is still a very high overhead in the collection mechanism. Do not use for performance- sensitive production scenarios, and always test first.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _example.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein

## SEE ALSO
trace(8), ustat(8)


bcc-phpstat

title: bcc-phpstat(8)

summary: Activity stats from

text:
# bcc-phpstat(8)

## NAME
ustat, javastat, nodestat, perlstat, phpstat, pythonstat, rubystat, tclstat - Activity stats from high-level languages.

## SYNOPSIS
javastat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

nodestat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

perlstat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

phpstat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

pythonstat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

rubystat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

tclstat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

ustat [-l {java,node,perl,php,python,ruby,tcl}] [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

## DESCRIPTION
This is "top" for high-level language events, such as garbage collections, exceptions, thread creations, object allocations, method calls, and more. The events are aggregated for each process and printed in a top-like table, which can be sorted by various fields. Not all language runtimes provide the same set of details.

This uses in-kernel eBPF maps to store per process summaries for efficiency.

This tool relies on USDT probes embedded in many high-level languages, such as Java, Node, Perl, PHP, Python, Ruby, and Tcl. It requires a runtime instrumented with these probes, which in some cases requires building from source with a USDT-specific flag, such as "--enable-dtrace" or "--with-dtrace". For Java, some probes are not enabled by default, and can be turned on by running the Java process with the "-XX:+ExtendedDTraceProbes" flag. For PHP processes, the environment variable USE_ZEND_DTRACE must be set to 1.

Newly-created processes will only be traced at the next interval. If you run this tool with a short interval (say, 1-5 seconds), this should be virtually unnoticeable. For longer intervals, you might miss processes that were started and terminated during the interval window.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-l {java,node,perl,php,python,ruby,tcl}

The language to trace. By default, all languages are traced.

-C

Do not clear the screen between updates.

-S {cload,excp,gc,method,objnew,thread}

Sort the output by the specified field.

-r MAXROWS

Do not print more than this number of rows.

-d

Print the resulting BPF program, for debugging purposes.

interval

Interval between updates, seconds.

count

Number of interval summaries.

## EXAMPLES
Summarize activity in high-level languages, 1 second refresh:

\# ustat

Don't clear the screen, and top 8 rows only:

\# ustat -Cr 8

5 second summaries, 10 times only:

\# ustat 5 10

## FIELDS
loadavg

The contents of /proc/loadavg

PID

Process ID.

CMDLINE

Process command line (often the second and following arguments will give you a hint as to which application is being run.

METHOD/s

Count of method invocations during interval.

GC/s

Count of garbage collections during interval.

OBJNEW/s

Count of objects allocated during interval.

CLOAD/s

Count of classes loaded during interval.

EXC/s

Count of exceptions thrown during interval.

THR/s

Count of threads created during interval.

## OVERHEAD
When using this tool with high-frequency events, such as method calls, a very significant slow-down can be expected. However, many of the high-level languages covered by this tool already have a fairly high per-method invocation cost, especially when running in interpreted mode. For the lower-frequency events, such as garbage collections or thread creations, the overhead should not be significant. Specifically, when probing Java processes and not using the "-XX:+ExtendedDTraceProbes" flag, the most expensive probes are not emitted, and the overhead should be acceptable.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _example.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein

## SEE ALSO
trace(8), argdist(8), tplist(8)


bcc-pidpersec

title: bcc-pidpersec(8)

text:
# bcc-pidpersec(8)

## NAME
pidpersec - Count new processes (via fork()). Uses Linux eBPF/bcc.

## SYNOPSIS
pidpersec

## DESCRIPTION
pidpersec shows how many new processes were created each second. There can be performance issues caused by many short-lived processes, which may not be visible in sampling tools like top(1). pidpersec provides one way to investigate this behavior.

This works by tracing the kernel sched_fork() function using dynamic tracing, and will need updating to match any changes to this function.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## EXAMPLES
Count new processes created each second:

\# pidpersec

## OVERHEAD
This traces the kernel fork function, and maintains an in-kernel count which is read asynchronously from user-space. As the rate of this is generally expected to be low (<< 1000/s), the overhead is also expected to be negligible.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
top(1)


bcc-profile

title: bcc-profile(8)

text:
# bcc-profile(8)

## NAME
profile - Profile CPU usage by sampling stack traces. Uses Linux eBPF/bcc.

## SYNOPSIS
profile [-adfh] [-p PID | -L TID] [-U | -K] [-F FREQUENCY | -c COUNT] [--stack-storage-size COUNT] [--cgroupmap CGROUPMAP] [--mntnsmap MAPPATH] [duration]

## DESCRIPTION
This is a CPU profiler. It works by taking samples of stack traces at timed intervals. It will help you understand and quantify CPU usage: which code is executing, and by how much, including both user-level and kernel code.

By default this samples at 49 Hertz (samples per second), across all CPUs. This frequency can be tuned using a command line option. The reason for 49, and not 50, is to avoid lock-step sampling.

This is also an efficient profiler, as stack traces are frequency counted in kernel context, rather than passing each stack to user space for frequency counting there. Only the unique stacks and counts are passed to user space at the end of the profile, greatly reducing the kernel<->user transfer.

## REQUIREMENTS
CONFIG_BPF and bcc.

This also requires Linux 4.9+ (BPF_PROG_TYPE_PERF_EVENT support). See tools/old for an older version that may work on Linux 4.6 - 4.8.

## OPTIONS
-h

Print usage message.

-p PID

Trace this process ID only (filtered in-kernel).

-L TID

Trace this thread ID only (filtered in-kernel).

-F frequency

Frequency to sample stacks.

-c count

Sample stacks every one in this many events.

-f

Print output in folded stack format.

-d

Include an output delimiter between kernel and user stacks (either "--", or, in folded mode, "-").

-U

Show stacks from user space only (no kernel space stacks).

-K

Show stacks from kernel space only (no user space stacks).

-I

Include CPU idle stacks (by default these are excluded).

--stack-storage-size COUNT

The maximum number of unique stack traces that the kernel will count (default 16384). If the sampled count exceeds this, a warning will be printed.

-C cpu

Collect stacks only from specified cpu.

--cgroupmap MAPPATH

Profile cgroups in this BPF map only (filtered in-kernel).

duration

Duration to trace, in seconds.

## EXAMPLES
Profile (sample) stack traces system-wide at 49 Hertz (samples per second) until Ctrl-C:

\# profile

Profile for 5 seconds only:

\# profile 5

Profile at 99 Hertz for 5 seconds only:

\# profile -F 99 5

Profile 1 in a million events for 5 seconds only:

\# profile -c 1000000 5

Profile process with PID 181 only:

\# profile -p 181

Profile thread with TID 181 only:

\# profile -L 181

Profile for 5 seconds and output in folded stack format (suitable as input for flame graphs), including a delimiter between kernel and user stacks:

\# profile -df 5

Profile kernel stacks only:

\# profile -K

Profile a set of cgroups only (see special_filtering.md from bcc sources for more details):

\# profile --cgroupmap /sys/fs/bpf/test01

## DEBUGGING
See "[unknown]" frames with bogus addresses? This can happen for different reasons. Your best approach is to get Linux perf to work first, and then to try this tool. Eg, "perf record -F 49 -a -g -- sleep 1; perf script", and to check for unknown frames there.

The most common reason for "[unknown]" frames is that the target software has not been compiled with frame pointers, and so we can't use that simple method for walking the stack. The fix in that case is to use software that does have frame pointers, eg, gcc -fno-omit-frame-pointer, or Java's -XX:+PreserveFramePointer.

Another reason for "[unknown]" frames is JIT compilers, which don't use a traditional symbol table. The fix in that case is to populate a /tmp/perf-PID.map file with the symbols, which this tool should read. How you do this depends on the runtime (Java, Node.js).

If you seem to have unrelated samples in the output, check for other sampling or tracing tools that may be running. The current version of this tool can include their events if profiling happened concurrently. Those samples may be filtered in a future version.

## OVERHEAD
This is an efficient profiler, as stack traces are frequency counted in kernel context, and only the unique stacks and their counts are passed to user space. Contrast this with the current "perf record -F 99 -a" method of profiling, which writes each sample to user space (via a ring buffer), and then to the file system (perf.data), which must be post-processed.

This uses perf_event_open to setup a timer which is instrumented by BPF, and for efficiency it does not initialize the perf ring buffer, so the redundant perf samples are not collected.

It's expected that the overhead while sampling at 49 Hertz (the default), across all CPUs, should be negligible. If you increase the sample rate, the overhead might begin to be measurable.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
offcputime(8)


bcc-pythoncalls

title: bcc-pythoncalls(8)

summary: Summarize method calls

text:
# bcc-pythoncalls(8)

## NAME
ucalls, javacalls, perlcalls, phpcalls, pythoncalls, rubycalls, tclcalls - Summarize method calls from high-level languages and Linux syscalls.

## SYNOPSIS
javacalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

perlcalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

phpcalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

pythoncalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

rubycalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

tclcalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

ucalls [-l {java,perl,php,python,ruby}] [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

## DESCRIPTION
This tool summarizes method calls from high-level languages such as Java, Perl, PHP, Python, Ruby, and Tcl. It can also trace Linux system calls. Whenever a method is invoked, ucalls records the call count and optionally the method's execution time (latency) and displays a summary.

This uses in-kernel eBPF maps to store per process summaries for efficiency.

This tool relies on USDT probes embedded in many high-level languages, such as Java, Perl, PHP, Python, Ruby, and Tcl. It requires a runtime instrumented with these probes, which in some cases requires building from source with a USDT-specific flag, such as "--enable-dtrace" or "--with-dtrace". For Java, method probes are not enabled by default, and can be turned on by running the Java process with the "-XX:+ExtendedDTraceProbes" flag. For PHP processes, the environment variable USE_ZEND_DTRACE must be set to 1.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-l {java,perl,php,python,ruby,tcl}

The language to trace. If not provided, only syscalls are traced (when the -S option is used).

-T TOP

Print only the top methods by frequency or latency.

-L

Collect method invocation latency (duration).

-S

Collect Linux syscalls frequency and timing.

-v

Print the resulting BPF program, for debugging purposes.

-m

Print times in milliseconds (the default is microseconds).

pid

The process id to trace.

interval

Print summary after this number of seconds and then exit. By default, wait for Ctrl+C to terminate.

## EXAMPLES
Trace the top 10 Ruby method calls:

\# ucalls -T 10 -l ruby 1344

Trace Python method calls and Linux syscalls including latency in milliseconds:

\# ucalls -l python -mL 2020

Trace only syscalls and print a summary after 10 seconds:

\# ucalls -S 788 10

## OVERHEAD
Tracing individual method calls will produce a considerable overhead in all high-level languages. For languages with just-in-time compilation, such as Java, the overhead can be more considerable than for interpreted languages. On the other hand, syscall tracing will typically be tolerable for most processes, unless they have a very unusual rate of system calls.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _example.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein

## SEE ALSO
ustat(8), argdist(8)


bcc-pythonflow

title: bcc-pythonflow(8)

summary: Print a flow graph of method

text:
# bcc-pythonflow(8)

## NAME
uflow, javaflow, perlflow, phpflow, pythonflow, rubyflow, tclflow - Print a flow graph of method calls in high-level languages.

## SYNOPSIS
javaflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

perlflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

phpflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

pythonflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

rubyflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

tclflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

uflow [-h] [-M METHOD] [-C CLAZZ] [-v] [-l {java,perl,php,python,ruby,tcl}] pid

## DESCRIPTION
uflow traces method calls and prints them in a flow graph that can facilitate debugging and diagnostics by following the program's execution (method flow).

This tool relies on USDT probes embedded in many high-level languages, such as Java, Perl, PHP, Python, Ruby, and Tcl. It requires a runtime instrumented with these probes, which in some cases requires building from source with a USDT-specific flag, such as "--enable-dtrace" or "--with-dtrace". For Java processes, the startup flag "-XX:+ExtendedDTraceProbes" is required. For PHP processes, the environment variable USE_ZEND_DTRACE must be set to 1.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-M METHOD

Print only method calls where the method name begins with this string.

-C CLAZZ

Print only method calls where the class name begins with this string. The class name interpretation strongly depends on the language. For example, in Java use "package/subpackage/ClassName" to refer to classes.

-v

Print the resulting BPF program, for debugging purposes.

{java,perl,php,python,ruby,tcl}

The language to trace.

pid

The process id to trace.

## EXAMPLES
Follow method flow in a Ruby process:

\# uflow ruby 148

Follow method flow in a Java process where the class name is java.lang.Thread:

\# uflow -C java/lang/Thread java 1802

## FIELDS
CPU

The CPU number on which the method was invoked. This is useful to easily see where the output skips to a different CPU.

PID

The process id.

TID

The thread id.

TIME

The duration of the method call.

METHOD

The method name.

## OVERHEAD
This tool has extremely high overhead because it prints every method call. For some scenarios, you might see lost samples in the output as the tool is unable to keep up with the rate of data coming from the kernel. Filtering by class or method prefix can help reduce the amount of data printed, but there is still a very high overhead in the collection mechanism. Do not use for performance- sensitive production scenarios, and always test first.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _example.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein

## SEE ALSO
trace(8), ustat(8)


bcc-pythongc

title: bcc-pythongc(8)

summary: Trace garbage collection events in

text:
# bcc-pythongc(8)

## NAME
ugc, javagc, nodegc, pythongc, rubygc - Trace garbage collection events in high-level languages.

## SYNOPSIS
javagc [-h] [-v] [-m] [-M MINIMUM] [-F FILTER] pid

nodegc [-h] [-v] [-m] [-M MINIMUM] [-F FILTER] pid

pythongc [-h] [-v] [-m] [-M MINIMUM] [-F FILTER] pid

rubygc [-h] [-v] [-m] [-M MINIMUM] [-F FILTER] pid

ugc [-h] [-v] [-m] [-M MINIMUM] [-F FILTER] [-l {java,node,python,ruby}] pid

## DESCRIPTION
This traces garbage collection events as they occur, including their duration and any additional information (such as generation collected or type of GC) provided by the respective language's runtime.

This tool relies on USDT probes embedded in many high-level languages, such as Java, Node, Python, and Ruby. It requires a runtime instrumented with these probes, which in some cases requires building from source with a USDT-specific flag, such as "--enable-dtrace" or "--with-dtrace".

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-v

Print the resulting BPF program, for debugging purposes.

-m

Print times in milliseconds. The default is microseconds.

-M MINIMUM

Display only collections that are longer than this threshold. The value is given in milliseconds. The default is to display all collections.

-F FILTER

Display only collections whose textual description matches (contains) this string. The default is to display all collections. Note that the filtering here is performed in user-space, and not as part of the BPF program. This means that if you have thousands of collection events, specifying this filter will not reduce the amount of data that has to be transferred from the BPF program to the user-space script.

{java,node,python,ruby}

The language to trace.

pid

The process id to trace.

## EXAMPLES
Trace garbage collections in a specific Node process:

\# ugc node 148

Trace garbage collections in a specific Java process, and print GC times in

milliseconds: # ugc -m java 6004

Trace garbage collections in a specific Java process, and display them only if

they are longer than 10ms and have the string "Tenured" in their detailed description: # ugc -M 10 -F Tenured java 6004

## FIELDS
START

The start time of the GC, in seconds from the beginning of the trace.

TIME

The duration of the garbage collection event.

DESCRIPTION

The runtime-provided description of this garbage collection event.

## OVERHEAD
Garbage collection events, even if frequent, should not produce a considerable overhead when traced because they are still not very common. Even hundreds of GCs per second (which is a very high rate) will still produce a fairly negligible overhead.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _example.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein

## SEE ALSO
trace(8), ustat(8), uobjnew(8)


bcc-pythonstat

title: bcc-pythonstat(8)

summary: Activity stats from

text:
# bcc-pythonstat(8)

## NAME
ustat, javastat, nodestat, perlstat, phpstat, pythonstat, rubystat, tclstat - Activity stats from high-level languages.

## SYNOPSIS
javastat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

nodestat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

perlstat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

phpstat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

pythonstat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

rubystat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

tclstat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

ustat [-l {java,node,perl,php,python,ruby,tcl}] [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

## DESCRIPTION
This is "top" for high-level language events, such as garbage collections, exceptions, thread creations, object allocations, method calls, and more. The events are aggregated for each process and printed in a top-like table, which can be sorted by various fields. Not all language runtimes provide the same set of details.

This uses in-kernel eBPF maps to store per process summaries for efficiency.

This tool relies on USDT probes embedded in many high-level languages, such as Java, Node, Perl, PHP, Python, Ruby, and Tcl. It requires a runtime instrumented with these probes, which in some cases requires building from source with a USDT-specific flag, such as "--enable-dtrace" or "--with-dtrace". For Java, some probes are not enabled by default, and can be turned on by running the Java process with the "-XX:+ExtendedDTraceProbes" flag. For PHP processes, the environment variable USE_ZEND_DTRACE must be set to 1.

Newly-created processes will only be traced at the next interval. If you run this tool with a short interval (say, 1-5 seconds), this should be virtually unnoticeable. For longer intervals, you might miss processes that were started and terminated during the interval window.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-l {java,node,perl,php,python,ruby,tcl}

The language to trace. By default, all languages are traced.

-C

Do not clear the screen between updates.

-S {cload,excp,gc,method,objnew,thread}

Sort the output by the specified field.

-r MAXROWS

Do not print more than this number of rows.

-d

Print the resulting BPF program, for debugging purposes.

interval

Interval between updates, seconds.

count

Number of interval summaries.

## EXAMPLES
Summarize activity in high-level languages, 1 second refresh:

\# ustat

Don't clear the screen, and top 8 rows only:

\# ustat -Cr 8

5 second summaries, 10 times only:

\# ustat 5 10

## FIELDS
loadavg

The contents of /proc/loadavg

PID

Process ID.

CMDLINE

Process command line (often the second and following arguments will give you a hint as to which application is being run.

METHOD/s

Count of method invocations during interval.

GC/s

Count of garbage collections during interval.

OBJNEW/s

Count of objects allocated during interval.

CLOAD/s

Count of classes loaded during interval.

EXC/s

Count of exceptions thrown during interval.

THR/s

Count of threads created during interval.

## OVERHEAD
When using this tool with high-frequency events, such as method calls, a very significant slow-down can be expected. However, many of the high-level languages covered by this tool already have a fairly high per-method invocation cost, especially when running in interpreted mode. For the lower-frequency events, such as garbage collections or thread creations, the overhead should not be significant. Specifically, when probing Java processes and not using the "-XX:+ExtendedDTraceProbes" flag, the most expensive probes are not emitted, and the overhead should be acceptable.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _example.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein

## SEE ALSO
trace(8), argdist(8), tplist(8)


bcc-readahead

title: bcc-readahead(8)

summary: Show performance of read-ahead cache

text:
# bcc-readahead(8)

## NAME
readahead - Show performance of read-ahead cache

## SYNOPSIS
readahead [-d DURATION]

## DESCRIPTION
The tool shows the performance of read-ahead caching on the system under a given load to investigate any caching issues. It shows a count of unused pages in the cache and also prints a histogram showing how long they have remained there.

This tool traces the __do_page_cache_readahead() kernel function to track entry and exit in the readahead mechanism in the kernel and then uses __page_cache_alloc() and mark_page_accessed() functions to calculate the age of the page in the cache as well as see how many are left unaccessed.

Since this uses BPF, only the root user can use this tool.

### NOTE ON KPROBES USAGE
Since the tool uses Kprobes, depending on your linux kernel's compilation, these functions may be inlined and hence not available for Kprobes. To see whether you have the functions available, check vmlinux source and binary to confirm whether inlining is happening or not. You can also check /proc/kallsyms on the host and verify if the target functions are present there before using this.

## REQUIREMENTS
CONFIG_BPF, bcc

## OPTIONS
-h Print usage message

-d DURATION

Trace the read-ahead caching system for DURATION seconds

## EXAMPLES
Trace for 30 seconds and show histogram of page age (ms) in read-ahead cache along with unused page count:

\# readahead -d 30

## OVERHEAD
The kernel functions instrumented by this program could be high-frequency depending on the profile of the application (for example sequential IO). We advise the users to measure and monitor the overhead before leaving this turned on in production environments.

## SOURCE
This originated as a bpftrace tool from the book "BPF Performance Tools", published by Addison Wesley (2019):

http://www.brendangregg.com/bpf-performance-tools-book.html

See the book for more documentation on this tool.

This version is in the BCC repository:

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Suchakra Sharma

## SEE ALSO
readahead(2), madvise(2)


bcc-reset-trace

title: bcc-reset-trace(8)

summary: reset the state of tracing.

text:
# bcc-reset-trace(8)

## NAME
reset-trace - reset the state of tracing.

## SYNOPSIS
reset-trace [-F] [-h] [-q] [-v]

## DESCRIPTION
You will probably never need this tool. If you kill -9 a bcc tool (plus other signals, like SIGTERM), or if a bcc tool crashes, then kernel tracing can be left in a semi-enabled state. It's not as bad as it sounds: there may just be overhead for writing to ring buffers that are never read. This tool can be used to clean up the tracing state, and reset and disable active tracing.

Make sure no other tracing sessions are active. This tool might stop them from functioning (perhaps ungracefully).

This specifically clears the state in at least the following files in /sys/kernel/debug/tracing: kprobe_events, uprobe_events, trace_pipe. Other tracing facilities (ftrace) are checked, and if not in an expected state, a note is printed. All tracing files can be reset with -F for force, but this will interfere with any other running tracing sessions (eg, ftrace).

## REQUIREMENTS
/sys/kernel/debug mounted as debugfs

## OPTIONS
-F

Force. Will reset all tracing facilities, including those not used by bcc (ftrace). You shouldn't need to use this.

-h

USAGE message.

-q

Quiet. No output while working.

-v

Verbose: print what it is doing.

## EXAMPLES
Reset the state of tracing:

\# reset-trace

Verbose:

\# reset-trace -v

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg


bcc-rubycalls

title: bcc-rubycalls(8)

summary: Summarize method calls

text:
# bcc-rubycalls(8)

## NAME
ucalls, javacalls, perlcalls, phpcalls, pythoncalls, rubycalls, tclcalls - Summarize method calls from high-level languages and Linux syscalls.

## SYNOPSIS
javacalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

perlcalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

phpcalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

pythoncalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

rubycalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

tclcalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

ucalls [-l {java,perl,php,python,ruby}] [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

## DESCRIPTION
This tool summarizes method calls from high-level languages such as Java, Perl, PHP, Python, Ruby, and Tcl. It can also trace Linux system calls. Whenever a method is invoked, ucalls records the call count and optionally the method's execution time (latency) and displays a summary.

This uses in-kernel eBPF maps to store per process summaries for efficiency.

This tool relies on USDT probes embedded in many high-level languages, such as Java, Perl, PHP, Python, Ruby, and Tcl. It requires a runtime instrumented with these probes, which in some cases requires building from source with a USDT-specific flag, such as "--enable-dtrace" or "--with-dtrace". For Java, method probes are not enabled by default, and can be turned on by running the Java process with the "-XX:+ExtendedDTraceProbes" flag. For PHP processes, the environment variable USE_ZEND_DTRACE must be set to 1.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-l {java,perl,php,python,ruby,tcl}

The language to trace. If not provided, only syscalls are traced (when the -S option is used).

-T TOP

Print only the top methods by frequency or latency.

-L

Collect method invocation latency (duration).

-S

Collect Linux syscalls frequency and timing.

-v

Print the resulting BPF program, for debugging purposes.

-m

Print times in milliseconds (the default is microseconds).

pid

The process id to trace.

interval

Print summary after this number of seconds and then exit. By default, wait for Ctrl+C to terminate.

## EXAMPLES
Trace the top 10 Ruby method calls:

\# ucalls -T 10 -l ruby 1344

Trace Python method calls and Linux syscalls including latency in milliseconds:

\# ucalls -l python -mL 2020

Trace only syscalls and print a summary after 10 seconds:

\# ucalls -S 788 10

## OVERHEAD
Tracing individual method calls will produce a considerable overhead in all high-level languages. For languages with just-in-time compilation, such as Java, the overhead can be more considerable than for interpreted languages. On the other hand, syscall tracing will typically be tolerable for most processes, unless they have a very unusual rate of system calls.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _example.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein

## SEE ALSO
ustat(8), argdist(8)


bcc-rubyflow

title: bcc-rubyflow(8)

summary: Print a flow graph of method

text:
# bcc-rubyflow(8)

## NAME
uflow, javaflow, perlflow, phpflow, pythonflow, rubyflow, tclflow - Print a flow graph of method calls in high-level languages.

## SYNOPSIS
javaflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

perlflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

phpflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

pythonflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

rubyflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

tclflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

uflow [-h] [-M METHOD] [-C CLAZZ] [-v] [-l {java,perl,php,python,ruby,tcl}] pid

## DESCRIPTION
uflow traces method calls and prints them in a flow graph that can facilitate debugging and diagnostics by following the program's execution (method flow).

This tool relies on USDT probes embedded in many high-level languages, such as Java, Perl, PHP, Python, Ruby, and Tcl. It requires a runtime instrumented with these probes, which in some cases requires building from source with a USDT-specific flag, such as "--enable-dtrace" or "--with-dtrace". For Java processes, the startup flag "-XX:+ExtendedDTraceProbes" is required. For PHP processes, the environment variable USE_ZEND_DTRACE must be set to 1.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-M METHOD

Print only method calls where the method name begins with this string.

-C CLAZZ

Print only method calls where the class name begins with this string. The class name interpretation strongly depends on the language. For example, in Java use "package/subpackage/ClassName" to refer to classes.

-v

Print the resulting BPF program, for debugging purposes.

{java,perl,php,python,ruby,tcl}

The language to trace.

pid

The process id to trace.

## EXAMPLES
Follow method flow in a Ruby process:

\# uflow ruby 148

Follow method flow in a Java process where the class name is java.lang.Thread:

\# uflow -C java/lang/Thread java 1802

## FIELDS
CPU

The CPU number on which the method was invoked. This is useful to easily see where the output skips to a different CPU.

PID

The process id.

TID

The thread id.

TIME

The duration of the method call.

METHOD

The method name.

## OVERHEAD
This tool has extremely high overhead because it prints every method call. For some scenarios, you might see lost samples in the output as the tool is unable to keep up with the rate of data coming from the kernel. Filtering by class or method prefix can help reduce the amount of data printed, but there is still a very high overhead in the collection mechanism. Do not use for performance- sensitive production scenarios, and always test first.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _example.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein

## SEE ALSO
trace(8), ustat(8)


bcc-rubygc

title: bcc-rubygc(8)

summary: Trace garbage collection events in

text:
# bcc-rubygc(8)

## NAME
ugc, javagc, nodegc, pythongc, rubygc - Trace garbage collection events in high-level languages.

## SYNOPSIS
javagc [-h] [-v] [-m] [-M MINIMUM] [-F FILTER] pid

nodegc [-h] [-v] [-m] [-M MINIMUM] [-F FILTER] pid

pythongc [-h] [-v] [-m] [-M MINIMUM] [-F FILTER] pid

rubygc [-h] [-v] [-m] [-M MINIMUM] [-F FILTER] pid

ugc [-h] [-v] [-m] [-M MINIMUM] [-F FILTER] [-l {java,node,python,ruby}] pid

## DESCRIPTION
This traces garbage collection events as they occur, including their duration and any additional information (such as generation collected or type of GC) provided by the respective language's runtime.

This tool relies on USDT probes embedded in many high-level languages, such as Java, Node, Python, and Ruby. It requires a runtime instrumented with these probes, which in some cases requires building from source with a USDT-specific flag, such as "--enable-dtrace" or "--with-dtrace".

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-v

Print the resulting BPF program, for debugging purposes.

-m

Print times in milliseconds. The default is microseconds.

-M MINIMUM

Display only collections that are longer than this threshold. The value is given in milliseconds. The default is to display all collections.

-F FILTER

Display only collections whose textual description matches (contains) this string. The default is to display all collections. Note that the filtering here is performed in user-space, and not as part of the BPF program. This means that if you have thousands of collection events, specifying this filter will not reduce the amount of data that has to be transferred from the BPF program to the user-space script.

{java,node,python,ruby}

The language to trace.

pid

The process id to trace.

## EXAMPLES
Trace garbage collections in a specific Node process:

\# ugc node 148

Trace garbage collections in a specific Java process, and print GC times in

milliseconds: # ugc -m java 6004

Trace garbage collections in a specific Java process, and display them only if

they are longer than 10ms and have the string "Tenured" in their detailed description: # ugc -M 10 -F Tenured java 6004

## FIELDS
START

The start time of the GC, in seconds from the beginning of the trace.

TIME

The duration of the garbage collection event.

DESCRIPTION

The runtime-provided description of this garbage collection event.

## OVERHEAD
Garbage collection events, even if frequent, should not produce a considerable overhead when traced because they are still not very common. Even hundreds of GCs per second (which is a very high rate) will still produce a fairly negligible overhead.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _example.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein

## SEE ALSO
trace(8), ustat(8), uobjnew(8)


bcc-rubyobjnew

title: bcc-rubyobjnew(8)

summary: Summarize object allocations in

text:
# bcc-rubyobjnew(8)

## NAME
uobjnew, cobjnew, javaobjnew, rubyobjnew, tclobjnew - Summarize object allocations in high-level languages.

## SYNOPSIS
cobjnew [-h] [-C TOP_COUNT] [-S TOP_SIZE] [-v] pid [interval]

javaobjnew [-h] [-C TOP_COUNT] [-S TOP_SIZE] [-v] pid [interval]

rubyobjnew [-h] [-C TOP_COUNT] [-S TOP_SIZE] [-v] pid [interval]

tclobjnew [-h] [-C TOP_COUNT] [-S TOP_SIZE] [-v] pid [interval]

uobjnew [-h] [-C TOP_COUNT] [-S TOP_SIZE] [-v] [-l {c,java,ruby,tcl}] pid [interval]

## DESCRIPTION
uobjnew traces object allocations in high-level languages (including "malloc") and prints summaries of the most frequently allocated types by number of objects or number of bytes.

This tool relies on USDT probes embedded in many high-level languages, such as C, Java, Ruby, and Tcl. It requires a runtime instrumented with these probes, which in some cases requires building from source with a USDT-specific flag, such as "--enable-dtrace" or "--with-dtrace". For Java, the Java process must be started with the "-XX:+ExtendedDTraceProbes" flag.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-C TOP_COUNT

Print the top object types sorted by number of instances.

-S TOP_SIZE

Print the top object types sorted by size.

-v

Print the resulting BPF program, for debugging purposes.

{c,java,ruby,tcl}

The language to trace.

pid

The process id to trace.

interval

Wait this many seconds and then print the summary and exit. By default, wait for Ctrl+C to exit.

## EXAMPLES
Trace object allocations in a Ruby process:

\# uobjnew ruby 148

Trace object allocations from "malloc" and print the top 10 by total size:

\# uobjnew -S 10 c 1788

## FIELDS
TYPE

The object type being allocated. For C (malloc), this is the block size.

ALLOCS

The number of objects allocated.

BYTES

The number of bytes allocated.

## OVERHEAD
Object allocation events are quite frequent, and therefore the overhead from running this tool can be considerable. Use with caution and make sure to test before using in a production environment. Nonetheless, even thousands of allocations per second will likely produce a reasonable overhead when investigating a problem.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _example.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein

## SEE ALSO
ustat(8), ugc(8), memleak(8)


bcc-rubystat

title: bcc-rubystat(8)

summary: Activity stats from

text:
# bcc-rubystat(8)

## NAME
ustat, javastat, nodestat, perlstat, phpstat, pythonstat, rubystat, tclstat - Activity stats from high-level languages.

## SYNOPSIS
javastat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

nodestat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

perlstat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

phpstat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

pythonstat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

rubystat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

tclstat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

ustat [-l {java,node,perl,php,python,ruby,tcl}] [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

## DESCRIPTION
This is "top" for high-level language events, such as garbage collections, exceptions, thread creations, object allocations, method calls, and more. The events are aggregated for each process and printed in a top-like table, which can be sorted by various fields. Not all language runtimes provide the same set of details.

This uses in-kernel eBPF maps to store per process summaries for efficiency.

This tool relies on USDT probes embedded in many high-level languages, such as Java, Node, Perl, PHP, Python, Ruby, and Tcl. It requires a runtime instrumented with these probes, which in some cases requires building from source with a USDT-specific flag, such as "--enable-dtrace" or "--with-dtrace". For Java, some probes are not enabled by default, and can be turned on by running the Java process with the "-XX:+ExtendedDTraceProbes" flag. For PHP processes, the environment variable USE_ZEND_DTRACE must be set to 1.

Newly-created processes will only be traced at the next interval. If you run this tool with a short interval (say, 1-5 seconds), this should be virtually unnoticeable. For longer intervals, you might miss processes that were started and terminated during the interval window.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-l {java,node,perl,php,python,ruby,tcl}

The language to trace. By default, all languages are traced.

-C

Do not clear the screen between updates.

-S {cload,excp,gc,method,objnew,thread}

Sort the output by the specified field.

-r MAXROWS

Do not print more than this number of rows.

-d

Print the resulting BPF program, for debugging purposes.

interval

Interval between updates, seconds.

count

Number of interval summaries.

## EXAMPLES
Summarize activity in high-level languages, 1 second refresh:

\# ustat

Don't clear the screen, and top 8 rows only:

\# ustat -Cr 8

5 second summaries, 10 times only:

\# ustat 5 10

## FIELDS
loadavg

The contents of /proc/loadavg

PID

Process ID.

CMDLINE

Process command line (often the second and following arguments will give you a hint as to which application is being run.

METHOD/s

Count of method invocations during interval.

GC/s

Count of garbage collections during interval.

OBJNEW/s

Count of objects allocated during interval.

CLOAD/s

Count of classes loaded during interval.

EXC/s

Count of exceptions thrown during interval.

THR/s

Count of threads created during interval.

## OVERHEAD
When using this tool with high-frequency events, such as method calls, a very significant slow-down can be expected. However, many of the high-level languages covered by this tool already have a fairly high per-method invocation cost, especially when running in interpreted mode. For the lower-frequency events, such as garbage collections or thread creations, the overhead should not be significant. Specifically, when probing Java processes and not using the "-XX:+ExtendedDTraceProbes" flag, the most expensive probes are not emitted, and the overhead should be acceptable.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _example.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein

## SEE ALSO
trace(8), argdist(8), tplist(8)


bcc-runqlat

title: bcc-runqlat(8)

text:
# bcc-runqlat(8)

## NAME
runqlat - Run queue (scheduler) latency as a histogram.

## SYNOPSIS
runqlat [-h] [-T] [-m] [-P] [--pidnss] [-L] [-p PID] [interval] [count]

## DESCRIPTION
This measures the time a task spends waiting on a run queue (or equivalent scheduler data structure) for a turn on-CPU, and shows this time as a histogram. This time should be small, but a task may need to wait its turn due to CPU load. The higher the CPU load, the longer a task will generally need to wait its turn.

This tool measures two types of run queue latency:

1. The time from a task being enqueued on a run queue to its context switch and execution. This traces ttwu_do_wakeup(), wake_up_new_task() -> finish_task_switch() with either raw tracepoints (if supported) or kprobes and instruments the run queue latency after a voluntary context switch.

2. The time from when a task was involuntary context switched and still in the runnable state, to when it next executed. This is instrumented from finish_task_switch() alone.

This tool uses in-kernel eBPF maps for storing timestamps and the histogram, for efficiency. Despite this, the overhead of this tool may become significant for some workloads: see the OVERHEAD section.

This works by tracing various kernel scheduler functions using dynamic tracing, and will need updating to match any changes to these functions.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print usage message.

-T

Include timestamps on output.

-m

Output histogram in milliseconds.

-P

Print a histogram for each PID.

--pidnss

Print a histogram for each PID namespace (short for PID namespaces). For container analysis.

-L

Print a histogram for each thread ID.

-p PID

Only show this PID (filtered in kernel for efficiency).

interval

Output interval, in seconds.

count

Number of outputs.

## EXAMPLES
Summarize run queue latency as a histogram:

\# runqlat

Print 1 second summaries, 10 times:

\# runqlat 1 10

Print 1 second summaries, using milliseconds as units for the histogram, and include timestamps on output:

\# runqlat -mT 1

Trace PID 186 only, 1 second summaries:

\# runqlat -P 185 1

## FIELDS
usecs

Microsecond range

msecs

Millisecond range

count

How many times a task event fell into this range

distribution

An ASCII bar chart to visualize the distribution (count column)

## OVERHEAD
This traces scheduler functions, which can become very frequent. While eBPF has very low overhead, and this tool uses in-kernel maps for efficiency, the frequency of scheduler events for some workloads may be high enough that the overhead of this tool becomes significant. Measure in a lab environment to quantify the overhead before use.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
runqlen(8), runqslower(8), pidstat(1)


bcc-runqlen

title: bcc-runqlen(8)

text:
# bcc-runqlen(8)

## NAME
runqlen - Scheduler run queue length as a histogram.

## SYNOPSIS
runqlen [-h] [-T] [-O] [-C] [interval] [count]

## DESCRIPTION
This program summarizes scheduler queue length as a histogram, and can also show run queue occupancy. It works by sampling the run queue length on all CPUs at 99 Hertz.

This tool can be used to identify imbalances, eg, when processes are bound to CPUs causing queueing, or interrupt mappings causing the same.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print usage message.

-T

Include timestamps on output.

-O

Report run queue occupancy.

-C

Report for each CPU.

interval

Output interval, in seconds.

count

Number of outputs.

## EXAMPLES
Summarize run queue length as a histogram:

\# runqlen

Print 1 second summaries, 10 times:

\# runqlen 1 10

Print output every second, with timestamps, and show each CPU separately:

\# runqlen -CT 1

Print run queue occupancy every second:

\# runqlen -O 1

Print run queue occupancy, with timestamps, for each CPU:

\# runqlen -COT 1

## FIELDS
runqlen

Scheduler run queue length: the number of threads (tasks) waiting to run, (excluding including the currently running task).

count

Number of samples at this queue length.

distribution

An ASCII bar chart to visualize the distribution (count column)

## OVERHEAD
This uses sampling at 99 Hertz (on all CPUs), and in-kernel summaries, which should make overhead negligible. This does not trace scheduler events, like runqlen does, which comes at a much higher overhead cost.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
runqlat(8), runqslower(8), pidstat(1)


bcc-runqslower

title: bcc-runqslower(8)

summary: Trace long process scheduling delays.

text:
# bcc-runqslower(8)

## NAME
runqslower - Trace long process scheduling delays.

## SYNOPSIS
runqslower [-p PID] [-t TID] [min_us]

## DESCRIPTION
This measures the time a task spends waiting on a run queue (or equivalent scheduler data structure) for a turn on-CPU, and shows occurrences of time exceeding passed threshold. This time should be small, but a task may need to wait its turn due to CPU load. The higher the CPU load, the longer a task will generally need to wait its turn.

This tool measures two types of run queue latency:

1. The time from a task being enqueued on a run queue to its context switch and execution. This traces ttwu_do_wakeup(), wake_up_new_task() -> finish_task_switch() with either raw tracepoints (if supported) or kprobes and instruments the run queue latency after a voluntary context switch.

2. The time from when a task was involuntary context switched and still in the runnable state, to when it next executed. This is instrumented from finish_task_switch() alone.

The overhead of this tool may become significant for some workloads: see the OVERHEAD section.

This works by tracing various kernel scheduler functions using dynamic tracing, and will need updating to match any changes to these functions.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print usage message.

-p PID

Only show this PID (filtered in kernel for efficiency).

-t TID

Only show this TID (filtered in kernel for efficiency).

min_us

Minimum scheduling delay in microseconds to output.

## EXAMPLES
Show scheduling delays longer than 10ms:

\# runqslower

Show scheduling delays longer than 1ms for process with PID 123:

\# runqslower -p 123 1000

## FIELDS
TIME

Time of when scheduling event occurred.

COMM

Process name.

PID

Process ID.

LAT(us)

Scheduling latency from time when task was ready to run to the time it was assigned to a CPU to run.

## OVERHEAD
This traces scheduler functions, which can become very frequent. While eBPF has very low overhead, and this tool uses in-kernel maps for efficiency, the frequency of scheduler events for some workloads may be high enough that the overhead of this tool becomes significant. Measure in a lab environment to quantify the overhead before use.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Ivan Babrou, original BCC Python version Andrii Nakryiko, CO-RE version

## SEE ALSO
runqlen(8), runqlat(8), pidstat(1)


bcc-shmsnoop

title: bcc-shmsnoop(8)

text:
# bcc-shmsnoop(8)

## NAME
shmsnoop - Trace System V shared memory syscalls. Uses Linux eBPF/bcc.

## SYNOPSIS
shmsnoop [-h] [-T] [-p] [-t] [-d DURATION] [-n NAME]

## DESCRIPTION
shmsnoop traces System V shared memory syscalls: shmget, shmat, shmdt, shmctl

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print usage message.

-T

Include a timestamp column.

-p PID

Trace this process ID only (filtered in-kernel).

-t TID

Trace this thread ID only (filtered in-kernel).

-d DURATION

Total duration of trace in seconds.

-n NAME

Only print command lines matching this command name (regex)

## EXAMPLES
Trace all shm* syscalls:

\# shmsnoop

Trace all shm* syscalls, and include timestamps:

\# shmsnoop -T

Only trace shm* syscalls where the process contains "server":

\# shmsnoop -n server

## FIELDS
TIME(s)

Time of shm syscall return, in seconds.

PID

Process ID

COMM

Process/command name.

RET

Return value of shm syscall.

ARGS

"arg: value" couples that represent given syscall arguments as described in their manpage

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Jiri Olsa

## SEE ALSO
opensnoop(1)


bcc-slabratetop

title: bcc-slabratetop(8)

text:
# bcc-slabratetop(8)

## NAME
slabratetop - Kernel SLAB/SLUB memory cache allocation rate top. Uses Linux BPF/bcc.

## SYNOPSIS
slabratetop [-h] [-C] [-r MAXROWS] [interval] [count]

## DESCRIPTION
This is top for the the rate of kernel SLAB/SLUB memory allocations. It works by tracing kmem_cache_alloc() calls, a commonly used interface for kernel memory allocation (SLAB or SLUB). It summarizes the rate and total bytes allocated of these calls per interval: the activity. Compare this to slabtop(1), which shows the current static volume of the caches.

This tool uses kernel dynamic tracing of the kmem_cache_alloc() function.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-C

Don't clear the screen.

-r MAXROWS

Maximum number of rows to print. Default is 20.

interval

Interval between updates, seconds.

count

Number of interval summaries.

## EXAMPLES
Summarize active kernel SLAB/SLUB calls (kmem_cache_alloc()), showing the top 20 caches every second:

\# slabratetop

Don't clear the screen, and top 8 rows only:

\# slabratetop -Cr 8

5 second summaries, 10 times only:

\# slabratetop 5 10

## FIELDS
loadavg:

The contents of /proc/loadavg

CACHE

Kernel cache name.

ALLOCS

Allocations (number of calls).

BYTES

Total bytes allocated.

## OVERHEAD
If kmem_cache_alloc() is called at a high rate (eg, >100k/second) the overhead of this tool might begin to be measurable. The rate can be seen in the ALLOCS column of the output.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
slabtop(1)


bcc-sofdsnoop

title: bcc-sofdsnoop(8)

summary: traces FDs passed by sockets

text:
# bcc-sofdsnoop(8)

## NAME
sofdsnoop - traces FDs passed by sockets

## SYNOPSIS
usage: sofdsnoop [-h] [-T] [-p PID] [-t TID] [-n NAME] [-d DURATION]

## DESCRIPTION
Trace file descriptors passed via socket

### optional arguments:
-h, --help

show this help message and exit

-T, --timestamp

include timestamp on output

-p PID, --pid PID

trace this PID only

-t TID, --tid TID

trace this TID only

-n NAME, --name NAME

only print process names containing this name

-d DURATION, --duration DURATION

total duration of trace in seconds

## EXAMPLES
Trace passed file descriptors

\# sofdsnoop

Include timestamps

\# sofdsnoop -T

Only trace PID 181

\# sofdsnoop -p 181

Only trace TID 123

\# sofdsnoop -t 123

Trace for 10 seconds only

\# sofdsnoop -d 10

Only print process names containing "main"

\# sofdsnoop -n main

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.