bcc-softirqs

title: bcc-softirqs(8)

text:
# bcc-softirqs(8)

## NAME
softirqs - Measure soft IRQ (soft interrupt) event time. Uses Linux eBPF/bcc.

## SYNOPSIS
softirqs [-h] [-T] [-N] [-d] [interval] [count]

## DESCRIPTION
This summarizes the time spent servicing soft IRQs (soft interrupts), and can show this time as either totals or histogram distributions. A system-wide summary of this time is shown by the %soft column of mpstat(1), and soft IRQ event counts (but not times) are available in /proc/softirqs.

This tool uses the irq:softirq_enter and irq:softirq_exit kernel tracepoints, which is a stable tracing mechanism. BPF programs can attach to tracepoints from Linux 4.7 only. An older version of this tool is available in tools/old, and uses kprobes instead of tracepoints.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print usage message.

-T

Include timestamps on output.

-N

Output in nanoseconds

-d

Show IRQ time distribution as histograms

## EXAMPLES
Sum soft IRQ event time until Ctrl-C:

\# softirqs

Show soft IRQ event time as histograms:

\# softirqs -d

Print 1 second summaries, 10 times:

\# softirqs 1 10

1 second summaries, printed in nanoseconds, with timestamps:

\# softirqs -NT 1

## FIELDS
SOFTIRQ

The kernel function name that performs the soft IRQ action.

TOTAL_usecs

Total time spent in this soft IRQ function in microseconds.

TOTAL_nsecs

Total time spent in this soft IRQ function in nanoseconds.

usecs

Range of microseconds for this bucket.

nsecs

Range of nanoseconds for this bucket.

count

Number of soft IRQs in this time range.

distribution

ASCII representation of the distribution (the count column).

## OVERHEAD
This traces kernel functions and maintains in-kernel counts, which are asynchronously copied to user-space. While the rate of interrupts be very high (>1M/sec), this is a relatively efficient way to trace these events, and so the overhead is expected to be small for normal workloads, but could become noticeable for heavy workloads. Measure in a test environment before use.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHORS
Brendan Gregg, Sasha Goldshtein

## SEE ALSO
hardirqs(8)


bcc-solisten

package: bcc-tools

title: bcc-solisten(8)

summary: Trace listening socket

text:
# bcc-solisten(8)

## NAME
solisten - Trace listening socket

## SYNOPSIS
usage: solisten [-h] [--show-netns] [-p PID] [-n NETNS]

## DESCRIPTION
All IPv4 and IPv6 listen attempts are traced, even if they ultimately fail or the listening program is not willing to accept().

### optional arguments:
-h, --help

show this help message and exit

--show-netns

show network namespace

-p PID, --pid PID

trace this PID only

-n NETNS, --netns NETNS

trace this Network Namespace only

## EXAMPLES
Stream socket listen:

\# solisten

Stream socket listen for specified PID only

\# solisten -p 1234

Stream socket listen for the specified network namespace ID only

\# solisten --netns 4242

Show network ns ID (useful for containers)

\# solisten --show-netns

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.


bcc-spfdsnoop

title: bcc-spfdsnoop(8)

text:
# bcc-spfdsnoop(8)

## NAME
sofdsnoop - Trace FDs passed through unix sockets. Uses Linux eBPF/bcc.

## SYNOPSIS
sofdsnoop [-h] [-T] [-p PID] [-t TID] [-n NAME] [-d DURATION]

## DESCRIPTION
sofdsnoop traces FDs passed through unix sockets

Every file descriptor that is passed via unix sockets os displayed on separate line together with process info (TID/COMM columns), ACTION details (SEND/RECV), file descriptor number (FD) and its translation to file if available (NAME).

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print usage message.

-T

Include a timestamp column.

-p PID

Trace this process ID only (filtered in-kernel).

-t TID

Trace this thread ID only (filtered in-kernel).

-d DURATION

Total duration of trace in seconds.

-n NAME

Only print command lines matching this command name (regex)

## EXAMPLES
Trace all sockets:

\# sofdsnoop

Trace all sockets, and include timestamps:

\# sofdsnoop -T

Only trace sockets where the process contains "server":

\# sofdsnoop -n server

## FIELDS
TIME(s)

Time of SEDN/RECV actions, in seconds.

ACTION

Operation on the fd SEND/RECV.

TID

Process TID

COMM

Parent process/command name.

SOCKET

The socket carrier.

FD

file descriptor number

NAME

file name for SEND lines

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Jiri Olsa

## SEE ALSO
opensnoop(1)


bcc-sslsniff

title: bcc-sslsniff(8)

text:
# bcc-sslsniff(8)

## NAME
sslsniff - Print data passed to OpenSSL, GnuTLS or NSS. Uses Linux eBPF/bcc.

## SYNOPSIS
sslsniff [-h] [-p PID] [-c COMM] [-o] [-g] [-n] [-d] [--hexdump]

## DESCRIPTION
sslsniff prints data sent to write/send and read/recv functions of OpenSSL, GnuTLS and NSS, allowing us to read plain text content before encryption (when writing) and after decryption (when reading).

This works reading the second parameter of both functions (*buf).

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## EXAMPLES
Print all calls to SSL write/send and read/recv system-wide:

\# sslsniff

## FIELDS
FUNC

Which function is being called (write/send or read/recv)

TIME

Time of the command, in seconds.

COMM

Entered command.

PID

Process ID calling SSL.

LEN

Bytes written or read by SSL functions.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHORS
Adrian Lopez and Mark Drayton

## SEE ALSO
trace(8)


bcc-stackcount

title: bcc-stackcount(8)

text:
# bcc-stackcount(8)

## NAME
stackcount - Count function calls and their stack traces. Uses Linux eBPF/bcc.

## SYNOPSIS
stackcount [-h] [-p PID] [-c CPU] [-i INTERVAL] [-D DURATION] [-T]

[-r] [-s] [-P] [-K] [-U] [-v] [-d] [-f] [--debug] pattern

## DESCRIPTION
stackcount traces functions and frequency counts them with their entire stack trace, kernel stack and user stack, summarized in-kernel for efficiency. This allows higher frequency events to be studied. The output consists of unique stack traces, and their occurrence counts. In addition to kernel and user functions, kernel tracepoints and USDT tracepoint are also supported.

The pattern is a string with optional '*' wildcards, similar to file globbing. If you'd prefer to use regular expressions, use the -r option.

This tool only works on Linux 4.6+. Stack traces are obtained using the new `BPF_STACK_TRACE` APIs. For kernels older than 4.6, see the version under tools/old.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print usage message.

-r

Allow regular expressions for the search pattern. The default allows "*" wildcards only.

-s

Show address offsets.

-P

Display stacks separately for each process.

-K

Show kernel stack only.

-U

Show user stack only.

-T

Include a timestamp with interval output.

-v

Show raw addresses.

-d

Print a delimiter ("--") in-between the kernel and user stacks.

--debug

Print the source of the BPF program when loading it (for debugging purposes).

-i interval

Summary interval, in seconds.

-D duration

Total duration of trace, in seconds. -f Folded output format.

-p PID

Trace this process ID only (filtered in-kernel).

-c CPU

Trace this CPU only (filtered in-kernel).

pattern

A function name, or a search pattern. Can include wildcards ("*"). If the -r option is used, can include regular expressions.

## EXAMPLES
Count kernel and user stack traces for submit_bio():

\# stackcount submit_bio

Count stacks with a delimiter for submit_bio():

\# stackcount -d submit_bio

Count kernel stack trace only for submit_bio():

\# stackcount -K submit_bio

Count user stack trace only for submit_bio():

\# stackcount -U submit_bio

Count stack traces for ip_output():

\# stackcount ip_output

Show symbol offsets:

\# stackcount -s ip_output

Show offsets and raw addresses (verbose):

\# stackcount -sv ip_output

Count stacks for kernel functions matching tcp_send*:

\# stackcount 'tcp_send*'

Same as previous, but using regular expressions:

\# stackcount -r '^tcp_send.*'

Output every 5 seconds, with timestamps:

\# stackcount -Ti 5 ip_output

Only count stacks when PID 185 is on-CPU:

\# stackcount -p 185 ip_output

Only count stacks for CPU 1:

\# stackcount -c 1 put_prev_entity

Count user stacks for dynamic heap allocations with malloc in PID 185:

\# stackcount -p 185 c:malloc

Count user stacks for thread creation (USDT tracepoint) in PID 185:

\# stackcount -p 185 u:pthread:pthread_create

Count stacks for context switch events using a kernel tracepoint:

\# stackcount t:sched:sched_switch

## OVERHEAD
This summarizes unique stack traces in-kernel for efficiency, allowing it to trace a higher rate of function calls than methods that post-process in user space. The stack trace data is only copied to user space when the output is printed, which usually only happens once. The stack walking also happens in an optimized code path in the kernel thanks to the new BPF_STACK_TRACE table APIs, which should be more efficient than the manual walker in the eBPF tracer which older versions of this script used. With this in mind, call rates of < 10,000/sec would incur negligible overhead. Test before production use. You can also use funccount to get a handle on function call rates first.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg, Sasha Goldshtein

## SEE ALSO
stacksnoop(8), funccount(8)


bcc-statsnoop

title: bcc-statsnoop(8)

text:
# bcc-statsnoop(8)

## NAME
statsnoop - Trace stat() syscalls. Uses Linux eBPF/bcc.

## SYNOPSIS
statsnoop [-h] [-t] [-x] [-p PID]

## DESCRIPTION
statsnoop traces the different stat() syscalls, showing which processes are attempting to read information about which files. This can be useful for determining the location of config and log files, or for troubleshooting applications that are failing, especially on startup.

This works by tracing various kernel sys_stat() functions using dynamic tracing, and will need updating to match any changes to these functions.

This makes use of a Linux 4.4 feature (bpf_perf_event_output()); for kernels older than 4.4, see the version under tools/old, which uses an older mechanism.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print usage message.

-t

Include a timestamp column: in seconds since the first event, with decimal places.

-x

Only print failed stats.

-p PID

Trace this process ID only (filtered in-kernel).

## EXAMPLES
Trace all stat() syscalls:

\# statsnoop

Trace all stat() syscalls, and include timestamps:

\# statsnoop -t

Trace only stat() syscalls that failed:

\# statsnoop -x

Trace PID 181 only:

\# statsnoop -p 181

## FIELDS
TIME(s)

Time of the call, in seconds.

PID

Process ID

COMM

Process name

FD

File descriptor (if success), or -1 (if failed)

ERR

Error number (see the system's errno.h)

PATH

Open path

## OVERHEAD
This traces the kernel stat function and prints output for each event. As the rate of this is generally expected to be low (< 1000/s), the overhead is also expected to be negligible. If you have an application that is calling a high rate of stat()s, then test and understand overhead before use.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
opensnoop(1)


bcc-swapin

title: bcc-swapin(8)

text:
# bcc-swapin(8)

## NAME
swapin - Count swapins by process. Uses BCC/eBPF.

## SYNOPSIS
swapin

## DESCRIPTION
This tool counts swapins by process, to show which process is affected by swapping (if swap devices are in use). This can explain a significant source of application latency, if it has began swapping due to memory pressure on the system.

This works by tracing the swap_readpage() kernel funciton using dynamic instrumentation. This tool may need maintenance to keep working if that function changes in later kernels.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and BCC.

## EXAMPLES
Count swapins by process, showing per-second summaries.

\# swapin

## FIELDS
1st

The process name.

2nd

The process ID.

3rd

The count of swapins during that interval.

## OVERHEAD
The rate of swapins should be low (bounded by swapin device IOPS), such that the overhead of this tool is expected to be negligible.

## SOURCE
This originated as a bpftrace tool from the book "BPF Performance Tools", published by Addison Wesley (2019):

http://www.brendangregg.com/bpf-performance-tools-book.html

See the book for more documentation on this tool.

This version is in the BCC repository:

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
swapon(8)


bcc-syncsnoop

title: bcc-syncsnoop(8)

text:
# bcc-syncsnoop(8)

## NAME
syncsnoop - Trace sync() syscall. Uses Linux eBPF/bcc.

## SYNOPSIS
syncsnoop

## DESCRIPTION
syncsnoop traces calls to sync(), which flushes file system buffers to storage devices. These calls can cause performance perturbations, and it can be useful to know if they are happening and how frequently.

This works by tracing the kernel sys_sync() function using dynamic tracing, and will need updating to match any changes to this function.

This makes use of a Linux 4.4 feature (bpf_perf_event_output()); for kernels older than 4.4, see the version under tools/old, which uses an older mechanism.

This program is also a basic example of eBPF/bcc.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## EXAMPLES
Trace calls to sync():

\# syncsnoop

## FIELDS
TIME(s)

Time of the call, in seconds.

CALL

Call traced.

## OVERHEAD
This traces the kernel sync function and prints output for each event. As the rate of this is generally expected to be low (<< 100/s), the overhead is also expected to be negligible.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
iostat(1)


bcc-syscount

title: bcc-syscount(8)

summary: Summarize syscall counts and latencies.

text:
# bcc-syscount(8)

## NAME
syscount - Summarize syscall counts and latencies.

## SYNOPSIS
syscount [-h] [-p PID] [-i INTERVAL] [-d DURATION] [-T TOP] [-x] [-e ERRNO] [-L] [-m] [-P] [-l]

## DESCRIPTION
This tool traces syscall entry and exit tracepoints and summarizes either the number of syscalls of each type, or the number of syscalls per process. It can also collect latency (invocation time) for each syscall or each process.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc. Linux 4.7+ is required to attach a BPF program to the raw_syscalls:sys_{enter,exit} tracepoints, used by this tool.

## OPTIONS
-h

Print usage message.

-p PID

Trace only this process.

-i INTERVAL

Print the summary at the specified interval (in seconds).

-d DURATION

Total duration of trace (in seconds).

-T TOP

Print only this many entries. Default: 10.

-x

Trace only failed syscalls (i.e., the return value from the syscall was < 0).

-e ERRNO

Trace only syscalls that failed with that error (e.g. -e EPERM or -e 1).

-m

Display times in milliseconds. Default: microseconds.

-P

Summarize by process and not by syscall.

-l

List the syscalls recognized by the tool (hard-coded list). Syscalls beyond this list will still be displayed, as "[unknown: nnn]" where nnn is the syscall number.

## EXAMPLES
Summarize all syscalls by syscall:

\# syscount

Summarize all syscalls by process:

\# syscount -P

Summarize only failed syscalls:

\# syscount -x

Summarize only syscalls that failed with EPERM:

\# syscount -e EPERM

Trace PID 181 only:

\# syscount -p 181

Summarize syscalls counts and latencies:

\# syscount -L

## FIELDS
PID

Process ID

COMM

Process name

SYSCALL

Syscall name, or "[unknown: nnn]" for syscalls that aren't recognized

COUNT

The number of events

TIME

The total elapsed time (in us or ms)

## OVERHEAD
For most applications, the overhead should be manageable if they perform 1000's or even 10,000's of syscalls per second. For higher rates, the overhead may become considerable. For example, tracing a loop of 4 million calls to geteuid(), slows it down by 1.85x when tracing only syscall counts, and slows it down by more than 5x when tracing syscall counts and latencies. However, this represents a rate of >3.5 million syscalls per second, which should not be typical.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein

## SEE ALSO
funccount(8), ucalls(8), argdist(8), trace(8), funclatency(8)


bcc-tclcalls

title: bcc-tclcalls(8)

summary: Summarize method calls

text:
# bcc-tclcalls(8)

## NAME
ucalls, javacalls, perlcalls, phpcalls, pythoncalls, rubycalls, tclcalls - Summarize method calls from high-level languages and Linux syscalls.

## SYNOPSIS
javacalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

perlcalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

phpcalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

pythoncalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

rubycalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

tclcalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

ucalls [-l {java,perl,php,python,ruby}] [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

## DESCRIPTION
This tool summarizes method calls from high-level languages such as Java, Perl, PHP, Python, Ruby, and Tcl. It can also trace Linux system calls. Whenever a method is invoked, ucalls records the call count and optionally the method's execution time (latency) and displays a summary.

This uses in-kernel eBPF maps to store per process summaries for efficiency.

This tool relies on USDT probes embedded in many high-level languages, such as Java, Perl, PHP, Python, Ruby, and Tcl. It requires a runtime instrumented with these probes, which in some cases requires building from source with a USDT-specific flag, such as "--enable-dtrace" or "--with-dtrace". For Java, method probes are not enabled by default, and can be turned on by running the Java process with the "-XX:+ExtendedDTraceProbes" flag. For PHP processes, the environment variable USE_ZEND_DTRACE must be set to 1.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-l {java,perl,php,python,ruby,tcl}

The language to trace. If not provided, only syscalls are traced (when the -S option is used).

-T TOP

Print only the top methods by frequency or latency.

-L

Collect method invocation latency (duration).

-S

Collect Linux syscalls frequency and timing.

-v

Print the resulting BPF program, for debugging purposes.

-m

Print times in milliseconds (the default is microseconds).

pid

The process id to trace.

interval

Print summary after this number of seconds and then exit. By default, wait for Ctrl+C to terminate.

## EXAMPLES
Trace the top 10 Ruby method calls:

\# ucalls -T 10 -l ruby 1344

Trace Python method calls and Linux syscalls including latency in milliseconds:

\# ucalls -l python -mL 2020

Trace only syscalls and print a summary after 10 seconds:

\# ucalls -S 788 10

## OVERHEAD
Tracing individual method calls will produce a considerable overhead in all high-level languages. For languages with just-in-time compilation, such as Java, the overhead can be more considerable than for interpreted languages. On the other hand, syscall tracing will typically be tolerable for most processes, unless they have a very unusual rate of system calls.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _example.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein

## SEE ALSO
ustat(8), argdist(8)


bcc-tclflow

title: bcc-tclflow(8)

summary: Print a flow graph of method

text:
# bcc-tclflow(8)

## NAME
uflow, javaflow, perlflow, phpflow, pythonflow, rubyflow, tclflow - Print a flow graph of method calls in high-level languages.

## SYNOPSIS
javaflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

perlflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

phpflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

pythonflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

rubyflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

tclflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

uflow [-h] [-M METHOD] [-C CLAZZ] [-v] [-l {java,perl,php,python,ruby,tcl}] pid

## DESCRIPTION
uflow traces method calls and prints them in a flow graph that can facilitate debugging and diagnostics by following the program's execution (method flow).

This tool relies on USDT probes embedded in many high-level languages, such as Java, Perl, PHP, Python, Ruby, and Tcl. It requires a runtime instrumented with these probes, which in some cases requires building from source with a USDT-specific flag, such as "--enable-dtrace" or "--with-dtrace". For Java processes, the startup flag "-XX:+ExtendedDTraceProbes" is required. For PHP processes, the environment variable USE_ZEND_DTRACE must be set to 1.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-M METHOD

Print only method calls where the method name begins with this string.

-C CLAZZ

Print only method calls where the class name begins with this string. The class name interpretation strongly depends on the language. For example, in Java use "package/subpackage/ClassName" to refer to classes.

-v

Print the resulting BPF program, for debugging purposes.

{java,perl,php,python,ruby,tcl}

The language to trace.

pid

The process id to trace.

## EXAMPLES
Follow method flow in a Ruby process:

\# uflow ruby 148

Follow method flow in a Java process where the class name is java.lang.Thread:

\# uflow -C java/lang/Thread java 1802

## FIELDS
CPU

The CPU number on which the method was invoked. This is useful to easily see where the output skips to a different CPU.

PID

The process id.

TID

The thread id.

TIME

The duration of the method call.

METHOD

The method name.

## OVERHEAD
This tool has extremely high overhead because it prints every method call. For some scenarios, you might see lost samples in the output as the tool is unable to keep up with the rate of data coming from the kernel. Filtering by class or method prefix can help reduce the amount of data printed, but there is still a very high overhead in the collection mechanism. Do not use for performance- sensitive production scenarios, and always test first.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _example.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein

## SEE ALSO
trace(8), ustat(8)


bcc-tclobjnew

title: bcc-tclobjnew(8)

summary: Summarize object allocations in

text:
# bcc-tclobjnew(8)

## NAME
uobjnew, cobjnew, javaobjnew, rubyobjnew, tclobjnew - Summarize object allocations in high-level languages.

## SYNOPSIS
cobjnew [-h] [-C TOP_COUNT] [-S TOP_SIZE] [-v] pid [interval]

javaobjnew [-h] [-C TOP_COUNT] [-S TOP_SIZE] [-v] pid [interval]

rubyobjnew [-h] [-C TOP_COUNT] [-S TOP_SIZE] [-v] pid [interval]

tclobjnew [-h] [-C TOP_COUNT] [-S TOP_SIZE] [-v] pid [interval]

uobjnew [-h] [-C TOP_COUNT] [-S TOP_SIZE] [-v] [-l {c,java,ruby,tcl}] pid [interval]

## DESCRIPTION
uobjnew traces object allocations in high-level languages (including "malloc") and prints summaries of the most frequently allocated types by number of objects or number of bytes.

This tool relies on USDT probes embedded in many high-level languages, such as C, Java, Ruby, and Tcl. It requires a runtime instrumented with these probes, which in some cases requires building from source with a USDT-specific flag, such as "--enable-dtrace" or "--with-dtrace". For Java, the Java process must be started with the "-XX:+ExtendedDTraceProbes" flag.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-C TOP_COUNT

Print the top object types sorted by number of instances.

-S TOP_SIZE

Print the top object types sorted by size.

-v

Print the resulting BPF program, for debugging purposes.

{c,java,ruby,tcl}

The language to trace.

pid

The process id to trace.

interval

Wait this many seconds and then print the summary and exit. By default, wait for Ctrl+C to exit.

## EXAMPLES
Trace object allocations in a Ruby process:

\# uobjnew ruby 148

Trace object allocations from "malloc" and print the top 10 by total size:

\# uobjnew -S 10 c 1788

## FIELDS
TYPE

The object type being allocated. For C (malloc), this is the block size.

ALLOCS

The number of objects allocated.

BYTES

The number of bytes allocated.

## OVERHEAD
Object allocation events are quite frequent, and therefore the overhead from running this tool can be considerable. Use with caution and make sure to test before using in a production environment. Nonetheless, even thousands of allocations per second will likely produce a reasonable overhead when investigating a problem.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _example.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein

## SEE ALSO
ustat(8), ugc(8), memleak(8)


bcc-tclstat

title: bcc-tclstat(8)

summary: Activity stats from

text:
# bcc-tclstat(8)

## NAME
ustat, javastat, nodestat, perlstat, phpstat, pythonstat, rubystat, tclstat - Activity stats from high-level languages.

## SYNOPSIS
javastat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

nodestat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

perlstat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

phpstat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

pythonstat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

rubystat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

tclstat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

ustat [-l {java,node,perl,php,python,ruby,tcl}] [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

## DESCRIPTION
This is "top" for high-level language events, such as garbage collections, exceptions, thread creations, object allocations, method calls, and more. The events are aggregated for each process and printed in a top-like table, which can be sorted by various fields. Not all language runtimes provide the same set of details.

This uses in-kernel eBPF maps to store per process summaries for efficiency.

This tool relies on USDT probes embedded in many high-level languages, such as Java, Node, Perl, PHP, Python, Ruby, and Tcl. It requires a runtime instrumented with these probes, which in some cases requires building from source with a USDT-specific flag, such as "--enable-dtrace" or "--with-dtrace". For Java, some probes are not enabled by default, and can be turned on by running the Java process with the "-XX:+ExtendedDTraceProbes" flag. For PHP processes, the environment variable USE_ZEND_DTRACE must be set to 1.

Newly-created processes will only be traced at the next interval. If you run this tool with a short interval (say, 1-5 seconds), this should be virtually unnoticeable. For longer intervals, you might miss processes that were started and terminated during the interval window.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-l {java,node,perl,php,python,ruby,tcl}

The language to trace. By default, all languages are traced.

-C

Do not clear the screen between updates.

-S {cload,excp,gc,method,objnew,thread}

Sort the output by the specified field.

-r MAXROWS

Do not print more than this number of rows.

-d

Print the resulting BPF program, for debugging purposes.

interval

Interval between updates, seconds.

count

Number of interval summaries.

## EXAMPLES
Summarize activity in high-level languages, 1 second refresh:

\# ustat

Don't clear the screen, and top 8 rows only:

\# ustat -Cr 8

5 second summaries, 10 times only:

\# ustat 5 10

## FIELDS
loadavg

The contents of /proc/loadavg

PID

Process ID.

CMDLINE

Process command line (often the second and following arguments will give you a hint as to which application is being run.

METHOD/s

Count of method invocations during interval.

GC/s

Count of garbage collections during interval.

OBJNEW/s

Count of objects allocated during interval.

CLOAD/s

Count of classes loaded during interval.

EXC/s

Count of exceptions thrown during interval.

THR/s

Count of threads created during interval.

## OVERHEAD
When using this tool with high-frequency events, such as method calls, a very significant slow-down can be expected. However, many of the high-level languages covered by this tool already have a fairly high per-method invocation cost, especially when running in interpreted mode. For the lower-frequency events, such as garbage collections or thread creations, the overhead should not be significant. Specifically, when probing Java processes and not using the "-XX:+ExtendedDTraceProbes" flag, the most expensive probes are not emitted, and the overhead should be acceptable.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _example.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein

## SEE ALSO
trace(8), argdist(8), tplist(8)


bcc-tcpaccept

title: bcc-tcpaccept(8)

text:
# bcc-tcpaccept(8)

## NAME
tcpaccept - Trace TCP passive connections (accept()). Uses Linux eBPF/bcc.

## SYNOPSIS
tcpaccept [-h] [-T] [-t] [-p PID] [-P PORTS] [--cgroupmap MAPPATH] [--mntnsmap MAPPATH]

## DESCRIPTION
This tool traces passive TCP connections (eg, via an accept() syscall; connect() are active connections). This can be useful for general troubleshooting to see what new connections the local server is accepting.

This uses dynamic tracing of the kernel inet_csk_accept() socket function (from tcp_prot.accept), and will need to be modified to match kernel changes.

This tool only traces successful TCP accept()s. Connection attempts to closed ports will not be shown (those can be traced via other functions).

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print usage message.

-T

Include a time column on output (HH:MM:SS).

-t

Include a timestamp column.

-p PID

Trace this process ID only (filtered in-kernel).

-P PORTS

Comma-separated list of local ports to trace (filtered in-kernel).

--cgroupmap MAPPATH

Trace cgroups in this BPF map only (filtered in-kernel).

--mntnsmap MAPPATH

Trace mount namespaces in this BPF map only (filtered in-kernel).

## EXAMPLES
Trace all passive TCP connections (accept()s):

\# tcpaccept

Trace all TCP accepts, and include timestamps:

\# tcpaccept -t

Trace connections to local ports 80 and 81 only:

\# tcpaccept -P 80,81

Trace PID 181 only:

\# tcpaccept -p 181

Trace a set of cgroups only (see special_filtering.md from bcc sources for more details):

\# tcpaccept --cgroupmap /sys/fs/bpf/test01

## FIELDS
TIME

Time of the event, in HH:MM:SS format.

TIME(s)

Time of the event, in seconds.

PID

Process ID

COMM

Process name

IP

IP address family (4 or 6)

RADDR

Remote IP address.

RPORT

Remote port

LADDR

Local IP address.

LPORT

Local port

## OVERHEAD
This traces the kernel inet_csk_accept function and prints output for each event. The rate of this depends on your server application. If it is a web or proxy server accepting many tens of thousands of connections per second, then the overhead of this tool may be measurable (although, still a lot better than tracing every packet). If it is less than a thousand a second, then the overhead is expected to be negligible. Test and understand this overhead before use.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
tcptracer(8), tcpconnect(8), funccount(8), tcpdump(8)


bcc-tcpconnect

title: bcc-tcpconnect(8)

text:
# bcc-tcpconnect(8)

## NAME
tcpconnect - Trace TCP active connections (connect()). Uses Linux eBPF/bcc.

## SYNOPSIS
tcpconnect [-h] [-c] [-t] [-p PID] [-P PORT] [-L] [-u UID] [-U] [--cgroupmap MAPPATH] [--mntnsmap MAPPATH] [-d]

## DESCRIPTION
This tool traces active TCP connections (eg, via a connect() syscall; accept() are passive connections). This can be useful for general troubleshooting to see what connections are initiated by the local server.

All connection attempts are traced, even if they ultimately fail.

This works by tracing the kernel tcp_v4_connect() and tcp_v6_connect() functions using dynamic tracing, and will need updating to match any changes to these functions.

When provided with the -d or --dns option, this tool will also correlate connect calls with the most recent DNS query that matches the IP connected. This feature works by tracing the kernel udp_recvmsg() function to collect DNS responses.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

If using the -d or --dns option, you must have the dnslib and cachetools python packages installed. You can install them with pip3 or with apt on Ubuntu 18.04+ using the python3-dnslib and python3-cachetools packages.

## OPTIONS
-h

Print usage message.

-t

Include a timestamp column.

-c

Count connects per src ip and dest ip/port.

-p PID

Trace this process ID only (filtered in-kernel).

-P PORT

Comma-separated list of destination ports to trace (filtered in-kernel).

-L

Include a LPORT column.

-U

Include a UID column.

-u UID

Trace this UID only (filtered in-kernel).

--cgroupmap MAPPATH

Trace cgroups in this BPF map only (filtered in-kernel).

--mntnsmap MAPPATH

Trace mount namespaces in this BPF map only (filtered in-kernel).

-d

Shows the most recent DNS query for the IP address in the connect call. This is likely related to the TCP connection details in the other columns, but is not guaranteed. This feature works by tracing the udp_recvmsg kernel function and tracking DNS responses received by the server. It only supports UDP DNS packets up to 512 bytes in length. The python code keeps a cache of 10k DNS responses in memory for up 24 hours. If the time difference in milliseconds between when the system received a DNS response and when a connect syscall was traced using an IP in that DNS response is greater than 100ms, this tool will report this delta after the query. These deltas should be relatively short for most applications. A long delay between the response and connect could be either anomalous activity or indicate a misattribution between the DNS name requested and the IP that the connect syscall is using.

The -d option may not be used with the count feature (option -c)

## EXAMPLES
Trace all active TCP connections:

\# tcpconnect

Trace all TCP connects, and include timestamps:

\# tcpconnect -t

Trace all TCP connects, and include most recent matching DNS query for each connected IP

\# tcpconnect -d

Trace PID 181 only:

\# tcpconnect -p 181

Trace ports 80 and 81 only:

\# tcpconnect -P 80,81

Trace all TCP connects, and include LPORT:

\# tcpconnect -L

Trace all TCP connects, and include UID:

\# tcpconnect -U

Trace UID 1000 only:

\# tcpconnect -u 1000

Count connects per src ip and dest ip/port:

\# tcpconnect -c

Trace a set of cgroups only (see special_filtering.md from bcc sources for more details):

\# tcpconnect --cgroupmap /sys/fs/bpf/test01

Trace a set of mount namespaces only (see special_filtering.md from bcc sources for more details):

\# tcpconnect --mntnsmap /sys/fs/bpf/mnt_ns_set

## FIELDS
TIME(s)

Time of the call, in seconds.

UID

User ID

PID

Process ID

COMM

Process name

IP

IP address family (4 or 6)

SADDR

Source IP address.

LPORT

Source port

DADDR

Destination IP address.

DPORT

Destination port

CONNECTS

Accumulated active connections since start.

QUERY

Shows the most recent DNS query for the IP address in the connect call. This is likely related to the TCP connection details in the other columns, but is not guaranteed.

## OVERHEAD
This traces the kernel tcp_v[46]_connect functions and prints output for each event. As the rate of this is generally expected to be low (< 1000/s), the overhead is also expected to be negligible. If you have an application that is calling a high rate of connect()s, such as a proxy server, then test and understand this overhead before use.

If you are using the -d option to track DNS requests, this tool will trace the udp_recvmsg function and generate an event for any packets from UDP port 53. This event contains up to 512 bytes of the UDP packet payload. Typical applications do not extensively use UDP, so the performance overhead of tracing udp_recvmsg is expected to be negligible, However, if you have an application that receives many UDP packets, then you should test and understand the overhead of tracing every received UDP message. Furthermore, performance overhead of running this tool on a DNS server is expected to be higher than average because all DNS response packets will be copied to userspace.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
tcptracer(8), tcpaccept(8), funccount(8), tcpdump(8)


bcc-tcpconnlat

title: bcc-tcpconnlat(8)

text:
# bcc-tcpconnlat(8)

## NAME
tcpconnlat - Trace TCP active connection latency. Uses Linux eBPF/bcc.

## SYNOPSIS
tcpconnlat [-h] [-t] [-p PID] [-L] [-v] [min_ms]

## DESCRIPTION
This tool traces active TCP connections (eg, via a connect() syscall), and shows the latency (time) for the connection as measured locally: the time from SYN sent to the response packet. This is a useful performance metric that typically spans kernel TCP/IP processing and the network round trip time (not application runtime).

All connection attempts are traced, even if they ultimately fail (RST packet in response).

This tool works by use of kernel dynamic tracing of TCP/IP functions, and will need updating to match any changes to these functions. This tool should be updated in the future to use static tracepoints, once they are available.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print usage message.

-t

Include a timestamp column.

-p PID

Trace this process ID only (filtered in-kernel).

-L

Include a LPORT column.

-v

Print the resulting BPF program, for debugging purposes.

min_ms

Minimum duration to trace, in milliseconds.

## EXAMPLES
Trace all active TCP connections, and show connection latency (SYN->response round trip):

\# tcpconnlat

Include timestamps:

\# tcpconnlat -t

Trace PID 181 only:

\# tcpconnlat -p 181

Trace connects, and include LPORT:

\# tcpconnlat -L

Trace connects with latency longer than 10 ms:

\# tcpconnlat 10

Print the BPF program:

\# tcpconnlat -v

## FIELDS
TIME(s)

Time of the response packet, in seconds.

PID

Process ID that initiated the connection.

COMM

Process name that initiated the connection.

IP

IP address family (4 or 6).

SADDR

Source IP address.

DADDR

Destination IP address.

LPORT

Source port

DPORT

Destination port

LAT(ms)

The time from when a TCP connect was issued (measured in-kernel) to when a response packet was received for this connection (can be SYN,ACK, or RST, etc). This time spans kernel to kernel latency, involving kernel TCP/IP processing and the network round trip in between. This typically does not include time spent by the application processing the new connection.

## OVERHEAD
This traces the kernel tcp_v[46]_connect functions and prints output for each event. As the rate of this is generally expected to be low (< 1000/s), the overhead is also expected to be negligible. If you have an application that is calling a high rate of connects()s, such as a proxy server, then test and understand this overhead before use.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
tcpconnect(8), tcpaccept(8), funccount(8), tcpdump(8)


bcc-tcpdrop

title: bcc-tcpdrop(8)

text:
# bcc-tcpdrop(8)

## NAME
tcpdrop - Trace kernel-based TCP packet drops with details. Uses Linux eBPF/bcc.

## SYNOPSIS
tcpdrop [-h]

## DESCRIPTION
This tool traces TCP packets or segments that were dropped by the kernel, and shows details from the IP and TCP headers, the socket state, and the kernel stack trace. This is useful for debugging cases of high kernel drops, which can cause timer-based retransmits and performance issues.

This tool works using dynamic tracing of the tcp_drop() kernel function, which requires a recent kernel version.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print usage message. tcpdrop

## FIELDS
TIME

Time of the drop, in HH:MM:SS format.

PID

Process ID that was on-CPU during the drop. This may be unrelated, as drops can occur on the receive interrupt and be unrelated to the PID that was interrupted.

IP

IP address family (4 or 6)

SADDR

Source IP address.

SPORT

Source TCP port.

DADDR

Destination IP address.

DPORT

Destionation TCP port.

STATE

TCP session state ("ESTABLISHED", etc).

FLAGS

TCP flags ("SYN", etc).

## OVERHEAD
This traces the kernel tcp_drop() function, which should be low frequency, and therefore the overhead of this tool should be negligible.

As always, test and understand this tools overhead for your types of workloads before production use.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
tcplife(8), tcpaccept(8), tcpconnect(8), tcptop(8)


bcc-tcplife

title: bcc-tcplife(8)

text:
# bcc-tcplife(8)

## NAME
tcplife - Trace TCP sessions and summarize lifespan. Uses Linux eBPF/bcc.

## SYNOPSIS
tcplife [-h] [-T] [-t] [-w] [-s] [-p PID] [-D PORTS] [-L PORTS]

## DESCRIPTION
This tool traces TCP sessions that open and close while tracing, and prints a line of output to summarize each one. This includes the IP addresses, ports, duration, and throughput for the session. This is useful for workload characterisation and flow accounting: identifying what connections are happening, with the bytes transferred.

This tool works using the sock:inet_sock_set_state tracepoint if it exists, added to Linux 4.16, and switches to using kernel dynamic tracing for older kernels. Only TCP state changes are traced, so it is expected that the overhead of this tool is much lower than typical send/receive tracing.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print usage message.

-s

Comma separated values output (parseable).

-t

Include a timestamp column (seconds).

-T

Include a time column (HH:MM:SS).

-w

Wide column output (fits IPv6 addresses).

-p PID

Trace this process ID only (filtered in-kernel).

-L PORTS

Comma-separated list of local ports to trace (filtered in-kernel).

-D PORTS

Comma-separated list of destination ports to trace (filtered in-kernel).

## EXAMPLES
Trace all TCP sessions, and summarize lifespan and throughput:

\# tcplife

Include a timestamp column, and wide column output:

\# tcplife -tw

Trace PID 181 only:

\# tcplife -p 181

Trace connections to local ports 80 and 81 only:

\# tcplife -L 80,81

Trace connections to remote port 80 only:

\# tcplife -D 80

## FIELDS
TIME

Time of the call, in HH:MM:SS format.

TIME(s)

Time of the call, in seconds.

PID

Process ID

COMM

Process name

IP

IP address family (4 or 6)

LADDR

Local IP address.

RADDR

Remote IP address.

LPORT

Local port.

RPORT

Remote port.

TX_KB

Total transmitted Kbytes.

RX_KB

Total received Kbytes.

MS

Lifespan of the session, in milliseconds.

## OVERHEAD
This traces the kernel TCP set state function, which should be called much less often than send/receive tracing, and therefore have lower overhead. The overhead of the tool is relative to the rate of new TCP sessions: if this is high, over 10,000 per second, then there may be noticeable overhead just to print out 10k lines of formatted output per second.

You can find out the rate of new TCP sessions using "sar -n TCP 1", and adding the active/s and passive/s columns.

As always, test and understand this tools overhead for your types of workloads before production use.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
tcpaccept(8), tcpconnect(8), tcptop(8)


bcc-tcpretrans

title: bcc-tcpretrans(8)

text:
# bcc-tcpretrans(8)

## NAME
tcpretrans - Trace or count TCP retransmits and TLPs. Uses Linux eBPF/bcc.

## SYNOPSIS
tcpretrans [-h] [-l] [-c]

## DESCRIPTION
This traces TCP retransmits, showing address, port, and TCP state information, and sometimes the PID (although usually not, since retransmits are usually sent by the kernel on timeouts). To keep overhead very low, only the TCP retransmit functions are traced. This does not trace every packet (like tcpdump(8) or a packet sniffer). Optionally, it can count retransmits over a user signalled interval to spot potentially dropping network paths the flows are traversing.

This uses dynamic tracing of the kernel tcp_retransmit_skb() and tcp_send_loss_probe() functions, and will need to be updated to match kernel changes to these functions.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print usage message.

-l

Include tail loss probe attempts (in some cases the kernel may not complete the TLP send).

-c

Count occurring retransmits per flow.

## EXAMPLES
Trace TCP retransmits:

\# tcpretrans

Trace TCP retransmits and TLP attempts:

\# tcpretrans -l

## FIELDS
TIME

Time of the retransmit.

PID

Process ID that was on-CPU. This is less useful than it might sound, as it may usually be 0, for the kernel, for timer-based retransmits.

IP

IP address family (4 or 6).

LADDR

Local IP address.

LPORT

Local port.

T>

Type of event: R> == retransmit, L> == tail loss probe.

RADDR

Remote IP address.

RPORT

Remote port.

STATE

TCP session state.

RETRANSMITS

Accumulated occurred retransmits since start.

## OVERHEAD
Should be negligible: TCP retransmit events should be low (<1000/s), and the low overhead this tool adds to each event should make the cost negligible.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
tcpconnect(8), tcpaccept(8)


bcc-tcprtt

title: bcc-tcprtt(8)

text:
# bcc-tcprtt(8)

## NAME
tcprtt - Trace TCP RTT of established connections. Uses Linux eBPF/bcc.

## SYNOPSIS
tcprtt [-h] [-T] [-D] [-m] [-i INTERVAL] [-d DURATION] [-b] [-B]

## DESCRIPTION
This tool traces established connections RTT(round-trip time) to analyze the quality of network. This can be useful for general troubleshooting to distinguish the network latency is from user process or physical network.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print usage message.

-T

Include a time column on output (HH:MM:SS).

-D

Show debug infomation of bpf text.

-m

Output histogram in milliseconds.

-i INTERVAL

Print output every interval seconds.

-d DURATION

Total duration of trace in seconds.

-p LPORT

Filter for local port.

-P RPORT

Filter for remote port.

-a LADDR

Filter for local address.

-A RADDR

Filter for remote address.

-b

Show sockets histogram by local address.

-B

Show sockets histogram by remote address.

## EXAMPLES
Trace TCP RTT and print 1 second summaries, 10 times:

\# tcprtt -i 1 -d 10

Summarize in millisecond, and timestamps:

\# tcprtt -m -T

Only trace TCP RTT for remote address 192.168.1.100 and remote port 80:

\# tcprtt -i 1 -d 10 -A 192.168.1.100 -P 80

Trace local port and show a breakdown of remote hosts RTT:

\# tcprtt -i 3 --lport 80 --byraddr

## OVERHEAD
This traces the kernel tcp_rcv_established function and collects TCP RTT. The rate of this depends on your server application. If it is a web or proxy server accepting many tens of thousands of connections per second.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
zhenwei pi

## SEE ALSO
tcptracer(8), tcpconnect(8), funccount(8), tcpdump(8)


bcc-tcpstates

title: bcc-tcpstates(8)

text:
# bcc-tcpstates(8)

## NAME
tcpstates - Trace TCP session state changes with durations. Uses Linux eBPF/bcc.

## SYNOPSIS
tcpstates [-h] [-T] [-t] [-w] [-s] [-D PORTS] [-L PORTS] [-Y]

## DESCRIPTION
This tool traces TCP session state changes while tracing, and prints details including the duration in each state. This can help explain the latency of TCP connections: whether the time is spent in the ESTABLISHED state (data transfer), or initialization state (SYN_SENT), etc.

This tool works using the sock:inet_sock_set_state tracepoint, which was added to Linux 4.16. Linux 4.16 also included extra state transitions so that all TCP transitions could be observed by this tracepoint.

Only TCP state changes are traced, so it is expected that the overhead of this tool is much lower than typical send/receive tracing.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc, and the sock:inet_sock_set_state tracepoint.

## OPTIONS
-h

Print usage message.

-s

Comma separated values output (parseable).

-t

Include a timestamp column (seconds).

-T

Include a time column (HH:MM:SS).

-w

Wide column output (fits IPv6 addresses).

-L PORTS

Comma-separated list of local ports to trace (filtered in-kernel).

-D PORTS

Comma-separated list of destination ports to trace (filtered in-kernel).

-Y

Log session state changes to the systemd journal.

## EXAMPLES
Trace all TCP sessions, and show all state changes:

\# tcpstates

Include a timestamp column, and wide column output:

\# tcpstates -tw

Trace connections to local ports 80 and 81 only:

\# tcpstates -L 80,81

Trace connections to remote port 80 only:

\# tcpstates -D 80

## FIELDS
TIME

Time of the change, in HH:MM:SS format.

TIME(s)

Time of the change, in seconds.

C-PID

The current on-CPU process ID. This may show the process that owns the TCP session if the state change executes in synchronous process context, else it is likely to show the kernel (asynchronous state change).

C-COMM

The current on-CPU process name. This may show the process that owns the TCP session if the state change executes in synchronous process context, else it is likely to show the kernel (asynchronous state change).

IP

IP address family (4 or 6)

LADDR

Local IP address.

RADDR

Remote IP address.

LPORT

Local port.

RPORT

Remote port.

OLDSTATE

Previous TCP state.

NEWSTATE

New TCP state.

MS

Duration of this state.

## OVERHEAD
This traces the kernel TCP set state function, which should be called much less often than send/receive tracing, and therefore have lower overhead. The overhead of the tool is relative to the rate of new TCP sessions: if this is high, over 10,000 per second, then there may be noticeable overhead just to print out 10k lines of formatted output per second.

You can find out the rate of new TCP sessions using "sar -n TCP 1", and adding the active/s and passive/s columns.

As always, test and understand this tools overhead for your types of workloads before production use.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
tcpaccept(8), tcpconnect(8), tcptop(8), tcplife(8)


bcc-tcpsubnet

title: bcc-tcpsubnet(8)

text:
# bcc-tcpsubnet(8)

## NAME
tcpsubnet - Summarize and aggregate IPv4 TCP traffic by subnet.

## SYNOPSIS
tcpsubnet [-h] [-v] [--ebpf] [-J] [-f FORMAT] [-i INTERVAL] [subnets]

## DESCRIPTION
This tool summarizes and aggregates IPv4 TCP sent to the subnets passed in argument and prints to stdout on a fixed interval.

This uses dynamic tracing of kernel TCP send/receive functions, and will need to be updated to match kernel changes.

The traced data is summarized in-kernel using a BPF map to reduce overhead. At very high TCP event rates, the overhead may still be measurable. See the OVERHEAD section for more details.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print USAGE message.

-v

Run in verbose mode. Will output subnet evaluation and the BPF program

-J

Format output in JSON.

-i

Interval between updates, seconds (default 1).

-f

Format output units. Supported values are bkmBKM. When using kmKM the output will be rounded to floor.

--ebpf

Prints the BPF program.

subnets

Comma separated list of subnets. Traffic will be categorized in theses subnets. Order matters. (default 127.0.0.1/32,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,0.0.0.0/0)

## EXAMPLES
Summarize TCP traffic by the default subnets:

\# tcpsubnet

Summarize all TCP traffic:

\# tcpsubnet 0.0.0.0/0

Summarize all TCP traffic and output in JSON and Kb:

\# tcpsubnet -J -fk 0.0.0.0/0

## FIELDS
(Standad output) Left hand side column:

Subnet

(Standard output) Right hand side column:

Aggregate traffic in units passed as argument

(JSON output) date

Current date formatted in the system locale

(JSON output) time

Current time formatted in the system locale

(JSON output) entries

Map of subnets to aggregates. Values will be in format passed to -f

## OVERHEAD
This traces all tcp_sendmsg function calls in the TCP/IP stack. It summarizes data in-kernel to reduce overhead. A simple iperf test (v2.0.5) with the default values shows a loss of ~5% throughput. On 10 runs without tcpsubnet running the average throughput was 32.42Gb/s, with tcpsubnet enabled it was 31.26Gb/s. This is not meant to be used as a long running service. Use it for troubleshooting or for a controlled interval. As always, try it out in a test environment first.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Rodrigo Manyari

## INSPIRATION
tcptop(8) by Brendan Gregg

## SEE ALSO
netlink(7)


bcc-tcpsynbl

title: bcc-tcpsynbl(8)

text:
# bcc-tcpsynbl(8)

## NAME
tcpsynbl - Show the TCP SYN backlog as a histogram. Uses BCC/eBPF.

## SYNOPSIS
tcpsynbl

## DESCRIPTION
This tool shows the TCP SYN backlog size during SYN arrival as a histogram. This lets you see how close your applications are to hitting the backlog limit and dropping SYNs (causing performance issues with SYN retransmits), and is a measure of workload saturation. The histogram shown is measured at the time of SYN received, and a separate histogram is shown for each backlog limit.

This works by tracing the tcp_v4_syn_recv_sock() and tcp_v6_syn_recv_sock() kernel functions using dynamic instrumentation. Since these functions may change in future kernels, this tool may need maintenance to keep working.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and BCC.

## EXAMPLES
Show the TCP SYN backlog as a histogram.

\# tcpsynbl

## FIELDS
backlog

The backlog size when a SYN was received.

count

The number of times this backlog size was encountered.

distribution

An ASCII visualization of the count column.

## OVERHEAD
Inbound SYNs should be relatively low compared to packets and other events, so the overhead of this tool is expected to be negligible.

## SOURCE
This originated as a bpftrace tool from the book "BPF Performance Tools", published by Addison Wesley (2019):

http://www.brendangregg.com/bpf-performance-tools-book.html

See the book for more documentation on this tool.

This version is in the BCC repository:

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
tcptop(8)


bcc-tcptop

title: bcc-tcptop(8)

text:
# bcc-tcptop(8)

## NAME
tcptop - Summarize TCP send/recv throughput by host. Top for TCP.

## SYNOPSIS
tcptop [-h] [-C] [-S] [-p PID] [--cgroupmap MAPPATH]

[--mntnsmap MAPPATH] [interval] [count]

## DESCRIPTION
This is top for TCP sessions.

This summarizes TCP send/receive Kbytes by host, and prints a summary that refreshes, along other system-wide metrics.

This uses dynamic tracing of kernel TCP send/receive functions, and will need to be updated to match kernel changes.

The traced TCP functions are usually called at a lower rate than per-packet functions, and therefore have lower overhead. The traced data is summarized in-kernel using a BPF map to further reduce overhead. At very high TCP event rates, the overhead may still be measurable. See the OVERHEAD section for more details.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print USAGE message.

-C

Don't clear the screen.

-S

Don't print the system summary line (load averages).

-p PID

Trace this PID only.

--cgroupmap MAPPATH

Trace cgroups in this BPF map only (filtered in-kernel).

--mntnsmap MAPPATH

Trace mount namespaces in this BPF map only (filtered in-kernel).

interval

Interval between updates, seconds (default 1).

count

Number of interval summaries (default is many).

## EXAMPLES
Summarize TCP throughput by active sessions, 1 second refresh:

\# tcptop

Don't clear the screen (rolling output), and 5 second summaries:

\# tcptop -C 5

Trace PID 181 only, and don't clear the screen:

\# tcptop -Cp 181

Trace a set of cgroups only (see special_filtering.md from bcc sources for more details):

\# tcptop --cgroupmap /sys/fs/bpf/test01

## FIELDS
loadavg:

The contents of /proc/loadavg

PID

Process ID.

COMM

Process name.

LADDR

Local address (IPv4), and TCP port

RADDR

Remote address (IPv4), and TCP port

LADDR6

Source address (IPv6), and TCP port

RADDR6

Destination address (IPv6), and TCP port

RX_KB

Received Kbytes

TX_KB

Transmitted Kbytes

## OVERHEAD
This traces all send/receives in TCP, high in the TCP/IP stack (close to the application) which are usually called at a lower rate than per-packet functions, lowering overhead. It also summarizes data in-kernel to further reduce overhead. These techniques help, but there may still be measurable overhead at high send/receive rates, eg, ~13% of one CPU at 100k events/sec. use funccount to count the kprobes in the tool to find out this rate, as the overhead is relative to the rate. Some sample production servers tested found total TCP event rates of 4k to 15k per second, and the CPU overhead at these rates ranged from 0.5% to 2.0% of one CPU. If your send/receive rate is low (eg, <1000/sec) then the overhead is expected to be negligible; Test in a lab environment first.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## INSPIRATION
top(1) by William LeFebvre

## SEE ALSO
tcpconnect(8), tcpaccept(8)


bcc-tcptracer

title: bcc-tcptracer(8)

text:
# bcc-tcptracer(8)

## NAME
tcptracer - Trace TCP established connections. Uses Linux eBPF/bcc.

## SYNOPSIS
tcptracer [-h] [-v] [-p PID] [-N NETNS] [--cgroupmap MAPPATH] [--mntnsmap MAPPATH]

## DESCRIPTION
This tool traces established TCP connections that open and close while tracing, and prints a line of output per connect, accept and close events. This includes the type of event, PID, IP addresses and ports.

This tool works by using kernel dynamic tracing, and will need to be updated if the kernel implementation changes. Only established TCP connections are traced, so it is expected that the overhead of this tool is rather low.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print usage message.

-v

Print full lines, with long event type names and network namespace numbers.

-p PID

Trace this process ID only (filtered in-kernel).

-N NETNS

Trace this network namespace only (filtered in-kernel).

--cgroupmap MAPPATH

Trace cgroups in this BPF map only (filtered in-kernel).

--mntnsmap MAPPATH

Trace mount namespaces in the map (filtered in-kernel).

## EXAMPLES
Trace all TCP established connections:

\# tcptracer

Trace all TCP established connections with verbose lines:

\# tcptracer -v

Trace PID 181 only:

\# tcptracer -p 181

Trace connections in network namespace 4026531969 only:

\# tcptracer -N 4026531969

Trace a set of cgroups only (see special_filtering.md from bcc sources for more details):

\# tcptracer --cgroupmap /sys/fs/bpf/test01

## FIELDS
TYPE

Type of event. In non-verbose mode: C for connect, A for accept, X for close.

PID

Process ID

COMM

Process name

IP

IP address family (4 or 6)

SADDR

Source IP address.

DADDR

Destination IP address.

SPORT

Source port.

DPORT

Destination port.

NETNS

Network namespace where the event originated.

## OVERHEAD
This traces the kernel inet accept function, and the TCP connect, close, and set state functions. However, it only prints information for connections that are established, so it shouldn't have a huge overhead.

As always, test and understand this tools overhead for your types of workloads before production use.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Iago López Galeiras

## SEE ALSO
tcpaccept(8), tcpconnect(8), tcptop(8), tcplife(8)


bcc-threadsnoop

title: bcc-threadsnoop(8)

text:
# bcc-threadsnoop(8)

## NAME
threadsnoop - Trace thread creation via pthread_create(). Uses BCC/eBPF.

## SYNOPSIS
threadsnoop

## DESCRIPTION
threadsnoop traces calls to pthread_create(), showing this path of thread creation. This can be used for workload characterization and discovery, and is a companion to execsnoop(8) which traces execve(2).

This works by tracing the pthread_create() from libpthread.so.0. The path to this library may need adjusting in the tool source to match your system.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and BCC.

## EXAMPLES
Trace calls pthread_create():

\# threadsnoop

## FIELDS
TIME(ms)

Elapsed time since the tool began tracing (in milliseconds).

PID

The process ID.

COMM

The process (thread) name.

FUNC

The name of the start routine, if the symbol is available, else a hex address for the start routine address.

## OVERHEAD
Thread creation is expected to be low (<< 1000/s), so the overhead of this tool is expected to be negligible.

## SOURCE
This originated as a bpftrace tool from the book "BPF Performance Tools", published by Addison Wesley (2019):

http://www.brendangregg.com/bpf-performance-tools-book.html

See the book for more documentation on this tool.

This version is in the BCC repository:

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
execsnoop(8)


bcc-tplist

title: bcc-tplist(8)

text:
# bcc-tplist(8)

## NAME
tplist - Display kernel tracepoints or USDT probes and their formats.

## SYNOPSIS
tplist [-p PID] [-l LIB] [-v] [filter]

## DESCRIPTION
tplist lists all kernel tracepoints, and can optionally print out the tracepoint format; namely, the variables that you can trace when the tracepoint is hit. tplist can also list USDT probes embedded in a specific library or executable, and can list USDT probes for all the libraries loaded by a specific process. These features are usually used in conjunction with the argdist and/or trace tools.

On a typical system, accessing the tracepoint list and format requires root. However, accessing USDT probes does not require root.

## OPTIONS
-p PID

Display the USDT probes from all the libraries loaded by the specified process.

-l LIB

Display the USDT probes from the specified library or executable. If the librar or executable can be found in the standard paths, a full path is not required.

-v

Increase the verbosity level. Can be used to display the variables, locations, and arguments of tracepoints and USDT probes.

[filter]

A wildcard expression that specifies which tracepoints or probes to print. For example, block:* will print all block tracepoints (block:block_rq_complete, etc.). Regular expressions are not supported.

## EXAMPLES
Print all kernel tracepoints:

\# tplist

Print all net tracepoints with their format:

\# tplist -v 'net:*'

Print all USDT probes in libpthread:

$ tplist -l pthread

Print all USDT probes in process 4717 from the libc provider:

$ tplist -p 4717 'libc:*'

Print all the USDT probes in the node executable:

$ tplist -l node

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein


bcc-trace

title: bcc-trace(8)

text:
# bcc-trace(8)

## NAME
trace - Trace a function and print its arguments or return value, optionally evaluating a filter. Uses Linux eBPF/bcc.

## SYNOPSIS
trace [-h] [-b BUFFER_PAGES] [-p PID] [-L TID] [-v] [-Z STRING_SIZE] [-S] [-s SYM_FILE_LIST]

[-M MAX_EVENTS] [-t] [-u] [-T] [-C] [-K] [-U] [-a] [-I header]

probe [probe ...]

## DESCRIPTION
trace probes functions you specify and displays trace messages if a particular condition is met. You can control the message format to display function arguments and return values.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print usage message.

-p PID

Trace only functions in the process PID.

-L TID

Trace only functions in the thread TID.

-v

Display the generated BPF program, for debugging purposes.

-z STRING_SIZE

When collecting string arguments (of type char*), collect up to STRING_SIZE characters. Longer strings will be truncated.

-s SYM_FILE_LIST

When collecting stack trace in build id format, use the coma separated list for symbol resolution.

-S

If set, trace messages from trace's own process. By default, this is off to avoid tracing storms -- for example, if you trace the write system call, and consider that trace is writing to the standard output.

-M MAX_EVENTS

Print up to MAX_EVENTS trace messages and then exit.

-t

Print times relative to the beginning of the trace (offsets), in seconds.

-u

Print UNIX timestamps instead of offsets from trace beginning, requires -t.

-T

Print the time column.

-C

Print CPU id.

-c CGROUP_PATH

Trace only functions in processes under CGROUP_PATH hierarchy.

-n NAME

Only print process names containing this name.

-f MSG_FILTER

Only print message of event containing this string.

-B

Treat argument of STRCMP helper as a binary value

-K

Print the kernel stack for each event.

-U

Print the user stack for each event. -a Print virtual address in kernel and user stacks.

-I header

Additional header files to include in the BPF program. This is needed if your filter or print expressions use types or data structures that are not available in the standard headers. For example: 'linux/mm.h'

probe [probe ...]

One or more probes that attach to functions, filter conditions, and print information. See PROBE SYNTAX below.

## PROBE SYNTAX
The general probe syntax is as follows:

[{p,r}]:[library]:function[+offset][(signature)] [(predicate)] ["format string"[, arguments]]

{t:category:event,u:library:probe} [(predicate)] ["format string"[, arguments]]

{[{p,r}],t,u}

Probe type - "p" for function entry, "r" for function return, "t" for kernel tracepoint, "u" for USDT probe. The default probe type is "p".

[library]

Library containing the probe. Specify the full path to the .so or executable file where the function to probe resides. Alternatively, you can specify just the lib name: for example, "c" refers to libc. If no library name is specified, the kernel is assumed. Also, you can specify an executable name (without a full path) if it is in the PATH. For example, "bash".

category

The tracepoint category. For example, "sched" or "irq".

function

The function to probe. offset The offset after the address of the function where the probe should injected. For example "kfree_skb+56" in decimal or hexadecimal "kfree_skb+0x38" format. Only works with kprobes and uprobes. Zero if omitted.

signature

The optional signature of the function to probe. This can make it easier to access the function's arguments, instead of using the "arg1", "arg2" etc. argument specifiers. For example, "(struct timespec *ts)" in the signature position lets you use "ts" in the filter or print expressions.

event

The tracepoint event. For example, "block_rq_complete".

probe

The USDT probe name. For example, "pthread_create".

[(predicate)]

The filter applied to the captured data. Only if the filter evaluates as true, the trace message will be printed. The filter can use any valid C expression that refers to the argument values: arg1, arg2, etc., or to the return value retval in a return probe. If necessary, use C cast operators to coerce the arguments to the desired type. For example, if arg1 is of type int, use the expression ((int)arg1 < 0) to trace only invocations where arg1 is negative. Note that only arg1-arg6 are supported, and only if the function is using the standard x86_64 convention where the first six arguments are in the RDI, RSI, RDX, RCX, R8, R9 registers. If no predicate is specified, all function invocations are traced. The predicate expression may also use the STRCMP pseudo-function to compare a predefined string to a string argument. For example: STRCMP("test", arg1). The order of arguments is important: the first argument MUST be a quoted literal string, and the second argument can be a runtime string, most typically an argument.

["format string"[, arguments]]

A printf-style format string that will be used for the trace message. You can use the following format specifiers: %s, %d, %u, %lld, %llu, %hd, %hu, %c, %x, %llx -- with the same semantics as printf's. Make sure to pass the exact number of arguments as there are placeholders in the format string. The format specifier replacements may be any C expressions, and may refer to the same special keywords as in the predicate (arg1, arg2, etc.). In addition to the above format specifiers, you can also use %K and %U when the expression is an address that potentially points to executable code (i.e., a symbol). trace will resolve %K specifiers to a kernel symbol, such as vfs__read, and will resolve %U specifiers to a user-space symbol in that process, such as sprintf.

In tracepoints, both the predicate and the arguments may refer to the tracepoint format structure, which is stored in the special "args" variable. For example, the block:block_rq_complete tracepoint can print or filter by args->nr_sector. To discover the format of your tracepoint, use the tplist tool.

In USDT probes, the arg1, ..., argN variables refer to the probe's arguments. To determine which arguments your probe has, use the tplist tool.

The predicate expression and the format specifier replacements for printing may also use the following special keywords: $pid, $tgid to refer to the current process' pid and tgid; $uid, $gid to refer to the current user's uid and gid; $cpu to refer to the current processor number.

## EXAMPLES
Trace all invocations of the open system call with the name of the file being opened:

\# trace '::do_sys_open "%s", arg2'

Trace all invocations of the read system call where the number of bytes requested is greater than 20,000:

\# trace '::sys_read (arg3 > 20000) "read %d bytes", arg3'

Trace all malloc calls and print the size of the requested allocation:

\# trace ':c:malloc "size = %d", arg1'

Trace returns from the readline function in bash and print the return value as a string:

\# trace 'r:bash:readline "%s", retval'

Trace the block:block_rq_complete tracepoint and print the number of sectors completed:

\# trace 't:block:block_rq_complete "%d sectors", args->nr_sector'

Trace the pthread_create USDT probe from the pthread library and print the address of the thread's start function:

\# trace 'u:pthread:pthread_create "start addr = %llx", arg3'

Trace the nanosleep system call and print the sleep duration in nanoseconds:

\# trace 'p::SyS_nanosleep(struct timespec *ts) sleep for %lld ns , ts->tv_nsec'

Trace the inet_pton system call using build id mechanism and print the stack

\# trace -s /lib/x86_64-linux-gnu/libc.so.6,/bin/ping 'p:c:inet_pton' -U

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein


bcc-ttysnoop

title: bcc-ttysnoop(8)

text:
# bcc-ttysnoop(8)

## NAME
ttysnoop - Watch output from a tty or pts device. Uses Linux eBPF/bcc.

## SYNOPSIS
ttysnoop [-h] [-C] device

## DESCRIPTION
ttysnoop watches a tty or pts device, and prints the same output that is appearing on that device. It can be used to mirror the output from a shell session, or the system console.

This works by use of kernel dynamic tracing of the tty_write() function. This tool will need updating in case that kernel function changes in a future kernel version.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-C

Don't clear the screen.

device

Either a path to a tty device (eg, /dev/tty0) or a pts number (eg, the "3" from /dev/pts/3).

## EXAMPLES
Snoop output from /dev/pts/2

\# ttysnoop /dev/pts/2

Snoop output from /dev/pts/2 (shortcut)

\# ttysnoop 2

Snoop output from the system console

\# ttysnoop /dev/console

Snoop output from /dev/tty0

\# ttysnoop /dev/tty0

## OVERHEAD
As the rate of tty_write() is expected to be very low (<100/s), the overhead of this tool is expected to be negligible.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
opensnoop(1)


bcc-ucalls

title: bcc-ucalls(8)

summary: Summarize method calls

text:
# bcc-ucalls(8)

## NAME
ucalls, javacalls, perlcalls, phpcalls, pythoncalls, rubycalls, tclcalls - Summarize method calls from high-level languages and Linux syscalls.

## SYNOPSIS
javacalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

perlcalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

phpcalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

pythoncalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

rubycalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

tclcalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

ucalls [-l {java,perl,php,python,ruby}] [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

## DESCRIPTION
This tool summarizes method calls from high-level languages such as Java, Perl, PHP, Python, Ruby, and Tcl. It can also trace Linux system calls. Whenever a method is invoked, ucalls records the call count and optionally the method's execution time (latency) and displays a summary.

This uses in-kernel eBPF maps to store per process summaries for efficiency.

This tool relies on USDT probes embedded in many high-level languages, such as Java, Perl, PHP, Python, Ruby, and Tcl. It requires a runtime instrumented with these probes, which in some cases requires building from source with a USDT-specific flag, such as "--enable-dtrace" or "--with-dtrace". For Java, method probes are not enabled by default, and can be turned on by running the Java process with the "-XX:+ExtendedDTraceProbes" flag. For PHP processes, the environment variable USE_ZEND_DTRACE must be set to 1.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-l {java,perl,php,python,ruby,tcl}

The language to trace. If not provided, only syscalls are traced (when the -S option is used).

-T TOP

Print only the top methods by frequency or latency.

-L

Collect method invocation latency (duration).

-S

Collect Linux syscalls frequency and timing.

-v

Print the resulting BPF program, for debugging purposes.

-m

Print times in milliseconds (the default is microseconds).

pid

The process id to trace.

interval

Print summary after this number of seconds and then exit. By default, wait for Ctrl+C to terminate.

## EXAMPLES
Trace the top 10 Ruby method calls:

\# ucalls -T 10 -l ruby 1344

Trace Python method calls and Linux syscalls including latency in milliseconds:

\# ucalls -l python -mL 2020

Trace only syscalls and print a summary after 10 seconds:

\# ucalls -S 788 10

## OVERHEAD
Tracing individual method calls will produce a considerable overhead in all high-level languages. For languages with just-in-time compilation, such as Java, the overhead can be more considerable than for interpreted languages. On the other hand, syscall tracing will typically be tolerable for most processes, unless they have a very unusual rate of system calls.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _example.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein

## SEE ALSO
ustat(8), argdist(8)


bcc-uflow

title: bcc-uflow(8)

summary: Print a flow graph of method

text:
# bcc-uflow(8)

## NAME
uflow, javaflow, perlflow, phpflow, pythonflow, rubyflow, tclflow - Print a flow graph of method calls in high-level languages.

## SYNOPSIS
javaflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

perlflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

phpflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

pythonflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

rubyflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

tclflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

uflow [-h] [-M METHOD] [-C CLAZZ] [-v] [-l {java,perl,php,python,ruby,tcl}] pid

## DESCRIPTION
uflow traces method calls and prints them in a flow graph that can facilitate debugging and diagnostics by following the program's execution (method flow).

This tool relies on USDT probes embedded in many high-level languages, such as Java, Perl, PHP, Python, Ruby, and Tcl. It requires a runtime instrumented with these probes, which in some cases requires building from source with a USDT-specific flag, such as "--enable-dtrace" or "--with-dtrace". For Java processes, the startup flag "-XX:+ExtendedDTraceProbes" is required. For PHP processes, the environment variable USE_ZEND_DTRACE must be set to 1.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-M METHOD

Print only method calls where the method name begins with this string.

-C CLAZZ

Print only method calls where the class name begins with this string. The class name interpretation strongly depends on the language. For example, in Java use "package/subpackage/ClassName" to refer to classes.

-v

Print the resulting BPF program, for debugging purposes.

{java,perl,php,python,ruby,tcl}

The language to trace.

pid

The process id to trace.

## EXAMPLES
Follow method flow in a Ruby process:

\# uflow ruby 148

Follow method flow in a Java process where the class name is java.lang.Thread:

\# uflow -C java/lang/Thread java 1802

## FIELDS
CPU

The CPU number on which the method was invoked. This is useful to easily see where the output skips to a different CPU.

PID

The process id.

TID

The thread id.

TIME

The duration of the method call.

METHOD

The method name.

## OVERHEAD
This tool has extremely high overhead because it prints every method call. For some scenarios, you might see lost samples in the output as the tool is unable to keep up with the rate of data coming from the kernel. Filtering by class or method prefix can help reduce the amount of data printed, but there is still a very high overhead in the collection mechanism. Do not use for performance- sensitive production scenarios, and always test first.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _example.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein

## SEE ALSO
trace(8), ustat(8)


bcc-ugc

title: bcc-ugc(8)

summary: Trace garbage collection events in

text:
# bcc-ugc(8)

## NAME
ugc, javagc, nodegc, pythongc, rubygc - Trace garbage collection events in high-level languages.

## SYNOPSIS
javagc [-h] [-v] [-m] [-M MINIMUM] [-F FILTER] pid

nodegc [-h] [-v] [-m] [-M MINIMUM] [-F FILTER] pid

pythongc [-h] [-v] [-m] [-M MINIMUM] [-F FILTER] pid

rubygc [-h] [-v] [-m] [-M MINIMUM] [-F FILTER] pid

ugc [-h] [-v] [-m] [-M MINIMUM] [-F FILTER] [-l {java,node,python,ruby}] pid

## DESCRIPTION
This traces garbage collection events as they occur, including their duration and any additional information (such as generation collected or type of GC) provided by the respective language's runtime.

This tool relies on USDT probes embedded in many high-level languages, such as Java, Node, Python, and Ruby. It requires a runtime instrumented with these probes, which in some cases requires building from source with a USDT-specific flag, such as "--enable-dtrace" or "--with-dtrace".

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-v

Print the resulting BPF program, for debugging purposes.

-m

Print times in milliseconds. The default is microseconds.

-M MINIMUM

Display only collections that are longer than this threshold. The value is given in milliseconds. The default is to display all collections.

-F FILTER

Display only collections whose textual description matches (contains) this string. The default is to display all collections. Note that the filtering here is performed in user-space, and not as part of the BPF program. This means that if you have thousands of collection events, specifying this filter will not reduce the amount of data that has to be transferred from the BPF program to the user-space script.

{java,node,python,ruby}

The language to trace.

pid

The process id to trace.

## EXAMPLES
Trace garbage collections in a specific Node process:

\# ugc node 148

Trace garbage collections in a specific Java process, and print GC times in

milliseconds: # ugc -m java 6004

Trace garbage collections in a specific Java process, and display them only if

they are longer than 10ms and have the string "Tenured" in their detailed description: # ugc -M 10 -F Tenured java 6004

## FIELDS
START

The start time of the GC, in seconds from the beginning of the trace.

TIME

The duration of the garbage collection event.

DESCRIPTION

The runtime-provided description of this garbage collection event.

## OVERHEAD
Garbage collection events, even if frequent, should not produce a considerable overhead when traced because they are still not very common. Even hundreds of GCs per second (which is a very high rate) will still produce a fairly negligible overhead.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _example.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein

## SEE ALSO
trace(8), ustat(8), uobjnew(8)


bcc-uobjnew

title: bcc-uobjnew(8)

summary: Summarize object allocations in

text:
# bcc-uobjnew(8)

## NAME
uobjnew, cobjnew, javaobjnew, rubyobjnew, tclobjnew - Summarize object allocations in high-level languages.

## SYNOPSIS
cobjnew [-h] [-C TOP_COUNT] [-S TOP_SIZE] [-v] pid [interval]

javaobjnew [-h] [-C TOP_COUNT] [-S TOP_SIZE] [-v] pid [interval]

rubyobjnew [-h] [-C TOP_COUNT] [-S TOP_SIZE] [-v] pid [interval]

tclobjnew [-h] [-C TOP_COUNT] [-S TOP_SIZE] [-v] pid [interval]

uobjnew [-h] [-C TOP_COUNT] [-S TOP_SIZE] [-v] [-l {c,java,ruby,tcl}] pid [interval]

## DESCRIPTION
uobjnew traces object allocations in high-level languages (including "malloc") and prints summaries of the most frequently allocated types by number of objects or number of bytes.

This tool relies on USDT probes embedded in many high-level languages, such as C, Java, Ruby, and Tcl. It requires a runtime instrumented with these probes, which in some cases requires building from source with a USDT-specific flag, such as "--enable-dtrace" or "--with-dtrace". For Java, the Java process must be started with the "-XX:+ExtendedDTraceProbes" flag.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-C TOP_COUNT

Print the top object types sorted by number of instances.

-S TOP_SIZE

Print the top object types sorted by size.

-v

Print the resulting BPF program, for debugging purposes.

{c,java,ruby,tcl}

The language to trace.

pid

The process id to trace.

interval

Wait this many seconds and then print the summary and exit. By default, wait for Ctrl+C to exit.

## EXAMPLES
Trace object allocations in a Ruby process:

\# uobjnew ruby 148

Trace object allocations from "malloc" and print the top 10 by total size:

\# uobjnew -S 10 c 1788

## FIELDS
TYPE

The object type being allocated. For C (malloc), this is the block size.

ALLOCS

The number of objects allocated.

BYTES

The number of bytes allocated.

## OVERHEAD
Object allocation events are quite frequent, and therefore the overhead from running this tool can be considerable. Use with caution and make sure to test before using in a production environment. Nonetheless, even thousands of allocations per second will likely produce a reasonable overhead when investigating a problem.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _example.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein

## SEE ALSO
ustat(8), ugc(8), memleak(8)


bcc-ustat

title: bcc-ustat(8)

summary: Activity stats from

text:
# bcc-ustat(8)

## NAME
ustat, javastat, nodestat, perlstat, phpstat, pythonstat, rubystat, tclstat - Activity stats from high-level languages.

## SYNOPSIS
javastat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

nodestat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

perlstat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

phpstat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

pythonstat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

rubystat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

tclstat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

ustat [-l {java,node,perl,php,python,ruby,tcl}] [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

## DESCRIPTION
This is "top" for high-level language events, such as garbage collections, exceptions, thread creations, object allocations, method calls, and more. The events are aggregated for each process and printed in a top-like table, which can be sorted by various fields. Not all language runtimes provide the same set of details.

This uses in-kernel eBPF maps to store per process summaries for efficiency.

This tool relies on USDT probes embedded in many high-level languages, such as Java, Node, Perl, PHP, Python, Ruby, and Tcl. It requires a runtime instrumented with these probes, which in some cases requires building from source with a USDT-specific flag, such as "--enable-dtrace" or "--with-dtrace". For Java, some probes are not enabled by default, and can be turned on by running the Java process with the "-XX:+ExtendedDTraceProbes" flag. For PHP processes, the environment variable USE_ZEND_DTRACE must be set to 1.

Newly-created processes will only be traced at the next interval. If you run this tool with a short interval (say, 1-5 seconds), this should be virtually unnoticeable. For longer intervals, you might miss processes that were started and terminated during the interval window.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-l {java,node,perl,php,python,ruby,tcl}

The language to trace. By default, all languages are traced.

-C

Do not clear the screen between updates.

-S {cload,excp,gc,method,objnew,thread}

Sort the output by the specified field.

-r MAXROWS

Do not print more than this number of rows.

-d

Print the resulting BPF program, for debugging purposes.

interval

Interval between updates, seconds.

count

Number of interval summaries.

## EXAMPLES
Summarize activity in high-level languages, 1 second refresh:

\# ustat

Don't clear the screen, and top 8 rows only:

\# ustat -Cr 8

5 second summaries, 10 times only:

\# ustat 5 10

## FIELDS
loadavg

The contents of /proc/loadavg

PID

Process ID.

CMDLINE

Process command line (often the second and following arguments will give you a hint as to which application is being run.

METHOD/s

Count of method invocations during interval.

GC/s

Count of garbage collections during interval.

OBJNEW/s

Count of objects allocated during interval.

CLOAD/s

Count of classes loaded during interval.

EXC/s

Count of exceptions thrown during interval.

THR/s

Count of threads created during interval.

## OVERHEAD
When using this tool with high-frequency events, such as method calls, a very significant slow-down can be expected. However, many of the high-level languages covered by this tool already have a fairly high per-method invocation cost, especially when running in interpreted mode. For the lower-frequency events, such as garbage collections or thread creations, the overhead should not be significant. Specifically, when probing Java processes and not using the "-XX:+ExtendedDTraceProbes" flag, the most expensive probes are not emitted, and the overhead should be acceptable.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _example.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein

## SEE ALSO
trace(8), argdist(8), tplist(8)


bcc-uthreads

title: bcc-uthreads(8)

text:
# bcc-uthreads(8)

## NAME
uthreads, cthreads, javathreads - Trace thread creation events in Java or pthreads.

## SYNOPSIS
cthreads [-h] [-v] pid javathreads [-h] [-v] pid uthreads [-h] [-l {c,java,none}] [-v] pid

## DESCRIPTION
This traces thread creation events in Java processes, or pthread creation events in any process. When a thread is created, its name or start address is printed.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-l {c,java,none}

The language to trace. C and none select tracing pthreads only, regardless of the runtime being traced.

-v

Print the resulting BPF program, for debugging purposes.

pid

The process id to trace.

## EXAMPLES
Trace Java thread creations:

\# uthreads -l java 148

Trace pthread creations:

\# uthreads 1802

## FIELDS
TIME

The event's time in seconds from the beginning of the trace.

ID

The thread's ID. The information in this column depends on the runtime.

TYPE

Event type -- thread start, stop, or pthread event.

DESCRIPTION

The thread's name or start address function name.

## OVERHEAD
Thread start and stop events are usually not very frequent, which makes this tool's overhead negligible.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _example.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein

## SEE ALSO
ustat(8), trace(8)


bcc-vfscount

title: bcc-vfscount(8)

summary: Count VFS calls (vfs_*). Uses Linux eBPF/bcc.

text:
# bcc-vfscount(8)

## NAME
vfscount - Count VFS calls ("vfs_*"). Uses Linux eBPF/bcc.

## SYNOPSIS
vfscount [duration]

## DESCRIPTION
This counts VFS calls. This can be useful for general workload characterization of these operations.

This works by tracing all kernel functions beginning with "vfs_" using dynamic tracing. This may match more functions than you are interested in measuring: Edit the script to customize which functions to trace.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
duration

duration of the trace in seconds.

## EXAMPLES
Count some VFS calls until Ctrl-C is hit:

\# vfscount

Count some VFS calls in ten seconds

\# vfscount 10

## FIELDS
ADDR

Address of the instruction pointer that was traced (only useful if the FUNC column is suspicious and you would like to double check the translation).

FUNC

Kernel function name

COUNT

Number of calls while tracing

## OVERHEAD
This traces kernel vfs functions and maintains in-kernel counts, which are asynchronously copied to user-space. While the rate of VFS operations can be very high (>1M/sec), this is a relatively efficient way to trace these events, and so the overhead is expected to be small for normal workloads. Measure in a test environment, and if overheads are an issue, edit the script to reduce the types of vfs functions traced (currently all beginning with "vfs_").

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
vfsstat(8)