bpftrace

package: bpftrace

title: bpftrace(8)

summary: a high\-level tracing language

text:
# bpftrace(8)

## NAME
bpftrace - a high-level tracing language

## SYNOPSIS
bpftrace [OPTIONS] FILENAME

bpftrace [OPTIONS] -e 'program code'

When FILENAME is "-", bpftrace will read program code from stdin.

A program will continue running until Ctrl-C is hit, or an exit function is called. When a program exits, all populated maps are printed (more details below).

## DESCRIPTION
bpftrace is a high-level tracing language for Linux. bpftrace uses LLVM as a backend to compile scripts to eBPF-bytecode and makes use of libbpf and bcc for interacting with the Linux BPF subsystem, as well as existing Linux tracing capabilities.

## EXAMPLES
Trace processes calling sleep

```
# bpftrace -e 'kprobe:do_nanosleep { printf("%d sleeping\n", pid); }'
```

Trace processes calling sleep while spawning sleep 5 as a child process

```
# bpftrace -e 'kprobe:do_nanosleep { printf("%d sleeping\n", pid); }' -c 'sleep 5'
```

List all probes with "sleep" in their name

```
# bpftrace -l '*sleep*'
```

List all the probes attached in the program

```
# bpftrace -l -e 'kprobe:do_nanosleep { printf("%d sleeping\n", pid); }'
```

## OPTIONS

### -B MODE
Set the output buffer mode (applies to terminal and file output -o).

Valid values are

none No buffering. Each I/O is written as soon as possible

line Data is written on the first newline or when the buffer is full. This is the default mode.

full Data is written once the buffer is full.

### -c COMMAND
Run COMMAND as a child process. When the child terminates bpftrace will also terminate, as if 'exit()' had been called. If bpftrace terminates before the child process does the child process will be terminated with a SIGTERM. If used, 'USDT' probes will only be attached to the child process. To avoid a race condition when using 'USDTs', the child is stopped after 'execve' using 'ptrace(2)' and continued when all 'USDT' probes are attached. The child process runs with the same privileges as bpftrace itself (usually root).

Unless otherwise specified, bpftrace does not perform any implicit filtering. Therefore, if you are only interested in events in COMMAND, you may want to filter based on the child PID. The child PID is available to programs as the 'cpid' builtin. For example, you could add the predicate /pid == cpid/ to probes with userspace context.

### -d STAGE
Enable debug mode. For more details see the Debug Output section.

### --debuginfo DIR[:DIR]
Add the directory DIR to the search path for DWARF debug information. Paths may be absolute or relative to the traced binary’s location. Specify multiple paths either by separating them with a colon (:) or by repeating the option, e.g. --debuginfo=/bin/debug:./lib/debug:.. and --debuginfo=/bin/debug --debuginfo=./lib/debug, respectively. Debug files in these paths are matched by build ID when available; prefix a path with + to enable CRC32 validation when needed. Files that do not match are skipped. By default, bpftrace searches under standard debug paths, including .debug relative to the traced binary and /usr/lib/debug; it may also query available debuginfod servers.

Split debuginfo format (.dwo, .dwp) is not yet supported for this option, but continues to work with the default probe binary path.

### --dry-run
Terminate execution right after attaching all the probes. Useful for testing that the script can be parsed, loaded, and attached, without actually running it.

### -e PROGRAM
Execute PROGRAM instead of reading the program from a file or stdin.

### -f FORMAT
Set the output format.

Valid values are

json

text

The JSON output is compatible with NDJSON and JSON Lines, meaning each line of the streamed output is a single blob of valid JSON.

### --fmt FILENAME
Output standard format for the bpftrace file FILENAME.

### -h, --help
Print the help summary.

### -I DIR
Add the directory DIR to the search path for C headers. This option can be used multiple times. For more details see the Preprocessor Options section.

### --include FILENAME
Add FILENAME as an include for the pre-processor. This is equal to adding '#include FILENAME' at the top of the program. This option can be used multiple times. For more details see the Preprocessor Options section.

### --info
Print detailed information about features supported by the kernel and the bpftrace build.

### -k
This flag enables runtime warnings for errors from 'probe_read_*' and map lookup BPF helpers. When errors occur bpftrace will log an error containing the source location and the error code:

```
stdin:48-57: WARNING: Failed to probe_read_user_str: Bad address (-14)
u:lib.so:"fn(char const*)" { printf("arg0:%s\n", str(arg0));}

                                                 ~~~~~~~~~
```

### -l [SEARCH|FILENAME]
List all probes that match the SEARCH pattern. If the pattern is omitted all probes will be listed. This pattern supports wildcards in the same way that probes do. E.g. '-l kprobe:*file*' to list all 'kprobes' with 'file' in the name. This can be used with a program, which will list all probes in that program. For more details see the Listing Probes section.

### --no-feature feature,feature,...
Disable use of detected features, valid values are

uprobe_multi to disable uprobe_multi link

kprobe_multi to disable kprobe_multi link

kprobe_session to disable automatic collapse of kprobe/kretprobe into kprobe session

### --no-warnings
Suppress all warning messages created by bpftrace.

### --probe-filter REGEX
Only attach and run probes whose name matches the regular expression REGEX. Probes that do not match are not loaded into the kernel. Special probes (BEGIN and END) are not affected by this filter. If no probes match, bpftrace exits with an error.

### --traceable-functions FILENAME
Specify the file containing the list of traceable kernel functions. If not set, bpftrace uses /sys/kernel/tracing/available_filter_functions, which depends on dynamic ftrace. The format is the same as in available_filter_functions: each line is either "symbol" or "symbol [module]".

### -o FILENAME
Write bpftrace tracing output to FILENAME instead of stdout. This doesn’t include child process (-c option) output. Errors are still written to stderr.

### -p PID
Attach to the process with or filter actions by PID. If the process terminates, bpftrace will also terminate. When using USDT, uprobes, uretprobes, hardware, software, profile, interval, or watchpoint probes they will be attached to only this process. For all other probes, except begin/end, the pid will act like a predicate to filter out events not from that pid. For listing uprobes/uretprobes set the target to '*' and the process’s address space will be searched for the symbols.

### -q
Keep messages quiet.

### --unsafe
Some calls, like 'system', are marked as unsafe as they can have dangerous side effects ('system("rm -rf")') and are disabled by default. This flag allows their use.

In addition, it makes kprobe/kretprobe checks less restrictive. It can be used to probe functions that bpftrace reports as not traceable, but are supported if the kernel is configured to allow probing notrace functions.

### --usdt-file-activation
Activate usdt semaphores based on file path.

### -V, --version
Print bpftrace version information.

### -v
Enable verbose messages. For more details see the Verbose Output section.

### Program Options
You can also pass custom options to a bpftrace program/script itself via positional or named parameters. Positional parameters can be placed before or after a double dash but named parameters can ONLY come after a double dash; e.g.

```
# bpftrace -e 'begin { print(($1, $2, getopt("aa", 1), getopt("bb"))); }' p1 -- --aa=20 --bb p2
// (p1, p2, 20, true) is printed
```

or

```
# bpftrace myscript.bt -- p1 --aa=20 --bb p2
```

In these examples there are two positional parameters (p1, p2) and two named parameters (aa, which is set to 20, and bb, which is set to true). Named program parameters require the = to set their value unless they are boolean parameters (like 'bb' above). Read about how to access positional and named parameters here <https://github.com/bpftrace/bpftrace/blob/master/docs/language.md#command-line-parameters>.

## THE LANGUAGE
Syntax, types, and concepts for bpftrace are available here <https://github.com/bpftrace/bpftrace/blob/master/docs/language.md>.

## PROBES
bpftrace supports various probe types which allow the user to attach BPF programs to different types of events. Each probe starts with a provider (e.g. kprobe) followed by a colon (:) separated list of options. The amount of options and their meaning depend on the provider. Optionally, the attachpoint specified can be prefixed by a user-provided name followed by equal sign (e.g. myname=kprobe:...); this is used for certain advanced features and is typically omitted. Full list of probe types <https://github.com/bpftrace/bpftrace/blob/master/docs/language.md#probes>.

## STANDARD LIBRARY
The standard library of all builtins <https://github.com/bpftrace/bpftrace/blob/master/docs/stdlib.md#builtins>, functions <https://github.com/bpftrace/bpftrace/blob/master/docs/stdlib.md#functions>, and map functions <https://github.com/bpftrace/bpftrace/blob/master/docs/stdlib.md#map-functions> is available here <https://github.com/bpftrace/bpftrace/blob/master/docs/stdlib.md>.

## CONFIGURATION

### Config Variables
Some behavior can only be controlled through config variables, which are available here <https://github.com/bpftrace/bpftrace/blob/master/docs/language.md#config-variables>. These can be set via the Config Block directly in a script (before any probes) or via their environment variable equivalent, which is upper case and includes the BPFTRACE_ prefix e.g. stack_mode's environment variable would be BPFTRACE_STACK_MODE.

### Environment Variables
These are not available as part of the standard set of Config Variables above and can only be set as environment variables.

### BPFTRACE_BTF
Default: None

The path to a BTF file. By default, bpftrace searches several locations to find a BTF file. See src/btf.cpp for the details.

### BPFTRACE_KERNEL_BUILD
Default: /lib/modules/$(uname -r)

Only used with BPFTRACE_KERNEL_SOURCE if it is out-of-tree Linux kernel build.

### BPFTRACE_KERNEL_SOURCE
Default: /lib/modules/$(uname -r)

bpftrace requires kernel headers for certain features, which are searched for in this directory.

### BPFTRACE_VMLINUX
Default: None

This specifies the vmlinux path used for kernel symbol resolution when attaching kprobe to offset. If this value is not given, bpftrace searches vmlinux from pre defined locations. See src/attached_probe.cpp:find_vmlinux() for details.

### BPFTRACE_COLOR
Default: auto

Colorize the bpftrace log output message. Valid values are auto, always and never.

## OPTIONS EXPANDED

### Debug Output
The -d STAGE option produces debug output. It prints the output of the bpftrace execution stage given by the STAGE argument. The option can be used multiple times (with different stage names) and the special value all prints the output of all the supported stages. The option also takes multiple stages in one invocation as comma separated values.

Note: This is primarily used for bpftrace developers.

The supported options are:

ast

Prints the Abstract Syntax Tree (AST) after every pass.

codegen

Prints the unoptimized LLVM IR as produced by CodegenLLVM.

codegen-opt

Prints the optimized LLVM IR, i.e. the code which will be compiled into BPF bytecode.

dis

Disassembles and prints out the generated bytecode that libbpf will see. Only available in debug builds.

libbpf

Captures and prints libbpf log for all libbpf operations that bpftrace uses.

verifier

Captures and prints the BPF verifier log.

all

Prints the output of all of the above stages.

### Listing Probes
Probe listing is the method to discover which probes are supported by the current system. Listing supports the same syntax as normal attachment does and alternatively can be combined with -e or filename args to see all the probes that a program would attach to.

```
# bpftrace -l 'kprobe:*'
# bpftrace -l 't:syscalls:*openat*
# bpftrace -l 'kprobe:tcp*,trace
# bpftrace -l 'k:*socket*,tracepoint:syscalls:*tcp*'
# bpftrace -l -e 'tracepoint:xdp:mem_* { exit(); }'
# bpftrace -l my_script.bt
# bpftrace -lv 'enum cpu_usage_stat'
```

The verbose flag (-v) can be specified to inspect arguments (args) for providers that support it:

```
# bpftrace -l 'fexit:tcp_reset,tracepoint:syscalls:sys_enter_openat' -v
fexit:tcp_reset

    struct sock * sk

    struct sk_buff * skb
tracepoint:syscalls:sys_enter_openat

    int __syscall_nr

    int dfd

    const char * filename

    int flags

    umode_t mode
# bpftrace -l 'uprobe:/bin/bash:rl_set_prompt' -v    # works only if /bin/bash has DWARF
uprobe:/bin/bash:rl_set_prompt

    const char *prompt
# bpftrace -lv 'struct css_task_iter'
struct css_task_iter {

        struct cgroup_subsys *ss;

        unsigned int flags;

        struct list_head *cset_pos;

        struct list_head *cset_head;

        struct list_head *tcset_pos;

        struct list_head *tcset_head;

        struct list_head *task_pos;

        struct list_head *cur_tasks_head;

        struct css_set *cur_cset;

        struct css_set *cur_dcset;

        struct task_struct *cur_task;

        struct list_head iters_node;
};
```

### Preprocessor Options
The -I option can be used to add directories to the list of directories that bpftrace uses to look for headers. Can be defined multiple times.

```
# cat program.bt
#include <foo.h>
begin { @ = FOO }
# bpftrace program.bt
definitions.h:1:10: fatal error: 'foo.h' file not found
# /tmp/include
foo.h
# bpftrace -I /tmp/include program.bt
Attached 1 probe
```

The --include option can be used to include headers by default. Can be defined multiple times. Headers are included in the order they are defined, and they are included before any other include in the program being executed.

```
# bpftrace --include linux/path.h --include linux/dcache.h \

    -e 'kprobe:vfs_open { printf("open path: %s\n", str(((struct path *)arg0).dentry.d_name.name)); }'
Attached 1 probe
open path: .com.google.Chrome.ASsbu2
open path: .com.google.Chrome.gimc10
open path: .com.google.Chrome.R1234s
```

### Verbose Output
The -v option prints more information about the program as it is run:

```
# bpftrace -v -e 'tracepoint:syscalls:sys_enter_nanosleep { printf("%s is sleeping.\n", comm); }'
AST node count: 7
Attached 1 probe
load tracepoint:syscalls:sys_enter_nanosleep, with BTF, with func_infos: Success
Program ID: 111
Attaching tracepoint:syscalls:sys_enter_nanosleep
iscsid is sleeping.
iscsid is sleeping.
[...]
```

## TERMINOLOGY
BPF

Berkeley Packet Filter: a kernel technology originally developed for optimizing the processing of packet filters (eg, tcpdump expressions).

BPF map

A BPF memory object, which is used by bpftrace to create many higher-level objects.

BTF

BPF Type Format: the metadata format which encodes the debug info related to BPF program/map.

dynamic tracing

Also known as dynamic instrumentation, this is a technology that can instrument any software event, such as function calls and returns, by live modification of instruction text. Target software usually does not need special capabilities to support dynamic tracing, other than a symbol table that bpftrace can read. Since this instruments all software text, it is not considered a stable API, and the target functions may not be documented outside of their source code.

eBPF

Enhanced BPF: a kernel technology that extends BPF so that it can execute more generic programs on any events, such as the bpftrace programs listed below. It makes use of the BPF sandboxed virtual machine environment. Also note that eBPF is often just referred to as BPF.

kprobes

A Linux kernel technology for providing dynamic tracing of kernel functions.

probe

An instrumentation point in software or hardware, that generates events that can execute bpftrace programs.

static tracing

Hard-coded instrumentation points in code. Since these are fixed, they may be provided as part of a stable API, and documented.

tracepoints

A Linux kernel technology for providing static tracing.

uprobes

A Linux kernel technology for providing dynamic tracing of user-level functions.

USDT

User Statically-Defined Tracing: static tracing points for user-level software. Some applications support USDT.

## PROGRAM FILES
Programs saved as files are often called scripts and can be executed by specifying their file name. It is convention to use the .bt file extension but it is not required.

For example, listing the sleepers.bt file using cat:

```
# cat sleepers.bt
tracepoint:syscalls:sys_enter_nanosleep {

  printf("%s is sleeping.\n", comm);
}
```

And then calling it:

```
# bpftrace sleepers.bt
Attached 1 probe
iscsid is sleeping.
iscsid is sleeping.
```

It can also be made executable to run stand-alone. Start by adding an interpreter line at the top (#!) with either the path to your installed bpftrace (/usr/local/bin is the default) or the path to env (usually just /usr/bin/env) followed by bpftrace (so it will find bpftrace in your $PATH):

```
#!/usr/local/bin/bpftrace
tracepoint:syscalls:sys_enter_nanosleep {

  printf("%s is sleeping.\n", comm);
}
```

Then make it executable:

```
# chmod 755 sleepers.bt
# ./sleepers.bt
Attached 1 probe
iscsid is sleeping.
iscsid is sleeping.
```


bpftrace-bashreadline.bt

title: bpftrace-bashreadline.bt(8)

text:
# bpftrace-bashreadline.bt(8)

## NAME
bashreadline.bt - Print bash commands system wide. Uses bpftrace/eBPF.

## SYNOPSIS
bashreadline.bt

## DESCRIPTION
bashreadline traces the return of the readline() function using uretprobes, to show the bash commands that were entered interactively, system wide. The entered command may fail: this is just showing what was entered.

This program is also a basic example of bpftrace and uretprobes.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bpftrace.

## EXAMPLES
Trace bash commands system wide:

\# bashreadline.bt

## FIELDS
TIME

A timestamp on the output, in "HH:MM:SS" format.

PID

The process ID for bash.

COMMAND

Entered command.

## OVERHEAD
As the rate of interactive bash commands is expected to be very low (<<100/s), the overhead of this program is expected to be negligible.

## SOURCE
This is from bpftrace.

https://github.com/bpftrace/bpftrace

Also look in the bpftrace distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

This is a bpftrace version of the bcc tool of the same name. The bcc tool may provide more options and customizations.

https://github.com/iovisor/bcc

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
opensnoop.bt(8)

branch: sisyphus


bpftrace-biolatency.bt

title: bpftrace-biolatency.bt(8)

text:
# bpftrace-biolatency.bt(8)

## NAME
biolatency.bt - Block I/O latency as a histogram. Uses bpftrace/eBPF.

## SYNOPSIS
biolatency.bt

## DESCRIPTION
This tool summarizes time (latency) spent in block device I/O (disk I/O) as a power-of-2 histogram. This allows the distribution to be studied, including modes and outliers. There are often two modes, one for device cache hits and one for cache misses, which can be shown by this tool. Latency outliers will also be shown.

The original tool, which is retained as "biolatency-kp.bt", currently works by dynamic tracing of the blk_account*() kernel functions, which will need updating to match any changes to these functions in future kernels versions.

The updated version of the tool utilizes tracepoints instead of kprobes so that it can be compatible with a wide range of kernel versions.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bpftrace.

## EXAMPLES
Trace block device I/O (disk I/O), and print a latency histogram on Ctrl-C:

\# biolatency.bt

## FIELDS
1st, 2nd

This is a range of latency, in microseconds (shown in "[...)" set notation).

3rd

A column showing the count of operations in this range.

4th

This is an ASCII histogram representing the count column.

## OVERHEAD
Since block device I/O usually has a relatively low frequency (< 10,000/s), the overhead for this tool is expected to be negligible. For high IOPS storage systems, test and quantify before use.

## SOURCE
This is from bpftrace.

https://github.com/bpftrace/bpftrace

Also look in the bpftrace distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

This is a bpftrace version of the bcc tool of the same name. The bcc tool may provide more options and customizations.

https://github.com/iovisor/bcc

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
biosnoop.bt(8)


bpftrace-biosnoop.bt

title: bpftrace-biosnoop.bt(8)

text:
# bpftrace-biosnoop.bt(8)

## NAME
biosnoop.bt - Block I/O tracing tool, showing per I/O latency. Uses bpftrace/eBPF.

## SYNOPSIS
biosnoop.bt

## DESCRIPTION
This is a basic block I/O (disk I/O) tracing tool, showing each I/O event along with the issuing process ID, and the I/O latency. This can be used to investigate disk I/O performance issues.

This tool currently works by dynamic tracing of the blk_account*() kernel functions, which will need updating to match any changes to these functions in future kernels versions.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bpftrace.

## EXAMPLES
Trace block I/O events, printing per-line summaries:

\# biosnoop.bt

## FIELDS
TIME

Time of the I/O completion, in milliseconds since program start.

DISK ID

Major and minor device numbers of the disk device.

COMM

Issuing process name. This often identifies the issuing application process, but I/O may be initiated from kernel threads only.

PID

Issuing process ID. This often identifies the issuing application process, but I/O may be initiated from kernel threads only.

LAT(ms)

Per I/O latency.

## OVERHEAD
Since block device I/O usually has a relatively low frequency (< 10,000/s), the overhead for this tool is expected to be negligible. For high IOPS storage systems, test and quantify before use.

## SOURCE
This is from bpftrace.

https://github.com/bpftrace/bpftrace

Also look in the bpftrace distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

This is a bpftrace version of the bcc tool of the same name. The bcc tool provides more fields.

https://github.com/iovisor/bcc

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
opensnoop.bt(8)


bpftrace-biostacks.bt

title: bpftrace-biostacks.bt(8)

text:
# bpftrace-biostacks.bt(8)

## NAME
biostacks - Show disk I/O latency with initialization stacks. Uses bpftrace/eBPF.

## SYNOPSIS
biostacks

## DESCRIPTION
This tool shows disk I/O latency histograms for each block I/O initialization path. This can help reveal the reason for different latencies, as some may be created by log flushing, others by application reads, etc.

This works by attaching to block_io_start and block_rq_issue tracepoints.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bpftrace.

## EXAMPLES
Trace disk I/O latency with initialization stacks:

\# biostacks.bt

## FIELDS
0th

An initialization kernel stack trace (shown in "@[...]") is printed before each I/O histogram.

1st, 2nd

This is a range of I/O latency, in microseconds (shown in "[...)" set notation).

3rd

A column showing the count of I/O in this range.

4th

This is an ASCII histogram representing the count column.

## OVERHEAD
The rate of biostacks should be low (bounded by device IOPS), such that the overhead of this tool is expected to be negligible.

## SOURCE
This tool originated from the book "BPF Performance Tools", published by Addison Wesley (2019):

http://www.brendangregg.com/bpf-performance-tools-book.html

See the book for more documentation on this tool.

This version is in the bpftrace repository:

https://github.com/bpftrace/bpftrace

Also look in the bpftrace distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
biosnoop.bt(8)


bpftrace-bitesize.bt

title: bpftrace-bitesize.bt(8)

text:
# bpftrace-bitesize.bt(8)

## NAME
bitesize.bt - Show disk I/O size as a histogram. Uses bpftrace/eBPF.

## SYNOPSIS
bitesize.bt

## DESCRIPTION
This can be used to characterize the distribution of block device (disk) I/O sizes. To study block device I/O in more detail, see biosnoop.bt(8).

This uses the tracepoint:block:block_rq_issue tracepoint, and is a simple example of bpftrace.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bpftrace.

## EXAMPLES
Trace block I/O and summarize as a histogram by process:

\# bitesize.bt

## FIELDS
0th

A process name (shown in "@[...]") is printed before each I/O histogram.

1st, 2nd

This is a range of I/O sizes, in Kbytes (shown in "[...)" set notation).

3rd

A column showing the count of I/O in this range.

4th

This is an ASCII histogram representing the count column.

## OVERHEAD
Since block device I/O usually has a relatively low frequency (< 10,000/s), the overhead for this tool is expected to be low or negligible. For high IOPS storage systems, test and quantify before use.

## SOURCE
This is from bpftrace.

https://github.com/bpftrace/bpftrace

Also look in the bpftrace distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

This is a bpftrace version of the bcc tool of the same name.

https://github.com/iovisor/bcc

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
biosnoop.bt(8)


bpftrace-capable.bt

title: bpftrace-capable.bt(8)

text:
# bpftrace-capable.bt(8)

## NAME
capable.bt - Trace security capability checks (cap_capable()).

## SYNOPSIS
capable.bt

## DESCRIPTION
This traces security capability checks in the kernel, and prints details for each call. This can be useful for general debugging, and also security enforcement: determining a white list of capabilities an application needs.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF, bpftrace.

## EXAMPLES
Trace all capability checks system-wide:

\# capable.bt

## FIELDS
TIME(s)

Time of capability check: HH:MM:SS.

UID

User ID.

PID

Process ID.

COMM

Process name.

CAP

Capability number.

NAME

Capability name. See capabilities(7) for descriptions.

AUDIT

Whether this was an audit event.

## OVERHEAD
This adds low-overhead instrumentation to capability checks, which are expected to be low frequency, however, that depends on the application. Test in a lab environment before use.

## SOURCE
This is from bpftrace.

https://github.com/bpftrace/bpftrace

Also look in the bpftrace distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

This is a bpftrace version of the bcc tool of the same name. The bcc tool provides options to customize the output.

https://github.com/iovisor/bcc

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
capabilities(7)


bpftrace-cpuwalk.bt

title: bpftrace-cpuwalk.bt(8)

text:
# bpftrace-cpuwalk.bt(8)

## NAME
cpuwalk.bt - Sample which CPUs are executing processes.. Uses bpftrace/eBPF.

## SYNOPSIS
cpuwalk.bt

## DESCRIPTION
This tool samples CPUs at 99 Hertz, then prints a histogram showing which CPUs were active. 99 Hertz is used to avoid lockstep sampling that would skew results. This tool can help identify if your application's workload is evenly using the CPUs, or if there is an imbalance problem.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bpftrace.

## EXAMPLES
Sample CPUs and print a summary on Ctrl-C:

\# cpuwalk.bt

## FIELDS
1st, 2nd

The CPU is shown in the first field, after the "[". Disregard the second field.

3rd

A column showing the number of samples for this CPU.

4th

This is an ASCII histogram representing the count column.

## OVERHEAD
This should be negligible.

## SOURCE
This is from bpftrace.

https://github.com/bpftrace/bpftrace

Also look in the bpftrace distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
mpstat(1)


bpftrace-dcsnoop.bt

title: bpftrace-dcsnoop.bt(8)

text:
# bpftrace-dcsnoop.bt(8)

## NAME
dcsnoop.bt - Trace directory entry cache (dcache) lookups. Uses bpftrace/eBPF.

## SYNOPSIS
dcsnoop.bt

## DESCRIPTION
By default, this traces every dcache lookup, and shows the process performing the lookup and the filename requested.

The output of this tool can be verbose, and is intended for further investigations of dcache performance beyond dcstat(8), which prints per-second summaries.

This uses kernel dynamic tracing of the d_lookup() function, and will need and will need updating to match any changes to this function.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## EXAMPLES
Trace all dcache lookups:

\# dcsnoop.bt

## FIELDS
TIME(ms)

Time of lookup, in milliseconds.

PID

Process ID.

COMM

Process name.

T

Type: R == reference, M == miss. A miss will print two lines, one for the reference, and one for the miss.

FILE

The file name component that was being looked up. This contains trailing pathname components (after '/'), which will be the subject of subsequent lookups.

## OVERHEAD
File name lookups can be frequent (depending on the workload), and this tool prints a line for each failed lookup, and with -a, each reference as well. The output may be verbose, and the incurred overhead, while optimized to some extent, may still be from noticeable to significant. This is only really intended for deeper investigations beyond dcstat(8), when absolutely necessary. Measure and quantify the overhead in a test environment before use.

## SOURCE
This is from bpftrace.

https://github.com/bpftrace/bpftrace

Also look in the bpftrace distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

This is a bpftrace version of the bcc tool of the same name. The bcc tool may provide more options and customizations.

https://github.com/iovisor/bcc

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
dcstat(8)


bpftrace-execsnoop.bt

title: bpftrace-execsnoop.bt(8)

text:
# bpftrace-execsnoop.bt(8)

## NAME
execsnoop.bt - Trace new processes via exec() syscalls. Uses bpftrace/eBPF.

## SYNOPSIS
execsnoop.bt

## DESCRIPTION
This traces when processes call exec() (execve()). It is handy for identifying new processes created via the usual fork()->exec() sequence. Note that the return value is not currently traced, so the exec() may have failed.

This tool is useful for debugging shell scripts, including application startup. It is also useful for identifying a type of performance issue: a flood of short-lived processes, that end quickly and aren't readily visible in top(1).

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bpftrace.

## EXAMPLES
Trace all new processes calling execve():

\# execsnoop.bt

## FIELDS
TIME

Time of the exec() call, in milliseconds since program start.

PID

Process ID

PPID

Parent Process ID

ARGS

Process name and arguments (16 word maximum).

## OVERHEAD
This traces the execve() tracepoint and prints output for each event. As the rate of this is generally expected to be low (< 100/s), the overhead is also expected to be negligible. If you have an application that is spawning a high rate of new processes for a reason (large build process), this could cause a small amount of overhead: test and understand overhead before use.

## SOURCE
This is from bpftrace.

https://github.com/bpftrace/bpftrace

Also look in the bpftrace distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

This is a bpftrace version of the bcc tool of the same name. The bcc tool provides more fields and options to customize the output.

https://github.com/iovisor/bcc

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
opensnoop.bt(8)


bpftrace-gethostlatency.bt

title: bpftrace-gethostlatency.bt(8)

text:
# bpftrace-gethostlatency.bt(8)

## NAME
gethostlatency.bt - Show latency for getaddrinfo/gethostbyname[2] calls. Uses bpftrace/eBPF.

## SYNOPSIS
gethostlatency.bt

## DESCRIPTION
This traces and prints when gethostbyaddr(), getaddrinfo(), gethostbyname(), and gethostbyname2() are called, system wide, and shows the responsible PID and command name, latency of the call (duration) in milliseconds, and the host string.

This tool can be useful for identifying DNS latency, by identifying which remote host name lookups were slow, and by how much.

This tool currently uses dynamic tracing of user-level functions and registers, and may need modifications to match your software and processor architecture.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## EXAMPLES
Trace host lookups (getaddrinfo/gethostbyname[2]) system wide:

\# gethostlatency.bt

## FIELDS
TIME

Time of the command (HH:MM:SS).

PID

Process ID of the client performing the call.

COMM

Process (command) name of the client performing the call.

LATms

Latency of the call, in milliseconds.

HOST

Host name string: the target of the lookup.

## OVERHEAD
The rate of lookups should be relatively low, so the overhead is not expected to be a problem.

## SOURCE
This is from bpftrace.

https://github.com/bpftrace/bpftrace

Also look in the bpftrace distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

This is a bpftrace version of the bcc tool of the same name. The bcc tool provides command line options.

https://github.com/iovisor/bcc

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
tcpdump(8), gethostbyname(3), getaddrinfo(3)


bpftrace-killsnoop.bt

title: bpftrace-killsnoop.bt(8)

text:
# bpftrace-killsnoop.bt(8)

## NAME
killsnoop.bt - Trace signals issued by the kill(),tkill(),tgkill() syscall. Uses bpftrace/eBPF.

## SYNOPSIS
killsnoop.bt

## DESCRIPTION
killsnoop traces the kill(),tkill(),tgkill() syscall, to show signals sent via this method. This may be useful to troubleshoot failing applications, where an unknown mechanism is sending signals.

This works by tracing the kill(),tkill(),tgkill() syscall tracepoints.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bpftrace.

## EXAMPLES
Trace all kill(),tkill(),tgkill() syscalls:

\# killsnoop.bt

## FIELDS
TIME

Time of the kill call.

PID

Source process ID

COMM

Source process name

SIG

Signal number or name. See signal(7).

TPID

Target process ID

RESULT

Result. 0 == success, a negative value (of the error code) for failure.

## OVERHEAD
This traces the kernel kill function and prints output for each event. As the rate of this is generally expected to be low (< 100/s), the overhead is also expected to be negligible. If you have an application that is calling a very high rate of kill()|tkill()|tgkill()s for some reason, then test and understand overhead before use.

## SOURCE
This is from bpftrace.

https://github.com/bpftrace/bpftrace

Also look in the bpftrace distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

This is a bpftrace version of the bcc tool of the same name. The bcc tool may provide more options and customizations.

https://github.com/iovisor/bcc

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg, Rong Tao

## SEE ALSO
opensnoop.bt(8)


bpftrace-loads.bt

title: bpftrace-loads.bt(8)

text:
# bpftrace-loads.bt(8)

## NAME
loads.bt - Prints load averages. Uses bpftrace/eBPF.

## SYNOPSIS
loads.bt

## DESCRIPTION
These are the same load averages printed by "uptime", but to three decimal places instead of two (not that it really matters). This is really a demonstration of fetching and processing a kernel structure from bpftrace.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bpftrace.

## EXAMPLES
Print system load averages every second:

\# loads.bt

## FIELDS
HH:MM:SS

Each output line includes time of printing in "HH:MM:SS" format.

load averages:

These are exponentially-damped moving sum averages of the system loads. Load is a measurement of demand on system resources, which include CPUs and other resources that are accessed with the kernel in an uninterruptible state (TASK_UNINTERRUPTIBLE), which includes types of disk I/O and lock accesses. Linux load averages originally reflected CPU demand only, as it does in other OSes, but this was changed in Linux 0.99.14. This demand measurement reflects not just the utilized resource, but also the queued demand (a saturation measurement). Finally, the three numbers are called the "one-", "five-", and "fifteen-minute" load averages, however these times are constants used in the exponentially-damping equation, and the load averages reflect load beyond these times. Were you expecting an accurate description of load averages in the man page of a bpftrace tool?

## OVERHEAD
Other than bpftrace startup time, negligible.

## SOURCE
This is from bpftrace.

https://github.com/bpftrace/bpftrace

Also look in the bpftrace distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## REFERENCE
For more on load averages, see:

http://www.brendangregg.com/blog/2017-08-08/linux-load-averages.html

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
uptime(1)


bpftrace-mdflush.bt

title: bpftrace-mdflush.bt(8)

text:
# bpftrace-mdflush.bt(8)

## NAME
mdflush.bt - Trace md flush events. Uses bpftrace/eBPF.

## SYNOPSIS
mdflush.bt

## DESCRIPTION
This tool traces flush events by md, the Linux multiple device driver (software RAID). The timestamp and md device for the flush are printed. Knowing when these flushes happen can be useful for correlation with unexplained spikes in disk latency.

This works by tracing the kernel md_flush_request() function using kernel dynamic tracing, and will need updating to match any changes to this function.

Note that the flushes themselves are likely to originate from higher in the I/O stack, such as from the file systems.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bpftrace.

## EXAMPLES
Trace md flush events:

\# mdflush.bt

## FIELDS
TIME

Time of the flush event (HH:MM:SS).

PID

The process ID that was on-CPU when the event was issued. This may identify the cause of the flush (eg, the "sync" command), but will often identify a kernel worker thread that was managing I/O.

COMM

The command name for the PID.

DEVICE

The md device name.

## OVERHEAD
Expected to be negligible.

## SOURCE
This is from bpftrace.

https://github.com/bpftrace/bpftrace

Also look in the bpftrace distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

This is a bpftrace version of the bcc tool of the same name. The bcc tool may provide more options and customizations.

https://github.com/iovisor/bcc

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
biosnoop.bt(8)


bpftrace-naptime.bt

title: bpftrace-naptime.bt(8)

text:
# bpftrace-naptime.bt(8)

## NAME
naptime.bt - Trace voluntary sleep calls. Uses bpftrace/eBPF.

## SYNOPSIS
naptime.bt

## DESCRIPTION
This tool traces application sleeps, and can be used for debugging high latency that may be caused by deliberate sleeps placed in application routines, especially administration scripts.

This tool works by tracing the nanosleep(2) syscall using the syscall tracepoints.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bpftrace.

## EXAMPLES
Trace application sleeps via nanosleep(2):

\# naptime.bt

## FIELDS
TIME

A timestamp in HH:MM:SS format.

PPID

Parent process ID.

PCOMM

Parent process name.

PID

The sleeping process ID.

COMM

The sleeping process name.

SECONDS

The requested duration of the sleep.

## OVERHEAD
nanosleep(2) calls are expected to be low frequency (<< 100/s), so the overhead of this tool is expected to be negligible.

## SOURCE
This tool originated from the book "BPF Performance Tools", published by Addison Wesley (2019):

http://www.brendangregg.com/bpf-performance-tools-book.html

See the book for more documentation on this tool.

This version is in the bpftrace repository:

https://github.com/bpftrace/bpftrace

Also look in the bpftrace distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
capable.bt(8)


bpftrace-oomkill.bt

title: bpftrace-oomkill.bt(8)

summary: Trace OOM killer. Uses bpftrace/eBPF.

text:
# bpftrace-oomkill.bt(8)

## NAME
oomkill.bt - Trace OOM killer. Uses bpftrace/eBPF.

## SYNOPSIS
oomkill.bt

## DESCRIPTION
This traces the kernel out-of-memory killer, and prints basic details, including the system load averages at the time of the OOM kill. This can provide more context on the system state at the time: was it getting busier or steady, based on the load averages? This tool may also be useful to customize for investigations; for example, by adding other task_struct details at the time of OOM, or by adding other commands to run at the shell.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bpftrace.

## EXAMPLES
Trace OOM kill events:

\# oomkill.bt

## FIELDS
Triggered by ...

The process ID and process name of the task that was running when another task was OOM killed.

CGROUP ...

If the OOM occurs in the memory cgroup, display the memory cgroup information.

OOM kill of ...

The process ID and name of the target process that was OOM killed.

loadavg

Contents of /proc/loadavg. The first three numbers are 1, 5, and 15 minute load averages (where the average is an exponentially damped moving sum, and those numbers are constants in the equation); then there is the number of running tasks, a slash, and the total number of tasks; and then the last number is the last PID to be created.

## OVERHEAD
Negligible.

## SOURCE
This is from bpftrace.

https://github.com/bpftrace/bpftrace

Also look in the bpftrace distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

This is a bpftrace version of the bcc tool of the same name. The bcc tool may provide more options and customizations.

https://github.com/iovisor/bcc

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg, Rong Tao

## SEE ALSO
dmesg(1)


bpftrace-opensnoop.bt

title: bpftrace-opensnoop.bt(8)

text:
# bpftrace-opensnoop.bt(8)

## NAME
opensnoop.bt - Trace open(),openat(),openat2() syscalls. Uses bpftrace/eBPF.

## SYNOPSIS
opensnoop.bt -- [--depth=<N>] [--errname]

## DESCRIPTION
opensnoop traces the open(),openat(),openat2() syscall, showing which processes are attempting to open which files. This can be useful for determining the location of config and log files, or for troubleshooting applications that are failing, specially on startup.

This works by tracing the open(),openat(),openat2() syscall tracepoint.

Since this uses BPF, only the root user can use this tool.

## OPTIONS
--depth=<N>

Used to set the maximum supported path depth, default is 35.

--errname

Show error message instead of errno.

## REQUIREMENTS
CONFIG_BPF and bcc.

## EXAMPLES
Trace all open(),openat(),openat2() syscalls:

\# opensnoop.bt

## FIELDS
PID Process ID

COMM

Process name

FD

File descriptor (if success), or -1 (if failed)

ERR

Error number (see the system's errno.h)

PATH

Open path

## OVERHEAD
This traces the open tracepoint and prints output for each event. As the rate of this is generally expected to be low (< 1000/s), the overhead is also expected to be negligible. If you have an application that is calling a high rate of open()s, then test and understand overhead before use.

## SOURCE
This is from bpftrace.

https://github.com/bpftrace/bpftrace

Also look in the bpftrace distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

This is a bpftrace version of the bcc tool of the same name. The bcc tool may provide more options and customizations.

https://github.com/iovisor/bcc

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHORS
Brendan Gregg, Rong Tao <rongtao@cestc.cn>

## SEE ALSO
statsnoop.bt(8), execsnoop.bt(8)


bpftrace-pidpersec.bt

title: bpftrace-pidpersec.bt(8)

text:
# bpftrace-pidpersec.bt(8)

## NAME
pidpersec.bt - Count new processes (via fork()). Uses bpftrace/eBPF.

## SYNOPSIS
pidpersec.bt

## DESCRIPTION
pidpersec shows how many new processes were created each second. There can be performance issues caused by many short-lived processes, which may not be visible in sampling tools like top(1). pidpersec provides one way to investigate this behavior.

This works by tracing the tracepoint:sched:sched_process_fork tracepoint.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bpftrace.

## EXAMPLES
Count new processes, printing per-second summaries until Ctrl-C is hit:

\# pidpersec.bt

## FIELDS
1st

Count of processes (after "@")

## OVERHEAD
This traces kernel forks, and maintains an in-kernel count which is read asynchronously from user-space. As the rate of this is generally expected to be low (<< 1000/s), the overhead is also expected to be negligible.

## SOURCE
This is from bpftrace.

https://github.com/bpftrace/bpftrace

Also look in the bpftrace distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

This is a bpftrace version of the bcc tool of the same name. The bcc tool may provide more options and customizations.

https://github.com/iovisor/bcc

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
top(1)


bpftrace-runqlat.bt

title: bpftrace-runqlat.bt(8)

text:
# bpftrace-runqlat.bt(8)

## NAME
runqlat.bt - CPU scheduler run queue latency as a histogram. Uses bpftrace/eBPF.

## SYNOPSIS
runqlat.bt

## DESCRIPTION
This traces time spent waiting in the CPU scheduler for a turn on-CPU. This metric is often called run queue latency, or scheduler latency. This tool shows this latency as a power-of-2 histogram in nanoseconds, allowing multimodal distributions to be studied, as well as latency outliers.

This tool uses the sched tracepoints.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bpftrace.

## EXAMPLES
Trace CPU run queue latency system wide, printing a histogram on Ctrl-C:

\# runqlat.bt

## FIELDS
1st, 2nd

This is a range of latency, in microseconds (shown in "[...)" set notation).

3rd

A column showing the count of scheduler events in this range.

4th

This is an ASCII histogram representing the count column.

## OVERHEAD
This traces scheduler functions, which can become very frequent. While eBPF has very low overhead, and this tool uses in-kernel maps for efficiency, the frequency of scheduler events for some workloads may be high enough that the overhead of this tool becomes significant. Measure in a lab environment to quantify the overhead before use.

## SOURCE
This is from bpftrace.

https://github.com/bpftrace/bpftrace

Also look in the bpftrace distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

This is a bpftrace version of the bcc tool of the same name. The bcc tool may provide more options and customizations.

https://github.com/iovisor/bcc

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
runqlen.bt(8), mpstat(1), pidstat(1), uptime(1)


bpftrace-runqlen.bt

title: bpftrace-runqlen.bt(8)

text:
# bpftrace-runqlen.bt(8)

## NAME
runqlen.bt - CPU scheduler run queue length as a histogram. Uses bpftrace/eBPF.

## SYNOPSIS
runqlen.bt

## DESCRIPTION
This program summarizes scheduler queue length as a histogram, and can also show run queue occupancy. It works by sampling the run queue length on all CPUs at 99 Hertz.

This tool can be used to identify imbalances, eg, when processes are bound to CPUs causing queueing, or interrupt mappings causing the same.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bpftrace.

## EXAMPLES
Trace CPU run queue length system wide, printing a histogram on Ctrl-C:

\# runqlen.bt

## FIELDS
1st, 2nd

The run queue length is shown in the first field (after "[").

3rd

A column showing the count of samples in for that length.

4th

This is an ASCII histogram representing the count column.

## OVERHEAD
This samples scheduler structs at 99 Hertz across all CPUs. Relatively, this is a low rate of events, and the overhead of this tool is expected to be near zero.

## SOURCE
This is from bpftrace.

https://github.com/bpftrace/bpftrace

Also look in the bpftrace distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

This is a bpftrace version of the bcc tool of the same name. The bcc tool may provide more options and customizations.

https://github.com/iovisor/bcc

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
runqlat.bt(8), mpstat(1), pidstat(1), uptime(1)


bpftrace-setuids.bt

title: bpftrace-setuids.bt(8)

text:
# bpftrace-setuids.bt(8)

## NAME
setuids.bt - Trace setuid family of syscalls. Uses bpftrace/eBPF.

## SYNOPSIS
setuids.bt

## DESCRIPTION
This tool traces privilege escalation via setuid syscalls, and can be used for debugging, whitelist creation, and intrusion detection.

It works by tracing the setuid(2), setfsuid(2), and retresuid(2) syscalls using the syscall tracepoints.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bpftrace.

## EXAMPLES
Trace setuid syscalls:

\# setuids.bt

## FIELDS
PID

The calling process ID.

COMM

The calling process (thread) name.

UID

The UID of the caller.

SYSCALL

The syscall name.

ARGS

The arguments to the syscall

(RET)

The return value for the syscall: 0 == success, other numbers indicate an error code.

## OVERHEAD
setuid calls are expected to be low frequency (<< 100/s), so the overhead of this tool is expected to be negligible.

## SOURCE
This tool originated from the book "BPF Performance Tools", published by Addison Wesley (2019):

http://www.brendangregg.com/bpf-performance-tools-book.html

See the book for more documentation on this tool.

This version is in the bpftrace repository:

https://github.com/bpftrace/bpftrace

Also look in the bpftrace distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
capable.bt(8)


bpftrace-ssllatency.bt

title: bpftrace-ssllatency.bt(8)

text:
# bpftrace-ssllatency.bt(8)

## NAME
ssllatency.bt - Show SSL/TLS handshake latency histogram. Uses bpftrace/eBPF.

## SYNOPSIS
ssllatency.bt

## DESCRIPTION
ssllatency shows latency distribution for OpenSSL handshake functions. This is useful for performance analysis with different crypto cipher suite, async SSL acceleration by CPU or offload device, etc.

This tool works by dynamic tracing the uprobes in OpenSSL and related crypto libs, and may need updating to match future changes to these functions.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bpftrace.

## EXAMPLES
Trace SSL/TLS handshake latency, and print a histogram on Ctrl-C:

\# ssllatency.bt

## FIELDS
0th

A function name is shown in "@hist[...]" followed by latency histogram and "@stat[...]" followed by total call count, average, total latency in microseconds. Non-zero failed calls are traced separately (in "@histF[]" and "@statF[]") for some functions.

1st, 2nd

This is a range of latency, in microseconds (shown in "[...)" set notation).

3rd

A column showing the count of operations in this range.

4th

This is an ASCII histogram representing the count column.

## OVERHEAD
SSL/TLS handshake usually contains network latency and the traced crypto functions are CPU intensive tasks, so call frequency should be low and the overhead of this tool is expected to be negligible.

## SOURCE
This is from bpftrace.

https://github.com/bpftrace/bpftrace

Also look in the bpftrace distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

There is a bcc tool sslsniff that can show SSL/TLS handshake event latency before sniffing the plaintext in SSL_read/write. This tool provides more detailed crypto latency distribution during the handshake event.

https://github.com/iovisor/bcc

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Tao Xu

## SEE ALSO
sslsnoop.bt(8)


bpftrace-sslsnoop.bt

title: bpftrace-sslsnoop.bt(8)

text:
# bpftrace-sslsnoop.bt(8)

## NAME
sslsnoop.bt - Show SSL/TLS handshake events. Uses bpftrace/eBPF.

## SYNOPSIS
sslsnoop.bt

## DESCRIPTION
sslsnoop traces OpenSSL handshake functions, and shows latency and return value. This can be used to analyze SSL/TLS performance.

This tool works by dynamic tracing the uprobes in OpenSSL and related crypto libs, and may need updating to match future changes to these functions.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bpftrace.

## EXAMPLES
Trace SSL/TLS handshake events, printing per-line summaries:

\# sslsnoop.bt

## FIELDS
TIME(us)

Time of the call completion, in microseconds since program start.

TID

Thread ID.

COMM

Process name.

LAT(us)

Latency of the call, in microseconds.

RET

Return value of the call.

FUNC

Function name.

## OVERHEAD
SSL/TLS handshake usually contains network latency and the traced crypto functions are CPU intensive tasks, so call frequency should be low and the overhead of this tool is expected to be negligible.

## SOURCE
This is from bpftrace.

https://github.com/bpftrace/bpftrace

Also look in the bpftrace distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

There is a bcc tool sslsniff that can show SSL/TLS handshake event latency before sniffing the plaintext in SSL_read/write. This tool provides more detailed crypto latency distribution during the handshake event.

https://github.com/iovisor/bcc

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Tao Xu

## SEE ALSO
biosnoop.bt(8)


bpftrace-statsnoop.bt

title: bpftrace-statsnoop.bt(8)

text:
# bpftrace-statsnoop.bt(8)

## NAME
statsnoop.bt - Trace stat() syscalls. Uses bpftrace/eBPF.

## SYNOPSIS
statsnoop.bt

## DESCRIPTION
statsnoop traces the stat() syscall, showing which processes are attempting to stat which files. This can be useful for determining the location of config and log files, or for troubleshooting applications that are failing, specially on startup.

This traces the tracepoints for statfs(), statx(), newstat(), and newlstat(). These aren't the only the stat syscalls: if you are missing activity, you may need to add more variants.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## EXAMPLES
Trace all stat() syscalls:

\# statsnoop.bt

## FIELDS
PID Process ID

COMM

Process name

ERR

Error number (see the system's errno.h)

PATH

Stat path

## OVERHEAD
This traces the stat tracepoints and prints output for each event. As the rate of this is generally expected to be low (< 1000/s), the overhead is also expected to be negligible. If you have an application that is calling a high rate of stat()s, then test and understand overhead before use.

## SOURCE
This is from bpftrace.

https://github.com/bpftrace/bpftrace

Also look in the bpftrace distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

This is a bpftrace version of the bcc tool of the same name. The bcc tool may provide more options and customizations.

https://github.com/iovisor/bcc

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
opensnoop.bt(8), execsnoop.bt(8)


bpftrace-swapin.bt

title: bpftrace-swapin.bt(8)

text:
# bpftrace-swapin.bt(8)

## NAME
swapin.bt - Count swapins by process. Uses bpftrace/eBPF.

## SYNOPSIS
swapin

## DESCRIPTION
This tool counts swapins by process, to show which process is affected by swapping (if swap devices are in use). This can explain a significant source of application latency, if it has began swapping due to memory pressure on the system.

This works by tracing the swap_readpage() kernel function using dynamic instrumentation. This tool may need maintenance to keep working if that function changes in later kernels.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bpftrace.

## EXAMPLES
Count swapins by process, showing per-second summaries.

\# swapin.bt

## FIELDS
1st

The process name.

2nd

The process ID.

3rd

The count of swapins during that interval.

## OVERHEAD
The rate of swapins should be low (bounded by swapin device IOPS), such that the overhead of this tool is expected to be negligible.

## SOURCE
This tool originated from the book "BPF Performance Tools", published by Addison Wesley (2019):

http://www.brendangregg.com/bpf-performance-tools-book.html

See the book for more documentation on this tool.

This version is in the bpftrace repository:

https://github.com/bpftrace/bpftrace

Also look in the bpftrace distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
swapon(8)


bpftrace-syncsnoop.bt

title: bpftrace-syncsnoop.bt(8)

text:
# bpftrace-syncsnoop.bt(8)

## NAME
syncsnoop.bt - Trace the sync() variety of syscalls. Uses bpftrace/eBPF.

## SYNOPSIS
syncsnoop.bt

## DESCRIPTION
syncsnoop traces calls to sync() syscalls (sync(), fsync(), msync(), etc), which flushes file system cache and buffers to storage devices. These calls can cause performance perturbations, and it can be useful to know if they are happening, when they happen, and how frequently.

This works by tracing the sync() variety of syscalls via tracepoints.

This program is also a basic example of eBPF/bcc.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bpftrace.

## EXAMPLES
Trace calls to sync() syscalls:

\# syncsnoop.bt

## FIELDS
TIME

A timestamp on the output, in "HH:MM:SS" format.

PID

The process ID that was on-CPU during the event.

COMM

The process name that was on-CPU during the event.

EVENT

The tracepoint name for the sync event.

## OVERHEAD
This traces sync syscalls and prints output for each event. As the rate of this is generally expected to be low (<< 100/s), the overhead is also expected to be negligible.

## SOURCE
This is from bpftrace.

https://github.com/bpftrace/bpftrace

Also look in the bpftrace distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

This is a bpftrace version of the bcc tool of the same name.

https://github.com/iovisor/bcc

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
iostat(1)


bpftrace-syscount.bt

title: bpftrace-syscount.bt(8)

summary: Count system calls. Uses bpftrace/eBPF.

text:
# bpftrace-syscount.bt(8)

## NAME
syscount.bt - Count system calls. Uses bpftrace/eBPF.

## SYNOPSIS
syscount.bt -- [--sysname]

## DESCRIPTION
This counts system calls (syscalls), printing a summary of the top ten syscall IDs, and the top ten process names making syscalls. This can be helpful for characterizing the kernel and resource workload, and finding applications who are using syscalls inefficiently.

This works by using the tracepoint:raw_syscalls:sys_enter tracepoint.

Since this uses BPF, only the root user can use this tool.

## OPTIONS
--sysname Show the syscall name instead of syscall number.

## REQUIREMENTS
CONFIG_BPF and bpftrace.

## EXAMPLES
Count system calls until Ctrl-C is hit:

\# syscount.bt

## OUTPUT
Top 10 syscalls IDs:

This shows the syscall ID number (in @syscall[]) followed by a count for this syscall during tracing. syscount.bt defaults to outputting the syscall number.

Top 10 syscalls NAMEs:

This shows the syscall ID name (in @syscallname[]) followed by a count for this syscall during tracing. Use the --sysname parameter to output the system call name.

Top 10 processes:

This shows the process name (in @process[]) followed by a count of syscalls during tracing.

## OVERHEAD
For most applications, the overhead should be manageable if they perform 1000's or even 10,000's of syscalls per second. For higher rates, the overhead may become considerable. For example, tracing a microbenchmark loop of 4 million calls to geteuid(), slows it down by 2.4x. However, this represents tracing a workload that has a syscall rate of over 4 million syscalls per second per CPU, which should not be typical (in one large cloud production environment, rates of between 10k and 50k are typical, where the application overhead is expected to be closer to 1%).

For comparison, strace(1) in its current ptrace-based implementation (which it has had for decades) runs the same geteuid() workload 102x slower (that's one hundred and two times slower).

## SOURCE
This is from bpftrace.

https://github.com/bpftrace/bpftrace

Also look in the bpftrace distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

This is a bpftrace version of the bcc tool of the same name. The bcc version provides different command line options, and translates the syscall IDs to their syscall names.

https://github.com/iovisor/bcc

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
strace(1)


bpftrace-tcpaccept.bt

title: bpftrace-tcpaccept.bt(8)

text:
# bpftrace-tcpaccept.bt(8)

## NAME
tcpaccept.bt - Trace TCP passive connections (accept()). Uses bpftrace/eBPF

## SYNOPSIS
tcpaccept.bt

## DESCRIPTION
This tool traces passive TCP connections (eg, via an accept() syscall; connect() are active connections). This can be useful for general troubleshooting to see what new connections the local server is accepting.

This uses dynamic tracing of the kernel inet_csk_accept() socket function (from tcp_prot.accept), and will need to be modified to match kernel changes.

This tool only traces successful TCP accept()s. Connection attempts to closed ports will not be shown (those can be traced via other functions).

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bpftrace.

## EXAMPLES
Trace all passive TCP connections (accept()s):

\# tcpaccept.bt

## FIELDS
TIME(s)

Time of the call, in HH:MM:SS format.

PID

Process ID

COMM

Process name

RADDR

Remote IP address.

RPORT

Remote port.

LADDR

Local IP address.

LPORT

Local port

BL

Current accept backlog vs maximum backlog

## OVERHEAD
This traces the kernel inet_csk_accept function and prints output for each event. The rate of this depends on your server application. If it is a web or proxy server accepting many tens of thousands of connections per second, then the overhead of this tool may be measurable (although, still a lot better than tracing every packet). If it is less than a thousand a second, then the overhead is expected to be negligible. Test and understand this overhead before use.

## SOURCE
This is from bpftrace

https://github.com/bpftrace/bpftrace

Also look in the bpftrace distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

This is a bpftrace version of the bcc tool of the same name. The bcc tool may provide more options and customizations.

https://github.com/iovisor/bcc

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg, adapted for bpftrace by Dale Hamel

## SEE ALSO
tcpconnect.bt(8), funccount(8), tcpdump(8)


bpftrace-tcpconnect.bt

title: bpftrace-tcpconnect.bt(8)

text:
# bpftrace-tcpconnect.bt(8)

## NAME
tcpconnect.bt - Trace TCP active connections (connect()). Uses Linux bpftrace/eBPF

## SYNOPSIS
tcpconnect.bt

## DESCRIPTION
This tool traces active TCP connections (eg, via a connect() syscall; accept() are passive connections). This can be useful for general troubleshooting to see what connections are initiated by the local server.

All connection attempts are traced, even if they ultimately fail.

This works by tracing the kernel tcp_v4_connect() and tcp_v6_connect() functions using dynamic tracing, and will need updating to match any changes to these functions.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bpftrace.

## EXAMPLES
Trace all active TCP connections:

\# tcpconnect.bt

## FIELDS
TIME(s)

Time of the call, in HH:MM:SS format.

PID

Process ID

COMM

Process name

SADDR

Source IP address.

SPORT

Source port.

DADDR

Destination IP address.

DPORT

Destination port

## OVERHEAD
This traces the kernel tcp_v[46]_connect functions and prints output for each event. As the rate of this is generally expected to be low (< 1000/s), the overhead is also expected to be negligible. If you have an application that is calling a high rate of connects()s, such as a proxy server, then test and understand this overhead before use.

## SOURCE
This is from bpftrace

https://github.com/bpftrace/bpftrace

Also look in the bpftrace distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

This is a bpftrace version of the bcc tool of the same name. The bcc tool may provide more options and customizations.

https://github.com/iovisor/bcc

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg, adapted for bpftrace by Dale Hamel

## SEE ALSO
tcpaccept.bt(8), funccount(8), tcpdump(8)


bpftrace-tcpdrop.bt

title: bpftrace-tcpdrop.bt(8)

text:
# bpftrace-tcpdrop.bt(8)

## NAME
tcpdrop.bt - Trace kernel-based TCP packet drops with details. Uses Linux bpftrace/eBPF

## SYNOPSIS
tcpdrop.bt

## DESCRIPTION
This tool traces TCP packets or segments that were dropped by the kernel, and shows details from the IP and TCP headers, the socket state, and the kernel stack trace. This is useful for debugging cases of high kernel drops, which can cause timer-based retransmits and performance issues.

This tool works using dynamic tracing of the tcp_drop() kernel function, which requires a recent kernel version.

This tool is limited to ipv4, and cannot parse tcpflags as bpftrace currently cannot parse socket buffers in the way that bcc can.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bpftrace.

## EXAMPLES
Trace all tcp drops:

\# tcpdrop.bt

## FIELDS
TIME

Time of the call, in HH:MM:SS format.

PID

Process ID that was on-CPU during the drop. This may be unrelated, as drops can occur on the receive interrupt and be unrelated to the PID that was interrupted.

COMM

Process name

SADDR:SPORT

Source IP address and source TCP port.

DADDR:DPORT

Destination IP address and destination TCP port.

STATE

TCP session state ("ESTABLISHED", etc).

## OVERHEAD
This traces the kernel tcp_drop() function, which should be low frequency, and therefore the overhead of this tool should be negligible.

As always, test and understand this tools overhead for your types of workloads before production use.

## SOURCE
This is from bpftrace

https://github.com/bpftrace/bpftrace

Also look in the bpftrace distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

This is a bpftrace version of the bcc tool of the same name. The bcc tool may provide more options and customizations.

https://github.com/iovisor/bcc

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg, adapted for bpftrace by Dale Hamel

## SEE ALSO
tcplife.bt(8), tcpaccept.bt(8), tcpconnect.bt(8), tcptop(8)


bpftrace-tcplife.bt

title: bpftrace-tcplife.bt(8)

text:
# bpftrace-tcplife.bt(8)

## NAME
tcplife.bt - Trace TCP session lifespans with connection details. Uses bpftrace/eBPF.

## SYNOPSIS
tcplife

## DESCRIPTION
This tool shows the lifespan of TCP sessions that open and close while tracing, and shows the duration and throughput statistics. For efficiency, this tool only instruments TCP state changes, rather than all packets.

This tool works by using the sock:inet_sock_set_state tracepoint, which was added in Linux 4.16.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF, bpftrace, and the sock:inet_sock_set_state tracepoint (Linux 4.16+).

## EXAMPLES
Show TCP sessions with details:

\# tcplife.bt

## FIELDS
PID

Process ID

COMM

Process name

LADDR

Local IP address.

LPORT

Local port.

RADDR

Remote IP address.

RPORT

Remote port.

TX_KB

Total transmitted Kbytes.

RX_KB

Total received Kbytes.

MS

Lifespan of the session, in milliseconds.

## OVERHEAD
This traces the kernel TCP set state function, which should be called much less often than send/receive tracing, and therefore have lower overhead. The overhead of the tool is relative to the rate of new TCP sessions: if this is high, over 10,000 per second, then there may be noticeable overhead just to print out 10k lines of formatted output per second.

You can find out the rate of new TCP sessions using "sar -n TCP 1", and adding the active/s and passive/s columns.

As always, test and understand this tools overhead for your types of workloads before production use.

## SOURCE
This tool originated from BCC:

https://github.com/iovisor/bcc

The BCC version has many command line options for customizing the output.

This bpftrace version originated from the book "BPF Performance Tools", published by Addison Wesley (2019):

http://www.brendangregg.com/bpf-performance-tools-book.html

See the book for more documentation on this tool.

This bpftrace version is in the bpftrace repository:

https://github.com/bpftrace/bpftrace

Also look in the bpftrace distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
tcptop(8)


bpftrace-tcpretrans.bt

title: bpftrace-tcpretrans.bt(8)

text:
# bpftrace-tcpretrans.bt(8)

## NAME
tcpretrans.bt - Trace or count TCP retransmits. Uses Linux bpftrace/eBPF

## SYNOPSIS
tcpretrans.bt

## DESCRIPTION
This traces TCP retransmits, showing address, port, and TCP state information, and sometimes the PID (although usually not, since retransmits are usually sent by the kernel on timeouts). To keep overhead very low, only the TCP retransmit functions are traced. This does not trace every packet (like tcpdump(8) or a packet sniffer). Optionally, it can count retransmits over a user signalled interval to spot potentially dropping network paths the flows are traversing.

This uses dynamic tracing of the kernel tcp_retransmit_skb() and tcp_send_loss_probe() functions, and will need to be updated to match kernel changes to these functions.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc. CONFIG_BPF and bpftrace.

## EXAMPLES
Trace TCP retransmits:

\# tcpretrans.bt

## FIELDS
TIME

Time of the call, in HH:MM:SS format.

PID

Process ID that was on-CPU. This is less useful than it might sound, as it may usually be 0, for the kernel, for timer-based retransmits.

LADDR:LPORT

Local IP address and local port.

RADDR:RPORT

Remote IP address and remote port.

STATE

TCP session state.

## OVERHEAD
Should be negligible: TCP retransmit events should be low (<1000/s), and the low overhead this tool adds to each event should make the cost negligible.

## SOURCE
This is from bpftrace

https://github.com/bpftrace/bpftrace

Also look in the bpftrace distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

This is a bpftrace version of the bcc tool of the same name. The bcc tool may provide more options and customizations.

https://github.com/iovisor/bcc

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg, adapted for bpftrace by Dale Hamel

## SEE ALSO
tcpconnect.bt(8), tcpaccept.bt(8)


bpftrace-tcpsynbl.bt

title: bpftrace-tcpsynbl.bt(8)

text:
# bpftrace-tcpsynbl.bt(8)

## NAME
tcpsynbl.bt - Show the TCP SYN backlog as a histogram. Uses bpftrace/eBPF.

## SYNOPSIS
tcpsynbl

## DESCRIPTION
This tool shows the TCP SYN backlog size during SYN arrival as a histogram. This lets you see how close your applications are to hitting the backlog limit and dropping SYNs (causing performance issues with SYN retransmits), and is a measure of workload saturation. The histogram shown is measured at the time of SYN received, and a separate histogram is shown for each backlog limit.

This works by tracing the tcp_v4_syn_recv_sock() and tcp_v6_syn_recv_sock() kernel functions using dynamic instrumentation. Since these functions may change in future kernels, this tool may need maintenance to keep working.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bpftrace.

## EXAMPLES
Show the TCP SYN backlog as a histogram.

\# tcpsynbl.bt

## FIELDS
backlog

The backlog size when a SYN was received.

count

The number of times this backlog size was encountered.

distribution

An ASCII visualization of the count column.

## OVERHEAD
Inbound SYNs should be relatively low compared to packets and other events, so the overhead of this tool is expected to be negligible.

## SOURCE
This tool originated from the book "BPF Performance Tools", published by Addison Wesley (2019):

http://www.brendangregg.com/bpf-performance-tools-book.html

See the book for more documentation on this tool.

This version is in the bpftrace repository:

https://github.com/bpftrace/bpftrace

Also look in the bpftrace distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
tcptop(8)


bpftrace-threadsnoop.bt

title: bpftrace-threadsnoop.bt(8)

text:
# bpftrace-threadsnoop.bt(8)

## NAME
threadsnoop.bt - Trace thread creation via pthread_create(). Uses bpftrace/eBPF.

## SYNOPSIS
threadsnoop.bt

## DESCRIPTION
threadsnoop traces calls to pthread_create(), showing this path of thread creation. This can be used for workload characterization and discovery, and is a companion to execsnoop.bt(8) which traces execve(2).

This works by tracing the pthread_create() from libpthread.so.0. The path to this library may need adjusting in the tool source to match your system.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bpftrace.

## EXAMPLES
Trace calls pthread_create():

\# threadsnoop.bt

## FIELDS
TIME(ms)

Elapsed time since the tool began tracing (in milliseconds).

PID

The process ID.

COMM

The process (thread) name.

FUNC

The name of the start routine, if the symbol is available, else a hex address for the start routine address.

## OVERHEAD
Thread creation is expected to be low (<< 1000/s), so the overhead of this tool is expected to be negligible.

## SOURCE
This tool originated from the book "BPF Performance Tools", published by Addison Wesley (2019):

http://www.brendangregg.com/bpf-performance-tools-book.html

See the book for more documentation on this tool.

This version is in the bpftrace repository:

https://github.com/bpftrace/bpftrace

Also look in the bpftrace distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
execsnoop.bt(8)


bpftrace-undump.bt

title: bpftrace-undump.bt(8)

text:
# bpftrace-undump.bt(8)

## NAME
undump.bt - Catch UNIX domain socket packages. Uses bpftrace/eBPF.

## SYNOPSIS
undump.bt

## DESCRIPTION
undump.bt tracked reception of UNIX domain sockets.

This program is also a basic example of bpftrace and kprobes.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bpftrace.

## EXAMPLES
Trace reception of UNIX domain sockets:

\# undump.bt

## FIELDS
TIME

A timestamp on the output, in "HH:MM:SS" format.

COMM

The process COMM.

PID

The process ID.

SIZE

The size of the received packet, in bytes.

DATA

Display received packets in hex or string.

## OVERHEAD
The overhead of this program mainly comes from the data packets received by the terminal output.

## SOURCE
This is from bpftrace.

https://github.com/bpftrace/bpftrace

Also look in the bpftrace distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

This is a bpftrace version of the bcc examples/tracing of the same name. The bcc tool may provide more options and customizations.

https://github.com/iovisor/bcc

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Rong Tao

## SEE ALSO
opensnoop.bt(8)


bpftrace-vfscount.bt

title: bpftrace-vfscount.bt(8)

summary: Count VFS calls (vfs_*). Uses bpftrace/eBPF.

text:
# bpftrace-vfscount.bt(8)

## NAME
vfscount.bt - Count VFS calls ("vfs_*"). Uses bpftrace/eBPF.

## SYNOPSIS
vfscount.bt

## DESCRIPTION
This counts VFS calls. This can be useful for general workload characterization of these operations.

This works by tracing all kernel functions beginning with "vfs_" using dynamic tracing. This may match more functions than you are interested in measuring: Edit the script to customize which functions to trace.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bpftrace.

## EXAMPLES
Count all VFS calls until Ctrl-C is hit:

\# vfscount.bt

## FIELDS
1st

Kernel function name (in @[])

2nd

Number of calls while tracing

## OVERHEAD
This traces kernel vfs functions and maintains in-kernel counts, which are asynchronously copied to user-space. While the rate of VFS operations can be very high (>1M/sec), this is a relatively efficient way to trace these events, and so the overhead is expected to be small for normal workloads. Measure in a test environment, and if overheads are an issue, edit the script to reduce the types of vfs functions traced (currently all beginning with "vfs_").

## SOURCE
This is from bpftrace.

https://github.com/bpftrace/bpftrace

Also look in the bpftrace distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

This is a bpftrace version of the bcc tool of the same name. The bcc tool may provide more options and customizations.

https://github.com/iovisor/bcc

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
vfsstat.bt(8)


bpftrace-vfsstat.bt

title: bpftrace-vfsstat.bt(8)

text:
# bpftrace-vfsstat.bt(8)

## NAME
vfsstat.bt - Count key VFS calls. Uses bpftrace/eBPF.

## SYNOPSIS
vfsstat.bt

## DESCRIPTION
This traces some common VFS calls and prints per-second summaries. This can be useful for general workload characterization, and looking for patterns in operation usage over time.

This works by tracing some kernel vfs functions using dynamic tracing, and will need updating to match any changes to these functions. Edit the script to customize which functions are traced. Also see vfscount, which is more easily customized to trace multiple functions.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bpftrace.

## EXAMPLES
Count some VFS calls, printing per-second summaries until Ctrl-C is hit:

\# vfsstat.bt

## FIELDS
HH:MM:SS

Each output summary is prefixed by the time of printing in "HH:MM:SS" format.

1st

Kernel function name (in @[])

2nd

Number of calls while tracing

## OVERHEAD
This traces various kernel vfs functions and maintains in-kernel counts, which are asynchronously copied to user-space. While the rate of VFS operations can be very high (>1M/sec), this is a relatively efficient way to trace these events, and so the overhead is expected to be small for normal workloads. Measure in a test environment.

## SOURCE
This is from bpftrace.

https://github.com/bpftrace/bpftrace

Also look in the bpftrace distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

This is a bpftrace version of the bcc tool of the same name. The bcc tool may provide more options and customizations.

https://github.com/iovisor/bcc

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
vfscount.bt(8)


bpftrace-writeback.bt

title: bpftrace-writeback.bt(8)

text:
# bpftrace-writeback.bt(8)

## NAME
writeback.bt - Trace file system writeback events with details. Uses bpftrace/eBPF.

## SYNOPSIS
writeback.bt

## DESCRIPTION
This traces when file system dirtied pages are flushed to disk by kernel writeback, and prints details including when the event occurred, and the duration of the event. This can be useful for correlating these times with other performance problems, and if there is a match, it would be a clue that the problem may be caused by writeback. How quickly the kernel does writeback can be tuned: see the kernel docs, eg, vm.dirty_writeback_centisecs.

This uses the tracepoint:writeback:writeback_start and tracepoint:writeback:writeback_written tracepoints.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bpftrace.

## EXAMPLES
Trace all writeback events with timestamps and latency details:

\# writeback.bt

## FIELDS
TIME

Time that the writeback event completed, in %H:%M:%S format.

DEVICE

Device name in major:minor number format.

PAGES

Pages written during writeback.

REASON

Reason for the writeback event. This may be "background", "vmscan", "sync", "periodic", etc.

ms

Duration of the writeback event in milliseconds.

## OVERHEAD
Since writeback events are expected to be infrequent (<10/sec), the overhead of this tool is expected to be negligible (near 0%).

## SOURCE
This is from bpftrace.

https://github.com/bpftrace/bpftrace

Also look in the bpftrace distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
biosnoop.bt(8)


bpftrace-xfsdist.bt

title: bpftrace-xfsdist.bt(8)

text:
# bpftrace-xfsdist.bt(8)

## NAME
xfsdist.bt - Summarize XFS operation latency. Uses bpftrace/eBPF.

## SYNOPSIS
xfsdist.bt

## DESCRIPTION
This tool summarizes time (latency) spent in common XFS file operations: reads, writes, opens, and syncs, and presents it as a power-of-2 histogram. It uses an in-kernel eBPF map to store the histogram for efficiency.

Since this works by tracing the xfs_file_operations interface functions, it will need updating to match any changes to these functions.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bpftrace.

## EXAMPLES
Trace XFS operation time, and print a summary on Ctrl-C:

\# xfsdist.bt

## FIELDS
0th

The operation name (shown in "@[...]") is printed before each I/O histogram.

1st, 2nd

This is a range of latency, in microseconds (shown in "[...)" set notation).

3rd

A column showing the count of operations in this range.

4th

This is an ASCII histogram representing the count column.

## OVERHEAD
This adds low-overhead instrumentation to these XFS operations, including reads and writes from the file system cache. Such reads and writes can be very frequent (depending on the workload; eg, 1M/sec), at which point the overhead of this tool may become noticeable. Measure and quantify before use.

## SOURCE
This is from bpftrace.

https://github.com/bpftrace/bpftrace

Also look in the bpftrace distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

This is a bpftrace version of the bcc tool of the same name. The bcc tool may provide more options and customizations.

https://github.com/iovisor/bcc

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
biolatency.bt(8)


compsize

package: btrfs-compsize

title: compsize(8)

text:
# compsize(8)

## NAME
compsize - calculate compression ratio of a set of files on btrfs

## SYNOPSIS
compsize file-or-dir [ file-or-dir ... ]

## DESCRIPTION
compsize takes a list of files on a btrfs filesystem (recursing directories) and measures used compression types and the effective compression ratio. Besides compression, compsize shows the effect of reflinks (cp --reflink, snapshots, deduplication), and certain types of btrfs waste.

The program gives a report similar to:

Processed 90319 files.

Type Perc Disk Usage Uncompressed Referenced

TOTAL 79% 1.4G 1.8G 1.9G

none 100% 1.0G 1.0G 1.0G

lzo 53% 446M 833M 843M

The fields above are:

Type

compression algorithm

Perc

disk usage/uncompressed (compression ratio)

Disk Usage

blocks on the disk; this is what storing these files actually costs you (save for RAID considerations)

Uncompressed

uncompressed extents; what you would need without compression - includes deduplication savings and pinned extent waste

Referenced

apparent file sizes (sans holes); this is what a traditional filesystem that supports holes and efficient tail packing, or tar -S, would need to store these files

Let's see this on an example: a file 128K big is stored as a single extent A which compressed to a single 4K page. It then receives a write of 32K at offset 32K, which also compressed to a single 4K page, stored as extent B.

The file now appears as:

+-------+-------+---------------+

extent A | used | waste | used |

+-------+-------+---------------+

extent B | used |

+-------+

The "waste" inside extent A can't be gotten rid until the whole extent is rewritten (for example by defrag). If compressed, the whole extent needs to be read every time that part of the file is being read, thus the "waste" is still required.

In this case, we have: Disk Usage: 8KB, Uncompressed: 160K, Referenced: 128K.

## OPTIONS
-b/--bytes

Show raw byte counts rather than human-friendly sizes.

-x/--one-file-system

Skip files and directories on different file systems.

## SIGNALS
USR1

Displays partial data for files processed so far.

## CAVEATS
Recently written files may show as not taking any space until they're actually allocated and compressed; this happens once they're synced or on natural writeout, typically on the order of 30 seconds.

The ioctls used by this program require root.

Inline extents are considered to be always unique, even if they share the same bytes on the disk.

This program doesn't currently support filesystems above 8TB on 32-bit machines but neither do other btrfs tools.


btrfs

package: btrfs-progs

title: btrfs(8)

summary: a toolbox to manage btrfs filesystems

text:
# btrfs(8)

## NAME
btrfs - a toolbox to manage btrfs filesystems

## SYNOPSIS
btrfs [global options] <group> [<group>...] <command> [options] [<args>]

## DESCRIPTION
The btrfs utility is a toolbox for managing btrfs filesystems. There are command groups to work with subvolumes, devices, for whole filesystem or other specific actions. See section COMMANDS.

There are also standalone tools for some tasks like btrfs-convert(8) <> or btrfstune(8) <> that were separate historically and/or haven't been merged to the main utility. See section STANDALONE TOOLS for more details.

For other topics (mount options, etc) please refer to the separate manual page btrfs(5) <>.

## COMMAND SYNTAX
Any command name can be shortened so long as the shortened form is unambiguous, however, it is recommended to use full command names in scripts. All command groups have their manual page named btrfs-<group>.

For example: it is possible to run btrfs sub snaps instead of btrfs subvolume snapshot. But btrfs file s is not allowed, because file s may be interpreted both as filesystem show and as filesystem sync.

If the command name is ambiguous, the list of conflicting options is printed.

Sizes, both upon input and output, can be expressed in either SI or IEC-I units (see numfmt(1) <https://man7.org/linux/man-pages/man1/numfmt.1.html>) with the suffix B appended. All numbers will be formatted according to the rules of the C locale (ignoring the shell locale, see locale(7) <https://man7.org/linux/man-pages/man7/locale.7.html>).

For an overview of a given command use btrfs command --help or btrfs [command...] help --full to print all available options for all commands.

There are global options that are passed between btrfs and the group name and affect behaviour not specific to the command, e.g. verbosity or the type of the output:

```
btrfs -q subvolume create ...
btrfs --dry-run subvolume create ...
```

--format <format>

if supported by the command, print subcommand output in that format (text, json)

-v|--verbose

increase verbosity of the subcommand

-q|--quiet

print only errors

--log <level>

set log level (default, info, verbose, debug, quiet)

The remaining options are relevant only for the main tool:

--help

print condensed help for all subcommands

--version

print version string

## COMMANDS
balance

Balance btrfs filesystem chunks across single or several devices. See btrfs-balance(8) <> for details.

check

Do off-line check on a btrfs filesystem. See btrfs-check(8) <> for details.

device

Manage devices managed by btrfs, including add/delete/scan and so on. See btrfs-device(8) <> for details.

filesystem

Manage a btrfs filesystem, including label setting/sync and so on. See btrfs-filesystem(8) <> for details.

inspect-internal

Debug tools for developers/hackers. See btrfs-inspect-internal(8) <> for details.

property

Get/set a property from/to a btrfs object. See btrfs-property(8) <> for details.

qgroup

Manage quota group(qgroup) for btrfs filesystem. See btrfs-qgroup(8) <> for details.

quota

Manage quota on btrfs filesystem like enabling/rescan and etc. See btrfs-quota(8) <> and btrfs-qgroup(8) <> for details.

receive

Receive subvolume data from stdin/file for restore and etc. See btrfs-receive(8) <> for details.

replace

Replace btrfs devices. See btrfs-replace(8) <> for details.

rescue

Try to rescue damaged btrfs filesystem. See btrfs-rescue(8) <> for details.

restore

Try to restore files from a damaged btrfs filesystem. See btrfs-restore(8) <> for details.

scrub

Scrub a btrfs filesystem. See btrfs-scrub(8) <> for details.

send

Send subvolume data to stdout/file for backup and etc. See btrfs-send(8) <> for details.

subvolume

Create/delete/list/manage btrfs subvolume. See btrfs-subvolume(8) <> for details.

## STANDALONE TOOLS
New functionality could be provided using a standalone tool. If the functionality proves to be useful, then the standalone tool is declared obsolete and its functionality is copied to the main tool. Obsolete tools are removed after a long (years) depreciation period.

Tools that are still in active use without an equivalent in btrfs:

btrfs-convert

in-place conversion from ext2/3/4 filesystems to btrfs

btrfstune

tweak some filesystem properties on a unmounted filesystem

btrfs-select-super

rescue tool to overwrite primary superblock from a spare copy

btrfs-find-root

rescue helper to find tree roots in a filesystem

For space-constrained environments, it's possible to build a single binary with functionality of several standalone tools. This is following the concept of busybox where the file name selects the functionality. This works for symlinks or hardlinks. The full list can be obtained by btrfs help --box.

## EXIT STATUS
btrfs returns a zero exit status if it succeeds. Non zero is returned in case of failure.

## AVAILABILITY
btrfs is part of btrfs-progs. Please refer to the documentation at <https://btrfs.readthedocs.io>.

## SEE ALSO
btrfs(5) <>, btrfs-balance(8) <>, btrfs-check(8) <>, btrfs-convert(8) <>, btrfs-device(8) <>, btrfs-filesystem(8) <>, btrfs-inspect-internal(8) <>, btrfs-property(8) <>, btrfs-qgroup(8) <>, btrfs-quota(8) <>, btrfs-receive(8) <>, btrfs-replace(8) <>, btrfs-rescue(8) <>, btrfs-restore(8) <>, btrfs-scrub(8) <>, btrfs-send(8) <>, btrfs-subvolume(8) <>, btrfstune(8) <>, mkfs.btrfs(8) <>