bcc-biotop

title: bcc-biotop(8)

summary: Block device (disk) I/O by process top.

text:
# bcc-biotop(8)

## NAME
biotop - Block device (disk) I/O by process top.

## SYNOPSIS
biotop [-h] [-C] [-r MAXROWS] [-p PID] [interval] [count]

## DESCRIPTION
This is top for disks.

This traces block device I/O (disk I/O), and prints a per-process summary every interval (by default, 1 second). The summary is sorted on the top disk consumers by throughput (Kbytes). The PID and process name shown are measured from when the I/O was first created, which usually identifies the responsible process.

For efficiency, this uses in-kernel eBPF maps to cache process details (PID and comm) by I/O request, as well as a starting timestamp for calculating I/O latency, and the final summary.

This works by tracing various kernel blk_*() functions using dynamic tracing, and will need updating to match any changes to these functions.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-C

Don't clear the screen.

-r MAXROWS

Maximum number of rows to print. Default is 20.

-p PID

Trace this PID only.

interval

Interval between updates, seconds.

count

Number of interval summaries.

## EXAMPLES
Summarize block device I/O by process, 1 second screen refresh:

\# biotop

Don't clear the screen:

\# biotop -C

5 second summaries, 10 times only:

\# biotop 5 10

## FIELDS
loadavg:

The contents of /proc/loadavg

PID

Cached process ID, if present. This usually (but isn't guaranteed) to identify the responsible process for the I/O.

COMM

Cached process name, if present. This usually (but isn't guaranteed) to identify the responsible process for the I/O.

D

Direction: R == read, W == write. This is a simplification.

MAJ

Major device number.

MIN

Minor device number.

DISK

Disk device name.

I/O

Number of I/O during the interval.

Kbytes

Total Kbytes for these I/O, during the interval.

AVGms

Average time for the I/O (latency) from the issue to the device, to its completion, in milliseconds.

## OVERHEAD
Since block device I/O usually has a relatively low frequency (< 10,000/s), the overhead for this tool is expected to be low or negligible. For high IOPS storage systems, test and quantify before use.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg, Rocky Xing

## INSPIRATION
top(1) by William LeFebvre

## SEE ALSO
biosnoop(8), biolatency(8), iostat(1)


bcc-bitesize

package: bcc-tools

title: bcc-bitesize(8)

summary: Summarize block device I/O size as a histogram \- Linux eBPF/bcc.

text:
# bcc-bitesize(8)

## NAME
bitesize - Summarize block device I/O size as a histogram - Linux eBPF/bcc.

## SYNOPSIS
bitesize

## DESCRIPTION
Show I/O distribution for requested block sizes, by process name.

This works by tracing block:block_rq_issue and prints a histogram of I/O size.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## EXAMPLES
Count I/O size per process until Ctrl-C is hit:

\# bitesize

## FIELDS
Kbtes

Size in kilobytes of range

count

How many I/O fell into this range

distribution

An ASCII bar chart to visualize the distribution (count column)

## OVERHEAD
This traces a block I/O tracepoint to update a histogram, which is asynchronously copied to user-space. This method is very efficient, and the overhead for most storage I/O rates (< 10k IOPS) should be negligible. If you have a higher IOPS storage environment, test and quantify the overhead before use.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Allan McAleavy

## SEE ALSO
https://github.com/brendangregg/systemtap-lwtools/blob/master/disk/bitesize-nd.stp

branch: sisyphus


bcc-bpflist

title: bcc-bpflist(8)

text:
# bcc-bpflist(8)

## NAME
bpflist - Display processes currently using BPF programs and maps.

## SYNOPSIS
bpflist [-v]

## DESCRIPTION
This tool displays processes currently using BPF programs and maps, and optionally also kprobes and uprobes on the system. This is useful to understand which BPF programs are loaded on the system.

Currently, for lack of a better alternative, this tool pipes into 'ls' and parses its output to snoop for BPF file descriptors in all running processes. In the future, when BPF accounting is provided by the kernel, this tool should use these accounting features.

Only the root user can use this tool, because it accesses debugfs.

## REQUIREMENTS
bcc, debugfs

## OPTIONS
-h Print usage message.

-v

Count kprobes and uprobes as well as BPF programs. Repeating verbose mode twice also prints the kprobe and uprobe definitions in addition to counting them.

## EXAMPLES
Display processes currently using BPF programs:

\# bpflist

Also count kprobes and uprobes:

\# bpflist -v

## FIELDS
PID

Process ID.

COMM

Process comm.

TYPE

The type of the data displayed: BPF program, BPF map, kprobe, or uprobe.

COUNT

The number of items of this type that belong to the specified process.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein


bcc-bps

title: bcc-bps(8)

text:
# bcc-bps(8)

## NAME
bps - List all BPF programs. 'ps' for BPF programs.

## SYNOPSIS
bps [bpf-prog-id]

## DESCRIPTION
bps lists all BPF programs loaded into the kernel. It is similar to the ps command but for the BPF programs.

Each loaded bpf program is identified by an unique integer (i.e. bpf-prog-id or simply BID). If a bpf-prog-id is specified, the maps used by bpf-prog-id will also be listed.

## EXAMPLES
List all BPF programs loaded into the kernel:

bps

Show the details and maps of BID 6:

bps 6

## BPF PROGRAM FIELDS
BID

BPF program ID. It ends with '-' if it is not jitted.

TYPE

The type of a BPF program. e.g. kprobe, tracepoint, xdp...etc.

UID

The user ID that loaded the BPF program.

\#MAPS

Total number of maps used by a BPF program.

LoadTime

When was the BPF program loaded?

NAME

The name of a BPF program. The user space library (like bcc ) usually uses the C function name of the original BPF's source code as the program name. It could be empty if the user space did not provide a name.

## BPF MAP FIELDS
MID

BPF map ID.

TYPE

The type of a BPF map. e.g. hash, array, stack trace...etc.

FLAGS

The flags used to create the BP map.

KeySz

The key size of a BPF map.

ValueSz

The value size of a BPF map.

MaxEnts

The maximum number of entries of a map.

NAME

The name of a BPF map. The user space library (like bcc ) usually uses the C variable name of the BPF map as its name. It could be empty if the user space did not provide a name.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Martin Lau


bcc-btrfsdist

title: bcc-btrfsdist(8)

text:
# bcc-btrfsdist(8)

## NAME
btrfsdist - Summarize btrfs operation latency. Uses Linux eBPF/bcc.

## SYNOPSIS
btrfsdist [-h] [-T] [-N] [-d] [interval] [count]

## DESCRIPTION
This tool summarizes time (latency) spent in common btrfs file operations: reads, writes, opens, and syncs, and presents it as a power-of-2 histogram. It uses an in-kernel eBPF map to store the histogram for efficiency.

Since this works by tracing the btrfs_file_operations interface functions, it will need updating to match any changes to these functions.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print usage message.

-T

Don't include timestamps on interval output.

-m

Output in milliseconds.

-p PID

Trace this PID only.

## EXAMPLES
Trace btrfs operation time, and print a summary on Ctrl-C:

\# btrfsdist

Trace PID 181 only:

\# btrfsdist -p 181

Print 1 second summaries, 10 times:

\# btrfsdist 1 10

1 second summaries, printed in milliseconds

\# btrfsdist -m 1

## FIELDS
msecs

Range of milliseconds for this bucket.

usecs

Range of microseconds for this bucket.

count

Number of operations in this time range.

distribution

ASCII representation of the distribution (the count column).

## OVERHEAD
This adds low-overhead instrumentation to btrfs writes and fsyncs, as well as all system reads and opens (due to the current implementation of the btrfs_file_operations interface). Particularly, all reads and writes from the file system cache will incur extra overhead while tracing. Such reads and writes can be very frequent (depending on the workload; eg, 1M/sec), at which point the overhead of this tool may become noticeable. Measure and quantify before use.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
btrfsslower(8)


bcc-btrfsslower

title: bcc-btrfsslower(8)

text:
# bcc-btrfsslower(8)

## NAME
btrfsslower - Trace slow btrfs file operations, with per-event details.

## SYNOPSIS
btrfsslower [-h] [-j] [-p PID] [min_ms] [-d DURATION]

## DESCRIPTION
This tool traces common btrfs file operations: reads, writes, opens, and syncs. It measures the time spent in these operations, and prints details for each that exceeded a threshold.

WARNING: See the OVERHEAD section.

By default, a minimum millisecond threshold of 10 is used. If a threshold of 0 is used, all events are printed (warning: verbose).

Since this works by tracing the btrfs_file_operations interface functions, it will need updating to match any changes to these functions.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-p PID Trace this PID only.

min_ms

Minimum I/O latency (duration) to trace, in milliseconds. Default is 10 ms.

-d DURATION

Total duration of trace in seconds.

## EXAMPLES
Trace synchronous file reads and writes slower than 10 ms:

\# btrfsslower

Trace slower than 1 ms:

\# btrfsslower 1

Trace slower than 1 ms, and output just the fields in parsable format (csv):

\# btrfsslower -j 1

Trace all file reads and writes (warning: the output will be verbose):

\# btrfsslower 0

Trace slower than 1 ms, for PID 181 only:

\# btrfsslower -p 181 1

Trace for 10 seconds only:

\# btrfsslower -d 10

## FIELDS
TIME(s)

Time of I/O completion since the first I/O seen, in seconds.

COMM

Process name.

PID

Process ID.

T

Type of operation. R == read, W == write, O == open, S == fsync.

OFF_KB

File offset for the I/O, in Kbytes.

BYTES

Size of I/O, in bytes.

LAT(ms)

Latency (duration) of I/O, measured from when it was issued by VFS to the filesystem, to when it completed. This time is inclusive of block device I/O, file system CPU cycles, file system locks, run queue latency, etc. It's a more accurate measure of the latency suffered by applications performing file system I/O, than to measure this down at the block device interface.

FILENAME

A cached kernel file name (comes from dentry->d_name.name).

ENDTIME_us

Completion timestamp, microseconds (-j only).

OFFSET_b

File offset, bytes (-j only).

LATENCY_us

Latency (duration) of the I/O, in microseconds (-j only).

## OVERHEAD
This adds low-overhead instrumentation to btrfs writes and fsyncs, as well as all system reads and opens (due to the current implementation of the btrfs_file_operations interface). Particularly, all reads and writes from the file system cache will incur extra overhead while tracing. Such reads and writes can be very frequent (depending on the workload; eg, 1M/sec), at which point the overhead of this tool may become noticeable. Measure and quantify before use. If this continues to be a problem, consider switching to a tool that prints in-kernel summaries only, such as btrfsdist(8).

Note that the overhead of this tool should be less than fileslower(8), as this tool targets btrfs functions only, and not all file read/write paths (which can include socket I/O).

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
btrfsdist(8), biosnoop(8), funccount(8), fileslower(8)


bcc-cachestat

title: bcc-cachestat(8)

text:
# bcc-cachestat(8)

## NAME
cachestat - Statistics for linux page cache hit/miss ratios. Uses Linux eBPF/bcc.

## SYNOPSIS
cachestat [-T] [interval [count]]

## DESCRIPTION
This traces four kernel functions and prints per-second summaries. This can be useful for general workload characterization, and looking for patterns in operation usage over time.

This works by tracing kernel page cache functions using dynamic tracing, and will need updating to match any changes to these functions. Edit the script to customize which functions are traced.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## EXAMPLES
Print summaries every second:

\# cachestat

Print summaries every second with timestamp:

\# cachestat -T

Print output every five seconds, three times:

\# cachestat 5 3

Print output with timestamp every five seconds, three times:

\# cachestat -T 5 3

## FIELDS
TIME

Timestamp.

HITS

Number of page cache hits.

MISSES

Number of page cache misses.

DIRTIES

Number of dirty pages added to the page cache.

HITRATIO

The hit ratio as a percentage.

READ_HIT%

Read hit percent of page cache usage.

WRITE_HIT%

Write hit percent of page cache usage.

BUFFERS_MB

Buffers size taken from /proc/meminfo.

CACHED_MB

Cached amount of data in current page cache taken from /proc/meminfo.

## OVERHEAD
This traces various kernel page cache functions and maintains in-kernel counts, which are asynchronously copied to user-space. While the rate of operations can be very high (>1G/sec) we can have up to 34% overhead, this is still a relatively efficient way to trace these events, and so the overhead is expected to be small for normal workloads. Measure in a test environment.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Allan McAleavy

## SEE ALSO
https://github.com/brendangregg/perf-tools/blob/master/fs/cachestat


bcc-cachetop

title: bcc-cachetop(8)

text:
# bcc-cachetop(8)

## NAME
cachetop - Statistics for linux page cache hit/miss ratios per processes. Uses Linux eBPF/bcc.

## SYNOPSIS
cachetop [-p PID] [interval]

## DESCRIPTION
This traces four kernel functions and prints per-processes summaries every interval seconds. This can be useful for processes workload characterization, and looking for patterns in operation usage over time. It provides a top-like interface which by default sorts by HITS in ascending order.

This works by tracing kernel page cache functions using dynamic tracing, and will need updating to match any changes to these functions. Edit the script to customize which functions are traced.

Since this uses BPF, only the root user can use this tool.

## OPTIONS
-p PID

Trace this PID only.

## KEYBINDINGS
The following keybindings can be used to control the output of cachetop.

<

Use the previous column for sorting.

>

Use the next column for sorting.

r

Toggle sorting order (default ascending).

q

Quit cachetop.

## REQUIREMENTS
CONFIG_BPF and bcc.

## EXAMPLES
Update summaries every five second:

\# cachetop

Print summaries each second:

\# cachetop 1

## FIELDS
PID

Process ID of the process causing the cache activity.

UID

User ID of the process causing the cache activity.

CMD

Name of the process.

HITS

Number of page cache hits.

MISSES

Number of page cache misses.

DIRTIES

Number of dirty pages added to the page cache.

READ_HIT%

Read hit percent of page cache usage.

WRITE_HIT%

Write hit percent of page cache usage.

BUFFERS_MB

Buffers size taken from /proc/meminfo.

CACHED_MB

Cached amount of data in current page cache taken from /proc/meminfo.

## OVERHEAD
This traces various kernel page cache functions and maintains in-kernel counts, which are asynchronously copied to user-space. While the rate of operations can be very high (>1G/sec) we can have up to 34% overhead, this is still a relatively efficient way to trace these events, and so the overhead is expected to be small for normal workloads. Measure in a test environment.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Emmanuel Bretelle, Rocky Xing

## SEE ALSO
cachestat (8)


bcc-capable

title: bcc-capable(8)

text:
# bcc-capable(8)

## NAME
capable - Trace security capability checks (cap_capable()).

## SYNOPSIS
capable [-h] [-v] [-p PID] [-K] [-U] [-x] [--cgroupmap MAPPATH]

[--mntnsmap MAPPATH] [--unique]

## DESCRIPTION
This traces security capability checks in the kernel, and prints details for each call. This can be useful for general debugging, and also security enforcement: determining a white list of capabilities an application needs.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF, bcc.

## OPTIONS
-h USAGE message.

-v

Include non-audit capability checks. These are those deemed not interesting and not necessary to audit, such as CAP_SYS_ADMIN checks on memory allocation to affect the behavior of overcommit.

-K

Include kernel stack traces to the output.

-U

Include user-space stack traces to the output.

-x

Show extra fields in TID and INSETID columns.

--cgroupmap MAPPATH

Trace cgroups in this BPF map only (filtered in-kernel).

--mntnsmap MAPPATH

Trace mount namespaces in this BPF map only (filtered in-kernel).

--unique

Don't repeat stacks for the same PID or cgroup.

## EXAMPLES
Trace all capability checks system-wide:

\# capable

Trace capability checks for PID 181:

\# capable -p 181

Trace capability checks in a set of cgroups only (see special_filtering.md

from bcc sources for more details): # capable --cgroupmap /sys/fs/bpf/test01

## FIELDS
TIME(s)

Time of capability check: HH:MM:SS.

UID

User ID.

PID

Process ID.

COMM

Process name. CAP Capability number. NAME Capability name. See capabilities(7) for descriptions.

AUDIT

Whether this was an audit event. Use -v to include non-audit events. INSETID Whether the INSETID bit was set (Linux >= 5.1).

## OVERHEAD
This adds low-overhead instrumentation to capability checks, which are expected to be low frequency, however, that depends on the application. Test in a lab environment before use.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
capabilities(7)


bcc-cobjnew

title: bcc-cobjnew(8)

summary: Summarize object allocations in

text:
# bcc-cobjnew(8)

## NAME
uobjnew, cobjnew, javaobjnew, rubyobjnew, tclobjnew - Summarize object allocations in high-level languages.

## SYNOPSIS
cobjnew [-h] [-C TOP_COUNT] [-S TOP_SIZE] [-v] pid [interval]

javaobjnew [-h] [-C TOP_COUNT] [-S TOP_SIZE] [-v] pid [interval]

rubyobjnew [-h] [-C TOP_COUNT] [-S TOP_SIZE] [-v] pid [interval]

tclobjnew [-h] [-C TOP_COUNT] [-S TOP_SIZE] [-v] pid [interval]

uobjnew [-h] [-C TOP_COUNT] [-S TOP_SIZE] [-v] [-l {c,java,ruby,tcl}] pid [interval]

## DESCRIPTION
uobjnew traces object allocations in high-level languages (including "malloc") and prints summaries of the most frequently allocated types by number of objects or number of bytes.

This tool relies on USDT probes embedded in many high-level languages, such as C, Java, Ruby, and Tcl. It requires a runtime instrumented with these probes, which in some cases requires building from source with a USDT-specific flag, such as "--enable-dtrace" or "--with-dtrace". For Java, the Java process must be started with the "-XX:+ExtendedDTraceProbes" flag.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-C TOP_COUNT

Print the top object types sorted by number of instances.

-S TOP_SIZE

Print the top object types sorted by size.

-v

Print the resulting BPF program, for debugging purposes.

{c,java,ruby,tcl}

The language to trace.

pid

The process id to trace.

interval

Wait this many seconds and then print the summary and exit. By default, wait for Ctrl+C to exit.

## EXAMPLES
Trace object allocations in a Ruby process:

\# uobjnew ruby 148

Trace object allocations from "malloc" and print the top 10 by total size:

\# uobjnew -S 10 c 1788

## FIELDS
TYPE

The object type being allocated. For C (malloc), this is the block size.

ALLOCS

The number of objects allocated.

BYTES

The number of bytes allocated.

## OVERHEAD
Object allocation events are quite frequent, and therefore the overhead from running this tool can be considerable. Use with caution and make sure to test before using in a production environment. Nonetheless, even thousands of allocations per second will likely produce a reasonable overhead when investigating a problem.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _example.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein

## SEE ALSO
ustat(8), ugc(8), memleak(8)


bcc-compactsnoop

title: bcc-compactsnoop(8)

text:
# bcc-compactsnoop(8)

## NAME
compactsnoop - Trace compact zone events. Uses Linux eBPF/bcc.

## SYNOPSIS
compactsnoop [-h] [-T] [-p PID] [-d DURATION] [-K] [-e]

## DESCRIPTION
compactsnoop traces the compact zone events, showing which processes are allocing pages with memory compaction. This can be useful for discovering when compact_stall (/proc/vmstat) continues to increase, whether it is caused by some critical processes or not.

This works by tracing the compact zone events using raw_tracepoints and one kretprobe.

For the Centos 7.6 (3.10.x kernel), see the version under tools/old, which uses an older memory compaction mechanism.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print usage message.

-T

Include a timestamp column.

-p PID

Trace this process ID only (filtered in-kernel).

-d DURATION

Total duration of trace in seconds.

-K

Output kernel stack trace

-e

Show extended fields.

## EXAMPLES
Trace all compact zone events:

\# compactsnoop

Trace all compact zone events, for 10 seconds only:

\# compactsnoop -d 10

## FIELDS
TIME(s)

Time of the call, in seconds.

COMM

Process name

PID

Process ID

NODE

Memory node

ZONE

Zone of the node (such as DMA, DMA32, NORMAL eg)

ORDER

Shows which order alloc cause memory compaction, -1 means all orders (eg: write to /proc/sys/vm/compact_memory)

MODE

SYNC OR ASYNC

FRAGIDX (extra column)

The FRAGIDX is short for fragmentation index, which only makes sense if an allocation of a requested size would fail. If that is true, the fragmentation index indicates whether external fragmentation or a lack of memory was the problem. The value can be used to determine if page reclaim or compaction should be used.

Index is between 0 and 1 so return within 3 decimal places

0 => allocation would fail due to lack of memory

1 => allocation would fail due to fragmentation

MIN (extra column)

The min watermark of the zone

LOW (extra column)

The low watermark of the zone

HIGH (extra column)

The high watermark of the zone

FREE (extra column)

The nr_free_pages of the zone

LAT(ms)

compact zone's latency

STATUS

The compaction's result.

For (CentOS 7.6's kernel), the status include:

"skipped" (COMPACT_SKIPPED): compaction didn't start as it was not possible or direct reclaim was more suitable

"continue" (COMPACT_CONTINUE): compaction should continue to another pageblock

"partial" (COMPACT_PARTIAL): direct compaction partially compacted a zone and there are suitable pages

"complete" (COMPACT_COMPLETE): The full zone was compacted

For (kernel 4.7 and above):

"not_suitable_zone" (COMPACT_NOT_SUITABLE_ZONE): For more detailed tracepoint output - internal to compaction

"skipped" (COMPACT_SKIPPED): compaction didn't start as it was not possible or direct reclaim was more suitable

"deferred" (COMPACT_DEFERRED): compaction didn't start as it was deferred due to past failures

"no_suitable_page" (COMPACT_NOT_SUITABLE_PAGE): For more detailed tracepoint output - internal to compaction

"continue" (COMPACT_CONTINUE): compaction should continue to another pageblock

"complete" (COMPACT_COMPLETE): The full zone was compacted scanned but wasn't successful to compact suitable pages.

"partial_skipped" (COMPACT_PARTIAL_SKIPPED): direct compaction has scanned part of the zone but wasn't successful to compact suitable pages.

"contended" (COMPACT_CONTENDED): compaction terminated prematurely due to lock contentions

"success" (COMPACT_SUCCESS): direct compaction terminated after concluding that the allocation should now succeed

## OVERHEAD
This traces the kernel compact zone kprobe/kretprobe or raw_tracepoints and prints output for each event. As the rate of this is generally expected to be low (< 1000/s), the overhead is also expected to be negligible.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Wenbo Zhang


bcc-cpudist

title: bcc-cpudist(8)

text:
# bcc-cpudist(8)

## NAME
cpudist - On- and off-CPU task time as a histogram.

## SYNOPSIS
cpudist [-h] [-O] [-T] [-m] [-P] [-L] [-p PID] [-I] [-e] [interval] [count]

## DESCRIPTION
This measures the time a task spends on the CPU before being descheduled, and shows the times as a histogram. Tasks that spend a very short time on the CPU can be indicative of excessive context-switches and poor workload distribution, and possibly point to a shared source of contention that keeps tasks switching in and out as it becomes available (such as a mutex).

Similarly, the tool can also measure the time a task spends off-CPU before it is scheduled again. This can be helpful in identifying long blocking and I/O operations, or alternatively very short descheduling times due to short-lived locks or timers.

By default CPU idle time are excluded by simply excluding PID 0.

This tool uses in-kernel eBPF maps for storing timestamps and the histogram, for efficiency. Despite this, the overhead of this tool may become significant for some workloads: see the OVERHEAD section.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print usage message.

-O

Measure off-CPU time instead of on-CPU time.

-T

Include timestamps on output.

-m

Output histogram in milliseconds.

-P

Print a histogram for each PID (tgid from the kernel's perspective).

-L

Print a histogram for each TID (pid from the kernel's perspective).

-p PID

Only show this PID (filtered in kernel for efficiency).

-I

Include CPU idle time (by default these are excluded).

-e

Show extension summary (average/total/count).

interval

Output interval, in seconds.

count

Number of outputs.

## EXAMPLES
Summarize task on-CPU time as a histogram:

\# cpudist

Summarize task off-CPU time as a histogram:

\# cpudist -O

Print 1 second summaries, 10 times:

\# cpudist 1 10

Print 1 second summaries, using milliseconds as units for the histogram, and include timestamps on output:

\# cpudist -mT 1

Trace PID 185 only, 1 second summaries:

\# cpudist -p 185 1

Include CPU idle time:

\# cpudist -I

Also show extension summary:

\# cpudist -e

## FIELDS
usecs

Microsecond range

msecs

Millisecond range

count

How many times a task event fell into this range

distribution

An ASCII bar chart to visualize the distribution (count column)

## OVERHEAD
This traces scheduler tracepoints, which can become very frequent. While eBPF has very low overhead, and this tool uses in-kernel maps for efficiency, the frequency of scheduler events for some workloads may be high enough that the overhead of this tool becomes significant. Measure in a lab environment to quantify the overhead before use.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _example.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein, Rocky Xing

## SEE ALSO
pidstat(1), runqlat(8)


bcc-cpuunclaimed

title: bcc-cpuunclaimed(8)

text:
# bcc-cpuunclaimed(8)

## NAME
cpuunclaimed - Sample CPU run queues and calculate unclaimed idle CPU. Uses Linux eBPF/bcc.

## SYNOPSIS
cpuunclaimed [-T] [-j] [-J] [interval [count]]

## DESCRIPTION
This tool samples the length of the run queues and determine when there are idle CPUs, yet queued threads waiting their turn. It reports the amount of idle (yet unclaimed by waiting threads) CPU as a system-wide percentage.

This situation can happen for a number of reasons:

-

An application has been bound to some, but not all, CPUs, and has runnable threads that cannot migrate to other CPUs due to this configuration.

-

CPU affinity: an optimization that leaves threads on CPUs where the CPU caches are warm, even if this means short periods of waiting while other CPUs are idle. The wait period is tunale (see sysctl, kernel.sched*).

-

Scheduler bugs.

An unclaimed idle of < 1% is likely to be CPU affinity, and not usually a cause for concern. By leaving the CPU idle, overall throughput of the system may be improved. This tool is best for identifying larger issues, > 2%, due to the coarseness of its 99 Hertz samples.

This is an experimental tool that currently works by use of sampling to keep overheads low. Tool assumptions:

-

CPU samples consistently fire around the same offset. There will sometimes be a lag as a sample is delayed by higher-priority interrupts, but it is assumed the subsequent samples will catch up to the expected offsets (as is seen in practice). You can use -J to inspect sample offsets. Some systems can power down CPUs when idle, and when they wake up again they may begin firing at a skewed offset: this tool will detect the skew, print an error, and exit.

-

All CPUs are online (see ncpu).

If this identifies unclaimed CPU, you can double check it by dumping raw samples (-j), as well as using other tracing tools to instrument scheduler events (although this latter approach has much higher overhead).

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## EXAMPLES
Sample and calculate unclaimed idle CPUs, output every 1 second (default:

\# cpuunclaimed

Print 5 second summaries, 10 times:

\# cpuunclaimed 5 10

Print 1 second summaries with timestamps:

\# cpuunclaimed -T 1

Raw dump of all samples (verbose), as comma-separated values:

\# cpuunclaimed -j

## FIELDS
%CPU

CPU utilization as a system-wide percentage.

unclaimed idle

Percentage of CPU resources that were idle when work was queued on other CPUs, as a system-wide percentage.

TIME

Time (HH:MM:SS)

TIMESTAMP_ns

Timestamp, nanoseconds.

CPU#

CPU ID.

OFFSET_ns_CPU#

Time offset that a sample fired within a sample group for this CPU.

## OVERHEAD
The overhead is expected to be low/negligible as this tool uses sampling at 99 Hertz (on all CPUs), which has a fixed and low cost, rather than sampling every scheduler event as many other approaches use (which can involve instrumenting millions of events per second). Sampled CPUs, run queue lengths, and timestamps are written to ring buffers that are periodically read by user space for reporting. Measure overhead in a test environment.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
runqlen(8)


bcc-criticalstat

title: bcc-criticalstat(8)

text:
# bcc-criticalstat(8)

## NAME
criticalstat - A tracer to find and report long atomic critical sections in kernel

## SYNOPSIS
criticalstat [-h] [-p] [-i] [-d DURATION]

## DESCRIPTION
criticalstat traces and reports occurrences of atomic critical sections in the kernel with useful stacktraces showing the origin of them. Such critical sections frequently occur due to use of spinlocks, or if interrupts or preemption were explicitly disabled by a driver. IRQ routines in Linux are also executed with interrupts disabled. There are many reasons. Such critical sections are a source of long latency/responsive issues for real-time systems.

This works by probing the preempt/irq and cpuidle tracepoints in the kernel. Since this uses BPF, only the root user can use this tool. Further, the kernel has to be built with certain CONFIG options enabled. See below.

## REQUIREMENTS
Enable following kernel configurations based on which kernel version you use.

- CONFIG_DEBUG_PREEMPT

- CONFIG_PREEMPT_TRACER

For kernel 4.19 and later:

- CONFIG_PREEMPTIRQ_TRACEPOINTS

- CONFIG_TRACE_IRQFLAGS

- CONFIG_TRACE_PREEMPT_TOGGLE

For kernel 4.15 to 4.18:

- CONFIG_PREEMPTIRQ_EVENTS

- CONFIG_PROVE_LOCKING

- CONFIG_DEBUG_PREEMPT Additionally, the following options should be turned off on older kernels:

- CONFIG_LOCKDEP

## OPTIONS
-h

Print usage message.

-p

Find long sections where preemption was disabled on local CPU.

-i

Find long sections where interrupt was disabled on local CPU.

-d DURATION

Only identify sections that are longer than DURATION in microseconds.

## EXAMPLES
Run with default options: irq disabled for more than 100 uS

\# criticalstat

Find sections with preemption disabled for more than 100 uS.

\# criticalstat -p

Find sections with IRQ disabled for more than 500 uS.

\# criticalstat -d 500

Find sections with preemption disabled for more than 500 uS.

\# criticalstat -p -d 500

## OVERHEAD
This tool can cause overhead if the application is spending a lot of time in kernel mode. The overhead is variable but can be 2-4% of performance degradation. If overhead is seen to be too much, please pass a higher DURATION to the -d option to filter more aggressively.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Joel Fernandes

## SEE ALSO
Linux kernel's preemptoff and irqoff tracers.


bcc-cthreads

title: bcc-cthreads(8)

text:
# bcc-cthreads(8)

## NAME
uthreads, cthreads, javathreads - Trace thread creation events in Java or pthreads.

## SYNOPSIS
cthreads [-h] [-v] pid javathreads [-h] [-v] pid uthreads [-h] [-l {c,java,none}] [-v] pid

## DESCRIPTION
This traces thread creation events in Java processes, or pthread creation events in any process. When a thread is created, its name or start address is printed.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-l {c,java,none}

The language to trace. C and none select tracing pthreads only, regardless of the runtime being traced.

-v

Print the resulting BPF program, for debugging purposes.

pid

The process id to trace.

## EXAMPLES
Trace Java thread creations:

\# uthreads -l java 148

Trace pthread creations:

\# uthreads 1802

## FIELDS
TIME

The event's time in seconds from the beginning of the trace.

ID

The thread's ID. The information in this column depends on the runtime.

TYPE

Event type -- thread start, stop, or pthread event.

DESCRIPTION

The thread's name or start address function name.

## OVERHEAD
Thread start and stop events are usually not very frequent, which makes this tool's overhead negligible.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _example.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein

## SEE ALSO
ustat(8), trace(8)


bcc-dbslower

title: bcc-dbslower(8)

text:
# bcc-dbslower(8)

## NAME
dbslower - Trace MySQL/PostgreSQL server queries slower than a threshold.

## SYNOPSIS
dbslower [-v] [-p PID [PID ...]] [-x PATH] [-m THRESHOLD] {mysql,postgres}

## DESCRIPTION
This traces queries served by a MySQL or PostgreSQL server, and prints those that exceed a latency (query time) threshold. By default a threshold of 1 ms is used.

This uses User Statically-Defined Tracing (USDT) probes, a feature added to MySQL and PostgreSQL for DTrace support, but which may not be enabled on a given installation. See requirements. Alternatively, MySQL queries can be traced without the USDT support using the -x option.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF, bcc, and MySQL server with USDT probe support (when configuring the build: -DENABLE_DTRACE=1) or PostgreSQL server with USDT probe support (when configuring the build: --enable-dtrace).

## OPTIONS
-h Print usage message.

-p PID

Trace this PID. If no PID is specified, the tool will attempt to automatically detect the MySQL or PostgreSQL processes running on the system.

-x PATH

Path to MySQL binary. This option allow to MySQL queries even when USDT probes aren't enabled on the MySQL server.

-m THRESHOLD

Minimum query latency (duration) to trace, in milliseconds. Default is 1 ms.

{mysql,postgres}

The database engine to trace.

## EXAMPLES
Trace MySQL server queries slower than 1 ms:

\# dbslower mysql

Trace slower than 10 ms for PostgreSQL in process 408:

\# dbslower postgres -p 408 -m 10

## FIELDS
TIME(s)

Time of query start, in seconds.

PID

Process ID of the traced server.

MS

Milliseconds for the query, from start to end.

QUERY

Query string, truncated to 256 characters.

## OVERHEAD
This adds low-overhead instrumentation to queries, and only emits output data from kernel to user-level if they query exceeds the threshold. If the server query rate is less than 1,000/sec, the overhead is expected to be negligible. If the query rate is higher, test to gauge overhead.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein, Brendan Gregg

## SEE ALSO
biosnoop(8), mysqld_qslower(8), dbstat(8)


bcc-dbstat

title: bcc-dbstat(8)

text:
# bcc-dbstat(8)

## NAME
dbstat - Collect histograms of MySQL/PostgreSQL query latencies.

## SYNOPSIS
dbstat [-v] [-p PID [PID ...]] [-m THRESHOLD] [-u] [-i INTERVAL] {mysql,postgres}

## DESCRIPTION
This traces queries served by a MySQL or PostgreSQL server, and collects a histogram of query latencies. The histogram is printed at the end of collection, or at specified intervals.

This uses User Statically-Defined Tracing (USDT) probes, a feature added to MySQL and PostgreSQL for DTrace support, but which may not be enabled on a given installation. See requirements.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF, bcc, and MySQL server with USDT probe support (when configuring the build: -DENABLE_DTRACE=1) or PostgreSQL server with USDT probe support (when configuring the build: --enable-dtrace).

## OPTIONS
-h Print usage message.

-p PID

Trace this PID. If no PID is specified, the tool will attempt to automatically detect the MySQL or PostgreSQL processes running on the system.

-m THRESHOLD

Minimum query latency (duration) to trace, in milliseconds. Default is all queries.

-u

Display query latencies in microseconds (default: milliseconds).

-i INTERVAL

Print summaries (histograms) at this interval, specified in seconds.

{mysql,postgres}

The database engine to trace.

## EXAMPLES
Display histogram of MySQL query latencies:

\# dbstat mysql

Display histogram of PostgreSQL query latencies slower than 10ms in pid 408:

\# dbstat postgres -p 408 -m 10

Display histogram of PostgreSQL query latencies at 3-second intervals:

\# dbstat postgres -i 3

## OVERHEAD
This adds low-overhead instrumentation to queries, and only emits output data from kernel to user-level if they query exceeds the threshold. If the server query rate is less than 1,000/sec, the overhead is expected to be negligible. If the query rate is higher, test to gauge overhead.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein

## SEE ALSO
dbslower(8)


bcc-dcsnoop

title: bcc-dcsnoop(8)

text:
# bcc-dcsnoop(8)

## NAME
dcsnoop - Trace directory entry cache (dcache) lookups. Uses Linux eBPF/bcc.

## SYNOPSIS
dcsnoop [-h] [-a]

## DESCRIPTION
By default, this traces every failed dcache lookup (cache miss), and shows the process performing the lookup and the filename requested. A -a option can be used to show all lookups, not just failed ones.

The output of this tool can be verbose, and is intended for further investigations of dcache performance beyond dcstat(8), which prints per-second summaries.

This uses kernel dynamic tracing of the d_lookup() function, and will need and will need updating to match any changes to this function.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print usage message.

-a

Trace references, not just failed lookups.

## EXAMPLES
Trace failed dcache lookups:

\# dcsnoop

Trace all dcache lookups:

\# dcsnoop -a

## FIELDS
TIME(s)

Time of lookup, in seconds.

PID

Process ID.

COMM

Process name.

T

Type: R == reference (only visible with -a), M == miss. A miss will print two lines, one for the reference, and one for the miss.

FILE

The file name component that was being looked up. This contains trailing pathname components (after '/'), which will be the subject of subsequent lookups.

## OVERHEAD
File name lookups can be frequent (depending on the workload), and this tool prints a line for each failed lookup, and with -a, each reference as well. The output may be verbose, and the incurred overhead, while optimized to some extent, may still be from noticeable to significant. This is only really intended for deeper investigations beyond dcstat(8), when absolutely necessary. Measure and quantify the overhead in a test environment before use.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
dcstat(1)


bcc-dcstat

title: bcc-dcstat(8)

text:
# bcc-dcstat(8)

## NAME
dcstat - Directory entry cache (dcache) stats. Uses Linux eBPF/bcc.

## SYNOPSIS
dcstat [interval [count]]

## DESCRIPTION
The Linux directory entry cache (dcache) improves the performance of file and directory name lookups. This tool provides per-second summary statistics of dcache performance.

This uses kernel dynamic tracing of kernel functions, lookup_fast() and d_lookup(), which will need to be modified to match kernel changes.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## EXAMPLES
Print summaries each second:

\# dcstat

Print output every five seconds, three times:

\# dcstat 5 3

## FIELDS
REFS/s

Number dcache lookups (references) per second.

SLOW/s

Number of dcache lookups that failed the lookup_fast() path and executed the lookup_slow() path instead.

MISS/s

Number of dcache misses (failed both fast and slow lookups).

HIT%

Percentage of dcache hits over total references.

## OVERHEAD
The overhead depends on the frequency of file and directory name lookups. While the per-event overhead is low, some applications may make over 100k lookups per second, and the low per-event overhead will begin to add up, and could begin to be measurable (over 10% CPU usage). Measure in a test environment.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
dcsnoop(8)


bcc-deadlock

title: bcc-deadlock(8)

text:
# bcc-deadlock(8)

## NAME
deadlock - Find potential deadlocks (lock order inversions) in a running program.

## SYNOPSIS
deadlock [-h] [--binary BINARY] [--dump-graph DUMP_GRAPH] [--verbose] [--lock-symbols LOCK_SYMBOLS] [--unlock-symbols UNLOCK_SYMBOLS] pid

## DESCRIPTION
deadlock finds potential deadlocks in a running process. The program attaches uprobes on `pthread_mutex_lock` and `pthread_mutex_unlock` by default to build a mutex wait directed graph, and then looks for a cycle in this graph. This graph has the following properties:

- Nodes in the graph represent mutexes.

- Edge (A, B) exists if there exists some thread T where lock(A) was called and lock(B) was called before unlock(A) was called.

If there is a cycle in this graph, this indicates that there is a lock order inversion (potential deadlock). If the program finds a lock order inversion, the program will dump the cycle of mutexes, dump the stack traces where each mutex was acquired, and then exit.

This program can only find potential deadlocks that occur while the program is tracing the process. It cannot find deadlocks that may have occurred before the program was attached to the process.

This tool does not work for shared mutexes or recursive mutexes.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc

## OPTIONS
-h, --help

show this help message and exit

--binary BINARY

If set, trace the mutexes from the binary at this path. For statically-linked binaries, this argument is not required. For dynamically-linked binaries, this argument is required and should be the path of the pthread library the binary is using. Example: /lib/x86_64-linux-gnu/libpthread.so.0

--dump-graph DUMP_GRAPH

If set, this will dump the mutex graph to the specified file.

--verbose

Print statistics about the mutex wait graph.

--lock-symbols LOCK_SYMBOLS

Comma-separated list of lock symbols to trace. Default is pthread_mutex_lock. These symbols cannot be inlined in the binary.

--unlock-symbols UNLOCK_SYMBOLS

Comma-separated list of unlock symbols to trace. Default is pthread_mutex_unlock. These symbols cannot be inlined in the binary.

-t THREADS, --threads THREADS

Specifies the maximum number of threads to trace. default 65536. Note. 40 bytes per thread.

-e EDGES, --edges EDGES

Specifies the maximum number of edge cases that can be recorded. default 65536. Note. 88 bytes per edge case.

pid

Pid to trace

## EXAMPLES
Find potential deadlocks in PID 181. The --binary argument is not needed for statically-linked binaries.

\# deadlock 181

Find potential deadlocks in PID 181. If the process was created from a dynamically-linked executable, the --binary argument is required and must be the path of the pthread library:

\# deadlock 181 --binary /lib/x86_64-linux-gnu/libpthread.so.0

Find potential deadlocks in PID 181. If the process was created from a statically-linked executable, optionally pass the location of the binary. On older kernels without https://lkml.org/lkml/2017/1/13/585, binaries that contain `:` in the path cannot be attached with uprobes. As a workaround, we can create a symlink to the binary, and provide the symlink name instead with the `--binary` option:

\# deadlock 181 --binary /usr/local/bin/lockinversion

Find potential deadlocks in PID 181 and dump the mutex wait graph to a file:

\# deadlock 181 --dump-graph graph.json

Find potential deadlocks in PID 181 and print mutex wait graph statistics:

\# deadlock 181 --verbose

Find potential deadlocks in PID 181 with custom mutexes:

\# deadlock 181 --lock-symbols custom_mutex1_lock,custom_mutex2_lock --unlock_symbols custom_mutex1_unlock,custom_mutex2_unlock

## OUTPUT
This program does not output any fields. Rather, it will keep running until it finds a potential deadlock, or the user hits Ctrl-C. If the program finds a potential deadlock, it will output the stack traces and lock order inversion in the following format and exit:

Potential Deadlock Detected!

Cycle in lock order graph: Mutex M0 => Mutex M1 => Mutex M0

Mutex M1 acquired here while holding Mutex M0 in Thread T:

[stack trace]

Mutex M0 previously acquired by the same Thread T here:

[stack trace]

Mutex M0 acquired here while holding Mutex M1 in Thread S:

[stack trace]

Mutex M1 previously acquired by the same Thread S here:

[stack trace]

Thread T created by Thread R here:

[stack trace]

Thread S created by Thread Q here:

[stack trace]

## OVERHEAD
This traces all mutex lock and unlock events and all thread creation events on the traced process. The overhead of this can be high if the process has many threads and mutexes. You should only run this on a process where the slowdown is acceptable.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Kenny Yu


bcc-dirtop

title: bcc-dirtop(8)

text:
# bcc-dirtop(8)

## NAME
dirtop - File reads and writes by directory. Top for directories.

## SYNOPSIS
dirtop -d directory1,directory2,... [-h] [-C] [-r MAXROWS] [-s {reads,writes,rbytes,wbytes}] [-p PID] [interval] [count]

## DESCRIPTION
This is top for directories.

This traces file reads and writes, and prints a per-directory summary every interval (by default, 1 second). By default the summary is sorted on the highest read throughput (Kbytes). Sorting order can be changed via -s option.

This uses in-kernel eBPF maps to store per process summaries for efficiency.

This script works by tracing the __vfs_read() and __vfs_write() functions using kernel dynamic tracing, which instruments explicit read and write calls. If files are read or written using another means (eg, via mmap()), then they will not be visible using this tool. Also, this tool will need updating to match any code changes to those vfs functions.

This should be useful for file system workload characterization when analyzing the performance of applications.

Note that tracing VFS level reads and writes can be a frequent activity, and this tool can begin to cost measurable overhead at high I/O rates.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-d

Defines a list of directories, comma separated, to observe. Wildcards are allowed if between single bracket.

-C

Don't clear the screen.

-r MAXROWS

Maximum number of rows to print. Default is 20.

-s {reads,writes,rbytes,wbytes}

Sort column. Default is rbytes (read throughput).

-p PID

Trace this PID only.

interval

Interval between updates, seconds.

count

Number of interval summaries.

## EXAMPLES
Summarize block device I/O by directory, 1 second screen refresh:

\# dirtop -d '/hdfs/uuid/*/yarn'

Don't clear the screen, and top 8 rows only:

\# dirtop -d '/hdfs/uuid/*/yarn' -Cr 8

5 second summaries, 10 times only:

\# dirtop -d '/hdfs/uuid/*/yarn' 5 10

Report read & write IOs generated in mutliple yarn and data directories:

\# dirtop -d '/hdfs/uuid/*/yarn,/hdfs/uuid/*/data'

## FIELDS
loadavg:

The contents of /proc/loadavg

READS

Count of reads during interval.

WRITES

Count of writes during interval.

R_Kb

Total read Kbytes during interval.

W_Kb

Total write Kbytes during interval.

PATH

The path were the IOs were accounted.

## OVERHEAD
Depending on the frequency of application reads and writes, overhead can become significant, in the worst case slowing applications by over 50%. Hopefully for real world workloads the overhead is much less -- test before use. The reason for the high overhead is that VFS reads and writes can be a frequent event, and despite the eBPF overhead being very small per event, if you multiply this small overhead by a million events per second, it becomes a million times worse. Literally. You can gauge the number of reads and writes using the vfsstat(8) tool, also from bcc.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Erwan Velu

## INSPIRATION
filetop(8) by Brendan Gregg

## SEE ALSO
vfsstat(8), vfscount(8), fileslower(8)


bcc-drsnoop

title: bcc-drsnoop(8)

text:
# bcc-drsnoop(8)

## NAME
drsnoop - Trace direct reclaim events. Uses Linux eBPF/bcc.

## SYNOPSIS
drsnoop [-h] [-T] [-U] [-p PID] [-t TID] [-u UID] [-d DURATION] [-n name] [-v]

## DESCRIPTION
drsnoop trace direct reclaim events, showing which processes are allocing pages with direct reclaiming. This can be useful for discovering when allocstall (/p- roc/vmstat) continues to increase, whether it is caused by some critical proc- esses or not.

This works by tracing the direct reclaim events using kernel tracepoints.

This makes use of a Linux 4.4 feature (bpf_perf_event_output()); for kernels older than 4.4, see the version under tools/old, which uses an older mechanism.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print usage message.

-T

Include a timestamp column.

-U

Show UID.

-p PID

Trace this process ID only (filtered in-kernel).

-t TID

Trace this thread ID only (filtered in-kernel).

-u UID

Trace this UID only (filtered in-kernel).

-d DURATION

Total duration of trace in seconds.

-n name

Only print processes where its name partially matches 'name' -v verbose Run in verbose mode. Will output system memory state

-v

show system memory state

## EXAMPLES
Trace all direct reclaim events:

\# drsnoop

Trace all direct reclaim events, for 10 seconds only:

\# drsnoop -d 10

Trace all direct reclaim events, and include timestamps:

\# drsnoop -T

Show UID:

\# drsnoop -U

Trace PID 181 only:

\# drsnoop -p 181

Trace UID 1000 only:

\# drsnoop -u 1000

Trace all direct reclaim events from processes where its name partially match-

es 'mond': # drnsnoop -n mond

## FIELDS
TIME(s)

Time of the call, in seconds.

UID

User ID

PID

Process ID

TID

Thread ID

COMM

Process name

## OVERHEAD
This traces the kernel direct reclaim tracepoints and prints output for each event. As the rate of this is generally expected to be low (< 1000/s), the overhead is also expected to be negligible.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Wenbo Zhang


bcc-execsnoop

title: bcc-execsnoop(8)

text:
# bcc-execsnoop(8)

## NAME
execsnoop - Trace new processes via exec() syscalls. Uses Linux eBPF/bcc.

## SYNOPSIS
execsnoop [-h] [-T] [-t] [-x] [--cgroupmap CGROUPMAP] [--mntnsmap MAPPATH] [-u USER] [-q] [-n NAME] [-l LINE] [-U] [--max-args MAX_ARGS]

## DESCRIPTION
execsnoop traces new processes, showing the filename executed and argument list.

It works by traces the execve() system call (commonly used exec() variant). This catches new processes that follow the fork->exec sequence, as well as processes that re-exec() themselves. Some applications fork() but do not exec(), eg, for worker processes, which won't be included in the execsnoop output.

This works by tracing the kernel sys_execve() function using dynamic tracing, and will need updating to match any changes to this function.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print usage message.

-T

Include a time column (HH:MM:SS).

-U

Include UID column.

-t

Include a timestamp column.

-u USER

Filter by UID (or username)

-x

Include failed exec()s

-q

Add "quotemarks" around arguments. Escape quotemarks in arguments with a backslash. For tracing empty arguments or arguments that contain whitespace.

-n NAME

Only print command lines matching this name (regex)

-l LINE

Only print commands where arg contains this line (regex)

--max-args MAXARGS

Maximum number of arguments parsed and displayed, defaults to 20

--cgroupmap MAPPATH

Trace cgroups in this BPF map only (filtered in-kernel).

--mntnsmap MAPPATH

Trace mount namespaces in this BPF map only (filtered in-kernel).

-P PPID

Trace this parent PID only.

## EXAMPLES
Trace all exec() syscalls:

\# execsnoop

Trace all exec() syscalls, and include timestamps:

\# execsnoop -t

Display process UID:

\# execsnoop -U

Trace only UID 1000:

\# execsnoop -u 1000

Trace only processes launched by root and display UID column:

\# execsnoop -Uu root

Include failed exec()s:

\# execsnoop -x

Put quotemarks around arguments.

\# execsnoop -q

Only trace exec()s where the filename contains "mount":

\# execsnoop -n mount

Only trace exec()s where argument's line contains "testpkg":

\# execsnoop -l testpkg

Trace a set of cgroups only (see special_filtering.md from bcc sources for more details):

\# execsnoop --cgroupmap /sys/fs/bpf/test01

## FIELDS
TIME

Time of exec() return, in HH:MM:SS format.

TIME(s)

Time of exec() return, in seconds.

UID

User ID

PCOMM

Parent process/command name.

PID

Process ID

PPID

Parent process ID

RET

Return value of exec(). 0 == successs. Failures are only shown when using the -x option.

ARGS

Filename for the exec(), followed be up to 19 arguments. An ellipsis "..." is shown if the argument list is known to be truncated.

## OVERHEAD
This traces the kernel execve function and prints output for each event. As the rate of this is generally expected to be low (< 1000/s), the overhead is also expected to be negligible. If you have an application that is calling a high rate of exec()s, then test and understand overhead before use.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg, Rocky Xing

## SEE ALSO
opensnoop(1)


bcc-exitsnoop

title: bcc-exitsnoop(8)

text:
# bcc-exitsnoop(8)

## NAME
exitsnoop - Trace all process termination (exit, fatal signal). Uses Linux eBPF/bcc.

## SYNOPSIS
exitsnoop [-h] [-t] [--utc] [-x] [-p PID] [--label LABEL] [--per-thread]

## DESCRIPTION
exitsnoop traces process termination, showing the command name and reason for termination, either an exit or a fatal signal.

It catches processes of all users, processes in containers, as well as processes that become zombie.

This works by tracing the kernel sched_process_exit() function using dynamic tracing, and will need updating to match any changes to this function.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print usage message.

-t

Include a timestamp column.

--utc

Include a timestamp column, use UTC timezone.

-x

Exclude successful exits, exit( 0 )

-p PID

Trace this process ID only (filtered in-kernel).

--label LABEL

Label each line with LABEL (default 'exit') in first column (2nd if timestamp is present).

--per-thread

Trace per thread termination

## EXAMPLES
Trace all process termination

\# exitsnoop

Trace all process termination, and include timestamps:

\# exitsnoop -t

Exclude successful exits, only include non-zero exit codes and fatal signals:

\# exitsnoop -x

Trace PID 181 only:

\# exitsnoop -p 181

Label each output line with 'EXIT':

\# exitsnoop --label EXIT

Trace per thread termination

\# exitsnoop --per-thread

## FIELDS
TIME-TZ

Time of process termination HH:MM:SS.sss with milliseconds, where TZ is the local time zone, 'UTC' with --utc option.

LABEL

The optional label if --label option is used. This is useful with the -t option for timestamps when the output of several tracing tools is sorted into one combined output.

PCOMM

Process/command name.

PID

Process ID

PPID

The process ID of the process that will be notified of PID termination.

TID

Thread ID.

EXIT_CODE

The exit code for exit() or the signal number for a fatal signal.

## OVERHEAD
This traces the kernel sched_process_exit() function and prints output for each event. As the rate of this is generally expected to be low (< 1000/s), the overhead is also expected to be negligible. If you have an application that has a high rate of process termination, then test and understand overhead before use.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Arturo Martin-de-Nicolas

## SEE ALSO
execsnoop(8)


title: bcc-ext4dist(8)

text:
# bcc-ext4dist(8)

## NAME
ext4dist - Summarize ext4 operation latency. Uses Linux eBPF/bcc.

## SYNOPSIS
ext4dist [-h] [-T] [-m] [-p PID] [interval] [count]

## DESCRIPTION
This tool summarizes time (latency) spent in common ext4 file operations: reads, writes, opens, and syncs, and presents it as a power-of-2 histogram. It uses an in-kernel eBPF map to store the histogram for efficiency.

Since this works by tracing the ext4_file_operations interface functions, it will need updating to match any changes to these functions.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print usage message.

-T

Don't include timestamps on interval output.

-m

Output in milliseconds.

-p PID

Trace this PID only.

## EXAMPLES
Trace ext4 operation time, and print a summary on Ctrl-C:

\# ext4dist

Trace PID 181 only:

\# ext4dist -p 181

Print 1 second summaries, 10 times:

\# ext4dist 1 10

1 second summaries, printed in milliseconds

\# ext4dist -m 1

## FIELDS
msecs

Range of milliseconds for this bucket.

usecs

Range of microseconds for this bucket.

count

Number of operations in this time range.

distribution

ASCII representation of the distribution (the count column).

## OVERHEAD
This adds low-overhead instrumentation to these ext4 operations, including reads and writes from the file system cache. Such reads and writes can be very frequent (depending on the workload; eg, 1M/sec), at which point the overhead of this tool may become noticeable. Measure and quantify before use.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
ext4snoop(8)


title: bcc-ext4slower(8)

text:
# bcc-ext4slower(8)

## NAME
ext4slower - Trace slow ext4 file operations, with per-event details.

## SYNOPSIS
ext4slower [-h] [-j] [-p PID] [min_ms]

## DESCRIPTION
This tool traces common ext4 file operations: reads, writes, opens, and syncs. It measures the time spent in these operations, and prints details for each that exceeded a threshold.

WARNING: See the OVERHEAD section.

By default, a minimum millisecond threshold of 10 is used. If a threshold of 0 is used, all events are printed (warning: verbose).

Since this works by tracing the ext4_file_operations interface functions, it will need updating to match any changes to these functions.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-p PID Trace this PID only.

min_ms

Minimum I/O latency (duration) to trace, in milliseconds. Default is 10 ms.

## EXAMPLES
Trace synchronous file reads and writes slower than 10 ms:

\# ext4slower

Trace slower than 1 ms:

\# ext4slower 1

Trace slower than 1 ms, and output just the fields in parsable format (csv):

\# ext4slower -j 1

Trace all file reads and writes (warning: the output will be verbose):

\# ext4slower 0

Trace slower than 1 ms, for PID 181 only:

\# ext4slower -p 181 1

## FIELDS
TIME(s)

Time of I/O completion since the first I/O seen, in seconds.

COMM

Process name.

PID

Process ID.

T

Type of operation. R == read, W == write, O == open, S == fsync.

OFF_KB

File offset for the I/O, in Kbytes.

BYTES

Size of I/O, in bytes.

LAT(ms)

Latency (duration) of I/O, measured from when it was issued by VFS to the filesystem, to when it completed. This time is inclusive of block device I/O, file system CPU cycles, file system locks, run queue latency, etc. It's a more accurate measure of the latency suffered by applications performing file system I/O, than to measure this down at the block device interface.

FILENAME

A cached kernel file name (comes from dentry->d_name.name).

ENDTIME_us

Completion timestamp, microseconds (-j only).

OFFSET_b

File offset, bytes (-j only).

LATENCY_us

Latency (duration) of the I/O, in microseconds (-j only).

## OVERHEAD
This adds low-overhead instrumentation to these ext4 operations, including reads and writes from the file system cache. Such reads and writes can be very frequent (depending on the workload; eg, 1M/sec), at which point the overhead of this tool (even if it prints no "slower" events) can begin to become significant. Measure and quantify before use. If this continues to be a problem, consider switching to a tool that prints in-kernel summaries only.

Note that the overhead of this tool should be less than fileslower(8), as this tool targets ext4 functions only, and not all file read/write paths (which can include socket I/O).

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
biosnoop(8), funccount(8), fileslower(8)


title: bcc-f2fsslower(8)

text:
# bcc-f2fsslower(8)

## NAME
f2fsslower - Trace slow f2fs file operations, with per-event details.

## SYNOPSIS
f2fsslower [-h] [-s] [-p PID] [min_ms]

## DESCRIPTION
This tool traces common f2fs file operations: reads, writes, opens, and syncs. It measures the time spent in these operations, and prints details for each that exceeded a threshold.

WARNING: See the OVERHEAD section.

By default, a minimum millisecond threshold of 10 is used. If a threshold of 0 is used, all events are printed (warning: verbose).

Since this works by tracing the f2fs_file_operations interface functions, it will need updating to match any changes to these functions.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-p PID Trace this PID only.

min_ms

Minimum I/O latency (duration) to trace, in milliseconds. Default is 10 ms.

## EXAMPLES
Trace synchronous file reads and writes slower than 10 ms:

\# f2fsslower

Trace slower than 1 ms:

\# f2fsslower 1

Trace slower than 1 ms, and output just the fields in parsable format (csv):

\# f2fsslower -s 1

Trace all file reads and writes (warning: the output will be verbose):

\# f2fsslower 0

Trace slower than 1 ms, for PID 181 only:

\# f2fsslower -p 181 1

## FIELDS
TIME(s)

Time of I/O completion since the first I/O seen, in seconds.

COMM

Process name.

PID

Process ID.

T

Type of operation. R == read, W == write, O == open, S == fsync.

OFF_KB

File offset for the I/O, in Kbytes.

BYTES

Size of I/O, in bytes.

LAT(ms)

Latency (duration) of I/O, measured from when it was issued by VFS to the filesystem, to when it completed. This time is inclusive of block device I/O, file system CPU cycles, file system locks, run queue latency, etc. It's a more accurate measure of the latency suffered by applications performing file system I/O, than to measure this down at the block device interface.

FILENAME

A cached kernel file name (comes from dentry->d_name.name).

ENDTIME_us

Completion timestamp, microseconds (-s only).

OFFSET_b

File offset, bytes (-s only).

LATENCY_us

Latency (duration) of the I/O, in microseconds (-s only).

## OVERHEAD
This adds low-overhead instrumentation to these f2fs operations, including reads and writes from the file system cache. Such reads and writes can be very frequent (depending on the workload; eg, 1M/sec), at which point the overhead of this tool (even if it prints no "slower" events) can begin to become significant. Measure and quantify before use. If this continues to be a problem, consider switching to a tool that prints in-kernel summaries only.

Note that the overhead of this tool should be less than fileslower(8), as this tool targets f2fs functions only, and not all file read/write paths (which can include socket I/O).

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Ting Zhang

## SEE ALSO
biosnoop(8), funccount(8), fileslower(8)


bcc-filegone

title: bcc-filegone(8)

text:
# bcc-filegone(8)

## NAME
filegone - Trace why file gone (deleted or renamed). Uses Linux eBPF/bcc.

## SYNOPSIS
filegone [-h] [-p PID]

## DESCRIPTION
This traces why file gone/vanished, providing information on who deleted or renamed the file.

This works by tracing the kernel vfs_unlink() , vfs_rmdir() , vfs_rename functions.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print usage message.

-p PID

Trace this process ID only (filtered in-kernel).

## EXAMPLES
Trace all file gone events

\# filegone

Trace file gone events caused by PID 181:

\# filegone -p 181

## FIELDS
TIME

Time of the event.

PID

Process ID that renamed/deleted the file.

COMM

Process name for the PID.

ACTION

action on file: 'DELETE' or 'RENAME'

FILE

Filename.

## OVERHEAD
This traces the kernel VFS file rename and delete functions and prints output for each event. As the rate of this is generally expected to be low (< 1000/s), the overhead is also expected to be negligible. This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Curu Wong

## SEE ALSO
filelife(8)


bcc-filelife

title: bcc-filelife(8)

text:
# bcc-filelife(8)

## NAME
filelife - Trace the lifespan of short-lived files. Uses Linux eBPF/bcc.

## SYNOPSIS
filelife [-h] [-p PID]

## DESCRIPTION
This traces the creation and deletion of files, providing information on who deleted the file, the file age, and the file name. The intent is to provide information on short-lived files, for debugging or performance analysis.

This works by tracing the kernel vfs_create() and vfs_delete() functions (and maybe more, see the source) using dynamic tracing, and will need updating to match any changes to these functions.

This makes use of a Linux 4.4 feature (bpf_perf_event_output()); for kernels older than 4.4, see the version under tools/old, which uses an older mechanism.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print usage message.

-p PID

Trace this process ID only (filtered in-kernel).

## EXAMPLES
Trace all short-lived files, and print details:

\# filelife

Trace all short-lived files created AND deleted by PID 181:

\# filelife -p 181

## FIELDS
TIME

Time of the deletion.

PID

Process ID that deleted the file.

COMM

Process name for the PID.

AGE(s)

Age of the file, from creation to deletion, in seconds.

FILE

Filename.

## OVERHEAD
This traces the kernel VFS file create and delete functions and prints output for each delete. As the rate of this is generally expected to be low (< 1000/s), the overhead is also expected to be negligible. This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
opensnoop(1)


bcc-fileslower

title: bcc-fileslower(8)

text:
# bcc-fileslower(8)

## NAME
fileslower - Trace slow synchronous file reads and writes.

## SYNOPSIS
fileslower [-h] [-p PID] [-a] [min_ms]

## DESCRIPTION
This script uses kernel dynamic tracing of synchronous reads and writes at the VFS interface, to identify slow file reads and writes for any file system.

This version traces __vfs_read() and __vfs_write() and only showing synchronous I/O (the path to new_sync_read() and new_sync_write()), and I/O with filenames. This approach provides a view of just two file system request types: file reads and writes. There are typically many others: asynchronous I/O, directory operations, file handle operations, file open()s, fflush(), etc.

WARNING: See the OVERHEAD section.

By default, a minimum millisecond threshold of 10 is used.

Since this works by tracing various kernel __vfs_*() functions using dynamic tracing, it will need updating to match any changes to these functions. A future version should switch to using FS tracepoints instead.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-p PID Trace this PID only.

-a

Include non-regular file types in output (sockets, FIFOs, etc).

min_ms

Minimum I/O latency (duration) to trace, in milliseconds. Default is 10 ms.

## EXAMPLES
Trace synchronous file reads and writes slower than 10 ms:

\# fileslower

Trace slower than 1 ms:

\# fileslower 1

Trace slower than 1 ms, for PID 181 only:

\# fileslower -p 181 1

## FIELDS
TIME(s)

Time of I/O completion since the first I/O seen, in seconds.

COMM

Process name.

PID

Process ID.

D

Direction of I/O. R == read, W == write.

BYTES

Size of I/O, in bytes.

LAT(ms)

Latency (duration) of I/O, measured from when the application issued it to VFS to when it completed. This time is inclusive of block device I/O, file system CPU cycles, file system locks, run queue latency, etc. It's a more accurate measure of the latency suffered by applications performing file system I/O, than to measure this down at the block device interface.

FILENAME

A cached kernel file name (comes from dentry->d_name.name).

## OVERHEAD
Depending on the frequency of application reads and writes, overhead can become severe, in the worst case slowing applications by 2x. In the best case, the overhead is negligible. Hopefully for real world workloads the overhead is often at the lower end of the spectrum -- test before use. The reason for high overhead is that this traces VFS reads and writes, which includes FS cache reads and writes, and can exceed one million events per second if the application is I/O heavy. While the instrumentation is extremely lightweight, and uses in-kernel eBPF maps for efficient timing and filtering, multiply that cost by one million events per second and that cost becomes a million times worse. You can get an idea of the possible cost by just counting the instrumented events using the bcc funccount tool, eg:

\# ./funccount.py -i 1 -r '^__vfs_(read|write)$'

This also costs overhead, but is somewhat less than fileslower.

If the overhead is prohibitive for your workload, I'd recommend moving down-stack a little from VFS into the file system functions (ext4, xfs, etc). Look for updates to bcc for specific file system tools that do this. The advantage of a per-file system approach is that we can trace post-cache, greatly reducing events and overhead. The disadvantage is needing custom tracing approaches for each different file system (whereas VFS is generic).

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
biosnoop(8), funccount(8)


bcc-filetop

title: bcc-filetop(8)

text:
# bcc-filetop(8)

## NAME
filetop - File reads and writes by filename and process. Top for files.

## SYNOPSIS
filetop [-h] [-a] [-C] [-r MAXROWS] [-s {reads,writes,rbytes,wbytes}] [-p PID] [--read-only] [--write-only] [interval] [count]

## DESCRIPTION
This is top for files.

This traces file reads and writes, and prints a per-file summary every interval (by default, 1 second). By default the summary is sorted on the highest read throughput (Kbytes). Sorting order can be changed via -s option. By default only IO on regular files is shown. The -a option will list all file types (sockets, FIFOs, etc). Only read or only write operations can be traced using --read-only or --write-only options respectively. By default both are traced. Using both options at once will raise an Exception.

This uses in-kernel eBPF maps to store per process summaries for efficiency.

This script works by tracing the __vfs_read() and __vfs_write() functions using kernel dynamic tracing, which instruments explicit read and write calls. If files are read or written using another means (eg, via mmap()), then they will not be visible using this tool. Also, this tool will need updating to match any code changes to those vfs functions.

This should be useful for file system workload characterization when analyzing the performance of applications.

Note that tracing VFS level reads and writes can be a frequent activity, and this tool can begin to cost measurable overhead at high I/O rates.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-a

Include non-regular file types (sockets, FIFOs, etc).

-C

Don't clear the screen.

-r MAXROWS

Maximum number of rows to print. Default is 20.

-s {reads,writes,rbytes,wbytes}

Sort column. Default is rbytes (read throughput).

-p PID

Trace this PID only.

--read-only

Trace read operations only

--write-only

Trace write operations only

interval

Interval between updates, seconds.

count

Number of interval summaries.

## EXAMPLES
Summarize block device I/O by process, 1 second screen refresh:

\# filetop

Don't clear the screen, and top 8 rows only:

\# filetop -Cr 8

5 second summaries, 10 times only:

\# filetop 5 10

## FIELDS
loadavg:

The contents of /proc/loadavg

PID

Process ID.

COMM

Process name.

READS

Count of reads during interval.

WRITES

Count of writes during interval.

R_Kb

Total read Kbytes during interval.

W_Kb

Total write Kbytes during interval.

T

Type of file: R == regular, S == socket, O == other (pipe, etc).

FILE

Filename.

## OVERHEAD
Depending on the frequency of application reads and writes, overhead can become significant, in the worst case slowing applications by over 50%. Hopefully for real world workloads the overhead is much less -- test before use. The reason for the high overhead is that VFS reads and writes can be a frequent event, and despite the eBPF overhead being very small per event, if you multiply this small overhead by a million events per second, it becomes a million times worse. Literally. You can gauge the number of reads and writes using the vfsstat(8) tool, also from bcc.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## INSPIRATION
top(1) by William LeFebvre

## SEE ALSO
vfsstat(8), vfscount(8), fileslower(8)


bcc-funccount

title: bcc-funccount(8)

text:
# bcc-funccount(8)

## NAME
funccount - Count function, tracepoint, and USDT probe calls matching a pattern. Uses Linux eBPF/bcc.

## SYNOPSIS
funccount [-h] [-p PID] [-i INTERVAL] [-d DURATION] [-T] [-r] [-c CPU] [-D] pattern

## DESCRIPTION
This tool is a quick way to determine which functions are being called, and at what rate. It uses in-kernel eBPF maps to count function calls.

WARNING: This uses dynamic tracing of (what can be many) functions, an activity that has had issues on some kernel versions (risk of panics or freezes). Test, and know what you are doing, before use.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
pattern Search pattern. Supports "*" wildcards. See EXAMPLES. You can also use -r for regular expressions.

-h

Print usage message.

-p PID

Trace this process ID only.

-i INTERVAL

Print output every interval seconds.

-d DURATION

Total duration of trace in seconds.

-T

Include timestamps on output.

-r

Use regular expressions for the search pattern.

-D

Print the BPF program before starting (for debugging purposes).

-c CPU

Trace on this CPU only.

## EXAMPLES
Count kernel functions beginning with "vfs_", until Ctrl-C is hit:

\# funccount 'vfs_*'

Count kernel functions beginning with "tcp_send", until Ctrl-C is hit:

\# funccount 'tcp_send*'

Print kernel functions beginning with "vfs_", every second:

\# funccount -i 1 'vfs_*'

Print kernel functions beginning with "vfs_", for ten seconds only:

\# funccount -d 10 'vfs_*'

Match kernel functions beginning with "vfs_", using regular expressions:

\# funccount -r '^vfs_.*'

Count vfs calls for process ID 181 only:

\# funccount -p 181 'vfs_*'

Count calls to the sched_fork tracepoint, indicating a fork() performed:

\# funccount t:sched:sched_fork

Count all GC USDT probes in the Node process:

\# funccount -p 185 u:node:gc*

Count all malloc() calls in libc:

\# funccount c:malloc

Count kernel functions beginning with "vfs_" on CPU 1 only:

\# funccount -c 1 'vfs_*'

## FIELDS
FUNC

Function name

COUNT

Number of calls while tracing

## OVERHEAD
This traces functions and maintains in-kernel counts, which are asynchronously copied to user-space. While the rate of calls be very high (>1M/sec), this is a relatively efficient way to trace these events, and so the overhead is expected to be small for normal workloads. Measure in a test environment before use.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg, Sasha Goldshtein

## SEE ALSO
stackcount(8) funclatency(8) vfscount(8)


bcc-funcinterval

title: bcc-funcinterval(8)

text:
# bcc-funcinterval(8)

## NAME
funcinterval - Time interval between the same function, tracepoint as a histogram.

## SYNOPSIS
funcinterval [-h] [-p PID] [-i INTERVAL] [-d DURATION] [-T] [-u] [-m] [-v] pattern

## DESCRIPTION
This tool times interval between the same function as a histogram.

eBPF/bcc is very suitable for platform performance tuning. By funclatency, we can profile specific functions to know how latency this function costs. However, sometimes performance drop is not about the latency of function but the interval between function calls. funcinterval is born for this purpose.

This tool uses in-kernel eBPF maps for storing timestamps and the histogram, for efficiency.

WARNING: This uses dynamic tracing of (what can be many) functions, an activity that has had issues on some kernel versions (risk of panics or freezes). Test, and know what you are doing, before use.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
pattern Function name. -h Print usage message.

-p PID

Trace this process ID only.

-i INTERVAL

Print output every interval seconds.

-d DURATION

Total duration of trace, in seconds.

-T

Include timestamps on output.

-u

Output histogram in microseconds.

-m

Output histogram in milliseconds.

-v

Print the BPF program (for debugging purposes).

## EXAMPLES
Time the interval of do_sys_open() kernel function as a histogram:

\# funcinterval do_sys_open

Time the interval of xhci_ring_ep_doorbell(), in microseconds:

\# funcinterval -u xhci_ring_ep_doorbell

Time the interval of do_nanosleep(), in milliseconds

\# funcinterval -m do_nanosleep

Output every 5 seconds, with timestamps:

\# funcinterval -mTi 5 vfs_read

Time process 181 only:

\# funcinterval -p 181 vfs_read

Time the interval of mm_vmscan_direct_reclaim_begin tracepoint:

\# funcinterval t:vmscan:mm_vmscan_direct_reclaim_begin

Time the interval of c:malloc used by top every 3 seconds:

\# funcinterval -p `pidof -s top` -i 3 c:malloc

Time /usr/local/bin/python main function:

\# funcinterval /usr/local/bin/python:main

## FIELDS
necs

Nanosecond range

usecs

Microsecond range

msecs

Millisecond range

count

How many calls fell into this range

distribution

An ASCII bar chart to visualize the distribution (count column)

## OVERHEAD
This traces kernel functions and maintains in-kernel timestamps and a histogram, which are asynchronously copied to user-space. While this method is very efficient, the rate of kernel functions can also be very high (>1M/sec), at which point the overhead is expected to be measurable. Measure in a test environment and understand overheads before use. You can also use funccount to measure the rate of kernel functions over a short duration, to set some expectations before use.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Edward Wu

## SEE ALSO
funclatency(8) funccount(8)


bcc-funclatency

title: bcc-funclatency(8)

text:
# bcc-funclatency(8)

## NAME
funclatency - Time functions and print latency as a histogram.

## SYNOPSIS
funclatency [-h] [-p PID] [-i INTERVAL] [-d DURATION] [-T] [-u] [-m] [-F] [-r] [-v] pattern

## DESCRIPTION
This tool traces function calls and times their duration (latency), and shows the latency distribution as a histogram. The time is measured from when the function is called to when it returns, and is inclusive of both on-CPU time and time spent blocked.

This tool uses in-kernel eBPF maps for storing timestamps and the histogram, for efficiency.

Currently nested or recursive functions are not supported properly, and timestamps will be overwritten, creating dubious output. Try to match single functions, or groups of functions that run at the same stack layer, and don't ultimately call each other.

WARNING: This uses dynamic tracing of (what can be many) functions, an activity that has had issues on some kernel versions (risk of panics or freezes). Test, and know what you are doing, before use.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
pattern Function name or search pattern. Supports "*" wildcards. See EXAMPLES. You can also use -r for regular expressions.

-h

Print usage message.

-p PID

Trace this process ID only.

-i INTERVAL

Print output every interval seconds.

-d DURATION

Total duration of trace, in seconds.

-l LEVEL

Set the level of nested or recursive functions.

-T

Include timestamps on output.

-u

Output histogram in microseconds.

-m

Output histogram in milliseconds.

-F

Print a separate histogram per function matched.

-r

Use regular expressions for the search pattern.

-v

Print the BPF program (for debugging purposes).

## EXAMPLES
Time the do_sys_open() kernel function, and print the distribution as a histogram:

\# funclatency do_sys_open

Time the read() function in libc across all processes on the system:

\# funclatency c:read

Time vfs_read(), and print the histogram in units of microseconds:

\# funclatency -u vfs_read

Time do_nanosleep(), and print the histogram in units of milliseconds:

\# funclatency -m do_nanosleep

Time libc open(), and print output every 2 seconds, for duration 10 seconds:

\# funclatency -i 2 -d 10 c:read

Time vfs_read(), and print output every 5 seconds, with timestamps:

\# funclatency -mTi 5 vfs_read

Time vfs_read() for process ID 181 only:

\# funclatency -p 181 vfs_read

Time both vfs_fstat() and vfs_fstatat() calls, by use of a wildcard:

\# funclatency 'vfs_fstat*'

Time both vfs_fstat* calls, and print a separate histogram for each:

\# funclatency -F 'vfs_fstat*'

## FIELDS
necs

Nanosecond range

usecs

Microsecond range

msecs

Millisecond range

count

How many calls fell into this range

distribution

An ASCII bar chart to visualize the distribution (count column)

## OVERHEAD
This traces kernel functions and maintains in-kernel timestamps and a histogram, which are asynchronously copied to user-space. While this method is very efficient, the rate of kernel functions can also be very high (>1M/sec), at which point the overhead is expected to be measurable. Measure in a test environment and understand overheads before use. You can also use funccount to measure the rate of kernel functions over a short duration, to set some expectations before use.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg, Sasha Goldshtein

## SEE ALSO
funccount(8)


bcc-funcslower

title: bcc-funcslower(8)

text:
# bcc-funcslower(8)

## NAME
funcslower - Trace slow kernel or user function calls.

## SYNOPSIS
funcslower [-hf] [-p PID] [-U | -K] [-m MIN_MS] [-u MIN_US] [-a ARGUMENTS] [-T] [-t] [-v] function [function ...]

## DESCRIPTION
This script traces a kernel or user function's entry and return points, and prints a message when the function's latency exceeded the specified threshold. Multiple functions are supported, and you can mix kernel functions with user functions in different libraries.

WARNING: See the OVERHEAD section.

By default, a minimum millisecond threshold of 1 is used. Recursive functions are not supported: only the inner-most recursive invocation will be traced.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-p PID Trace this PID only.

-m MIN_NS

Minimum duration to trace, in milliseconds. Default is 1 ms.

-u MIN_US

Minimum duration to trace, in microseconds.

-a ARGUMENTS

Print the function's arguments, up to 6.

-T

Print a HH:MM:SS timestamp with each entry.

-t

Print a seconds timestamp with each entry, at microsecond resolution.

-f

Print output in folded stack format.

-U

Show stacks from user space only (no kernel space stacks).

-K

Show stacks from kernel space only (no user space stacks).

-v

Print the resulting BPF program, for debugging purposes.

function

The function to trace -- multiple functions are supported. If a plain function name is provided, the function is assumed to be a kernel function. For user functions, provide the library name and the function name, e.g. bash:readline or c:malloc.

## EXAMPLES
Trace vfs_write calls slower than 1ms:

\# funcslower vfs_write

Trace open() calls in libc slower than 10us:

\# funcslower -u 10 c:open

Trace both malloc() and free() slower than 10us, in pid 135 only:

\# funcslower -p 135 -u 10 c:malloc c:free

Trace the write syscall and print its first 4 arguments:

\# funcslower -a 4 SyS_write

Trace opens from libc and print the user and kernel stack frames:

\# funcslower -UK c:open

## FIELDS
TIME

Time of the event as a human-readable HH:MM:SS format, or a timestamp in seconds at microsecond-accuracy from the first event seen.

COMM

Process name.

PID

Process ID.

LAT

Latency of the operation in either microseconds (us) or milliseconds (ms).

RVAL

The return value from the function. Often useful for diagnosing a relationship between slow and failed function calls.

FUNC

The function name, followed by its arguments if requested.

## OVERHEAD
Depending on the function(s) being traced, overhead can become severe. For example, tracing a common function like malloc() can slow down a C/C++ program by a factor of 2 or more. On the other hand, tracing a low-frequency event like the SyS_setreuid() function will probably not be as prohibitive, and in fact negligible for functions that are called up to 100-1000 times per second.

You should first use the funclatency and argdist tools for investigation, because they summarize data in-kernel and have a much lower overhead than this tool. To get a general idea of the number of times a particular function is called (and estimate the overhead), use the funccount tool, e.g.:

\# funccount c:open

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein

## SEE ALSO
funccount(8), funclatency(8), argdist(8), trace(8)


bcc-gethostlatency

title: bcc-gethostlatency(8)

text:
# bcc-gethostlatency(8)

## NAME
gethostlatency - Show latency for getaddrinfo/gethostbyname[2] calls. Uses Linux eBPF/bcc.

## SYNOPSIS
gethostlatency

## DESCRIPTION
This traces and prints when getaddrinfo(), gethostbyname(), and gethostbyname2() are called, system wide, and shows the responsible PID and command name, latency of the call (duration) in milliseconds, and the host string.

This tool can be useful for identifying DNS latency, by identifying which remote host name lookups were slow, and by how much.

This makes use of a Linux 4.4 feature (bpf_perf_event_output()); for kernels older than 4.4, see the version under tools/old, which uses an older mechanism

This tool currently uses dynamic tracing of user-level functions and registers, and may need modifications to match your software and processor architecture.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-p PID

Trace this process ID only.

## EXAMPLES
Trace host lookups (getaddrinfo/gethostbyname[2]) system wide:

\# gethostlatency

## FIELDS
TIME

Time of the command (HH:MM:SS).

PID

Process ID of the client performing the call.

COMM

Process (command) name of the client performing the call.

HOST

Host name string: the target of the lookup.

## OVERHEAD
The rate of lookups should be relatively low, so the overhead is not expected to be a problem.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
tcpdump(8)


bcc-hardirqs

title: bcc-hardirqs(8)

text:
# bcc-hardirqs(8)

## NAME
hardirqs - Measure hard IRQ (hard interrupt) event time. Uses Linux eBPF/bcc.

## SYNOPSIS
hardirqs [-h] [-T] [-N] [-C] [-d] [interval] [outputs]

## DESCRIPTION
This summarizes the time spent servicing hard IRQs (hard interrupts), and can show this time as either totals or histogram distributions. A system-wide summary of this time is shown by the %irq column of mpstat(1), and event counts (but not times) are shown by /proc/interrupts.

This tool uses the irq:irq_handler_entry and irq:irq_handler_exit kernel tracepoints, which is a stable tracing mechanism. BPF programs can attach to tracepoints from Linux 4.7 only. An older version of this tool is available in tools/old, and uses kprobes instead of tracepoints.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print usage message.

-T

Include timestamps on output.

-N

Output in nanoseconds.

-C

Count events only.

-d

Show IRQ time distribution as histograms.

-c CPU

Trace on this CPU only.

## EXAMPLES
Sum hard IRQ event time until Ctrl-C:

\# hardirqs

Show hard IRQ event time as histograms:

\# hardirqs -d

Print 1 second summaries, 10 times:

\# hardirqs 1 10

1 second summaries, printed in nanoseconds, with timestamps:

\# hardirqs -NT 1

Sum hard IRQ event time on CPU 1 until Ctrl-C:

\# hardirqs -c 1

## FIELDS
HARDIRQ

The irq action name for this hard IRQ.

TOTAL_usecs

Total time spent in this hard IRQ in microseconds.

TOTAL_nsecs

Total time spent in this hard IRQ in nanoseconds.

usecs

Range of microseconds for this bucket.

nsecs

Range of nanoseconds for this bucket.

count

Number of hard IRQs in this time range.

distribution

ASCII representation of the distribution (the count column).

## OVERHEAD
This traces kernel functions and maintains in-kernel counts, which are asynchronously copied to user-space. While the rate of interrupts be very high (>1M/sec), this is a relatively efficient way to trace these events, and so the overhead is expected to be small for normal workloads, but could become noticeable for heavy workloads. Measure in a test environment before use.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg, Hengqi Chen, Rocky Xing

## SEE ALSO
softirqs(8)