bcc-dirtop

title: bcc-dirtop(8)

text:
# bcc-dirtop(8)

## NAME
dirtop - File reads and writes by directory. Top for directories.

## SYNOPSIS
dirtop -d directory1,directory2,... [-h] [-C] [-r MAXROWS] [-s {reads,writes,rbytes,wbytes}] [-p PID] [interval] [count]

## DESCRIPTION
This is top for directories.

This traces file reads and writes, and prints a per-directory summary every interval (by default, 1 second). By default the summary is sorted on the highest read throughput (Kbytes). Sorting order can be changed via -s option.

This uses in-kernel eBPF maps to store per process summaries for efficiency.

This script works by tracing the __vfs_read() and __vfs_write() functions using kernel dynamic tracing, which instruments explicit read and write calls. If files are read or written using another means (eg, via mmap()), then they will not be visible using this tool. Also, this tool will need updating to match any code changes to those vfs functions.

This should be useful for file system workload characterization when analyzing the performance of applications.

Note that tracing VFS level reads and writes can be a frequent activity, and this tool can begin to cost measurable overhead at high I/O rates.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-d

Defines a list of directories, comma separated, to observe. Wildcards are allowed if between single bracket.

-C

Don't clear the screen.

-r MAXROWS

Maximum number of rows to print. Default is 20.

-s {reads,writes,rbytes,wbytes}

Sort column. Default is rbytes (read throughput).

-p PID

Trace this PID only.

interval

Interval between updates, seconds.

count

Number of interval summaries.

## EXAMPLES
Summarize block device I/O by directory, 1 second screen refresh:

\# dirtop -d '/hdfs/uuid/*/yarn'

Don't clear the screen, and top 8 rows only:

\# dirtop -d '/hdfs/uuid/*/yarn' -Cr 8

5 second summaries, 10 times only:

\# dirtop -d '/hdfs/uuid/*/yarn' 5 10

Report read & write IOs generated in mutliple yarn and data directories:

\# dirtop -d '/hdfs/uuid/*/yarn,/hdfs/uuid/*/data'

## FIELDS
loadavg:

The contents of /proc/loadavg

READS

Count of reads during interval.

WRITES

Count of writes during interval.

R_Kb

Total read Kbytes during interval.

W_Kb

Total write Kbytes during interval.

PATH

The path were the IOs were accounted.

## OVERHEAD
Depending on the frequency of application reads and writes, overhead can become significant, in the worst case slowing applications by over 50%. Hopefully for real world workloads the overhead is much less -- test before use. The reason for the high overhead is that VFS reads and writes can be a frequent event, and despite the eBPF overhead being very small per event, if you multiply this small overhead by a million events per second, it becomes a million times worse. Literally. You can gauge the number of reads and writes using the vfsstat(8) tool, also from bcc.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Erwan Velu

## INSPIRATION
filetop(8) by Brendan Gregg

## SEE ALSO
vfsstat(8), vfscount(8), fileslower(8)


bcc-drsnoop

package: bcc-tools

title: bcc-drsnoop(8)

text:
# bcc-drsnoop(8)

## NAME
drsnoop - Trace direct reclaim events. Uses Linux eBPF/bcc.

## SYNOPSIS
drsnoop [-h] [-T] [-U] [-p PID] [-t TID] [-u UID] [-d DURATION] [-n name] [-v]

## DESCRIPTION
drsnoop trace direct reclaim events, showing which processes are allocing pages with direct reclaiming. This can be useful for discovering when allocstall (/p- roc/vmstat) continues to increase, whether it is caused by some critical proc- esses or not.

This works by tracing the direct reclaim events using kernel tracepoints.

This makes use of a Linux 4.4 feature (bpf_perf_event_output()); for kernels older than 4.4, see the version under tools/old, which uses an older mechanism.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print usage message.

-T

Include a timestamp column.

-U

Show UID.

-p PID

Trace this process ID only (filtered in-kernel).

-t TID

Trace this thread ID only (filtered in-kernel).

-u UID

Trace this UID only (filtered in-kernel).

-d DURATION

Total duration of trace in seconds.

-n name

Only print processes where its name partially matches 'name' -v verbose Run in verbose mode. Will output system memory state

-v

show system memory state

## EXAMPLES
Trace all direct reclaim events:

\# drsnoop

Trace all direct reclaim events, for 10 seconds only:

\# drsnoop -d 10

Trace all direct reclaim events, and include timestamps:

\# drsnoop -T

Show UID:

\# drsnoop -U

Trace PID 181 only:

\# drsnoop -p 181

Trace UID 1000 only:

\# drsnoop -u 1000

Trace all direct reclaim events from processes where its name partially match-

es 'mond': # drnsnoop -n mond

## FIELDS
TIME(s)

Time of the call, in seconds.

UID

User ID

PID

Process ID

TID

Thread ID

COMM

Process name

## OVERHEAD
This traces the kernel direct reclaim tracepoints and prints output for each event. As the rate of this is generally expected to be low (< 1000/s), the overhead is also expected to be negligible.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Wenbo Zhang


bcc-execsnoop

title: bcc-execsnoop(8)

text:
# bcc-execsnoop(8)

## NAME
execsnoop - Trace new processes via exec() syscalls. Uses Linux eBPF/bcc.

## SYNOPSIS
execsnoop [-h] [-T] [-t] [-x] [--cgroupmap CGROUPMAP] [--mntnsmap MAPPATH] [-u USER] [-q] [-n NAME] [-l LINE] [-U] [--max-args MAX_ARGS]

## DESCRIPTION
execsnoop traces new processes, showing the filename executed and argument list.

It works by traces the execve() system call (commonly used exec() variant). This catches new processes that follow the fork->exec sequence, as well as processes that re-exec() themselves. Some applications fork() but do not exec(), eg, for worker processes, which won't be included in the execsnoop output.

This works by tracing the kernel sys_execve() function using dynamic tracing, and will need updating to match any changes to this function.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print usage message.

-T

Include a time column (HH:MM:SS).

-U

Include UID column.

-t

Include a timestamp column.

-u USER

Filter by UID (or username)

-x

Include failed exec()s

-q

Add "quotemarks" around arguments. Escape quotemarks in arguments with a backslash. For tracing empty arguments or arguments that contain whitespace.

-n NAME

Only print command lines matching this name (regex)

-l LINE

Only print commands where arg contains this line (regex)

--max-args MAXARGS

Maximum number of arguments parsed and displayed, defaults to 20

--cgroupmap MAPPATH

Trace cgroups in this BPF map only (filtered in-kernel).

--mntnsmap MAPPATH

Trace mount namespaces in this BPF map only (filtered in-kernel).

-P PPID

Trace this parent PID only.

## EXAMPLES
Trace all exec() syscalls:

\# execsnoop

Trace all exec() syscalls, and include timestamps:

\# execsnoop -t

Display process UID:

\# execsnoop -U

Trace only UID 1000:

\# execsnoop -u 1000

Trace only processes launched by root and display UID column:

\# execsnoop -Uu root

Include failed exec()s:

\# execsnoop -x

Put quotemarks around arguments.

\# execsnoop -q

Only trace exec()s where the filename contains "mount":

\# execsnoop -n mount

Only trace exec()s where argument's line contains "testpkg":

\# execsnoop -l testpkg

Trace a set of cgroups only (see special_filtering.md from bcc sources for more details):

\# execsnoop --cgroupmap /sys/fs/bpf/test01

## FIELDS
TIME

Time of exec() return, in HH:MM:SS format.

TIME(s)

Time of exec() return, in seconds.

UID

User ID

PCOMM

Parent process/command name.

PID

Process ID

PPID

Parent process ID

RET

Return value of exec(). 0 == successs. Failures are only shown when using the -x option.

ARGS

Filename for the exec(), followed be up to 19 arguments. An ellipsis "..." is shown if the argument list is known to be truncated.

## OVERHEAD
This traces the kernel execve function and prints output for each event. As the rate of this is generally expected to be low (< 1000/s), the overhead is also expected to be negligible. If you have an application that is calling a high rate of exec()s, then test and understand overhead before use.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg, Rocky Xing

## SEE ALSO
opensnoop(1)


bcc-exitsnoop

title: bcc-exitsnoop(8)

text:
# bcc-exitsnoop(8)

## NAME
exitsnoop - Trace all process termination (exit, fatal signal). Uses Linux eBPF/bcc.

## SYNOPSIS
exitsnoop [-h] [-t] [--utc] [-x] [-p PID] [--label LABEL] [--per-thread]

## DESCRIPTION
exitsnoop traces process termination, showing the command name and reason for termination, either an exit or a fatal signal.

It catches processes of all users, processes in containers, as well as processes that become zombie.

This works by tracing the kernel sched_process_exit() function using dynamic tracing, and will need updating to match any changes to this function.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print usage message.

-t

Include a timestamp column.

--utc

Include a timestamp column, use UTC timezone.

-x

Exclude successful exits, exit( 0 )

-p PID

Trace this process ID only (filtered in-kernel).

--label LABEL

Label each line with LABEL (default 'exit') in first column (2nd if timestamp is present).

--per-thread

Trace per thread termination

## EXAMPLES
Trace all process termination

\# exitsnoop

Trace all process termination, and include timestamps:

\# exitsnoop -t

Exclude successful exits, only include non-zero exit codes and fatal signals:

\# exitsnoop -x

Trace PID 181 only:

\# exitsnoop -p 181

Label each output line with 'EXIT':

\# exitsnoop --label EXIT

Trace per thread termination

\# exitsnoop --per-thread

## FIELDS
TIME-TZ

Time of process termination HH:MM:SS.sss with milliseconds, where TZ is the local time zone, 'UTC' with --utc option.

LABEL

The optional label if --label option is used. This is useful with the -t option for timestamps when the output of several tracing tools is sorted into one combined output.

PCOMM

Process/command name.

PID

Process ID

PPID

The process ID of the process that will be notified of PID termination.

TID

Thread ID.

EXIT_CODE

The exit code for exit() or the signal number for a fatal signal.

## OVERHEAD
This traces the kernel sched_process_exit() function and prints output for each event. As the rate of this is generally expected to be low (< 1000/s), the overhead is also expected to be negligible. If you have an application that has a high rate of process termination, then test and understand overhead before use.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Arturo Martin-de-Nicolas

## SEE ALSO
execsnoop(8)


title: bcc-ext4dist(8)

text:
# bcc-ext4dist(8)

## NAME
ext4dist - Summarize ext4 operation latency. Uses Linux eBPF/bcc.

## SYNOPSIS
ext4dist [-h] [-T] [-m] [-p PID] [interval] [count]

## DESCRIPTION
This tool summarizes time (latency) spent in common ext4 file operations: reads, writes, opens, and syncs, and presents it as a power-of-2 histogram. It uses an in-kernel eBPF map to store the histogram for efficiency.

Since this works by tracing the ext4_file_operations interface functions, it will need updating to match any changes to these functions.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print usage message.

-T

Don't include timestamps on interval output.

-m

Output in milliseconds.

-p PID

Trace this PID only.

## EXAMPLES
Trace ext4 operation time, and print a summary on Ctrl-C:

\# ext4dist

Trace PID 181 only:

\# ext4dist -p 181

Print 1 second summaries, 10 times:

\# ext4dist 1 10

1 second summaries, printed in milliseconds

\# ext4dist -m 1

## FIELDS
msecs

Range of milliseconds for this bucket.

usecs

Range of microseconds for this bucket.

count

Number of operations in this time range.

distribution

ASCII representation of the distribution (the count column).

## OVERHEAD
This adds low-overhead instrumentation to these ext4 operations, including reads and writes from the file system cache. Such reads and writes can be very frequent (depending on the workload; eg, 1M/sec), at which point the overhead of this tool may become noticeable. Measure and quantify before use.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
ext4snoop(8)


title: bcc-ext4slower(8)

text:
# bcc-ext4slower(8)

## NAME
ext4slower - Trace slow ext4 file operations, with per-event details.

## SYNOPSIS
ext4slower [-h] [-j] [-p PID] [min_ms]

## DESCRIPTION
This tool traces common ext4 file operations: reads, writes, opens, and syncs. It measures the time spent in these operations, and prints details for each that exceeded a threshold.

WARNING: See the OVERHEAD section.

By default, a minimum millisecond threshold of 10 is used. If a threshold of 0 is used, all events are printed (warning: verbose).

Since this works by tracing the ext4_file_operations interface functions, it will need updating to match any changes to these functions.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-p PID Trace this PID only.

min_ms

Minimum I/O latency (duration) to trace, in milliseconds. Default is 10 ms.

## EXAMPLES
Trace synchronous file reads and writes slower than 10 ms:

\# ext4slower

Trace slower than 1 ms:

\# ext4slower 1

Trace slower than 1 ms, and output just the fields in parsable format (csv):

\# ext4slower -j 1

Trace all file reads and writes (warning: the output will be verbose):

\# ext4slower 0

Trace slower than 1 ms, for PID 181 only:

\# ext4slower -p 181 1

## FIELDS
TIME(s)

Time of I/O completion since the first I/O seen, in seconds.

COMM

Process name.

PID

Process ID.

T

Type of operation. R == read, W == write, O == open, S == fsync.

OFF_KB

File offset for the I/O, in Kbytes.

BYTES

Size of I/O, in bytes.

LAT(ms)

Latency (duration) of I/O, measured from when it was issued by VFS to the filesystem, to when it completed. This time is inclusive of block device I/O, file system CPU cycles, file system locks, run queue latency, etc. It's a more accurate measure of the latency suffered by applications performing file system I/O, than to measure this down at the block device interface.

FILENAME

A cached kernel file name (comes from dentry->d_name.name).

ENDTIME_us

Completion timestamp, microseconds (-j only).

OFFSET_b

File offset, bytes (-j only).

LATENCY_us

Latency (duration) of the I/O, in microseconds (-j only).

## OVERHEAD
This adds low-overhead instrumentation to these ext4 operations, including reads and writes from the file system cache. Such reads and writes can be very frequent (depending on the workload; eg, 1M/sec), at which point the overhead of this tool (even if it prints no "slower" events) can begin to become significant. Measure and quantify before use. If this continues to be a problem, consider switching to a tool that prints in-kernel summaries only.

Note that the overhead of this tool should be less than fileslower(8), as this tool targets ext4 functions only, and not all file read/write paths (which can include socket I/O).

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
biosnoop(8), funccount(8), fileslower(8)


title: bcc-f2fsslower(8)

text:
# bcc-f2fsslower(8)

## NAME
f2fsslower - Trace slow f2fs file operations, with per-event details.

## SYNOPSIS
f2fsslower [-h] [-s] [-p PID] [min_ms]

## DESCRIPTION
This tool traces common f2fs file operations: reads, writes, opens, and syncs. It measures the time spent in these operations, and prints details for each that exceeded a threshold.

WARNING: See the OVERHEAD section.

By default, a minimum millisecond threshold of 10 is used. If a threshold of 0 is used, all events are printed (warning: verbose).

Since this works by tracing the f2fs_file_operations interface functions, it will need updating to match any changes to these functions.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-p PID Trace this PID only.

min_ms

Minimum I/O latency (duration) to trace, in milliseconds. Default is 10 ms.

## EXAMPLES
Trace synchronous file reads and writes slower than 10 ms:

\# f2fsslower

Trace slower than 1 ms:

\# f2fsslower 1

Trace slower than 1 ms, and output just the fields in parsable format (csv):

\# f2fsslower -s 1

Trace all file reads and writes (warning: the output will be verbose):

\# f2fsslower 0

Trace slower than 1 ms, for PID 181 only:

\# f2fsslower -p 181 1

## FIELDS
TIME(s)

Time of I/O completion since the first I/O seen, in seconds.

COMM

Process name.

PID

Process ID.

T

Type of operation. R == read, W == write, O == open, S == fsync.

OFF_KB

File offset for the I/O, in Kbytes.

BYTES

Size of I/O, in bytes.

LAT(ms)

Latency (duration) of I/O, measured from when it was issued by VFS to the filesystem, to when it completed. This time is inclusive of block device I/O, file system CPU cycles, file system locks, run queue latency, etc. It's a more accurate measure of the latency suffered by applications performing file system I/O, than to measure this down at the block device interface.

FILENAME

A cached kernel file name (comes from dentry->d_name.name).

ENDTIME_us

Completion timestamp, microseconds (-s only).

OFFSET_b

File offset, bytes (-s only).

LATENCY_us

Latency (duration) of the I/O, in microseconds (-s only).

## OVERHEAD
This adds low-overhead instrumentation to these f2fs operations, including reads and writes from the file system cache. Such reads and writes can be very frequent (depending on the workload; eg, 1M/sec), at which point the overhead of this tool (even if it prints no "slower" events) can begin to become significant. Measure and quantify before use. If this continues to be a problem, consider switching to a tool that prints in-kernel summaries only.

Note that the overhead of this tool should be less than fileslower(8), as this tool targets f2fs functions only, and not all file read/write paths (which can include socket I/O).

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Ting Zhang

## SEE ALSO
biosnoop(8), funccount(8), fileslower(8)


bcc-filegone

title: bcc-filegone(8)

text:
# bcc-filegone(8)

## NAME
filegone - Trace why file gone (deleted or renamed). Uses Linux eBPF/bcc.

## SYNOPSIS
filegone [-h] [-p PID]

## DESCRIPTION
This traces why file gone/vanished, providing information on who deleted or renamed the file.

This works by tracing the kernel vfs_unlink() , vfs_rmdir() , vfs_rename functions.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print usage message.

-p PID

Trace this process ID only (filtered in-kernel).

## EXAMPLES
Trace all file gone events

\# filegone

Trace file gone events caused by PID 181:

\# filegone -p 181

## FIELDS
TIME

Time of the event.

PID

Process ID that renamed/deleted the file.

COMM

Process name for the PID.

ACTION

action on file: 'DELETE' or 'RENAME'

FILE

Filename.

## OVERHEAD
This traces the kernel VFS file rename and delete functions and prints output for each event. As the rate of this is generally expected to be low (< 1000/s), the overhead is also expected to be negligible. This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Curu Wong

## SEE ALSO
filelife(8)


bcc-filelife

title: bcc-filelife(8)

text:
# bcc-filelife(8)

## NAME
filelife - Trace the lifespan of short-lived files. Uses Linux eBPF/bcc.

## SYNOPSIS
filelife [-h] [-p PID]

## DESCRIPTION
This traces the creation and deletion of files, providing information on who deleted the file, the file age, and the file name. The intent is to provide information on short-lived files, for debugging or performance analysis.

This works by tracing the kernel vfs_create() and vfs_delete() functions (and maybe more, see the source) using dynamic tracing, and will need updating to match any changes to these functions.

This makes use of a Linux 4.4 feature (bpf_perf_event_output()); for kernels older than 4.4, see the version under tools/old, which uses an older mechanism.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print usage message.

-p PID

Trace this process ID only (filtered in-kernel).

## EXAMPLES
Trace all short-lived files, and print details:

\# filelife

Trace all short-lived files created AND deleted by PID 181:

\# filelife -p 181

## FIELDS
TIME

Time of the deletion.

PID

Process ID that deleted the file.

COMM

Process name for the PID.

AGE(s)

Age of the file, from creation to deletion, in seconds.

FILE

Filename.

## OVERHEAD
This traces the kernel VFS file create and delete functions and prints output for each delete. As the rate of this is generally expected to be low (< 1000/s), the overhead is also expected to be negligible. This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
opensnoop(1)


bcc-fileslower

title: bcc-fileslower(8)

text:
# bcc-fileslower(8)

## NAME
fileslower - Trace slow synchronous file reads and writes.

## SYNOPSIS
fileslower [-h] [-p PID] [-a] [min_ms]

## DESCRIPTION
This script uses kernel dynamic tracing of synchronous reads and writes at the VFS interface, to identify slow file reads and writes for any file system.

This version traces __vfs_read() and __vfs_write() and only showing synchronous I/O (the path to new_sync_read() and new_sync_write()), and I/O with filenames. This approach provides a view of just two file system request types: file reads and writes. There are typically many others: asynchronous I/O, directory operations, file handle operations, file open()s, fflush(), etc.

WARNING: See the OVERHEAD section.

By default, a minimum millisecond threshold of 10 is used.

Since this works by tracing various kernel __vfs_*() functions using dynamic tracing, it will need updating to match any changes to these functions. A future version should switch to using FS tracepoints instead.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-p PID Trace this PID only.

-a

Include non-regular file types in output (sockets, FIFOs, etc).

min_ms

Minimum I/O latency (duration) to trace, in milliseconds. Default is 10 ms.

## EXAMPLES
Trace synchronous file reads and writes slower than 10 ms:

\# fileslower

Trace slower than 1 ms:

\# fileslower 1

Trace slower than 1 ms, for PID 181 only:

\# fileslower -p 181 1

## FIELDS
TIME(s)

Time of I/O completion since the first I/O seen, in seconds.

COMM

Process name.

PID

Process ID.

D

Direction of I/O. R == read, W == write.

BYTES

Size of I/O, in bytes.

LAT(ms)

Latency (duration) of I/O, measured from when the application issued it to VFS to when it completed. This time is inclusive of block device I/O, file system CPU cycles, file system locks, run queue latency, etc. It's a more accurate measure of the latency suffered by applications performing file system I/O, than to measure this down at the block device interface.

FILENAME

A cached kernel file name (comes from dentry->d_name.name).

## OVERHEAD
Depending on the frequency of application reads and writes, overhead can become severe, in the worst case slowing applications by 2x. In the best case, the overhead is negligible. Hopefully for real world workloads the overhead is often at the lower end of the spectrum -- test before use. The reason for high overhead is that this traces VFS reads and writes, which includes FS cache reads and writes, and can exceed one million events per second if the application is I/O heavy. While the instrumentation is extremely lightweight, and uses in-kernel eBPF maps for efficient timing and filtering, multiply that cost by one million events per second and that cost becomes a million times worse. You can get an idea of the possible cost by just counting the instrumented events using the bcc funccount tool, eg:

\# ./funccount.py -i 1 -r '^__vfs_(read|write)$'

This also costs overhead, but is somewhat less than fileslower.

If the overhead is prohibitive for your workload, I'd recommend moving down-stack a little from VFS into the file system functions (ext4, xfs, etc). Look for updates to bcc for specific file system tools that do this. The advantage of a per-file system approach is that we can trace post-cache, greatly reducing events and overhead. The disadvantage is needing custom tracing approaches for each different file system (whereas VFS is generic).

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
biosnoop(8), funccount(8)


bcc-filetop

title: bcc-filetop(8)

text:
# bcc-filetop(8)

## NAME
filetop - File reads and writes by filename and process. Top for files.

## SYNOPSIS
filetop [-h] [-a] [-C] [-r MAXROWS] [-s {reads,writes,rbytes,wbytes}] [-p PID] [--read-only] [--write-only] [interval] [count]

## DESCRIPTION
This is top for files.

This traces file reads and writes, and prints a per-file summary every interval (by default, 1 second). By default the summary is sorted on the highest read throughput (Kbytes). Sorting order can be changed via -s option. By default only IO on regular files is shown. The -a option will list all file types (sockets, FIFOs, etc). Only read or only write operations can be traced using --read-only or --write-only options respectively. By default both are traced. Using both options at once will raise an Exception.

This uses in-kernel eBPF maps to store per process summaries for efficiency.

This script works by tracing the __vfs_read() and __vfs_write() functions using kernel dynamic tracing, which instruments explicit read and write calls. If files are read or written using another means (eg, via mmap()), then they will not be visible using this tool. Also, this tool will need updating to match any code changes to those vfs functions.

This should be useful for file system workload characterization when analyzing the performance of applications.

Note that tracing VFS level reads and writes can be a frequent activity, and this tool can begin to cost measurable overhead at high I/O rates.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-a

Include non-regular file types (sockets, FIFOs, etc).

-C

Don't clear the screen.

-r MAXROWS

Maximum number of rows to print. Default is 20.

-s {reads,writes,rbytes,wbytes}

Sort column. Default is rbytes (read throughput).

-p PID

Trace this PID only.

--read-only

Trace read operations only

--write-only

Trace write operations only

interval

Interval between updates, seconds.

count

Number of interval summaries.

## EXAMPLES
Summarize block device I/O by process, 1 second screen refresh:

\# filetop

Don't clear the screen, and top 8 rows only:

\# filetop -Cr 8

5 second summaries, 10 times only:

\# filetop 5 10

## FIELDS
loadavg:

The contents of /proc/loadavg

PID

Process ID.

COMM

Process name.

READS

Count of reads during interval.

WRITES

Count of writes during interval.

R_Kb

Total read Kbytes during interval.

W_Kb

Total write Kbytes during interval.

T

Type of file: R == regular, S == socket, O == other (pipe, etc).

FILE

Filename.

## OVERHEAD
Depending on the frequency of application reads and writes, overhead can become significant, in the worst case slowing applications by over 50%. Hopefully for real world workloads the overhead is much less -- test before use. The reason for the high overhead is that VFS reads and writes can be a frequent event, and despite the eBPF overhead being very small per event, if you multiply this small overhead by a million events per second, it becomes a million times worse. Literally. You can gauge the number of reads and writes using the vfsstat(8) tool, also from bcc.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## INSPIRATION
top(1) by William LeFebvre

## SEE ALSO
vfsstat(8), vfscount(8), fileslower(8)


bcc-funccount

title: bcc-funccount(8)

text:
# bcc-funccount(8)

## NAME
funccount - Count function, tracepoint, and USDT probe calls matching a pattern. Uses Linux eBPF/bcc.

## SYNOPSIS
funccount [-h] [-p PID] [-i INTERVAL] [-d DURATION] [-T] [-r] [-c CPU] [-D] pattern

## DESCRIPTION
This tool is a quick way to determine which functions are being called, and at what rate. It uses in-kernel eBPF maps to count function calls.

WARNING: This uses dynamic tracing of (what can be many) functions, an activity that has had issues on some kernel versions (risk of panics or freezes). Test, and know what you are doing, before use.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
pattern Search pattern. Supports "*" wildcards. See EXAMPLES. You can also use -r for regular expressions.

-h

Print usage message.

-p PID

Trace this process ID only.

-i INTERVAL

Print output every interval seconds.

-d DURATION

Total duration of trace in seconds.

-T

Include timestamps on output.

-r

Use regular expressions for the search pattern.

-D

Print the BPF program before starting (for debugging purposes).

-c CPU

Trace on this CPU only.

## EXAMPLES
Count kernel functions beginning with "vfs_", until Ctrl-C is hit:

\# funccount 'vfs_*'

Count kernel functions beginning with "tcp_send", until Ctrl-C is hit:

\# funccount 'tcp_send*'

Print kernel functions beginning with "vfs_", every second:

\# funccount -i 1 'vfs_*'

Print kernel functions beginning with "vfs_", for ten seconds only:

\# funccount -d 10 'vfs_*'

Match kernel functions beginning with "vfs_", using regular expressions:

\# funccount -r '^vfs_.*'

Count vfs calls for process ID 181 only:

\# funccount -p 181 'vfs_*'

Count calls to the sched_fork tracepoint, indicating a fork() performed:

\# funccount t:sched:sched_fork

Count all GC USDT probes in the Node process:

\# funccount -p 185 u:node:gc*

Count all malloc() calls in libc:

\# funccount c:malloc

Count kernel functions beginning with "vfs_" on CPU 1 only:

\# funccount -c 1 'vfs_*'

## FIELDS
FUNC

Function name

COUNT

Number of calls while tracing

## OVERHEAD
This traces functions and maintains in-kernel counts, which are asynchronously copied to user-space. While the rate of calls be very high (>1M/sec), this is a relatively efficient way to trace these events, and so the overhead is expected to be small for normal workloads. Measure in a test environment before use.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg, Sasha Goldshtein

## SEE ALSO
stackcount(8) funclatency(8) vfscount(8)


bcc-funcinterval

title: bcc-funcinterval(8)

text:
# bcc-funcinterval(8)

## NAME
funcinterval - Time interval between the same function, tracepoint as a histogram.

## SYNOPSIS
funcinterval [-h] [-p PID] [-i INTERVAL] [-d DURATION] [-T] [-u] [-m] [-v] pattern

## DESCRIPTION
This tool times interval between the same function as a histogram.

eBPF/bcc is very suitable for platform performance tuning. By funclatency, we can profile specific functions to know how latency this function costs. However, sometimes performance drop is not about the latency of function but the interval between function calls. funcinterval is born for this purpose.

This tool uses in-kernel eBPF maps for storing timestamps and the histogram, for efficiency.

WARNING: This uses dynamic tracing of (what can be many) functions, an activity that has had issues on some kernel versions (risk of panics or freezes). Test, and know what you are doing, before use.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
pattern Function name. -h Print usage message.

-p PID

Trace this process ID only.

-i INTERVAL

Print output every interval seconds.

-d DURATION

Total duration of trace, in seconds.

-T

Include timestamps on output.

-u

Output histogram in microseconds.

-m

Output histogram in milliseconds.

-v

Print the BPF program (for debugging purposes).

## EXAMPLES
Time the interval of do_sys_open() kernel function as a histogram:

\# funcinterval do_sys_open

Time the interval of xhci_ring_ep_doorbell(), in microseconds:

\# funcinterval -u xhci_ring_ep_doorbell

Time the interval of do_nanosleep(), in milliseconds

\# funcinterval -m do_nanosleep

Output every 5 seconds, with timestamps:

\# funcinterval -mTi 5 vfs_read

Time process 181 only:

\# funcinterval -p 181 vfs_read

Time the interval of mm_vmscan_direct_reclaim_begin tracepoint:

\# funcinterval t:vmscan:mm_vmscan_direct_reclaim_begin

Time the interval of c:malloc used by top every 3 seconds:

\# funcinterval -p `pidof -s top` -i 3 c:malloc

Time /usr/local/bin/python main function:

\# funcinterval /usr/local/bin/python:main

## FIELDS
necs

Nanosecond range

usecs

Microsecond range

msecs

Millisecond range

count

How many calls fell into this range

distribution

An ASCII bar chart to visualize the distribution (count column)

## OVERHEAD
This traces kernel functions and maintains in-kernel timestamps and a histogram, which are asynchronously copied to user-space. While this method is very efficient, the rate of kernel functions can also be very high (>1M/sec), at which point the overhead is expected to be measurable. Measure in a test environment and understand overheads before use. You can also use funccount to measure the rate of kernel functions over a short duration, to set some expectations before use.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Edward Wu

## SEE ALSO
funclatency(8) funccount(8)


bcc-funclatency

title: bcc-funclatency(8)

text:
# bcc-funclatency(8)

## NAME
funclatency - Time functions and print latency as a histogram.

## SYNOPSIS
funclatency [-h] [-p PID] [-i INTERVAL] [-d DURATION] [-T] [-u] [-m] [-F] [-r] [-v] pattern

## DESCRIPTION
This tool traces function calls and times their duration (latency), and shows the latency distribution as a histogram. The time is measured from when the function is called to when it returns, and is inclusive of both on-CPU time and time spent blocked.

This tool uses in-kernel eBPF maps for storing timestamps and the histogram, for efficiency.

Currently nested or recursive functions are not supported properly, and timestamps will be overwritten, creating dubious output. Try to match single functions, or groups of functions that run at the same stack layer, and don't ultimately call each other.

WARNING: This uses dynamic tracing of (what can be many) functions, an activity that has had issues on some kernel versions (risk of panics or freezes). Test, and know what you are doing, before use.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
pattern Function name or search pattern. Supports "*" wildcards. See EXAMPLES. You can also use -r for regular expressions.

-h

Print usage message.

-p PID

Trace this process ID only.

-i INTERVAL

Print output every interval seconds.

-d DURATION

Total duration of trace, in seconds.

-l LEVEL

Set the level of nested or recursive functions.

-T

Include timestamps on output.

-u

Output histogram in microseconds.

-m

Output histogram in milliseconds.

-F

Print a separate histogram per function matched.

-r

Use regular expressions for the search pattern.

-v

Print the BPF program (for debugging purposes).

## EXAMPLES
Time the do_sys_open() kernel function, and print the distribution as a histogram:

\# funclatency do_sys_open

Time the read() function in libc across all processes on the system:

\# funclatency c:read

Time vfs_read(), and print the histogram in units of microseconds:

\# funclatency -u vfs_read

Time do_nanosleep(), and print the histogram in units of milliseconds:

\# funclatency -m do_nanosleep

Time libc open(), and print output every 2 seconds, for duration 10 seconds:

\# funclatency -i 2 -d 10 c:read

Time vfs_read(), and print output every 5 seconds, with timestamps:

\# funclatency -mTi 5 vfs_read

Time vfs_read() for process ID 181 only:

\# funclatency -p 181 vfs_read

Time both vfs_fstat() and vfs_fstatat() calls, by use of a wildcard:

\# funclatency 'vfs_fstat*'

Time both vfs_fstat* calls, and print a separate histogram for each:

\# funclatency -F 'vfs_fstat*'

## FIELDS
necs

Nanosecond range

usecs

Microsecond range

msecs

Millisecond range

count

How many calls fell into this range

distribution

An ASCII bar chart to visualize the distribution (count column)

## OVERHEAD
This traces kernel functions and maintains in-kernel timestamps and a histogram, which are asynchronously copied to user-space. While this method is very efficient, the rate of kernel functions can also be very high (>1M/sec), at which point the overhead is expected to be measurable. Measure in a test environment and understand overheads before use. You can also use funccount to measure the rate of kernel functions over a short duration, to set some expectations before use.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg, Sasha Goldshtein

## SEE ALSO
funccount(8)


bcc-funcslower

title: bcc-funcslower(8)

text:
# bcc-funcslower(8)

## NAME
funcslower - Trace slow kernel or user function calls.

## SYNOPSIS
funcslower [-hf] [-p PID] [-U | -K] [-m MIN_MS] [-u MIN_US] [-a ARGUMENTS] [-T] [-t] [-v] function [function ...]

## DESCRIPTION
This script traces a kernel or user function's entry and return points, and prints a message when the function's latency exceeded the specified threshold. Multiple functions are supported, and you can mix kernel functions with user functions in different libraries.

WARNING: See the OVERHEAD section.

By default, a minimum millisecond threshold of 1 is used. Recursive functions are not supported: only the inner-most recursive invocation will be traced.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-p PID Trace this PID only.

-m MIN_NS

Minimum duration to trace, in milliseconds. Default is 1 ms.

-u MIN_US

Minimum duration to trace, in microseconds.

-a ARGUMENTS

Print the function's arguments, up to 6.

-T

Print a HH:MM:SS timestamp with each entry.

-t

Print a seconds timestamp with each entry, at microsecond resolution.

-f

Print output in folded stack format.

-U

Show stacks from user space only (no kernel space stacks).

-K

Show stacks from kernel space only (no user space stacks).

-v

Print the resulting BPF program, for debugging purposes.

function

The function to trace -- multiple functions are supported. If a plain function name is provided, the function is assumed to be a kernel function. For user functions, provide the library name and the function name, e.g. bash:readline or c:malloc.

## EXAMPLES
Trace vfs_write calls slower than 1ms:

\# funcslower vfs_write

Trace open() calls in libc slower than 10us:

\# funcslower -u 10 c:open

Trace both malloc() and free() slower than 10us, in pid 135 only:

\# funcslower -p 135 -u 10 c:malloc c:free

Trace the write syscall and print its first 4 arguments:

\# funcslower -a 4 SyS_write

Trace opens from libc and print the user and kernel stack frames:

\# funcslower -UK c:open

## FIELDS
TIME

Time of the event as a human-readable HH:MM:SS format, or a timestamp in seconds at microsecond-accuracy from the first event seen.

COMM

Process name.

PID

Process ID.

LAT

Latency of the operation in either microseconds (us) or milliseconds (ms).

RVAL

The return value from the function. Often useful for diagnosing a relationship between slow and failed function calls.

FUNC

The function name, followed by its arguments if requested.

## OVERHEAD
Depending on the function(s) being traced, overhead can become severe. For example, tracing a common function like malloc() can slow down a C/C++ program by a factor of 2 or more. On the other hand, tracing a low-frequency event like the SyS_setreuid() function will probably not be as prohibitive, and in fact negligible for functions that are called up to 100-1000 times per second.

You should first use the funclatency and argdist tools for investigation, because they summarize data in-kernel and have a much lower overhead than this tool. To get a general idea of the number of times a particular function is called (and estimate the overhead), use the funccount tool, e.g.:

\# funccount c:open

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein

## SEE ALSO
funccount(8), funclatency(8), argdist(8), trace(8)


bcc-gethostlatency

title: bcc-gethostlatency(8)

text:
# bcc-gethostlatency(8)

## NAME
gethostlatency - Show latency for getaddrinfo/gethostbyname[2] calls. Uses Linux eBPF/bcc.

## SYNOPSIS
gethostlatency

## DESCRIPTION
This traces and prints when getaddrinfo(), gethostbyname(), and gethostbyname2() are called, system wide, and shows the responsible PID and command name, latency of the call (duration) in milliseconds, and the host string.

This tool can be useful for identifying DNS latency, by identifying which remote host name lookups were slow, and by how much.

This makes use of a Linux 4.4 feature (bpf_perf_event_output()); for kernels older than 4.4, see the version under tools/old, which uses an older mechanism

This tool currently uses dynamic tracing of user-level functions and registers, and may need modifications to match your software and processor architecture.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-p PID

Trace this process ID only.

## EXAMPLES
Trace host lookups (getaddrinfo/gethostbyname[2]) system wide:

\# gethostlatency

## FIELDS
TIME

Time of the command (HH:MM:SS).

PID

Process ID of the client performing the call.

COMM

Process (command) name of the client performing the call.

HOST

Host name string: the target of the lookup.

## OVERHEAD
The rate of lookups should be relatively low, so the overhead is not expected to be a problem.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
tcpdump(8)


bcc-hardirqs

title: bcc-hardirqs(8)

text:
# bcc-hardirqs(8)

## NAME
hardirqs - Measure hard IRQ (hard interrupt) event time. Uses Linux eBPF/bcc.

## SYNOPSIS
hardirqs [-h] [-T] [-N] [-C] [-d] [interval] [outputs]

## DESCRIPTION
This summarizes the time spent servicing hard IRQs (hard interrupts), and can show this time as either totals or histogram distributions. A system-wide summary of this time is shown by the %irq column of mpstat(1), and event counts (but not times) are shown by /proc/interrupts.

This tool uses the irq:irq_handler_entry and irq:irq_handler_exit kernel tracepoints, which is a stable tracing mechanism. BPF programs can attach to tracepoints from Linux 4.7 only. An older version of this tool is available in tools/old, and uses kprobes instead of tracepoints.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print usage message.

-T

Include timestamps on output.

-N

Output in nanoseconds.

-C

Count events only.

-d

Show IRQ time distribution as histograms.

-c CPU

Trace on this CPU only.

## EXAMPLES
Sum hard IRQ event time until Ctrl-C:

\# hardirqs

Show hard IRQ event time as histograms:

\# hardirqs -d

Print 1 second summaries, 10 times:

\# hardirqs 1 10

1 second summaries, printed in nanoseconds, with timestamps:

\# hardirqs -NT 1

Sum hard IRQ event time on CPU 1 until Ctrl-C:

\# hardirqs -c 1

## FIELDS
HARDIRQ

The irq action name for this hard IRQ.

TOTAL_usecs

Total time spent in this hard IRQ in microseconds.

TOTAL_nsecs

Total time spent in this hard IRQ in nanoseconds.

usecs

Range of microseconds for this bucket.

nsecs

Range of nanoseconds for this bucket.

count

Number of hard IRQs in this time range.

distribution

ASCII representation of the distribution (the count column).

## OVERHEAD
This traces kernel functions and maintains in-kernel counts, which are asynchronously copied to user-space. While the rate of interrupts be very high (>1M/sec), this is a relatively efficient way to trace these events, and so the overhead is expected to be small for normal workloads, but could become noticeable for heavy workloads. Measure in a test environment before use.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg, Hengqi Chen, Rocky Xing

## SEE ALSO
softirqs(8)


bcc-inject

title: bcc-inject(8)

text:
# bcc-inject(8)

## NAME
inject - injects appropriate error into function if input call chain and predicates are satisfied. Uses Linux eBPF/bcc.

## SYNOPSIS
inject -h [-I header] [-P probability] [-v] [-c count] <mode> <spec>

## DESCRIPTION
inject injects errors into specified kernel functionality when a given call chain and associated predicates are satisfied.

WARNING: This tool injects failures into key kernel functions and may crash the kernel. You should know what you're doing if you're using this tool.

This makes use of a Linux 4.16 feature (bpf_override_return())

Since this uses BPF, only the root user can use this tool.

## OPTIONS
-h

Print usage message.

-v

Display the generated BPF program, for debugging or modification.

-I header

Necessary headers to be included.

-P probability

Optional probability of failure, default 1.

-c count

Number of errors to inject before stopping, default never stops.

## MODE
kmalloc

Make the following function indicate failure

int should_failslab(struct kmem_cache *s, gfp_t gfpflags)

bio

Make the following function indicate failure

int should_fail_bio(struct bio *bio)

alloc_page

Make the following function indicate failure

bool should_fail_alloc_page(gfp_t gfp_mask, unsigned int order)

## SPEC
FUNCTION([ARGS])[(TEST)] [=> ...]

A list of predicates separated by "=>". A predicate is a function signature (name and arguments) in a call stack and a test on the function's arguments.

Missing predicates are implicitly true. Missing tests are implicitly true. Specifying the function arguments is optional if the test does not use them. If the error injection function is not listed as the first predicate, it is implicitly added.

Functions are listed in the reverse order that they are called, ie. if a() calls b(), the spec would be "b() => a()".

## REQUIREMENTS
CONFIG_BPF, CONFIG_BPF_KPROBE_OVERRIDE, bcc

## EXAMPLES

```
inject kmalloc -v 'SyS_mount()'
```

```
inject kmalloc -v 'mount_subtree() => btrfs_mount()'
```

```
inject -P 0.5 -c 100 alloc_page "should_fail_alloc_page(gfp_t gfp_mask, unsigned int order) (order == 1) => qlge_refill_bq()"
```

Please see the output of '-h' and tools/inject_example.txt for more examples.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Howard McLauchlan


bcc-javacalls

title: bcc-javacalls(8)

summary: Summarize method calls

text:
# bcc-javacalls(8)

## NAME
ucalls, javacalls, perlcalls, phpcalls, pythoncalls, rubycalls, tclcalls - Summarize method calls from high-level languages and Linux syscalls.

## SYNOPSIS
javacalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

perlcalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

phpcalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

pythoncalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

rubycalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

tclcalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

ucalls [-l {java,perl,php,python,ruby}] [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

## DESCRIPTION
This tool summarizes method calls from high-level languages such as Java, Perl, PHP, Python, Ruby, and Tcl. It can also trace Linux system calls. Whenever a method is invoked, ucalls records the call count and optionally the method's execution time (latency) and displays a summary.

This uses in-kernel eBPF maps to store per process summaries for efficiency.

This tool relies on USDT probes embedded in many high-level languages, such as Java, Perl, PHP, Python, Ruby, and Tcl. It requires a runtime instrumented with these probes, which in some cases requires building from source with a USDT-specific flag, such as "--enable-dtrace" or "--with-dtrace". For Java, method probes are not enabled by default, and can be turned on by running the Java process with the "-XX:+ExtendedDTraceProbes" flag. For PHP processes, the environment variable USE_ZEND_DTRACE must be set to 1.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-l {java,perl,php,python,ruby,tcl}

The language to trace. If not provided, only syscalls are traced (when the -S option is used).

-T TOP

Print only the top methods by frequency or latency.

-L

Collect method invocation latency (duration).

-S

Collect Linux syscalls frequency and timing.

-v

Print the resulting BPF program, for debugging purposes.

-m

Print times in milliseconds (the default is microseconds).

pid

The process id to trace.

interval

Print summary after this number of seconds and then exit. By default, wait for Ctrl+C to terminate.

## EXAMPLES
Trace the top 10 Ruby method calls:

\# ucalls -T 10 -l ruby 1344

Trace Python method calls and Linux syscalls including latency in milliseconds:

\# ucalls -l python -mL 2020

Trace only syscalls and print a summary after 10 seconds:

\# ucalls -S 788 10

## OVERHEAD
Tracing individual method calls will produce a considerable overhead in all high-level languages. For languages with just-in-time compilation, such as Java, the overhead can be more considerable than for interpreted languages. On the other hand, syscall tracing will typically be tolerable for most processes, unless they have a very unusual rate of system calls.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _example.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein

## SEE ALSO
ustat(8), argdist(8)


bcc-javaflow

title: bcc-javaflow(8)

summary: Print a flow graph of method

text:
# bcc-javaflow(8)

## NAME
uflow, javaflow, perlflow, phpflow, pythonflow, rubyflow, tclflow - Print a flow graph of method calls in high-level languages.

## SYNOPSIS
javaflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

perlflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

phpflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

pythonflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

rubyflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

tclflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

uflow [-h] [-M METHOD] [-C CLAZZ] [-v] [-l {java,perl,php,python,ruby,tcl}] pid

## DESCRIPTION
uflow traces method calls and prints them in a flow graph that can facilitate debugging and diagnostics by following the program's execution (method flow).

This tool relies on USDT probes embedded in many high-level languages, such as Java, Perl, PHP, Python, Ruby, and Tcl. It requires a runtime instrumented with these probes, which in some cases requires building from source with a USDT-specific flag, such as "--enable-dtrace" or "--with-dtrace". For Java processes, the startup flag "-XX:+ExtendedDTraceProbes" is required. For PHP processes, the environment variable USE_ZEND_DTRACE must be set to 1.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-M METHOD

Print only method calls where the method name begins with this string.

-C CLAZZ

Print only method calls where the class name begins with this string. The class name interpretation strongly depends on the language. For example, in Java use "package/subpackage/ClassName" to refer to classes.

-v

Print the resulting BPF program, for debugging purposes.

{java,perl,php,python,ruby,tcl}

The language to trace.

pid

The process id to trace.

## EXAMPLES
Follow method flow in a Ruby process:

\# uflow ruby 148

Follow method flow in a Java process where the class name is java.lang.Thread:

\# uflow -C java/lang/Thread java 1802

## FIELDS
CPU

The CPU number on which the method was invoked. This is useful to easily see where the output skips to a different CPU.

PID

The process id.

TID

The thread id.

TIME

The duration of the method call.

METHOD

The method name.

## OVERHEAD
This tool has extremely high overhead because it prints every method call. For some scenarios, you might see lost samples in the output as the tool is unable to keep up with the rate of data coming from the kernel. Filtering by class or method prefix can help reduce the amount of data printed, but there is still a very high overhead in the collection mechanism. Do not use for performance- sensitive production scenarios, and always test first.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _example.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein

## SEE ALSO
trace(8), ustat(8)


bcc-javagc

title: bcc-javagc(8)

summary: Trace garbage collection events in

text:
# bcc-javagc(8)

## NAME
ugc, javagc, nodegc, pythongc, rubygc - Trace garbage collection events in high-level languages.

## SYNOPSIS
javagc [-h] [-v] [-m] [-M MINIMUM] [-F FILTER] pid

nodegc [-h] [-v] [-m] [-M MINIMUM] [-F FILTER] pid

pythongc [-h] [-v] [-m] [-M MINIMUM] [-F FILTER] pid

rubygc [-h] [-v] [-m] [-M MINIMUM] [-F FILTER] pid

ugc [-h] [-v] [-m] [-M MINIMUM] [-F FILTER] [-l {java,node,python,ruby}] pid

## DESCRIPTION
This traces garbage collection events as they occur, including their duration and any additional information (such as generation collected or type of GC) provided by the respective language's runtime.

This tool relies on USDT probes embedded in many high-level languages, such as Java, Node, Python, and Ruby. It requires a runtime instrumented with these probes, which in some cases requires building from source with a USDT-specific flag, such as "--enable-dtrace" or "--with-dtrace".

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-v

Print the resulting BPF program, for debugging purposes.

-m

Print times in milliseconds. The default is microseconds.

-M MINIMUM

Display only collections that are longer than this threshold. The value is given in milliseconds. The default is to display all collections.

-F FILTER

Display only collections whose textual description matches (contains) this string. The default is to display all collections. Note that the filtering here is performed in user-space, and not as part of the BPF program. This means that if you have thousands of collection events, specifying this filter will not reduce the amount of data that has to be transferred from the BPF program to the user-space script.

{java,node,python,ruby}

The language to trace.

pid

The process id to trace.

## EXAMPLES
Trace garbage collections in a specific Node process:

\# ugc node 148

Trace garbage collections in a specific Java process, and print GC times in

milliseconds: # ugc -m java 6004

Trace garbage collections in a specific Java process, and display them only if

they are longer than 10ms and have the string "Tenured" in their detailed description: # ugc -M 10 -F Tenured java 6004

## FIELDS
START

The start time of the GC, in seconds from the beginning of the trace.

TIME

The duration of the garbage collection event.

DESCRIPTION

The runtime-provided description of this garbage collection event.

## OVERHEAD
Garbage collection events, even if frequent, should not produce a considerable overhead when traced because they are still not very common. Even hundreds of GCs per second (which is a very high rate) will still produce a fairly negligible overhead.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _example.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein

## SEE ALSO
trace(8), ustat(8), uobjnew(8)


bcc-javaobjnew

title: bcc-javaobjnew(8)

summary: Summarize object allocations in

text:
# bcc-javaobjnew(8)

## NAME
uobjnew, cobjnew, javaobjnew, rubyobjnew, tclobjnew - Summarize object allocations in high-level languages.

## SYNOPSIS
cobjnew [-h] [-C TOP_COUNT] [-S TOP_SIZE] [-v] pid [interval]

javaobjnew [-h] [-C TOP_COUNT] [-S TOP_SIZE] [-v] pid [interval]

rubyobjnew [-h] [-C TOP_COUNT] [-S TOP_SIZE] [-v] pid [interval]

tclobjnew [-h] [-C TOP_COUNT] [-S TOP_SIZE] [-v] pid [interval]

uobjnew [-h] [-C TOP_COUNT] [-S TOP_SIZE] [-v] [-l {c,java,ruby,tcl}] pid [interval]

## DESCRIPTION
uobjnew traces object allocations in high-level languages (including "malloc") and prints summaries of the most frequently allocated types by number of objects or number of bytes.

This tool relies on USDT probes embedded in many high-level languages, such as C, Java, Ruby, and Tcl. It requires a runtime instrumented with these probes, which in some cases requires building from source with a USDT-specific flag, such as "--enable-dtrace" or "--with-dtrace". For Java, the Java process must be started with the "-XX:+ExtendedDTraceProbes" flag.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-C TOP_COUNT

Print the top object types sorted by number of instances.

-S TOP_SIZE

Print the top object types sorted by size.

-v

Print the resulting BPF program, for debugging purposes.

{c,java,ruby,tcl}

The language to trace.

pid

The process id to trace.

interval

Wait this many seconds and then print the summary and exit. By default, wait for Ctrl+C to exit.

## EXAMPLES
Trace object allocations in a Ruby process:

\# uobjnew ruby 148

Trace object allocations from "malloc" and print the top 10 by total size:

\# uobjnew -S 10 c 1788

## FIELDS
TYPE

The object type being allocated. For C (malloc), this is the block size.

ALLOCS

The number of objects allocated.

BYTES

The number of bytes allocated.

## OVERHEAD
Object allocation events are quite frequent, and therefore the overhead from running this tool can be considerable. Use with caution and make sure to test before using in a production environment. Nonetheless, even thousands of allocations per second will likely produce a reasonable overhead when investigating a problem.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _example.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein

## SEE ALSO
ustat(8), ugc(8), memleak(8)


bcc-javastat

title: bcc-javastat(8)

summary: Activity stats from

text:
# bcc-javastat(8)

## NAME
ustat, javastat, nodestat, perlstat, phpstat, pythonstat, rubystat, tclstat - Activity stats from high-level languages.

## SYNOPSIS
javastat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

nodestat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

perlstat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

phpstat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

pythonstat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

rubystat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

tclstat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

ustat [-l {java,node,perl,php,python,ruby,tcl}] [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

## DESCRIPTION
This is "top" for high-level language events, such as garbage collections, exceptions, thread creations, object allocations, method calls, and more. The events are aggregated for each process and printed in a top-like table, which can be sorted by various fields. Not all language runtimes provide the same set of details.

This uses in-kernel eBPF maps to store per process summaries for efficiency.

This tool relies on USDT probes embedded in many high-level languages, such as Java, Node, Perl, PHP, Python, Ruby, and Tcl. It requires a runtime instrumented with these probes, which in some cases requires building from source with a USDT-specific flag, such as "--enable-dtrace" or "--with-dtrace". For Java, some probes are not enabled by default, and can be turned on by running the Java process with the "-XX:+ExtendedDTraceProbes" flag. For PHP processes, the environment variable USE_ZEND_DTRACE must be set to 1.

Newly-created processes will only be traced at the next interval. If you run this tool with a short interval (say, 1-5 seconds), this should be virtually unnoticeable. For longer intervals, you might miss processes that were started and terminated during the interval window.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-l {java,node,perl,php,python,ruby,tcl}

The language to trace. By default, all languages are traced.

-C

Do not clear the screen between updates.

-S {cload,excp,gc,method,objnew,thread}

Sort the output by the specified field.

-r MAXROWS

Do not print more than this number of rows.

-d

Print the resulting BPF program, for debugging purposes.

interval

Interval between updates, seconds.

count

Number of interval summaries.

## EXAMPLES
Summarize activity in high-level languages, 1 second refresh:

\# ustat

Don't clear the screen, and top 8 rows only:

\# ustat -Cr 8

5 second summaries, 10 times only:

\# ustat 5 10

## FIELDS
loadavg

The contents of /proc/loadavg

PID

Process ID.

CMDLINE

Process command line (often the second and following arguments will give you a hint as to which application is being run.

METHOD/s

Count of method invocations during interval.

GC/s

Count of garbage collections during interval.

OBJNEW/s

Count of objects allocated during interval.

CLOAD/s

Count of classes loaded during interval.

EXC/s

Count of exceptions thrown during interval.

THR/s

Count of threads created during interval.

## OVERHEAD
When using this tool with high-frequency events, such as method calls, a very significant slow-down can be expected. However, many of the high-level languages covered by this tool already have a fairly high per-method invocation cost, especially when running in interpreted mode. For the lower-frequency events, such as garbage collections or thread creations, the overhead should not be significant. Specifically, when probing Java processes and not using the "-XX:+ExtendedDTraceProbes" flag, the most expensive probes are not emitted, and the overhead should be acceptable.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _example.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein

## SEE ALSO
trace(8), argdist(8), tplist(8)


bcc-javathreads

title: bcc-javathreads(8)

text:
# bcc-javathreads(8)

## NAME
uthreads, cthreads, javathreads - Trace thread creation events in Java or pthreads.

## SYNOPSIS
cthreads [-h] [-v] pid javathreads [-h] [-v] pid uthreads [-h] [-l {c,java,none}] [-v] pid

## DESCRIPTION
This traces thread creation events in Java processes, or pthread creation events in any process. When a thread is created, its name or start address is printed.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-l {c,java,none}

The language to trace. C and none select tracing pthreads only, regardless of the runtime being traced.

-v

Print the resulting BPF program, for debugging purposes.

pid

The process id to trace.

## EXAMPLES
Trace Java thread creations:

\# uthreads -l java 148

Trace pthread creations:

\# uthreads 1802

## FIELDS
TIME

The event's time in seconds from the beginning of the trace.

ID

The thread's ID. The information in this column depends on the runtime.

TYPE

Event type -- thread start, stop, or pthread event.

DESCRIPTION

The thread's name or start address function name.

## OVERHEAD
Thread start and stop events are usually not very frequent, which makes this tool's overhead negligible.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _example.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein

## SEE ALSO
ustat(8), trace(8)


bcc-killsnoop

title: bcc-killsnoop(8)

text:
# bcc-killsnoop(8)

## NAME
killsnoop - Trace signals issued by the kill() syscall. Uses Linux eBPF/bcc.

## SYNOPSIS
killsnoop [-h] [-x] [-p PID] [-T PID] [-s SIGNAL]

## DESCRIPTION
killsnoop traces the kill() syscall, to show signals sent via this method. This may be useful to troubleshoot failing applications, where an unknown mechanism is sending signals.

This works by tracing the kernel sys_kill() function using dynamic tracing, and will need updating to match any changes to this function.

This makes use of a Linux 4.4 feature (bpf_perf_event_output()); for kernels older than 4.4, see the version under tools/old, which uses an older mechanism.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print usage message.

-x

Only print failed kill() syscalls.

-p PID

Trace this process ID only which is the sender of signal (filtered in-kernel).

-T PID

Trace this target process ID only which is the receiver of signal (filtered in-kernel).

-s SIGNAL

Trace this signal only (filtered in-kernel).

## EXAMPLES
Trace all kill() syscalls:

\# killsnoop

Trace only kill() syscalls that failed:

\# killsnoop -x

Trace PID 181 only:

\# killsnoop -p 181

Trace target PID 189 only:

\# killsnoop -T 189

Trace signal 9 only:

\# killsnoop -s 9

## FIELDS
TIME

Time of the kill call.

PID

Source process ID

COMM

Source process name

SIG

Signal number. See signal(7).

TPID

Target process ID

RES

Result. 0 == success, a negative value (of the error code) for failure.

## OVERHEAD
This traces the kernel kill function and prints output for each event. As the rate of this is generally expected to be low (< 100/s), the overhead is also expected to be negligible. If you have an application that is calling a very high rate of kill()s for some reason, then test and understand overhead before use.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
opensnoop(8), funccount(8)


bcc-klockstat

title: bcc-klockstat(8)

text:
# bcc-klockstat(8)

## NAME
klockstat - Traces kernel mutex lock events and display locks statistics. Uses Linux eBPF/bcc.

## SYNOPSIS
klockstat [-h] [-i] [-n] [-s] [-c] [-S FIELDS] [-p] [-t] [-d DURATION]

## DESCRIPTION
klockstat traces kernel mutex lock events and display locks statistics and displays following data:

Caller Avg Spin Count Max spin Total spin

psi_avgs_work+0x2e 3675 5 5468 18379

flush_to_ldisc+0x22 2833 2 4210 5667

n_tty_write+0x30c 3914 1 3914 3914

isig+0x5d 2390 1 2390 2390

tty_buffer_flush+0x2a 1604 1 1604 1604

commit_echoes+0x22 1400 1 1400 1400

n_tty_receive_buf_common+0x3b9 1399 1 1399 1399

Caller Avg Hold Count Max hold Total hold

flush_to_ldisc+0x22 42558 2 76135 85116

psi_avgs_work+0x2e 14821 5 20446 74106

n_tty_receive_buf_common+0x3b9 12300 1 12300 12300

n_tty_write+0x30c 10712 1 10712 10712

isig+0x5d 3362 1 3362 3362

tty_buffer_flush+0x2a 3078 1 3078 3078

commit_echoes+0x22 3017 1 3017 3017

Every caller to using kernel's mutex is displayed on every line.

First portion of lines show the lock acquiring data, showing the amount of time it took to acquired given lock.

'Caller' - symbol acquiring the mutex

'Average Spin' - average time to acquire the mutex

'Count' - number of times mutex was acquired

'Max spin' - maximum time to acquire the mutex

'Total spin' - total time spent in acquiring the mutex

Second portion of lines show the lock holding data, showing the amount of time it took to hold given lock.

'Caller' - symbol holding the mutex

'Average Hold' - average time mutex was held

'Count' - number of times mutex was held

'Max hold' - maximum time mutex was held

'Total hold' - total time spent in holding the mutex

This works by tracing mutex_lock/unlock kprobes, updating the lock stats in maps and processing them in the python part.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print usage message.

-i SEC

print summary at this interval (seconds)

-c CALLER

print locks taken by given caller filter (e.g., pipe_)

-S FIELDS

sort data on specific columns defined by FIELDS string (by default the data is sorted by Max columns) FIELDS string contains 1 or 2 fields describing columns to sort on for both acquiring and holding data. Following fields are available:

acq_max for 'Max spin' column

acq_total for 'Total spin' column

acq_count for 'Count' column

hld_max for 'Max hold' column

hld_total for 'Total hold' column

hld_count for 'Count' column

See EXAMPLES.

-n COUNT

print COUNT number of locks

-s COUNT

print COUNT number of stack entries

-p PID

Trace this process ID only (filtered in-kernel).

-t TID

Trace this thread ID only (filtered in-kernel).

-d DURATION

Total duration of trace in seconds.

--stack-storage-size STACK_STORAGE_SIZE

Change the number of unique stack traces that can be stored and displayed.

## EXAMPLES
Sort lock acquired results on acquired count:

\# klockstat -S acq_count

Sort lock held results on total held time:

\# klockstat -S hld_total

Combination of above:

\# klockstat -S acq_count,hld_total

Trace system wide:

\# klockstat

Trace for 5 seconds only:

\# klockstat -d 5

Display stats every 5 seconds

\# klockstat -i 5

Trace locks for PID 123:

\# klockstat -p 123

Trace locks for TID 321:

\# klockstat -t 321

Display stats only for lock callers with 'pipe_' substring

\# klockstat -c pipe_

Display 3 locks:

\# klockstat -n 3

Display 10 levels of stack for the most expensive lock:

\# klockstat -n 1 -s 10 Tracing lock events... Hit Ctrl-C to end. ^C

Caller Avg Spin Count Max spin Total spin

pipe_wait+0xa9 670 397691 17273 266775437

pipe_wait+0xa9

pipe_read+0x206

new_sync_read+0x12a

vfs_read+0x9d

ksys_read+0x5f

do_syscall_64+0x5b

entry_SYSCALL_64_after_hwframe+0x44

Caller Avg Hold Count Max hold Total hold

flush_to_ldisc+0x22 28381 3 65484 85144

flush_to_ldisc+0x22

process_one_work+0x1b0

worker_thread+0x50

kthread+0xfb

ret_from_fork+0x35

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## CREDITS
This tool is based on work of David Valin <dvalin@redhat.com> and his script.

## AUTHOR
Jiri Olsa


bcc-ksnoop

title: bcc-ksnoop(8)

text:
# bcc-ksnoop(8)

## NAME
KSNOOP - tool for tracing kernel function entry/return showing arguments/return values

## SYNOPSIS
ksnoop [OPTIONS] { COMMAND FUNC | help } OPTIONS := { { -V | --version } | { -h | --help } | { [-P | --pages] nr_pages} | { [-p | --pid] pid} | [{ -s | --stack }] | [{ -d | --debug }] }

COMMAND := { trace | info }

FUNC := { name | name(arg[,**arg]) }

## DESCRIPTION
ksnoop allows for inspection of arguments and return values associated with function entry/return. ksnoop info FUNC

Show function description, arguments and return value types.

ksnoop trace FUNC [FUNC]

Trace function entry and return, showing arguments and return values. A function name can simply be specified, or a function name along with named arguments, return values. return is used to specify the return value.

ksnoop requires the kernel to provide BTF for itself, and if tracing of module data is required, module BTF must be present also. Check /sys/kernel/btf to see if BTF is present.

ksnoop requires CAP_BPF and CAP_TRACING capabilities.

## OPTIONS
-h, --help

Show help information

-V, --version

Show version.

-d, --debug

Show debug output.

-p, --pid

Filter events by pid.

-P, --pages

Specify number of pages used per-CPU for perf event collection. Default is 8.

-s, --stack

Specified set of functions are traced if and only if they are encountered in the order specified.

## EXAMPLES
\# ksnoop info ip_send_skb

```
int  ip_send_skb(struct net  * net, struct sk_buff  * skb);
```

Show function description.

\# ksnoop trace ip_send_skb

```
          TIME  CPU      PID FUNCTION/ARGS
78101668506811    1     2813 ip_send_skb(

                                 net = *(0xffffffffb5959840)

                                  (struct net){

                                   .passive = (refcount_t){

                                    .refs = (atomic_t){

                                     .counter = (int)0x2,

                                    },

                                   },

                                   .dev_base_seq = (unsigned int)0x18,

                                   .ifindex = (int)0xf,

                                   .list = (struct list_head){

                                    .next = (struct list_head *)0xffff9895440dc120,

                                    .prev = (struct list_head *)0xffffffffb595a8d0,

                                   },

                                 ...
79561322965250    1     2813 ip_send_skb(

                                 return =

                                  (int)0x0

                             );
```

Show entry/return for ip_send_skb() with arguments, return values.

\# ksnoop trace "ip_send_skb(skb)"

```
         TIME  CPU      PID FUNCTION/ARGS
78142420834537    1     2813 ip_send_skb(

                                 skb = *(0xffff989750797c00)

                                  (struct sk_buff){

                                   (union){

                                    .sk = (struct sock *)0xffff98966ce19200,

                                    .ip_defrag_offset = (int)0x6ce19200,

                                   },

                                   (union){

                                    (struct){

                                     ._skb_refdst = (long unsigned int)0xffff98981dde2d80,

                                     .destructor = (void (*)(struct sk_buff *))0xffffffffb3e1beb0,

                                    },

                                ...
```

Show entry argument skb.

\# ksnoop trace "ip_send_skb(return)"

```
         TIME  CPU      PID FUNCTION/ARGS
78178228354796    1     2813 ip_send_skb(

                                 return =

                                  (int)0x0

                             );
```

Show return value from ip_send_skb().

\# ksnoop trace "ip_send_skb(skb->sk)"

```
          TIME  CPU      PID FUNCTION/ARGS
78207649138829    2     2813 ip_send_skb(

                                 skb->sk = *(0xffff98966ce19200)

                                  (struct sock){

                                   .__sk_common = (struct sock_common){

                                    (union){

                                     .skc_addrpair = (__addrpair)0x1701a8c017d38f8d,

                                     (struct){

                                      .skc_daddr = (__be32)0x17d38f8d,

                                      .skc_rcv_saddr = (__be32)0x1701a8c0,

                                     },

                                    },

                                  ...
```

Trace member information associated with argument. Only one level of membership is supported.

\# ksnoop -p 2813 "ip_rcv(dev)"

```
          TIME  CPU      PID FUNCTION/ARGS
78254803164920    1     2813 ip_rcv(

                                 dev = *(0xffff9895414cb000)

                                  (struct net_device){

                                   .name = (char[16])[

                                    'l',

                                    'o',

                                   ],

                                   .name_node = (struct netdev_name_node *)0xffff989541515ec0,

                                   .state = (long unsigned int)0x3,

                                 ...
```

Trace dev argument of ip_rcv(). Specify process id 2813 for events for that process only.

\# ksnoop -s tcp_sendmsg __tcp_transmit_skb ip_output

```
         TIME  CPU      PID FUNCTION/ARGS
71827770952903    1     4777 __tcp_transmit_skb(

                                 sk = *(0xffff9852460a2300)

                                  (struct sock){

                                   .__sk_common = (struct sock_common){

                                    (union){

                                     .skc_addrpair = (__addrpair)0x61b2af0a35cbfe0a,
```

Trace entry/return of tcp_sendmsg, __tcp_transmit_skb and ip_output when tcp_sendmsg leads to a call to __tcp_transmit_skb and that in turn leads to a call to ip_output; i.e. with a call graph matching the order specified. The order does not have to be direct calls, i.e. function A can call another function that calls function B.

\# ksnoop "ip_send_skb(skb->len > 100, skb)"

```
          TIME  CPU      PID FUNCTION/ARGS
39267395709745    1     2955 ip_send_skb(

                                 skb->len =

                                  (unsigned int)0x89,

                                 skb = *(0xffff89c8be81e500)

                                  (struct sk_buff){

                                   (union){

                                    .sk = (struct sock *)0xffff89c6c59e5580,

                                    .ip_defrag_offset = (int)0xc59e5580,

                                   },
```

Trace ip_send_skb() skbs which have len > 100.

## SEE ALSO
bpf(2),


bcc-kvmexit

title: bcc-kvmexit(8)

text:
# bcc-kvmexit(8)

## NAME
kvmexit - Display the exit_reason and its statistics of each vm exit.

## SYNOPSIS
kvmexit [-h] [-p PID [-v VCPU | -a] ] [-t TID | -T 'TID1,TID2'] [duration]

## DESCRIPTION
Considering virtual machines' frequent exits can cause performance problems, this tool aims to locate the frequent exited reasons and then find solutions to reduce or even avoid the exit, by displaying the detail exit reasons and the counts of each vm exit for all vms running on one physical machine.

This tool uses a PERCPU_ARRAY: pcpuArrayA and a percpu_hash: hashA to collaboratively store each kvm exit reason and its count. The reason is there exists a rule when one vcpu exits and re-enters, it tends to continue to run on the same physical cpu as the last cycle, which is also called 'cache hit'. Thus we turn to use a PERCPU_ARRAY to record the 'cache hit' situation to speed things up; and for other cases, then use a percpu_hash.

As RAW_TRACEPOINT_PROBE(kvm_exit) consumes less cpu cycles, when this tool is used, it firstly tries to employ raw tracepoints in modules, and if failes, then fall back to regular tracepoint.

Limitation: In view of the hardware-assisted virtualization technology of different architectures, currently we only adapt on vmx in intel.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

This also requires Linux 4.7+ (BPF_PROG_TYPE_TRACEPOINT support).

## OPTIONS
-h

Print usage message.

-p PID

Display process with this PID only, collpase all tids with exit reasons sorted in descending order.

-v VCPU

Display this VCPU only for this PID.

-a ALLTIDS

Display all TIDS for this PID.

-t TID

Display thread with this TID only with exit reasons sorted in descending order.

-T 'TID1,TID2'

Display threads for a union like {395490, 395491}.

duration

Duration of display, after sleeping several seconds.

## EXAMPLES
Display kvm exit reasons and statistics for all threads... Hit Ctrl-C to end:

\# kvmexit

Display kvm exit reasons and statistics for all threads after sleeping 6 secs:

\# kvmexit 6

Display kvm exit reasons and statistics for PID 1273795 after sleeping 5 secs:

\# kvmexit -p 1273795 5

Display kvm exit reasons and statistics for PID 1273795 and its all threads after sleeping 5 secs:

\# kvmexit -p 1273795 5 -a

Display kvm exit reasons and statistics for PID 1273795 VCPU 0... Hit Ctrl-C to end:

\# kvmexit -p 1273795 -v 0

Display kvm exit reasons and statistics for PID 1273795 VCPU 0 after sleeping 4 secs:

\# kvmexit -p 1273795 -v 0 4

Display kvm exit reasons and statistics for TID 1273819 after sleeping 10 secs:

\# kvmexit -t 1273819 10

Display kvm exit reasons and statistics for TIDS ['1273820', '1273819']... Hit Ctrl-C to end:

\# kvmexit -T '1273820,1273819'

## OVERHEAD
This traces the "kvm_exit" kernel function, records the exit reason and calculates its counts. Contrast with filling more vm-exit reason debug entries, this tool is more easily and flexibly: the bcc python logic could provide nice kernel aggregation and custom output, the bpf in-kernel percpu_array and percpu_cache further improves performance.

The impact of using this tool on the host should be negligible. While this tool is very efficient, it does affect the guest virtual machine itself, the average test results on guest vm are as follows:

| cpu cycles

no TP | 1127

regular TP | 1277 (13% downgrade)

RAW TP | 1187 (5% downgrade)

Host: echo 1 > /proc/sys/net/core/bpf_jit_enable

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Fei Li <lifei.shirley@bytedance.com>


bcc-llcstat

title: bcc-llcstat(8)

text:
# bcc-llcstat(8)

## NAME
llcstat - Summarize CPU cache references and misses by process. Uses Linux eBPF/bcc.

## SYNOPSIS
llcstat [-h] [-c SAMPLE_PERIOD] [duration]

## DESCRIPTION
llcstat instruments CPU cache references and cache misses system-side, and summarizes them by PID and CPU. These events have different meanings on different architecture. For x86-64, they mean misses and references to LLC. This can be useful to locate and debug performance issues caused by cache hit rate.

This works by sampling corresponding events defined in uapi/linux/perf_event.h, namely PERF_COUNT_HW_CACHE_REFERENCES and PERF_COUNT_HW_CACHE_MISSES, using BPF perf event tracing. Upon each sampled event, the attached BPF program records the PID and CPU ID on which the event happened, and stores it in table.

This makes use of a Linux 4.9 feature (BPF_PROG_TYPE_PERF_EVENT).

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print usage message.

-c SAMPLE_PERIOD

Sample one in this many cache reference and cache miss events.

-t

Summarize cache references and misses by PID/TID

duration

Duration to trace, in seconds.

## EXAMPLES
Sample one in 100 events, trace for 20 seconds:

\# llcstat -c 100 20

## FIELDS
PID

Process ID

NAME

Process name

CPU

CPU ID

REFERENCE

Number of cache reference events

MISS

Number of cache miss events

HIT%

Cache hit ratio

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Teng Qin

## SEE ALSO
Perf can be used as a generic event counter tool. An example for LLC:

\# perf top -e cache-misses -e cache-references -a -ns pid,cpu,comm


bcc-mdflush

title: bcc-mdflush(8)

text:
# bcc-mdflush(8)

## NAME
mdflush - Trace md flush events. Uses Linux eBPF/bcc.

## SYNOPSIS
mdflush

## DESCRIPTION
This tool traces flush events by md, the Linux multiple device driver (software RAID). The timestamp and md device for the flush are printed. Knowing when these flushes happen can be useful for correlation with unexplained spikes in disk latency.

This works by tracing the kernel md_flush_request() function using dynamic tracing, and will need updating to match any changes to this function.

Note that the flushes themselves are likely to originate from higher in the I/O stack, such as from the file systems.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## EXAMPLES
Trace md flush events:

\# mdflush

## FIELDS
TIME

Time of the flush event (HH:MM:SS).

PID

The process ID that was on-CPU when the event was issued. This may identify the cause of the flush (eg, the "sync" command), but will often identify a kernel worker thread that was managing I/O.

COMM

The command name for the PID.

DEVICE

The md device name.

## OVERHEAD
Expected to be negligible.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
biosnoop(8)


bcc-memleak

title: bcc-memleak(8)

text:
# bcc-memleak(8)

## NAME
memleak - Print a summary of outstanding allocations and their call stacks to detect memory leaks. Uses Linux eBPF/bcc.

## SYNOPSIS
memleak [-h] [-p PID] [-t] [-a] [-o OLDER] [-c COMMAND] [--combined-only] [--wa-missing-free] [-s SAMPLE_RATE] [-T TOP] [-z MIN_SIZE] [-Z MAX_SIZE] [-O OBJ] [INTERVAL] [COUNT]

## DESCRIPTION
memleak traces and matches memory allocation and deallocation requests, and collects call stacks for each allocation. memleak can then print a summary of which call stacks performed allocations that weren't subsequently freed.

When tracing a specific process, memleak instruments a list of allocation functions from libc, specifically: malloc, calloc, realloc, posix_memalign, valloc, memalign, pvalloc, aligned_alloc, and free. When tracing all processes, memleak instruments kmalloc/kfree, kmem_cache_alloc/kmem_cache_free, and also page allocations made by get_free_pages/free_pages.

memleak may introduce significant overhead when tracing processes that allocate and free many blocks very quickly. See the OVERHEAD section below.

This tool only works on Linux 4.6+. Stack traces are obtained using the new BPF_STACK_TRACE` APIs. For kernels older than 4.6, see the version under tools/old. Kernel memory allocations are intercepted through tracepoints, which are available on Linux 4.7+.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print usage message.

-p PID

Trace this process ID only (filtered in-kernel). This traces libc allocator.

-t

Print a trace of all allocation and free requests and results.

-a

Print a list of allocations that weren't freed (and their sizes) in addition to their call stacks.

-o OLDER

Print only allocations older than OLDER milliseconds. Useful to remove false positives. The default value is 500 milliseconds.

-c COMMAND

Run the specified command and trace its allocations only. This traces libc allocator.

--combined-only

Use statistics precalculated in kernel space. Amount of data to be pulled from kernel significantly decreases, at the cost of losing capabilities of time-based false positives filtering (-o).

--wa-missing-free

Make up the action of free to alleviate misjudgments when free is missing.

-s SAMPLE_RATE

Record roughly every SAMPLE_RATE-th allocation to reduce overhead.

-t TOP

Print only the top TOP stacks (sorted by size). The default value is 10.

-z MIN_SIZE

Capture only allocations that are larger than or equal to MIN_SIZE bytes.

-Z MAX_SIZE

Capture only allocations that are smaller than or equal to MAX_SIZE bytes.

-O OBJ

Attach to allocation functions in specified object instead of resolving libc. Ignored when kernel allocations are profiled.

INTERVAL

Print a summary of outstanding allocations and their call stacks every INTERVAL seconds. The default interval is 5 seconds.

COUNT

Print the outstanding allocations summary COUNT times and then exit.

## EXAMPLES
Print outstanding kernel allocation stacks every 3 seconds:

\# memleak 3

Print user outstanding allocation stacks and allocation details for the process 1005:

\# memleak -p 1005 -a

Sample roughly every 5th allocation (~20%) of the call stacks and print the top 5

stacks 10 times before quitting. # memleak -s 5 --top=5 10

Run ./allocs and print outstanding allocation stacks for that process:

\# memleak -c ./allocs

Capture only allocations between 16 and 32 bytes in size:

\# memleak -z 16 -Z 32

## OVERHEAD
memleak can have significant overhead if the target process or kernel performs allocations at a very high rate. Pathological cases may exhibit up to 100x degradation in running time. Most of the time, however, memleak shouldn't cause a significant slowdown. You can use the -s switch to reduce the overhead further by capturing only every N-th allocation. The -z and -Z switches can also reduce overhead by capturing only allocations of specific sizes.

Additionally, option --combined-only saves processing time by reusing already calculated allocation statistics from kernel. It's faster, but lacks information about particular allocations.

Also, option --wa-missing-free makes memleak more accuracy in the complicated environment.

To determine the rate at which your application is calling malloc/free, or the rate at which your kernel is calling kmalloc/kfree, place a probe with perf and collect statistics. For example, to determine how many calls to __kmalloc are placed in a typical period of 10 seconds:

\# perf probe '__kmalloc'

\# perf stat -a -e 'probe:__kmalloc' -- sleep 10

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein


bcc-mountsnoop

title: bcc-mountsnoop(8)

text:
# bcc-mountsnoop(8)

## NAME
mountsnoop - Trace mount() and umount() syscalls. Uses Linux eBPF/bcc.

## SYNOPSIS
mountsnoop

## DESCRIPTION
mountsnoop traces the mount() and umount() syscalls, showing which processes are mounting and unmounting filesystems in what mount namespaces. This can be useful for troubleshooting system and container setup.

This works by tracing the kernel sys_mount() and sys_umount() functions using dynamic tracing, and will need updating to match any changes to this function.

This makes use of a Linux 4.8 feature (bpf_get_current_task()).

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## FIELDS
COMM

Process name

PID

Process ID

TID

Thread ID

MNT_NS

Mount namespace inode number

CALL

System call, arguments, and return value

## OVERHEAD
This traces the kernel mount and umount functions and prints output for each event. As the rate of these calls is generally expected to be very low, the overhead is also expected to be negligible. If your system calls mount() and umount() at a high rate, then test and understand overhead before use.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Omar Sandoval

## SEE ALSO
mount(2) umount(2)


title: bcc-mysqld_qslower(8)

text:
# bcc-mysqld_qslower(8)

## NAME
mysqld_qslower - Trace MySQL server queries slower than a threshold.

## SYNOPSIS
mysqld_qslower PID [min_ms]

## DESCRIPTION
This traces queries served by a MySQL server, and prints those that exceed a custom latency (query duration) threshold. By default, a minimum threshold of 1 millisecond is used. If a threshold of 0 is used, all queries are printed.

This uses User Statically-Defined Tracing (USDT) probes, a feature added to MySQL for DTrace support, but which may not be enabled on a given MySQL installation. See requirements.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF, bcc, and MySQL server with USDT probe support (when configuring the build: -DENABLE_DTRACE=1).

## OPTIONS
PID Trace this mysqld PID.

min_ms

Minimum query latency (duration) to trace, in milliseconds. Default is 1 ms.

## EXAMPLES
Trace MySQL server queries slower than 1 ms for PID 1981:

\# mysqld_qslower 1981

Trace slower than 10 ms for PID 1981:

\# mysqld_qslower 1981 10

## FIELDS
TIME(s)

Time of query start, in seconds.

PID

Process ID of the traced server.

MS

Milliseconds for the query, from start to end.

QUERY

Query string, truncated to 128 characters.

## OVERHEAD
This adds low-overhead instrumentation to MySQL queries, and only emits output data from kernel to user-level if they query exceeds the threshold. If the server query rate is less than 1,000/sec, the overhead is expected to be negligible. If the query rate is higher, test to gauge overhead.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
biosnoop(8)


bcc-netqtop

title: bcc-netqtop(8)

text:
# bcc-netqtop(8)

## NAME
netqtop - Summarize PPS, BPS, average size of packets and packet counts ordered by packet sizes on each queue of a network interface.

## SYNOPSIS
netqtop [-n nic] [-i interval] [-t throughput]

## DESCRIPTION
netqtop accounts statistics of both transmitted and received packets on each queue of a specified network interface to help developers check if its traffic load is balanced. The result is displayed as a table with columns of PPS, BPS, average size and packet counts in range [0,64), [64, 5120), [512, 2048), [2048, 16K), [16K, 64K). This is printed every given interval (default 1) in seconds.

The tool uses the net:net_dev_start_xmit and net:netif_receive_skb kernel tracepoints. Since it uses tracepoint, the tool only works on Linux 4.7+.

netqtop introduces significant overhead while network traffic is large. See OVERHEAD section below.

## REQUIREMENTS
CONFIG_bpf and bcc

## OPTIONS
-n NIC

Specify the network interface card

-i INTERVAL

Print results every INTERVAL seconds. The default value is 1.

-t THROUGHPUT

Print BPS and PPS of each queue.

## EXAMPLES
Account statistics of eth0 and output every 2 seconds:

\# netqtop -n eth0 -i 1

## OVERHEAD
In performance test, netqtop introduces a overhead up to 30% PPS drop while printing interval is set to 1 second. So be mindful of potential packet drop when using this tool.

It also increases ping-pong latency by about 1 usec.

## SOURCE
This is from bcc

https://github.com/iovisor/bcc

Also look in the bcc distribution for a netqtop_example.txt file containing example usage, output and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development

## AUTHOR
Yolandajn