bcc-tcpconnect

title: bcc-tcpconnect(8)

text:
# bcc-tcpconnect(8)

## NAME
tcpconnect - Trace TCP active connections (connect()). Uses Linux eBPF/bcc.

## SYNOPSIS
tcpconnect [-h] [-c] [-t] [-p PID] [-P PORT] [-4 | -6] [-L] [-u UID] [-U] [--cgroupmap MAPPATH] [--mntnsmap MAPPATH] [-d]

## DESCRIPTION
This tool traces active TCP connections (eg, via a connect() syscall; accept() are passive connections). This can be useful for general troubleshooting to see what connections are initiated by the local server.

All connection attempts are traced, even if they ultimately fail.

This works by tracing the kernel tcp_v4_connect() and tcp_v6_connect() functions using dynamic tracing, and will need updating to match any changes to these functions.

When provided with the -d or --dns option, this tool will also correlate connect calls with the most recent DNS query that matches the IP connected. This feature works by tracing the kernel udp_recvmsg() function to collect DNS responses.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

If using the -d or --dns option, you must have the dnslib and cachetools python packages installed. You can install them with pip3 or with apt on Ubuntu 18.04+ using the python3-dnslib and python3-cachetools packages.

## OPTIONS
-h

Print usage message.

-t

Include a timestamp column.

-c

Count connects per src ip and dest ip/port.

-p PID

Trace this process ID only (filtered in-kernel).

-P PORT

Comma-separated list of destination ports to trace (filtered in-kernel).

-4

Trace IPv4 family only.

-6

Trace IPv6 family only.

-L

Include a LPORT column.

-U

Include a UID column.

-u UID

Trace this UID only (filtered in-kernel).

--cgroupmap MAPPATH

Trace cgroups in this BPF map only (filtered in-kernel).

--mntnsmap MAPPATH

Trace mount namespaces in this BPF map only (filtered in-kernel).

-d

Shows the most recent DNS query for the IP address in the connect call. This is likely related to the TCP connection details in the other columns, but is not guaranteed. This feature works by tracing the udp_recvmsg kernel function and tracking DNS responses received by the server. It only supports UDP DNS packets up to 512 bytes in length. The python code keeps a cache of 10k DNS responses in memory for up 24 hours. If the time difference in milliseconds between when the system received a DNS response and when a connect syscall was traced using an IP in that DNS response is greater than 100ms, this tool will report this delta after the query. These deltas should be relatively short for most applications. A long delay between the response and connect could be either anomalous activity or indicate a misattribution between the DNS name requested and the IP that the connect syscall is using.

The -d option may not be used with the count feature (option -c)

## EXAMPLES
Trace all active TCP connections:

\# tcpconnect

Trace all TCP connects, and include timestamps:

\# tcpconnect -t

Trace all TCP connects, and include most recent matching DNS query for each connected IP

\# tcpconnect -d

Trace PID 181 only:

\# tcpconnect -p 181

Trace ports 80 and 81 only:

\# tcpconnect -P 80,81

Trace IPv4 family only:

\# tcpconnect -4

Trace IPv6 family only:

\# tcpconnect -6

Trace all TCP connects, and include LPORT:

\# tcpconnect -L

Trace all TCP connects, and include UID:

\# tcpconnect -U

Trace UID 1000 only:

\# tcpconnect -u 1000

Count connects per src ip and dest ip/port:

\# tcpconnect -c

Trace a set of cgroups only (see special_filtering.md from bcc sources for more details):

\# tcpconnect --cgroupmap /sys/fs/bpf/test01

Trace a set of mount namespaces only (see special_filtering.md from bcc sources for more details):

\# tcpconnect --mntnsmap /sys/fs/bpf/mnt_ns_set

## FIELDS
TIME(s)

Time of the call, in seconds.

UID

User ID

PID

Process ID

COMM

Process name

IP

IP address family (4 or 6)

SADDR

Source IP address.

LPORT

Source port

DADDR

Destination IP address.

DPORT

Destination port

CONNECTS

Accumulated active connections since start.

QUERY

Shows the most recent DNS query for the IP address in the connect call. This is likely related to the TCP connection details in the other columns, but is not guaranteed.

## OVERHEAD
This traces the kernel tcp_v[46]_connect functions and prints output for each event. As the rate of this is generally expected to be low (< 1000/s), the overhead is also expected to be negligible. If you have an application that is calling a high rate of connect()s, such as a proxy server, then test and understand this overhead before use.

If you are using the -d option to track DNS requests, this tool will trace the udp_recvmsg function and generate an event for any packets from UDP port 53. This event contains up to 512 bytes of the UDP packet payload. Typical applications do not extensively use UDP, so the performance overhead of tracing udp_recvmsg is expected to be negligible, However, if you have an application that receives many UDP packets, then you should test and understand the overhead of tracing every received UDP message. Furthermore, performance overhead of running this tool on a DNS server is expected to be higher than average because all DNS response packets will be copied to userspace.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
tcptracer(8), tcpaccept(8), funccount(8), tcpdump(8)


bcc-tcpconnlat

package: bcc-tools

title: bcc-tcpconnlat(8)

text:
# bcc-tcpconnlat(8)

## NAME
tcpconnlat - Trace TCP active connection latency. Uses Linux eBPF/bcc.

## SYNOPSIS
tcpconnlat [-h] [-t] [-p PID] [-L] [-4 | -6] [-v] [min_ms]

## DESCRIPTION
This tool traces active TCP connections (eg, via a connect() syscall), and shows the latency (time) for the connection as measured locally: the time from SYN sent to the response packet. This is a useful performance metric that typically spans kernel TCP/IP processing and the network round trip time (not application runtime).

All connection attempts are traced, even if they ultimately fail (RST packet in response).

This tool works by use of kernel dynamic tracing of TCP/IP functions, and will need updating to match any changes to these functions. This tool should be updated in the future to use static tracepoints, once they are available.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print usage message.

-t

Include a timestamp column.

-p PID

Trace this process ID only (filtered in-kernel).

-L

Include a LPORT column.

-4

Trace IPv4 family only.

-6

Trace IPv6 family only.

-v

Print the resulting BPF program, for debugging purposes.

min_ms

Minimum duration to trace, in milliseconds.

## EXAMPLES
Trace all active TCP connections, and show connection latency (SYN->response round trip):

\# tcpconnlat

Include timestamps:

\# tcpconnlat -t

Trace PID 181 only:

\# tcpconnlat -p 181

Trace connects, and include LPORT:

\# tcpconnlat -L

Trace IPv4 family only:

\# tcpconnlat -4

Trace IPv6 family only:

\# tcpconnlat -6

Trace connects with latency longer than 10 ms:

\# tcpconnlat 10

Print the BPF program:

\# tcpconnlat -v

## FIELDS
TIME(s)

Time of the response packet, in seconds.

PID

Process ID that initiated the connection.

COMM

Process name that initiated the connection.

IP

IP address family (4 or 6).

SADDR

Source IP address.

DADDR

Destination IP address.

LPORT

Source port

DPORT

Destination port

LAT(ms)

The time from when a TCP connect was issued (measured in-kernel) to when a response packet was received for this connection (can be SYN,ACK, or RST, etc). This time spans kernel to kernel latency, involving kernel TCP/IP processing and the network round trip in between. This typically does not include time spent by the application processing the new connection.

## OVERHEAD
This traces the kernel tcp_v[46]_connect functions and prints output for each event. As the rate of this is generally expected to be low (< 1000/s), the overhead is also expected to be negligible. If you have an application that is calling a high rate of connects()s, such as a proxy server, then test and understand this overhead before use.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
tcpconnect(8), tcpaccept(8), funccount(8), tcpdump(8)

branch: sisyphus


bcc-tcpdrop

title: bcc-tcpdrop(8)

text:
# bcc-tcpdrop(8)

## NAME
tcpdrop - Trace kernel-based TCP packet drops with details. Uses Linux eBPF/bcc.

## SYNOPSIS
tcpdrop [-4 | -6] [-h]

## DESCRIPTION
This tool traces TCP packets or segments that were dropped by the kernel, and shows details from the IP and TCP headers, the socket state, and the kernel stack trace. This is useful for debugging cases of high kernel drops, which can cause timer-based retransmits and performance issues.

This tool works using dynamic tracing of the tcp_drop() kernel function, which requires a recent kernel version.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-4

Trace IPv4 family only.

-6

Trace IPv6 family only.

-h

Print usage message.

## EXAMPLES
Trace kernel-based TCP packet drops with details:

\# tcpdrop

Trace IPv4 family only:

\# tcpdrop -4

Trace IPv6 family only:

\# tcpdrop -6

## FIELDS
TIME

Time of the drop, in HH:MM:SS format.

PID

Process ID that was on-CPU during the drop. This may be unrelated, as drops can occur on the receive interrupt and be unrelated to the PID that was interrupted.

IP

IP address family (4 or 6)

SADDR

Source IP address.

SPORT

Source TCP port.

DADDR

Destination IP address.

DPORT

Destionation TCP port.

STATE

TCP session state ("ESTABLISHED", etc).

FLAGS

TCP flags ("SYN", etc).

## OVERHEAD
This traces the kernel tcp_drop() function, which should be low frequency, and therefore the overhead of this tool should be negligible.

As always, test and understand this tools overhead for your types of workloads before production use.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
tcplife(8), tcpaccept(8), tcpconnect(8), tcptop(8)


bcc-tcplife

title: bcc-tcplife(8)

text:
# bcc-tcplife(8)

## NAME
tcplife - Trace TCP sessions and summarize lifespan. Uses Linux eBPF/bcc.

## SYNOPSIS
tcplife [-h] [-T] [-t] [-w] [-s] [-p PID] [-D PORTS] [-L PORTS] [-4 | -6]

## DESCRIPTION
This tool traces TCP sessions that open and close while tracing, and prints a line of output to summarize each one. This includes the IP addresses, ports, duration, and throughput for the session. This is useful for workload characterisation and flow accounting: identifying what connections are happening, with the bytes transferred.

This tool works using the sock:inet_sock_set_state tracepoint if it exists, added to Linux 4.16, and switches to using kernel dynamic tracing for older kernels. Only TCP state changes are traced, so it is expected that the overhead of this tool is much lower than typical send/receive tracing.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print usage message.

-s

Comma separated values output (parseable).

-t

Include a timestamp column (seconds).

-T

Include a time column (HH:MM:SS).

-w

Wide column output (fits IPv6 addresses).

-p PID

Trace this process ID only (filtered in-kernel).

-L PORTS

Comma-separated list of local ports to trace (filtered in-kernel).

-D PORTS

Comma-separated list of destination ports to trace (filtered in-kernel).

-4

Trace IPv4 family only.

-6

Trace IPv6 family only.

## EXAMPLES
Trace all TCP sessions, and summarize lifespan and throughput:

\# tcplife

Include a timestamp column, and wide column output:

\# tcplife -tw

Trace PID 181 only:

\# tcplife -p 181

Trace connections to local ports 80 and 81 only:

\# tcplife -L 80,81

Trace connections to remote port 80 only:

\# tcplife -D 80

Trace IPv4 family only:

\# tcplife -4

Trace IPv6 family only:

\# tcplife -6

## FIELDS
TIME

Time of the call, in HH:MM:SS format.

TIME(s)

Time of the call, in seconds.

PID

Process ID

COMM

Process name

IP

IP address family (4 or 6)

LADDR

Local IP address.

RADDR

Remote IP address.

LPORT

Local port.

RPORT

Remote port.

TX_KB

Total transmitted Kbytes.

RX_KB

Total received Kbytes.

MS

Lifespan of the session, in milliseconds.

## OVERHEAD
This traces the kernel TCP set state function, which should be called much less often than send/receive tracing, and therefore have lower overhead. The overhead of the tool is relative to the rate of new TCP sessions: if this is high, over 10,000 per second, then there may be noticeable overhead just to print out 10k lines of formatted output per second.

You can find out the rate of new TCP sessions using "sar -n TCP 1", and adding the active/s and passive/s columns.

As always, test and understand this tools overhead for your types of workloads before production use.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
tcpaccept(8), tcpconnect(8), tcptop(8)


bcc-tcpretrans

title: bcc-tcpretrans(8)

text:
# bcc-tcpretrans(8)

## NAME
tcpretrans - Trace or count TCP retransmits and TLPs. Uses Linux eBPF/bcc.

## SYNOPSIS
tcpretrans [-h] [-s] [-l] [-c] [-4 | -6]

## DESCRIPTION
This traces TCP retransmits, showing address, port, and TCP state information, and sometimes the PID (although usually not, since retransmits are usually sent by the kernel on timeouts). To keep overhead very low, only the TCP retransmit functions are traced. This does not trace every packet (like tcpdump(8) or a packet sniffer). Optionally, it can count retransmits over a user signalled interval to spot potentially dropping network paths the flows are traversing.

This uses dynamic tracing of the kernel tcp_retransmit_skb() and tcp_send_loss_probe() functions, and will need to be updated to match kernel changes to these functions.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print usage message.

-s

Display TCP sequence numbers.

-l

Include tail loss probe attempts (in some cases the kernel may not complete the TLP send).

-c

Count occurring retransmits per flow.

-4

Trace IPv4 family only.

-6

Trace IPv6 family only.

## EXAMPLES
Trace TCP retransmits:

\# tcpretrans

Trace TCP retransmits and TLP attempts:

\# tcpretrans -l

Trace IPv4 family only:

\# tcpretrans -4

Trace IPv6 family only:

\# tcpretrans -6

## FIELDS
TIME

Time of the retransmit.

PID

Process ID that was on-CPU. This is less useful than it might sound, as it may usually be 0, for the kernel, for timer-based retransmits.

IP

IP address family (4 or 6).

LADDR

Local IP address.

LPORT

Local port.

T>

Type of event: R> == retransmit, L> == tail loss probe.

RADDR

Remote IP address.

RPORT

Remote port.

STATE

TCP session state.

SEQ

TCP sequence.

RETRANSMITS

Accumulated occurred retransmits since start.

## OVERHEAD
Should be negligible: TCP retransmit events should be low (<1000/s), and the low overhead this tool adds to each event should make the cost negligible.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
tcpconnect(8), tcpaccept(8)


bcc-tcprtt

title: bcc-tcprtt(8)

text:
# bcc-tcprtt(8)

## NAME
tcprtt - Trace TCP RTT of established connections. Uses Linux eBPF/bcc.

## SYNOPSIS
tcprtt [-h] [-T] [-D] [-m] [-p LPORT] [-P RPORT] [-a LADDR] [-A RADDR] [-i INTERVAL] [-d DURATION] [-b] [-B] [-e] [-4 | -6]

## DESCRIPTION
This tool traces established connections RTT(round-trip time) to analyze the quality of network. This can be useful for general troubleshooting to distinguish the network latency is from user process or physical network.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print usage message.

-T

Include a time column on output (HH:MM:SS).

-D

Show debug infomation of bpf text.

-m

Output histogram in milliseconds.

-i INTERVAL

Print output every interval seconds.

-d DURATION

Total duration of trace in seconds.

-p LPORT

Filter for local port.

-P RPORT

Filter for remote port.

-a LADDR

Filter for local address.

-A RADDR

Filter for remote address.

-b

Show sockets histogram by local address.

-B

Show sockets histogram by remote address.

-e

Show extension summary(average).

-4

Trace IPv4 family only.

-6

Trace IPv6 family only.

## EXAMPLES
Trace TCP RTT and print 1 second summaries, 10 times:

\# tcprtt -i 1 -d 10

Summarize in millisecond, and timestamps:

\# tcprtt -m -T

Only trace TCP RTT for remote address 192.168.1.100 and remote port 80:

\# tcprtt -i 1 -d 10 -A 192.168.1.100 -P 80

Trace local port and show a breakdown of remote hosts RTT:

\# tcprtt -i 3 --lport 80 --byraddr

Trace IPv4 family only:

\# tcprtt -4

Trace IPv6 family only:

\# tcprtt -6

## OVERHEAD
This traces the kernel tcp_rcv_established function and collects TCP RTT. The rate of this depends on your server application. If it is a web or proxy server accepting many tens of thousands of connections per second.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
zhenwei pi

## SEE ALSO
tcptracer(8), tcpconnect(8), funccount(8), tcpdump(8)


bcc-tcpstates

title: bcc-tcpstates(8)

text:
# bcc-tcpstates(8)

## NAME
tcpstates - Trace TCP session state changes with durations. Uses Linux eBPF/bcc.

## SYNOPSIS
tcpstates [-h] [-T] [-t] [-w] [-s] [-D PORTS] [-L PORTS] [-Y] [-4 | -6]

## DESCRIPTION
This tool traces TCP session state changes while tracing, and prints details including the duration in each state. This can help explain the latency of TCP connections: whether the time is spent in the ESTABLISHED state (data transfer), or initialization state (SYN_SENT), etc.

This tool works using the sock:inet_sock_set_state tracepoint, which was added to Linux 4.16. Linux 4.16 also included extra state transitions so that all TCP transitions could be observed by this tracepoint.

Only TCP state changes are traced, so it is expected that the overhead of this tool is much lower than typical send/receive tracing.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc, and the sock:inet_sock_set_state tracepoint.

## OPTIONS
-h

Print usage message.

-s

Comma separated values output (parseable).

-t

Include a timestamp column (seconds).

-T

Include a time column (HH:MM:SS).

-w

Wide column output (fits IPv6 addresses).

-L PORTS

Comma-separated list of local ports to trace (filtered in-kernel).

-D PORTS

Comma-separated list of destination ports to trace (filtered in-kernel).

-Y

Log session state changes to the systemd journal.

-4

Trace IPv4 family only.

-6

Trace IPv6 family only.

## EXAMPLES
Trace all TCP sessions, and show all state changes:

\# tcpstates

Include a timestamp column, and wide column output:

\# tcpstates -tw

Trace connections to local ports 80 and 81 only:

\# tcpstates -L 80,81

Trace connections to remote port 80 only:

\# tcpstates -D 80

Trace IPv4 family only:

\# tcpstates -4

Trace IPv6 family only:

\# tcpstates -6

## FIELDS
TIME

Time of the change, in HH:MM:SS format.

TIME(s)

Time of the change, in seconds.

C-PID

The current on-CPU process ID. This may show the process that owns the TCP session if the state change executes in synchronous process context, else it is likely to show the kernel (asynchronous state change).

C-COMM

The current on-CPU process name. This may show the process that owns the TCP session if the state change executes in synchronous process context, else it is likely to show the kernel (asynchronous state change).

IP

IP address family (4 or 6)

LADDR

Local IP address.

RADDR

Remote IP address.

LPORT

Local port.

RPORT

Remote port.

OLDSTATE

Previous TCP state.

NEWSTATE

New TCP state.

MS

Duration of this state.

## OVERHEAD
This traces the kernel TCP set state function, which should be called much less often than send/receive tracing, and therefore have lower overhead. The overhead of the tool is relative to the rate of new TCP sessions: if this is high, over 10,000 per second, then there may be noticeable overhead just to print out 10k lines of formatted output per second.

You can find out the rate of new TCP sessions using "sar -n TCP 1", and adding the active/s and passive/s columns.

As always, test and understand this tools overhead for your types of workloads before production use.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
tcpaccept(8), tcpconnect(8), tcptop(8), tcplife(8)


bcc-tcpsubnet

title: bcc-tcpsubnet(8)

text:
# bcc-tcpsubnet(8)

## NAME
tcpsubnet - Summarize and aggregate IPv4 TCP traffic by subnet.

## SYNOPSIS
tcpsubnet [-h] [-v] [-J] [-f FORMAT] [-i INTERVAL] [subnets]

## DESCRIPTION
This tool summarizes and aggregates IPv4 TCP sent to the subnets passed in argument and prints to stdout on a fixed interval.

This uses dynamic tracing of kernel TCP send/receive functions, and will need to be updated to match kernel changes.

The traced data is summarized in-kernel using a BPF map to reduce overhead. At very high TCP event rates, the overhead may still be measurable. See the OVERHEAD section for more details.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print USAGE message.

-v

Run in verbose mode. Will output subnet evaluation and the BPF program

-J

Format output in JSON.

-i

Interval between updates, seconds (default 1).

-f

Format output units. Supported values are bkmBKM. When using kmKM the output will be rounded to floor.

subnets

Comma separated list of subnets. Traffic will be categorized in theses subnets. Order matters. (default 127.0.0.1/32,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,0.0.0.0/0)

## EXAMPLES
Summarize TCP traffic by the default subnets:

\# tcpsubnet

Summarize all TCP traffic:

\# tcpsubnet 0.0.0.0/0

Summarize all TCP traffic and output in JSON and Kb:

\# tcpsubnet -J -fk 0.0.0.0/0

## FIELDS
(Standad output) Left hand side column:

Subnet

(Standard output) Right hand side column:

Aggregate traffic in units passed as argument

(JSON output) date

Current date formatted in the system locale

(JSON output) time

Current time formatted in the system locale

(JSON output) entries

Map of subnets to aggregates. Values will be in format passed to -f

## OVERHEAD
This traces all tcp_sendmsg function calls in the TCP/IP stack. It summarizes data in-kernel to reduce overhead. A simple iperf test (v2.0.5) with the default values shows a loss of ~5% throughput. On 10 runs without tcpsubnet running the average throughput was 32.42Gb/s, with tcpsubnet enabled it was 31.26Gb/s. This is not meant to be used as a long running service. Use it for troubleshooting or for a controlled interval. As always, try it out in a test environment first.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Rodrigo Manyari

## INSPIRATION
tcptop(8) by Brendan Gregg

## SEE ALSO
netlink(7)


bcc-tcpsynbl

title: bcc-tcpsynbl(8)

text:
# bcc-tcpsynbl(8)

## NAME
tcpsynbl - Show the TCP SYN backlog as a histogram. Uses BCC/eBPF.

## SYNOPSIS
tcpsynbl [-4 | -6]

## DESCRIPTION
This tool shows the TCP SYN backlog size during SYN arrival as a histogram. This lets you see how close your applications are to hitting the backlog limit and dropping SYNs (causing performance issues with SYN retransmits), and is a measure of workload saturation. The histogram shown is measured at the time of SYN received, and a separate histogram is shown for each backlog limit.

This works by tracing the tcp_v4_syn_recv_sock() and tcp_v6_syn_recv_sock() kernel functions using dynamic instrumentation. Since these functions may change in future kernels, this tool may need maintenance to keep working.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and BCC.

## OPTIONS
-h

Print usage message.

-4

Trace IPv4 family only.

-6

Trace IPv6 family only.

## EXAMPLES
Show the TCP SYN backlog as a histogram.

\# tcpsynbl

Trace IPv4 family only:

\# tcpsynbl -4

Trace IPv6 family only:

\# tcpsynbl -6

## FIELDS
backlog

The backlog size when a SYN was received.

count

The number of times this backlog size was encountered.

distribution

An ASCII visualization of the count column.

## OVERHEAD
Inbound SYNs should be relatively low compared to packets and other events, so the overhead of this tool is expected to be negligible.

## SOURCE
This originated as a bpftrace tool from the book "BPF Performance Tools", published by Addison Wesley (2019):

http://www.brendangregg.com/bpf-performance-tools-book.html

See the book for more documentation on this tool.

This version is in the BCC repository:

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
tcptop(8)


bcc-tcptop

title: bcc-tcptop(8)

text:
# bcc-tcptop(8)

## NAME
tcptop - Summarize TCP send/recv throughput by host. Top for TCP.

## SYNOPSIS
tcptop [-h] [-C] [-S] [-p PID] [--cgroupmap MAPPATH]

[--mntnsmap MAPPATH] [interval] [count] [-4 | -6]

## DESCRIPTION
This is top for TCP sessions.

This summarizes TCP send/receive Kbytes by host, and prints a summary that refreshes, along other system-wide metrics.

This uses dynamic tracing of kernel TCP send/receive functions, and will need to be updated to match kernel changes.

The traced TCP functions are usually called at a lower rate than per-packet functions, and therefore have lower overhead. The traced data is summarized in-kernel using a BPF map to further reduce overhead. At very high TCP event rates, the overhead may still be measurable. See the OVERHEAD section for more details.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print USAGE message.

-C

Don't clear the screen.

-S

Don't print the system summary line (load averages).

-p PID

Trace this PID only.

--cgroupmap MAPPATH

Trace cgroups in this BPF map only (filtered in-kernel).

--mntnsmap MAPPATH

Trace mount namespaces in this BPF map only (filtered in-kernel).

interval

Interval between updates, seconds (default 1).

count

Number of interval summaries (default is many).

-4

Trace IPv4 family only.

-6

Trace IPv6 family only.

## EXAMPLES
Summarize TCP throughput by active sessions, 1 second refresh:

\# tcptop

Don't clear the screen (rolling output), and 5 second summaries:

\# tcptop -C 5

Trace PID 181 only, and don't clear the screen:

\# tcptop -Cp 181

Trace a set of cgroups only (see special_filtering.md from bcc sources for more details):

\# tcptop --cgroupmap /sys/fs/bpf/test01

Trace IPv4 family only:

\# tcptop -4

Trace IPv6 family only:

\# tcptop -6

## FIELDS
loadavg:

The contents of /proc/loadavg

PID

Process ID.

COMM

Process name.

LADDR

Local address (IPv4), and TCP port

RADDR

Remote address (IPv4), and TCP port

LADDR6

Source address (IPv6), and TCP port

RADDR6

Destination address (IPv6), and TCP port

RX_KB

Received Kbytes

TX_KB

Transmitted Kbytes

## OVERHEAD
This traces all send/receives in TCP, high in the TCP/IP stack (close to the application) which are usually called at a lower rate than per-packet functions, lowering overhead. It also summarizes data in-kernel to further reduce overhead. These techniques help, but there may still be measurable overhead at high send/receive rates, eg, ~13% of one CPU at 100k events/sec. use funccount to count the kprobes in the tool to find out this rate, as the overhead is relative to the rate. Some sample production servers tested found total TCP event rates of 4k to 15k per second, and the CPU overhead at these rates ranged from 0.5% to 2.0% of one CPU. If your send/receive rate is low (eg, <1000/sec) then the overhead is expected to be negligible; Test in a lab environment first.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## INSPIRATION
top(1) by William LeFebvre

## SEE ALSO
tcpconnect(8), tcpaccept(8)


bcc-tcptracer

title: bcc-tcptracer(8)

text:
# bcc-tcptracer(8)

## NAME
tcptracer - Trace TCP established connections. Uses Linux eBPF/bcc.

## SYNOPSIS
tcptracer [-h] [-v] [-t] [-p PID] [-N NETNS] [--cgroupmap MAPPATH] [--mntnsmap MAPPATH] [-4 | -6]

## DESCRIPTION
This tool traces established TCP connections that open and close while tracing, and prints a line of output per connect, accept and close events. This includes the type of event, PID, IP addresses and ports.

This tool works by using kernel dynamic tracing, and will need to be updated if the kernel implementation changes. Only established TCP connections are traced, so it is expected that the overhead of this tool is rather low.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print usage message.

-v

Print full lines, with long event type names and network namespace numbers.

-t

Include timestamp on output

-p PID

Trace this process ID only (filtered in-kernel).

-N NETNS

Trace this network namespace only (filtered in-kernel).

--cgroupmap MAPPATH

Trace cgroups in this BPF map only (filtered in-kernel).

--mntnsmap MAPPATH

Trace mount namespaces in the map (filtered in-kernel).

-4

Trace IPv4 family only.

-6

Trace IPv6 family only.

## EXAMPLES
Trace all TCP established connections:

\# tcptracer

Trace all TCP established connections with verbose lines:

\# tcptracer -v

Trace PID 181 only:

\# tcptracer -p 181

Trace connections in network namespace 4026531969 only:

\# tcptracer -N 4026531969

Trace a set of cgroups only (see special_filtering.md from bcc sources for more details):

\# tcptracer --cgroupmap /sys/fs/bpf/test01

Trace IPv4 family only:

\# tcptracer -4

Trace IPv6 family only:

\# tcptracer -6

## FIELDS
TYPE

Type of event. In non-verbose mode: C for connect, A for accept, X for close.

PID

Process ID

COMM

Process name

IP

IP address family (4 or 6)

SADDR

Source IP address.

DADDR

Destination IP address.

SPORT

Source port.

DPORT

Destination port.

NETNS

Network namespace where the event originated.

## OVERHEAD
This traces the kernel inet accept function, and the TCP connect, close, and set state functions. However, it only prints information for connections that are established, so it shouldn't have a huge overhead.

As always, test and understand this tools overhead for your types of workloads before production use.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Iago López Galeiras

## SEE ALSO
tcpaccept(8), tcpconnect(8), tcptop(8), tcplife(8)


bcc-threadsnoop

title: bcc-threadsnoop(8)

text:
# bcc-threadsnoop(8)

## NAME
threadsnoop - Trace thread creation via pthread_create(). Uses BCC/eBPF.

## SYNOPSIS
threadsnoop

## DESCRIPTION
threadsnoop traces calls to pthread_create(), showing this path of thread creation. This can be used for workload characterization and discovery, and is a companion to execsnoop(8) which traces execve(2).

This works by tracing the pthread_create() from libpthread.so.0. The path to this library may need adjusting in the tool source to match your system.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and BCC.

## EXAMPLES
Trace calls pthread_create():

\# threadsnoop

## FIELDS
TIME(ms)

Elapsed time since the tool began tracing (in milliseconds).

PID

The process ID.

COMM

The process (thread) name.

FUNC

The name of the start routine, if the symbol is available, else a hex address for the start routine address.

## OVERHEAD
Thread creation is expected to be low (<< 1000/s), so the overhead of this tool is expected to be negligible.

## SOURCE
This originated as a bpftrace tool from the book "BPF Performance Tools", published by Addison Wesley (2019):

http://www.brendangregg.com/bpf-performance-tools-book.html

See the book for more documentation on this tool.

This version is in the BCC repository:

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
execsnoop(8)


bcc-tplist

title: bcc-tplist(8)

text:
# bcc-tplist(8)

## NAME
tplist - Display kernel tracepoints or USDT probes and their formats.

## SYNOPSIS
tplist [-p PID] [-l LIB] [-v] [filter]

## DESCRIPTION
tplist lists all kernel tracepoints, and can optionally print out the tracepoint format; namely, the variables that you can trace when the tracepoint is hit. tplist can also list USDT probes embedded in a specific library or executable, and can list USDT probes for all the libraries loaded by a specific process. These features are usually used in conjunction with the argdist and/or trace tools.

On a typical system, accessing the tracepoint list and format requires root. However, accessing USDT probes does not require root.

## OPTIONS
-p PID

Display the USDT probes from all the libraries loaded by the specified process.

-l LIB

Display the USDT probes from the specified library or executable. If the librar or executable can be found in the standard paths, a full path is not required.

-v

Increase the verbosity level. Can be used to display the variables, locations, and arguments of tracepoints and USDT probes.

[filter]

A wildcard expression that specifies which tracepoints or probes to print. For example, block:* will print all block tracepoints (block:block_rq_complete, etc.). Regular expressions are not supported.

## EXAMPLES
Print all kernel tracepoints:

\# tplist

Print all net tracepoints with their format:

\# tplist -v 'net:*'

Print all USDT probes in libpthread:

$ tplist -l pthread

Print all USDT probes in process 4717 from the libc provider:

$ tplist -p 4717 'libc:*'

Print all the USDT probes in the node executable:

$ tplist -l node

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein


bcc-trace

title: bcc-trace(8)

text:
# bcc-trace(8)

## NAME
trace - Trace a function and print its arguments or return value, optionally evaluating a filter. Uses Linux eBPF/bcc.

## SYNOPSIS
trace [-h] [-b BUFFER_PAGES] [-p PID] [-L TID] [--uid UID] [-v] [-Z STRING_SIZE] [-S] [-M MAX_EVENTS] [-t]

[-u] [-T] [-C] [-c CGROUP_PATH] [-n NAME] [-f MSG_FILTER] [-B] [-s SYM_FILE_LIST] [-K] [-U] [-a]

[-I header] [-A]

probe [probe ...]

## DESCRIPTION
trace probes functions you specify and displays trace messages if a particular condition is met. You can control the message format to display function arguments and return values.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print usage message.

-p PID

Trace only functions in the process PID.

-L TID

Trace only functions in the thread TID.

--uid UID

Trace only functions from user UID.

-v

Display the generated BPF program, for debugging purposes.

-z STRING_SIZE

When collecting string arguments (of type char*), collect up to STRING_SIZE characters. Longer strings will be truncated.

-s SYM_FILE_LIST

When collecting stack trace in build id format, use the comma separated list for symbol resolution.

-S

If set, trace messages from trace's own process. By default, this is off to avoid tracing storms -- for example, if you trace the write system call, and consider that trace is writing to the standard output.

-M MAX_EVENTS

Print up to MAX_EVENTS trace messages and then exit.

-t

Print times relative to the beginning of the trace (offsets), in seconds.

-u

Print UNIX timestamps instead of offsets from trace beginning, requires -t.

-T

Print the time column.

-C

Print CPU id.

-c CGROUP_PATH

Trace only functions in processes under CGROUP_PATH hierarchy.

-n NAME

Only print process names containing this name.

-f MSG_FILTER

Only print message of event containing this string.

-B

Treat argument of STRCMP helper as a binary value

-K

Print the kernel stack for each event.

-U

Print the user stack for each event.

-a

Print virtual address in kernel and user stacks.

-I header

Additional header files to include in the BPF program. This is needed if your filter or print expressions use types or data structures that are not available in the standard headers. For example: 'linux/mm.h'

-A

Print aggregated amount of each trace. This should be used with -M/--max-events together.

probe [probe ...]

One or more probes that attach to functions, filter conditions, and print information. See PROBE SYNTAX below.

## PROBE SYNTAX
The general probe syntax is as follows:

[{p,r}]:[library]:function[+offset][(signature)] [(predicate)] ["format string"[, arguments]]

{t:category:event,u:library:probe} [(predicate)] ["format string"[, arguments]]

{[{p,r}],t,u}

Probe type - "p" for function entry, "r" for function return, "t" for kernel tracepoint, "u" for USDT probe. The default probe type is "p".

[library]

Library containing the probe. Specify the full path to the .so or executable file where the function to probe resides. Alternatively, you can specify just the lib name: for example, "c" refers to libc. If no library name is specified, the kernel is assumed. Also, you can specify an executable name (without a full path) if it is in the PATH. For example, "bash".

category

The tracepoint category. For example, "sched" or "irq".

function

The function to probe. offset The offset after the address of the function where the probe should injected. For example "kfree_skb+56" in decimal or hexadecimal "kfree_skb+0x38" format. Only works with kprobes and uprobes. Zero if omitted.

signature

The optional signature of the function to probe. This can make it easier to access the function's arguments, instead of using the "arg1", "arg2" etc. argument specifiers. For example, "(struct timespec *ts)" in the signature position lets you use "ts" in the filter or print expressions.

event

The tracepoint event. For example, "block_rq_complete".

probe

The USDT probe name. For example, "pthread_create".

[(predicate)]

The filter applied to the captured data. Only if the filter evaluates as true, the trace message will be printed. The filter can use any valid C expression that refers to the argument values: arg1, arg2, etc., or to the return value retval in a return probe. If necessary, use C cast operators to coerce the arguments to the desired type. For example, if arg1 is of type int, use the expression ((int)arg1 < 0) to trace only invocations where arg1 is negative. Note that only arg1-arg6 are supported, and only if the function is using the standard x86_64 convention where the first six arguments are in the RDI, RSI, RDX, RCX, R8, R9 registers. If no predicate is specified, all function invocations are traced. The predicate expression may also use the STRCMP pseudo-function to compare a predefined string to a string argument. For example: STRCMP("test", arg1). The order of arguments is important: the first argument MUST be a quoted literal string, and the second argument can be a runtime string, most typically an argument.

["format string"[, arguments]]

A printf-style format string that will be used for the trace message. You can use the following format specifiers: %s, %d, %u, %lld, %llu, %hd, %hu, %c, %x, %llx -- with the same semantics as printf's. Make sure to pass the exact number of arguments as there are placeholders in the format string. The format specifier replacements may be any C expressions, and may refer to the same special keywords as in the predicate (arg1, arg2, etc.). In addition to the above format specifiers, you can also use %K and %U when the expression is an address that potentially points to executable code (i.e., a symbol). trace will resolve %K specifiers to a kernel symbol, such as vfs__read, and will resolve %U specifiers to a user-space symbol in that process, such as sprintf.

In tracepoints, both the predicate and the arguments may refer to the tracepoint format structure, which is stored in the special "args" variable. For example, the block:block_rq_complete tracepoint can print or filter by args->nr_sector. To discover the format of your tracepoint, use the tplist tool.

In USDT probes, the arg1, ..., argN variables refer to the probe's arguments. To determine which arguments your probe has, use the tplist tool.

The predicate expression and the format specifier replacements for printing may also use the following special keywords: $pid, $tgid to refer to the current process' pid and tgid; $uid, $gid to refer to the current user's uid and gid; $cpu to refer to the current processor number.

## EXAMPLES
Trace all invocations of the open system call with the name of the file (from userspace) being opened:

\# trace '::do_sys_open "%s", arg2@user'

Trace all invocations of the read system call where the number of bytes requested is greater than 20,000:

\# trace '::sys_read (arg3 > 20000) "read %d bytes", arg3'

Trace all malloc calls and print the size of the requested allocation:

\# trace ':c:malloc "size = %d", arg1'

Trace returns from the readline function in bash and print the return value as a string:

\# trace 'r:bash:readline "%s", retval'

Trace the block:block_rq_complete tracepoint and print the number of sectors completed:

\# trace 't:block:block_rq_complete "%d sectors", args->nr_sector'

Trace the pthread_create USDT probe from the pthread library and print the address of the thread's start function:

\# trace 'u:pthread:pthread_create "start addr = %llx", arg3'

Trace the nanosleep system call and print the sleep duration in nanoseconds:

\# trace 'p::SyS_nanosleep(struct timespec *ts) sleep for %lld ns , ts->tv_nsec'

Trace the inet_pton system call using build id mechanism and print the stack

\# trace -s /lib/x86_64-linux-gnu/libc.so.6,/bin/ping 'p:c:inet_pton' -U

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein


bcc-ttysnoop

title: bcc-ttysnoop(8)

text:
# bcc-ttysnoop(8)

## NAME
ttysnoop - Watch output from a tty or pts device. Uses Linux eBPF/bcc.

## SYNOPSIS
ttysnoop [-h] [-C] device

## DESCRIPTION
ttysnoop watches a tty or pts device, and prints the same output that is appearing on that device. It can be used to mirror the output from a shell session, or the system console.

This works by use of kernel dynamic tracing of the tty_write() function. This tool will need updating in case that kernel function changes in a future kernel version.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-C

Don't clear the screen.

-s SIZE , --datasize SIZE

Size of the transmitting buffer (default 256).

-c COUNT, --datacount COUNT

Number of times ttysnop checks for SIZE bytes of data (default 16).

device

Either a path to a tty device (eg, /dev/tty0) or a pts number (eg, the "3" from /dev/pts/3).

## EXAMPLES
Snoop output from /dev/pts/2

\# ttysnoop /dev/pts/2

Snoop output from /dev/pts/2 (shortcut)

\# ttysnoop 2

Snoop output from the system console

\# ttysnoop /dev/console

Snoop output from /dev/tty0

\# ttysnoop /dev/tty0

## OVERHEAD
As the rate of tty_write() is expected to be very low (<100/s), the overhead of this tool is expected to be negligible.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
opensnoop(1)


bcc-ucalls

title: bcc-ucalls(8)

summary: Summarize method calls

text:
# bcc-ucalls(8)

## NAME
ucalls, javacalls, perlcalls, phpcalls, pythoncalls, rubycalls, tclcalls - Summarize method calls from high-level languages and Linux syscalls.

## SYNOPSIS
javacalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

perlcalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

phpcalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

pythoncalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

rubycalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

tclcalls [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

ucalls [-l {java,perl,php,python,ruby}] [-h] [-T TOP] [-L] [-S] [-v] [-m] pid [interval]

## DESCRIPTION
This tool summarizes method calls from high-level languages such as Java, Perl, PHP, Python, Ruby, and Tcl. It can also trace Linux system calls. Whenever a method is invoked, ucalls records the call count and optionally the method's execution time (latency) and displays a summary.

This uses in-kernel eBPF maps to store per process summaries for efficiency.

This tool relies on USDT probes embedded in many high-level languages, such as Java, Perl, PHP, Python, Ruby, and Tcl. It requires a runtime instrumented with these probes, which in some cases requires building from source with a USDT-specific flag, such as "--enable-dtrace" or "--with-dtrace". For Java, method probes are not enabled by default, and can be turned on by running the Java process with the "-XX:+ExtendedDTraceProbes" flag. For PHP processes, the environment variable USE_ZEND_DTRACE must be set to 1.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-l {java,perl,php,python,ruby,tcl}

The language to trace. If not provided, only syscalls are traced (when the -S option is used).

-T TOP

Print only the top methods by frequency or latency.

-L

Collect method invocation latency (duration).

-S

Collect Linux syscalls frequency and timing.

-v

Print the resulting BPF program, for debugging purposes.

-m

Print times in milliseconds (the default is microseconds).

pid

The process id to trace.

interval

Print summary after this number of seconds and then exit. By default, wait for Ctrl+C to terminate.

## EXAMPLES
Trace the top 10 Ruby method calls:

\# ucalls -T 10 -l ruby 1344

Trace Python method calls and Linux syscalls including latency in milliseconds:

\# ucalls -l python -mL 2020

Trace only syscalls and print a summary after 10 seconds:

\# ucalls -S 788 10

## OVERHEAD
Tracing individual method calls will produce a considerable overhead in all high-level languages. For languages with just-in-time compilation, such as Java, the overhead can be more considerable than for interpreted languages. On the other hand, syscall tracing will typically be tolerable for most processes, unless they have a very unusual rate of system calls.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _example.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein

## SEE ALSO
ustat(8), argdist(8)


bcc-uflow

title: bcc-uflow(8)

summary: Print a flow graph of method

text:
# bcc-uflow(8)

## NAME
uflow, javaflow, perlflow, phpflow, pythonflow, rubyflow, tclflow - Print a flow graph of method calls in high-level languages.

## SYNOPSIS
javaflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

perlflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

phpflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

pythonflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

rubyflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

tclflow [-h] [-M METHOD] [-C CLAZZ] [-v] pid

uflow [-h] [-M METHOD] [-C CLAZZ] [-v] [-l {java,perl,php,python,ruby,tcl}] pid

## DESCRIPTION
uflow traces method calls and prints them in a flow graph that can facilitate debugging and diagnostics by following the program's execution (method flow).

This tool relies on USDT probes embedded in many high-level languages, such as Java, Perl, PHP, Python, Ruby, and Tcl. It requires a runtime instrumented with these probes, which in some cases requires building from source with a USDT-specific flag, such as "--enable-dtrace" or "--with-dtrace". For Java processes, the startup flag "-XX:+ExtendedDTraceProbes" is required. For PHP processes, the environment variable USE_ZEND_DTRACE must be set to 1.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-M METHOD

Print only method calls where the method name begins with this string.

-C CLAZZ

Print only method calls where the class name begins with this string. The class name interpretation strongly depends on the language. For example, in Java use "package/subpackage/ClassName" to refer to classes.

-v

Print the resulting BPF program, for debugging purposes.

{java,perl,php,python,ruby,tcl}

The language to trace.

pid

The process id to trace.

## EXAMPLES
Follow method flow in a Ruby process:

\# uflow ruby 148

Follow method flow in a Java process where the class name is java.lang.Thread:

\# uflow -C java/lang/Thread java 1802

## FIELDS
CPU

The CPU number on which the method was invoked. This is useful to easily see where the output skips to a different CPU.

PID

The process id.

TID

The thread id.

TIME

The duration of the method call.

METHOD

The method name.

## OVERHEAD
This tool has extremely high overhead because it prints every method call. For some scenarios, you might see lost samples in the output as the tool is unable to keep up with the rate of data coming from the kernel. Filtering by class or method prefix can help reduce the amount of data printed, but there is still a very high overhead in the collection mechanism. Do not use for performance- sensitive production scenarios, and always test first.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _example.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein

## SEE ALSO
trace(8), ustat(8)


bcc-ugc

title: bcc-ugc(8)

summary: Trace garbage collection events in

text:
# bcc-ugc(8)

## NAME
ugc, javagc, nodegc, pythongc, rubygc - Trace garbage collection events in high-level languages.

## SYNOPSIS
javagc [-h] [-v] [-m] [-M MINIMUM] [-F FILTER] pid

nodegc [-h] [-v] [-m] [-M MINIMUM] [-F FILTER] pid

pythongc [-h] [-v] [-m] [-M MINIMUM] [-F FILTER] pid

rubygc [-h] [-v] [-m] [-M MINIMUM] [-F FILTER] pid

ugc [-h] [-v] [-m] [-M MINIMUM] [-F FILTER] [-l {java,node,python,ruby}] pid

## DESCRIPTION
This traces garbage collection events as they occur, including their duration and any additional information (such as generation collected or type of GC) provided by the respective language's runtime.

This tool relies on USDT probes embedded in many high-level languages, such as Java, Node, Python, and Ruby. It requires a runtime instrumented with these probes, which in some cases requires building from source with a USDT-specific flag, such as "--enable-dtrace" or "--with-dtrace".

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-v

Print the resulting BPF program, for debugging purposes.

-m

Print times in milliseconds. The default is microseconds.

-M MINIMUM

Display only collections that are longer than this threshold. The value is given in milliseconds. The default is to display all collections.

-F FILTER

Display only collections whose textual description matches (contains) this string. The default is to display all collections. Note that the filtering here is performed in user-space, and not as part of the BPF program. This means that if you have thousands of collection events, specifying this filter will not reduce the amount of data that has to be transferred from the BPF program to the user-space script.

{java,node,python,ruby}

The language to trace.

pid

The process id to trace.

## EXAMPLES
Trace garbage collections in a specific Node process:

\# ugc node 148

Trace garbage collections in a specific Java process, and print GC times in

milliseconds: # ugc -m java 6004

Trace garbage collections in a specific Java process, and display them only if

they are longer than 10ms and have the string "Tenured" in their detailed description: # ugc -M 10 -F Tenured java 6004

## FIELDS
START

The start time of the GC, in seconds from the beginning of the trace.

TIME

The duration of the garbage collection event.

DESCRIPTION

The runtime-provided description of this garbage collection event.

## OVERHEAD
Garbage collection events, even if frequent, should not produce a considerable overhead when traced because they are still not very common. Even hundreds of GCs per second (which is a very high rate) will still produce a fairly negligible overhead.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _example.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein

## SEE ALSO
trace(8), ustat(8), uobjnew(8)


bcc-uobjnew

title: bcc-uobjnew(8)

summary: Summarize object allocations in

text:
# bcc-uobjnew(8)

## NAME
uobjnew, cobjnew, javaobjnew, rubyobjnew, tclobjnew - Summarize object allocations in high-level languages.

## SYNOPSIS
cobjnew [-h] [-C TOP_COUNT] [-S TOP_SIZE] [-v] pid [interval]

javaobjnew [-h] [-C TOP_COUNT] [-S TOP_SIZE] [-v] pid [interval]

rubyobjnew [-h] [-C TOP_COUNT] [-S TOP_SIZE] [-v] pid [interval]

tclobjnew [-h] [-C TOP_COUNT] [-S TOP_SIZE] [-v] pid [interval]

uobjnew [-h] [-C TOP_COUNT] [-S TOP_SIZE] [-v] [-l {c,java,ruby,tcl}] pid [interval]

## DESCRIPTION
uobjnew traces object allocations in high-level languages (including "malloc") and prints summaries of the most frequently allocated types by number of objects or number of bytes.

This tool relies on USDT probes embedded in many high-level languages, such as C, Java, Ruby, and Tcl. It requires a runtime instrumented with these probes, which in some cases requires building from source with a USDT-specific flag, such as "--enable-dtrace" or "--with-dtrace". For Java, the Java process must be started with the "-XX:+ExtendedDTraceProbes" flag.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-C TOP_COUNT

Print the top object types sorted by number of instances.

-S TOP_SIZE

Print the top object types sorted by size.

-v

Print the resulting BPF program, for debugging purposes.

{c,java,ruby,tcl}

The language to trace.

pid

The process id to trace.

interval

Wait this many seconds and then print the summary and exit. By default, wait for Ctrl+C to exit.

## EXAMPLES
Trace object allocations in a Ruby process:

\# uobjnew ruby 148

Trace object allocations from "malloc" and print the top 10 by total size:

\# uobjnew -S 10 c 1788

## FIELDS
TYPE

The object type being allocated. For C (malloc), this is the block size.

ALLOCS

The number of objects allocated.

BYTES

The number of bytes allocated.

## OVERHEAD
Object allocation events are quite frequent, and therefore the overhead from running this tool can be considerable. Use with caution and make sure to test before using in a production environment. Nonetheless, even thousands of allocations per second will likely produce a reasonable overhead when investigating a problem.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _example.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein

## SEE ALSO
ustat(8), ugc(8), memleak(8)


bcc-ustat

title: bcc-ustat(8)

summary: Activity stats from

text:
# bcc-ustat(8)

## NAME
ustat, javastat, nodestat, perlstat, phpstat, pythonstat, rubystat, tclstat - Activity stats from high-level languages.

## SYNOPSIS
javastat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

nodestat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

perlstat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

phpstat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

pythonstat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

rubystat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

tclstat [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

ustat [-l {java,node,perl,php,python,ruby,tcl}] [-C] [-S {cload,excp,gc,method,objnew,thread}] [-r MAXROWS] [-d] [interval [count]]

## DESCRIPTION
This is "top" for high-level language events, such as garbage collections, exceptions, thread creations, object allocations, method calls, and more. The events are aggregated for each process and printed in a top-like table, which can be sorted by various fields. Not all language runtimes provide the same set of details.

This uses in-kernel eBPF maps to store per process summaries for efficiency.

This tool relies on USDT probes embedded in many high-level languages, such as Java, Node, Perl, PHP, Python, Ruby, and Tcl. It requires a runtime instrumented with these probes, which in some cases requires building from source with a USDT-specific flag, such as "--enable-dtrace" or "--with-dtrace". For Java, some probes are not enabled by default, and can be turned on by running the Java process with the "-XX:+ExtendedDTraceProbes" flag. For PHP processes, the environment variable USE_ZEND_DTRACE must be set to 1.

Newly-created processes will only be traced at the next interval. If you run this tool with a short interval (say, 1-5 seconds), this should be virtually unnoticeable. For longer intervals, you might miss processes that were started and terminated during the interval window.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-l {java,node,perl,php,python,ruby,tcl}

The language to trace. By default, all languages are traced.

-C

Do not clear the screen between updates.

-S {cload,excp,gc,method,objnew,thread}

Sort the output by the specified field.

-r MAXROWS

Do not print more than this number of rows.

-d

Print the resulting BPF program, for debugging purposes.

interval

Interval between updates, seconds.

count

Number of interval summaries.

## EXAMPLES
Summarize activity in high-level languages, 1 second refresh:

\# ustat

Don't clear the screen, and top 8 rows only:

\# ustat -Cr 8

5 second summaries, 10 times only:

\# ustat 5 10

## FIELDS
loadavg

The contents of /proc/loadavg

PID

Process ID.

CMDLINE

Process command line (often the second and following arguments will give you a hint as to which application is being run.

METHOD/s

Count of method invocations during interval.

GC/s

Count of garbage collections during interval.

OBJNEW/s

Count of objects allocated during interval.

CLOAD/s

Count of classes loaded during interval.

EXC/s

Count of exceptions thrown during interval.

THR/s

Count of threads created during interval.

## OVERHEAD
When using this tool with high-frequency events, such as method calls, a very significant slow-down can be expected. However, many of the high-level languages covered by this tool already have a fairly high per-method invocation cost, especially when running in interpreted mode. For the lower-frequency events, such as garbage collections or thread creations, the overhead should not be significant. Specifically, when probing Java processes and not using the "-XX:+ExtendedDTraceProbes" flag, the most expensive probes are not emitted, and the overhead should be acceptable.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _example.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein

## SEE ALSO
trace(8), argdist(8), tplist(8)


bcc-uthreads

title: bcc-uthreads(8)

text:
# bcc-uthreads(8)

## NAME
uthreads, cthreads, javathreads - Trace thread creation events in Java or pthreads.

## SYNOPSIS
cthreads [-h] [-v] pid javathreads [-h] [-v] pid uthreads [-h] [-l {c,java,none}] [-v] pid

## DESCRIPTION
This traces thread creation events in Java processes, or pthread creation events in any process. When a thread is created, its name or start address is printed.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-l {c,java,none}

The language to trace. C and none select tracing pthreads only, regardless of the runtime being traced.

-v

Print the resulting BPF program, for debugging purposes.

pid

The process id to trace.

## EXAMPLES
Trace Java thread creations:

\# uthreads -l java 148

Trace pthread creations:

\# uthreads 1802

## FIELDS
TIME

The event's time in seconds from the beginning of the trace.

ID

The thread's ID. The information in this column depends on the runtime.

TYPE

Event type -- thread start, stop, or pthread event.

DESCRIPTION

The thread's name or start address function name.

## OVERHEAD
Thread start and stop events are usually not very frequent, which makes this tool's overhead negligible.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _example.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Sasha Goldshtein

## SEE ALSO
ustat(8), trace(8)


bcc-vfscount

title: bcc-vfscount(8)

summary: Count VFS calls (vfs_*). Uses Linux eBPF/bcc.

text:
# bcc-vfscount(8)

## NAME
vfscount - Count VFS calls ("vfs_*"). Uses Linux eBPF/bcc.

## SYNOPSIS
vfscount [duration]

## DESCRIPTION
This counts VFS calls. This can be useful for general workload characterization of these operations.

This works by tracing all kernel functions beginning with "vfs_" using dynamic tracing. This may match more functions than you are interested in measuring: Edit the script to customize which functions to trace.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
duration

duration of the trace in seconds.

## EXAMPLES
Count some VFS calls until Ctrl-C is hit:

\# vfscount

Count some VFS calls in ten seconds

\# vfscount 10

## FIELDS
ADDR

Address of the instruction pointer that was traced (only useful if the FUNC column is suspicious and you would like to double check the translation).

FUNC

Kernel function name

COUNT

Number of calls while tracing

## OVERHEAD
This traces kernel vfs functions and maintains in-kernel counts, which are asynchronously copied to user-space. While the rate of VFS operations can be very high (>1M/sec), this is a relatively efficient way to trace these events, and so the overhead is expected to be small for normal workloads. Measure in a test environment, and if overheads are an issue, edit the script to reduce the types of vfs functions traced (currently all beginning with "vfs_").

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
vfsstat(8)


bcc-vfsstat

title: bcc-vfsstat(8)

text:
# bcc-vfsstat(8)

## NAME
vfsstat - Statistics for some common VFS calls. Uses Linux eBPF/bcc.

## SYNOPSIS
vfsstat [interval [count]]

## DESCRIPTION
This traces some common VFS calls and prints per-second summaries. This can be useful for general workload characterization, and looking for patterns in operation usage over time.

This works by tracing some kernel vfs functions using dynamic tracing, and will need updating to match any changes to these functions. Edit the script to customize which functions are traced. Also see vfscount, which is more easily customized to trace multiple functions.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## EXAMPLES
Print summaries each second:

\# vfsstat

Print output every five seconds, three times:

\# vfsstat 5 3

## FIELDS
READ/s

Number of vfs_read() calls as a per-second average.

WRITE/s

Number of vfs_write() calls as a per-second average.

CREATE/s

Number of vfs_create() calls as a per-second average.

OPEN/s

Number of vfs_open() calls as a per-second average.

FSYNC/s

Number of vfs_fsync() calls as a per-second average.

## OVERHEAD
This traces various kernel vfs functions and maintains in-kernel counts, which are asynchronously copied to user-space. While the rate of VFS operations can be very high (>1M/sec), this is a relatively efficient way to trace these events, and so the overhead is expected to be small for normal workloads. Measure in a test environment.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
vfscount(8)


bcc-virtiostat

title: bcc-virtiostat(8)

text:
# bcc-virtiostat(8)

## NAME
virtiostat - Trace VIRTIO devices input/output statistics. Uses Linux eBPF/bcc.

## SYNOPSIS
virtiostat [-h] [-T] [-D] [-d DRIVER] [-n DEVNAME] [INTERVAL] [COUNT]

## DESCRIPTION
This tool traces VIRTIO devices input/output statistics. It works in lower layer of VIRTIO base driver, so it could trace all the devices of VIRTIO family. For example, we can't get IO statistics of 9p-fs in a guest virtual machine by iostat command, but we can analyze IO statistics by virtiostat. The outputing result shows In/Out SGs(scatter list operation) to represent positive correlation IOPS, and In/Out BW to represent throughput.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print usage message.

-T

Include a time column on output (HH:MM:SS).

-D

Show debug infomation of bpf text.

-d DRIVER

Filter for driver name.

-n DEVNAME

Filter for device name.

INTERVAL

Print output every interval seconds.

COUNT

Total count of trace in seconds.

## EXAMPLES
Trace virtio device statistics and print 1 second summaries, 10 times:

\# virtiostat 1 10

Trace virtio block deivces only:

\# virtiostat -d virtio_blk

## OVERHEAD
This traces the kernel virtqueue_add_sgs, virtqueue_add_outbuf, virtqueue_add_inbuf, virtqueue_add_inbuf_ctx functions. The rate of this depends on all the VIRTIO devices IOPS.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
zhenwei pi

## SEE ALSO
iostat(1), iftop(8), funccount(8)


bcc-wakeuptime

title: bcc-wakeuptime(8)

text:
# bcc-wakeuptime(8)

## NAME
wakeuptime - Summarize sleep to wakeup time by waker kernel stack. Uses Linux eBPF/bcc.

## SYNOPSIS
wakeuptime [-h] [-u] [-p PID] [-v] [-f] [--stack-storage-size STACK_STORAGE_SIZE] [-m MIN_BLOCK_TIME] [-M MAX_BLOCK_TIME] [duration]

## DESCRIPTION
This program shows the kernel stack traces for threads that woke up other blocked threads, along with the process names of the waker and target, along with a sum of the time that the target was blocked: the "blocked time". It works by tracing when threads block and when they were then woken up, and measuring the time delta. This time measurement will be very similar to off-CPU time, however, off-CPU time may include a little extra time spent waiting on a run queue to be scheduled. The stack traces, process names, and time spent blocked is summarized in the kernel using an eBPF map for efficiency.

The output summary will help you identify reasons why threads were blocking by showing who woke them up, along with the time they were blocked. This spans all types of blocking activity: disk I/O, network I/O, locks, page faults, involuntary context switches, etc.

This can be used in conjunction with offcputime, which shows the stack trace of the blocked thread. wakeuptime shows the stack trace of the waker thread.

See http://www.brendangregg.com/FlameGraphs/offcpuflamegraphs.html

This tool only works on Linux 4.6+. It uses the new `BPF_STACK_TRACE` table APIs to generate the in-kernel stack traces.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print usage message.

-f

Print output in folded stack format.

-u

Only trace user threads (not kernel threads).

-v

Show raw addresses (for non-folded format).

-p PID

Trace this process ID only (filtered in-kernel).

--stack-storage-size STACK_STORAGE_SIZE

Change the number of unique stack traces that can be stored and displayed.

duration

Duration to trace, in seconds.

-m MIN_BLOCK_TIME

The amount of time in microseconds over which we store traces (default 1)

-M MAX_BLOCK_TIME

The amount of time in microseconds under which we store traces (default U64_MAX)

## EXAMPLES
Trace all thread blocking events, and summarize (in-kernel) by kernel stack trace and total blocked time:

\# wakeuptime

Trace user-mode target threads only:

\# wakeuptime -u

Trace for 5 seconds only:

\# wakeuptime 5

Trace for 5 seconds, and emit output in folded stack format (suitable for flame graphs):

\# wakeuptime -f 5

Trace PID 185 only:

\# wakeuptime -p 185

## OVERHEAD
This summarizes unique stack traces in-kernel for efficiency, allowing it to trace a higher rate of events than methods that post-process in user space. The stack trace and time data is only copied to user space once, when the output is printed. While these techniques greatly lower overhead, scheduler events are still a high frequency event, as they can exceed 1 million events per second, and so caution should still be used. Test before production use.

If the overhead is still a problem, take a look at the min block option. If your aim is to chase down longer blocking events, then this could be increased to filter shorter blocking events, further lowering overhead.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
offcputime(8), stackcount(8)


bcc-wqlat

title: bcc-wqlat(8)

text:
# bcc-wqlat(8)

## NAME
wqlat - Summarize kernel workqueue latency as a histogram.

## SYNOPSIS
wqlat [-h] [-T] [-N] [-W] [-w WQNAME] [interval [count]]

## DESCRIPTION
wqlat traces work's waiting on workqueue, and records the distribution of work's queuing latency (time). This is printed as a histogram either on Ctrl-C, or after a given interval in seconds.

This tool uses in-kernel eBPF maps for storing timestamps and the histogram, for efficiency.

This tool uses the workqueue:workqueue_queue_work and workqueue:workqueue_execute_start kernel tracepoints, which is a stable tracing mechanism. Please note BPF programs can attach to tracepoints from Linux 4.7 only, so this tools can only support kernel 4.7 or later version.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h Print usage message.

-T

Include timestamps on output.

-N

Output histogram in nanoseconds.

-W

Print a histogram per workqueue.

-w WQNAME

Trace this workqueue only

interval

Output interval, in seconds.

count

Number of outputs.

## EXAMPLES
Summarize kernel workqueue latency as a histogram:

\# wqlat

Print 1 second summaries, 10 times:

\# wqlat 1 10

Print 1 second summaries, using nanoseconds as units for the histogram, and

include timestamps on output: # wqlat -NT 1

Print 1 second summaries, 10 times per workqueue:

\# wqlat -W 1 10

Print 1 second summaries for workqueue nvmet_tcp_wq:

\# wqlat -w nvmet_tcp_wq 1

## FIELDS
usecs

Microsecond range

nsecs

Nanosecond range

count

How many works into this range

distribution

An ASCII bar chart to visualize the distribution (count column)

## OVERHEAD
This traces kernel functions and maintains in-kernel timestamps and a histogram, which are asynchronously copied to user-space. This method is very efficient, and the overhead for most workqueue scheduling rates (< 100k) should be negligible.If you have a higher workqueue scheduling, please test and quantify the overhead before use.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Ping Gan

## SEE ALSO
biolatency


bcc-xfsdist

title: bcc-xfsdist(8)

text:
# bcc-xfsdist(8)

## NAME
xfsdist - Summarize XFS operation latency. Uses Linux eBPF/bcc.

## SYNOPSIS
xfsdist [-h] [-T] [-m] [-p PID] [interval] [count]

## DESCRIPTION
This tool summarizes time (latency) spent in common XFS file operations: reads, writes, opens, and syncs, and presents it as a power-of-2 histogram. It uses an in-kernel eBPF map to store the histogram for efficiency.

Since this works by tracing the xfs_file_operations interface functions, it will need updating to match any changes to these functions.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print usage message.

-T

Don't include timestamps on interval output.

-m

Output in milliseconds.

-p PID

Trace this PID only.

## EXAMPLES
Trace XFS operation time, and print a summary on Ctrl-C:

\# xfsdist

Trace PID 181 only:

\# xfsdist -p 181

Print 1 second summaries, 10 times:

\# xfsdist 1 10

1 second summaries, printed in milliseconds

\# xfsdist -m 1

## FIELDS
msecs

Range of milliseconds for this bucket.

usecs

Range of microseconds for this bucket.

count

Number of operations in this time range.

distribution

ASCII representation of the distribution (the count column).

## OVERHEAD
This adds low-overhead instrumentation to these XFS operations, including reads and writes from the file system cache. Such reads and writes can be very frequent (depending on the workload; eg, 1M/sec), at which point the overhead of this tool may become noticeable. Measure and quantify before use.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
xfssnoop(8)


bcc-xfsslower

title: bcc-xfsslower(8)

text:
# bcc-xfsslower(8)

## NAME
xfsslower - Trace slow xfs file operations, with per-event details.

## SYNOPSIS
xfsslower [-h] [-j] [-p PID] [min_ms]

## DESCRIPTION
This tool traces common XFS file operations: reads, writes, opens, and syncs. It measures the time spent in these operations, and prints details for each that exceeded a threshold.

WARNING: See the OVERHEAD section.

By default, a minimum millisecond threshold of 10 is used. If a threshold of 0 is used, all events are printed (warning: verbose).

Since this works by tracing the xfs_file_operations interface functions, it will need updating to match any changes to these functions.

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-p PID Trace this PID only.

min_ms

Minimum I/O latency (duration) to trace, in milliseconds. Default is 10 ms.

## EXAMPLES
Trace synchronous file reads and writes slower than 10 ms:

\# xfsslower

Trace slower than 1 ms:

\# xfsslower 1

Trace slower than 1 ms, and output just the fields in parsable format (csv):

\# xfsslower -j 1

Trace all file reads and writes (warning: the output will be verbose):

\# xfsslower 0

Trace slower than 1 ms, for PID 181 only:

\# xfsslower -p 181 1

## FIELDS
TIME(s)

Time of I/O completion since the first I/O seen, in seconds.

COMM

Process name.

PID

Process ID.

T

Type of operation. R == read, W == write, O == open, S == fsync.

OFF_KB

File offset for the I/O, in Kbytes.

BYTES

Size of I/O, in bytes.

LAT(ms)

Latency (duration) of I/O, measured from when it was issued by VFS to the filesystem, to when it completed. This time is inclusive of block device I/O, file system CPU cycles, file system locks, run queue latency, etc. It's a more accurate measure of the latency suffered by applications performing file system I/O, than to measure this down at the block device interface.

FILENAME

A cached kernel file name (comes from dentry->d_name.name).

ENDTIME_us

Completion timestamp, microseconds (-j only).

OFFSET_b

File offset, bytes (-j only).

LATENCY_us

Latency (duration) of the I/O, in microseconds (-j only).

## OVERHEAD
This adds low-overhead instrumentation to these XFS operations, including reads and writes from the file system cache. Such reads and writes can be very frequent (depending on the workload; eg, 1M/sec), at which point the overhead of this tool (even if it prints no "slower" events) can begin to become significant. Measure and quantify before use. If this continues to be a problem, consider switching to a tool that prints in-kernel summaries only.

Note that the overhead of this tool should be less than fileslower(8), as this tool targets xfs functions only, and not all file read/write paths (which can include socket I/O).

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
biosnoop(8), funccount(8), fileslower(8)


bcc-zfsdist

title: bcc-zfsdist(8)

text:
# bcc-zfsdist(8)

## NAME
zfsdist - Summarize ZFS operation latency. Uses Linux eBPF/bcc.

## SYNOPSIS
zfsdist [-h] [-T] [-m] [-p PID] [interval] [count]

## DESCRIPTION
This tool summarizes time (latency) spent in common ZFS file operations: reads, writes, opens, and syncs, and presents it as a power-of-2 histogram. It uses an in-kernel eBPF map to store the histogram for efficiency.

This uses kernel dynamic tracing of the ZPL interface (ZFS POSIX Layer), and will need updates to match any changes to this interface.

This is intended to work with the ZFS on Linux project:

http://zfsonlinux.org

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-h

Print usage message.

-T

Don't include timestamps on interval output.

-m

Output in milliseconds.

-p PID

Trace this PID only.

## EXAMPLES
Trace ZFS operation time, and print a summary on Ctrl-C:

\# zfsdist

Trace PID 181 only:

\# zfsdist -p 181

Print 1 second summaries, 10 times:

\# zfsdist 1 10

1 second summaries, printed in milliseconds

\# zfsdist -m 1

## FIELDS
msecs

Range of milliseconds for this bucket.

usecs

Range of microseconds for this bucket.

count

Number of operations in this time range.

distribution

ASCII representation of the distribution (the count column).

## OVERHEAD
This adds low-overhead instrumentation to these ZFS operations, including reads and writes from the file system cache. Such reads and writes can be very frequent (depending on the workload; eg, 1M/sec), at which point the overhead of this tool may become noticeable. Measure and quantify before use.

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
zfssnoop(8)


bcc-zfsslower

title: bcc-zfsslower(8)

text:
# bcc-zfsslower(8)

## NAME
zfsslower - Trace slow zfs file operations, with per-event details.

## SYNOPSIS
zfsslower [-h] [-j] [-p PID] [min_ms]

## DESCRIPTION
This tool traces common ZFS file operations: reads, writes, opens, and syncs. It measures the time spent in these operations, and prints details for each that exceeded a threshold.

WARNING: See the OVERHEAD section.

By default, a minimum millisecond threshold of 10 is used. If a threshold of 0 is used, all events are printed (warning: verbose).

This uses kernel dynamic tracing of the ZPL interface (ZFS POSIX Layer), and will need updates to match any changes to this interface.

This is intended to work with the ZFS on Linux project:

http://zfsonlinux.org

Since this uses BPF, only the root user can use this tool.

## REQUIREMENTS
CONFIG_BPF and bcc.

## OPTIONS
-p PID Trace this PID only.

min_ms

Minimum I/O latency (duration) to trace, in milliseconds. Default is 10 ms.

## EXAMPLES
Trace synchronous file reads and writes slower than 10 ms:

\# zfsslower

Trace slower than 1 ms:

\# zfsslower 1

Trace slower than 1 ms, and output just the fields in parsable format (csv):

\# zfsslower -j 1

Trace all file reads and writes (warning: the output will be verbose):

\# zfsslower 0

Trace slower than 1 ms, for PID 181 only:

\# zfsslower -p 181 1

## FIELDS
TIME(s)

Time of I/O completion since the first I/O seen, in seconds.

COMM

Process name.

PID

Process ID.

T

Type of operation. R == read, W == write, O == open, S == fsync.

OFF_KB

File offset for the I/O, in Kbytes.

BYTES

Size of I/O, in bytes.

LAT(ms)

Latency (duration) of I/O, measured from when it was issued by VFS to the filesystem, to when it completed. This time is inclusive of block device I/O, file system CPU cycles, file system locks, run queue latency, etc. It's a more accurate measure of the latency suffered by applications performing file system I/O, than to measure this down at the block device interface.

FILENAME

A cached kernel file name (comes from dentry->d_name.name).

ENDTIME_us

Completion timestamp, microseconds (-j only).

OFFSET_b

File offset, bytes (-j only).

LATENCY_us

Latency (duration) of the I/O, in microseconds (-j only).

## OVERHEAD
This adds low-overhead instrumentation to these ZFS operations, including reads and writes from the file system cache. Such reads and writes can be very frequent (depending on the workload; eg, 1M/sec), at which point the overhead of this tool (even if it prints no "slower" events) can begin to become significant. Measure and quantify before use. If this continues to be a problem, consider switching to a tool that prints in-kernel summaries only.

Note that the overhead of this tool should be less than fileslower(8), as this tool targets zfs functions only, and not all file read/write paths (which can include socket I/O).

## SOURCE
This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

## OS
Linux

## STABILITY
Unstable - in development.

## AUTHOR
Brendan Gregg

## SEE ALSO
biosnoop(8), funccount(8), fileslower(8)


title: bgpq3(8)

text:
# bgpq3(8)

## NAME
bgpq3 — bgp filtering automation for cisco and juniper routers

## SYNOPSIS
bgpq3

[-h host] [-S sources] [-EP] [-f asn | -G asn] [-346AbDdJjX] [-r len] [-R len] [-m max] [-W len] OBJECTS [...]

## DESCRIPTION
The bgpq3 utility used to generate Cisco and Juniper prefix-lists, extended access-lists, policy-statement terms and as-path lists based on RADB data.

The options are as follows:

-3

assume that your device is asn32-safe.

-4

generate IPv4 prefix/access-lists (default).

-6

generate IPv6 prefix/access-lists (IPv4 by default).

-A

try to aggregate prefix-lists as much as possible (not all output formats supported).

-b

generate output in BIRD format (default: Cisco).

-d

enable some debugging output.

-D

use asdot notation for Cisco as-path access-lists.

-E

generate extended access-list (Cisco) or policy-statement term using route-filters (Juniper).

-f number

generate input as-path access-list.

-G number

generate output as-path access-list.

-h host

host running IRRD database (default: whois.radb.net).

-J

generate config for Juniper (default: Cisco).

-j

generate output in JSON format (default: Cisco).

-m len

maximum prefix-length of accepted prefixes (default: 32 for IPv4 and 128 for IPv6).

-M match

extra match conditions for Juniper route-filters.

-l name

name of generated entry.

-P

generate prefix-list (default, backward compatibility).

-r len

allow more specific routes starting with specified masklen too.

-R len

allow more specific routes up to specified masklen too.

-S sources

use specified sources only (default: RADB,RIPE,APNIC).

-T

disable pipelining.

-W len

generate as-path strings of no more than len items (use 0 for inifinity).

-X

generate config for Cisco IOS XR devices (plain IOS by default).

OBJECTS

means networks (in prefix format), autonomous systems, as-sets and route-sets.

## EXAMPLES
Generating named juniper prefix-filter for AS20597:

```
~>bgpq3 -Jl eltel AS20597
policy-options {
replace:
 prefix-list eltel {
    81.9.0.0/20;
    81.9.32.0/20;
    81.9.96.0/20;
    81.222.128.0/20;
    81.222.192.0/18;
    85.249.8.0/21;
    85.249.224.0/19;
    89.112.0.0/19;
    89.112.4.0/22;
    89.112.32.0/19;
    89.112.64.0/19;
    217.170.64.0/20;
    217.170.80.0/20;
 }
}
```

For Cisco we can use aggregation (-A) flag to make this prefix-filter more compact:

```
~>bgpq3 -Al eltel AS20597
no ip prefix-list eltel
ip prefix-list eltel permit 81.9.0.0/20
ip prefix-list eltel permit 81.9.32.0/20
ip prefix-list eltel permit 81.9.96.0/20
ip prefix-list eltel permit 81.222.128.0/20
ip prefix-list eltel permit 81.222.192.0/18
ip prefix-list eltel permit 85.249.8.0/21
ip prefix-list eltel permit 85.249.224.0/19
ip prefix-list eltel permit 89.112.0.0/18 ge 19 le 19
ip prefix-list eltel permit 89.112.4.0/22
ip prefix-list eltel permit 89.112.64.0/19
ip prefix-list eltel permit 217.170.64.0/19 ge 20 le 20
```

- you see, prefixes 89.112.0.0/19 and 89.112.32.0/19 now aggregated into single entry 89.112.0.0/18 ge 19 le 19.

Well, for Juniper we can generate even more interesting policy-options, using -M <extra match conditions>, -R <len> and hierarchical names:

```
policy-options {
 policy-statement eltel {
  term specifics {
replace:
   from {
    community blackhole;
    route-filter 81.9.0.0/20 prefix-length-range /29-/32;
    route-filter 81.9.32.0/20 prefix-length-range /29-/32;
    route-filter 81.9.96.0/20 prefix-length-range /29-/32;
    route-filter 81.222.128.0/20 prefix-length-range /29-/32;
    route-filter 81.222.192.0/18 prefix-length-range /29-/32;
    route-filter 85.249.8.0/21 prefix-length-range /29-/32;
    route-filter 85.249.224.0/19 prefix-length-range /29-/32;
    route-filter 89.112.0.0/17 prefix-length-range /29-/32;
    route-filter 217.170.64.0/19 prefix-length-range /29-/32;
   }
  }
 }
}
```

generated policy-option term now allows all specifics with prefix-length between /29 and /32 for eltel networks if they match with special community Of course, this version supports IPv6 (-6):

```
~>bgpq3 -6l as-retn-6 AS-RETN6
no ipv6 prefix-list as-retn-6
ipv6 prefix-list as-retn-6 permit 2001:7fb:fe00::/48
ipv6 prefix-list as-retn-6 permit 2001:7fb:fe01::/48
[....]
```

and support for ASN 32 is also here

```
~>bgpq3 -J3f 112 AS-SPACENET
policy-options {
replace:
 as-path-group NN {
  as-path a0 "^112(112)*$";
  as-path a1 "^112(.)*(1898|5539|8495|8763|8878|12136|12931|15909)$";
  as-path a2 "^112(.)*(21358|23456|23600|24151|25152|31529|34127|34906)$";
  as-path a3 "^112(.)*(35052|41720|43628|44450|196611)$";
 }
}
```

see AS196611 in the end of the list ? That's AS3.3 in 'asplain' notation.

For non-ASN32 capable routers you should not use switch -3, and the result will be next:

```
~>bgpq3 -f 112 AS-SPACENET
no ip as-path access-list NN
ip as-path access-list NN permit ^112(_112)*$
ip as-path access-list NN permit ^112(_[0-9]+)*_(1898|5539|8495|8763)$
ip as-path access-list NN permit ^112(_[0-9]+)*_(8878|12136|12931|15909)$
ip as-path access-list NN permit ^112(_[0-9]+)*_(21358|23456|23600|24151)$
ip as-path access-list NN permit ^112(_[0-9]+)*_(25152|31529|34127|34906)$
ip as-path access-list NN permit ^112(_[0-9]+)*_(35052|41720|43628|44450)$
```

AS196611 is no more in the list, however, AS23456 (transition AS) would be added to list if it were not present.

## DIAGNOSTICS
When everything is OK, bgpq3 generates access-list to standard output and exits with status == 0. In case of errors they are printed to stderr and program exits with non-zero status.

## SEE ALSO
http://www.radb.net/ Routing Arbiter project http://tools.ietf.org/html/draft-michaelson-4byte-as-representation-05 for information on 'asdot' and 'asplain' notations. http://www.cisco.com/en/US/docs/ios/12_0s/release/ntes/120SNEWF.html#wp3521658 for information on Cisco implementation of ASN32.

## AUTHOR
Alexandre Snarskii ⟨snar@snar.spb.ru⟩


ddns-confgen

package: bind

title: ddns-confgen(8)

summary: ddns key generation tool

text:
# ddns-confgen(8)

## NAME
ddns-confgen - ddns key generation tool

## SYNOPSIS
ddns-confgen [-a algorithm] [-h] [-k keyname] [-q] [-s name] [-z zone]

## DESCRIPTION
ddns-confgen is an utility that generates keys for use in TSIG signing. The resulting keys can be used, for example, to secure dynamic DNS updates to a zone, or for the rndc <#std-iscman-rndc> command channel.

The key name can specified using -k parameter and defaults to ddns-key. The generated key is accompanied by configuration text and instructions that can be used with nsupdate <#std-iscman-nsupdate> and named <#std-iscman-named> when setting up dynamic DNS, including an example update-policy statement. (This usage is similar to the rndc-confgen <#std-iscman-rndc-confgen> command for setting up command-channel security.)

Note that named <#std-iscman-named> itself can configure a local DDNS key for use with nsupdate -l <#cmdoption-nsupdate-l>; it does this when a zone is configured with update-policy local;. ddns-confgen is only needed when a more elaborate configuration is required: for instance, if nsupdate <#std-iscman-nsupdate> is to be used from a remote system.

## OPTIONS
-a algorithm

This option specifies the algorithm to use for the TSIG key. Available choices are: hmac-md5, hmac-sha1, hmac-sha224, hmac-sha256, hmac-sha384, and hmac-sha512. The default is hmac-sha256. Options are case-insensitive, and the "hmac-" prefix may be omitted.

-h

This option prints a short summary of options and arguments.

-k keyname

This option specifies the key name of the DDNS authentication key. The default is ddns-key when neither the -s nor -z option is specified; otherwise, the default is ddns-key as a separate label followed by the argument of the option, e.g., ddns-key.example.com. The key name must have the format of a valid domain name, consisting of letters, digits, hyphens, and periods.

-q

This option enables quiet mode, which prints only the key, with no explanatory text or usage examples. This is essentially identical to tsig-keygen <#std-iscman-tsig-keygen>.

-s name

This option generates a configuration example to allow dynamic updates of a single hostname. The example named.conf <#std-iscman-named.conf> text shows how to set an update policy for the specified name using the "name" nametype. The default key name is ddns-key.name. Note that the "self" nametype cannot be used, since the name to be updated may differ from the key name. This option cannot be used with the -z option.

-z zone

This option generates a configuration example to allow dynamic updates of a zone. The example named.conf <#std-iscman-named.conf> text shows how to set an update policy for the specified zone using the "zonesub" nametype, allowing updates to all subdomain names within that zone. This option cannot be used with the -s option.

## SEE ALSO
nsupdate(1) <#std-iscman-nsupdate>, named.conf(5) <#std-iscman-named.conf>, named(8) <#std-iscman-named>, BIND 9 Administrator Reference Manual.

## Author
Internet Systems Consortium

## Copyright
2026, Internet Systems Consortium


filter-a

title: filter-a(8)

text:
# filter-a(8)

## NAME
filter-a - filter A in DNS responses when AAAA is present

## SYNOPSIS
plugin query "filter-a.so" [{ parameters }];

## DESCRIPTION
filter-a.so is a query plugin module for named <#std-iscman-named>, enabling named <#std-iscman-named> to omit some IPv4 addresses when responding to clients.

For example:

```
plugin query "filter-a.so" {

        filter-a-on-v6 yes;

        filter-a-on-v4 yes;

        filter-a { 192.0.2.1; 2001:db8:2::1; };
};
```

This module is intended to aid transition from IPv4 to IPv6 by withholding IPv4 addresses from DNS clients which are not connected to the IPv4 Internet, when the name being looked up has an IPv6 address available. Use of this module is not recommended unless absolutely necessary.

Note: This mechanism can erroneously cause other servers not to give A records to their clients. If a recursing server with both IPv6 and IPv4 network connections queries an authoritative server using this mechanism via IPv6, it is denied A records even if its client is using IPv4.

## OPTIONS
filter-a

This option specifies a list of client addresses for which A filtering is to be applied. The default is any.

filter-a-on-v6

If set to yes, this option indicates that the DNS client is at an IPv6 address, in filter-a. If the response does not include DNSSEC signatures, then all A records are deleted from the response. This filtering applies to all responses, not only authoritative ones. If set to break-dnssec, then A records are deleted even when DNSSEC is enabled. As suggested by the name, this causes the response to fail to verify, because the DNSSEC protocol is designed to detect deletions.

This mechanism can erroneously cause other servers not to give A records to their clients. If a recursing server with both IPv6 and IPv4 network connections queries an authoritative server using this mechanism via IPv6, it is denied A records even if its client is using IPv4.

filter-a-on-v4

This option is identical to filter-a-on-v6, except that it filters A responses to queries from IPv4 clients instead of IPv6 clients. To filter all responses, set both options to yes.

## SEE ALSO
BIND 9 Administrator Reference Manual.

## Author
Internet Systems Consortium

## Copyright
2026, Internet Systems Consortium


filter-aaaa

title: filter-aaaa(8)

text:
# filter-aaaa(8)

## NAME
filter-aaaa - filter AAAA in DNS responses when A is present

## SYNOPSIS
plugin query "filter-aaaa.so" [{ parameters }];

## DESCRIPTION
filter-aaaa.so is a query plugin module for named <#std-iscman-named>, enabling named <#std-iscman-named> to omit some IPv6 addresses when responding to clients.

Until BIND 9.12, this feature was implemented natively in named <#std-iscman-named> and enabled with the filter-aaaa ACL and the filter-aaaa-on-v4 and filter-aaaa-on-v6 options. These options are now deprecated in named.conf <#std-iscman-named.conf> but can be passed as parameters to the filter-aaaa.so plugin, for example:

```
plugin query "filter-aaaa.so" {

        filter-aaaa-on-v4 yes;

        filter-aaaa-on-v6 yes;

        filter-aaaa { 192.0.2.1; 2001:db8:2::1; };
};
```

This module is intended to aid transition from IPv4 to IPv6 by withholding IPv6 addresses from DNS clients which are not connected to the IPv6 Internet, when the name being looked up has an IPv4 address available. Use of this module is not recommended unless absolutely necessary.

Note: This mechanism can erroneously cause other servers not to give AAAA records to their clients. If a recursing server with both IPv6 and IPv4 network connections queries an authoritative server using this mechanism via IPv4, it is denied AAAA records even if its client is using IPv6.

## OPTIONS
filter-aaaa

This option specifies a list of client addresses for which AAAA filtering is to be applied. The default is any.

filter-aaaa-on-v4

If set to yes, this option indicates that the DNS client is at an IPv4 address, in filter-aaaa. If the response does not include DNSSEC signatures, then all AAAA records are deleted from the response. This filtering applies to all responses, not only authoritative ones. If set to break-dnssec, then AAAA records are deleted even when DNSSEC is enabled. As suggested by the name, this causes the response to fail to verify, because the DNSSEC protocol is designed to detect deletions.

This mechanism can erroneously cause other servers not to give AAAA records to their clients. If a recursing server with both IPv6 and IPv4 network connections queries an authoritative server using this mechanism via IPv4, it is denied AAAA records even if its client is using IPv6.

filter-aaaa-on-v6

This option is identical to filter-aaaa-on-v4, except that it filters AAAA responses to queries from IPv6 clients instead of IPv4 clients. To filter all responses, set both options to yes.

## SEE ALSO
BIND 9 Administrator Reference Manual.

## Author
Internet Systems Consortium

## Copyright
2026, Internet Systems Consortium