rtmon

title: rtmon(8)

summary: listens to and monitors RTnetlink

text:
# rtmon(8)

## NAME
rtmon - listens to and monitors RTnetlink

## SYNOPSIS

rtmon [ OPTIONS ] file FILE [ all | OBJECTS ]

OPTIONS := { f[amily] { inet | inet6 | link | help } | -4 | -6 | -0 | -V[ersion] }

OBJECTS := [ link ] [ address ] [ route ]

## DESCRIPTION
This manual page documents briefly the rtmon command.

rtmon listens on netlink socket and monitors routing table changes.

rtmon can be started before the first network configuration command is issued. For example if you insert:

rtmon file /var/log/rtmon.log

in a startup script, you will be able to view the full history later. Certainly, it is possible to start rtmon at any time. It prepends the history with the state snapshot dumped at the moment of starting.

## OPTIONS
rtmon supports the following options:

-Version

Print version and exit.

help

Show summary of options.

file FILE [ all | OBJECTS ]

Log output to FILE. OBJECTS is the list of object types that we want to monitor. It may contain 'link', 'address', 'route' and 'all'. 'link' specifies the network device, 'address' the protocol (IP or IPv6) address on a device, 'route' the routing table entry and 'all' does what the name says.

-family [ inet | inet6 | link | help ]

Specify protocol family. 'inet' is IPv4, 'inet6' is IPv6, 'link' means that no networking protocol is involved and 'help' prints usage information.

-4

Use IPv4. Shortcut for -family inet.

-6

Use IPv6. Shortcut for -family inet6.

-0

Use a special family identifier meaning that no networking protocol is involved. Shortcut for -family link.

## USAGE EXAMPLES
\# rtmon file /var/log/rtmon.log

Log to file /var/log/rtmon.log, then run:

\# ip monitor file /var/log/rtmon.log

to display logged output from file.

## SEE ALSO
ip(8) ip-monitor(8)

## AUTHOR
rtmon was written by Alexey Kuznetsov <kuznet@ms2.inr.ac.ru>.

This manual page was written by Michael Prokop <mika@grml.org>, for the Debian project (but may be used by others).


ss

title: ss(8)

summary: another utility to investigate sockets

text:
# ss(8)

## NAME
ss - another utility to investigate sockets

## SYNOPSIS
ss [options] [ FILTER ]

## DESCRIPTION
ss is used to dump socket statistics. It allows showing information similar to netstat. It can display more TCP and state information than other tools.

## OPTIONS
When no option is used ss displays a list of open non-listening sockets (e.g. TCP/UNIX/UDP) that have established connection.

-h, --help

Show summary of options.

-V, --version

Output version information.

-H, --no-header

Suppress header line.

-Q, --no-queues

Suppress sending and receiving queue columns.

-O, --oneline

Print each socket's data on a single line.

-n, --numeric

Do not try to resolve service names. Show exact bandwidth values, instead of human-readable.

-r, --resolve

Try to resolve numeric address/ports.

-a, --all

Display both listening and non-listening (for TCP this means established connections) sockets.

-l, --listening

Display only listening sockets (these are omitted by default).

-B, --bound-inactive

Display only TCP bound but inactive (not listening, connecting, etc.) sockets (these are omitted by default).

-o, --options

Show timer information. For TCP protocol, the output format is:

timer:(<timer_name>,<expire_time>,<retrans>)

<timer_name>

the name of the timer, there are five kind of timer names:

on : means one of these timers: TCP retrans timer, TCP early retrans timer and tail loss probe timer

keepalive: tcp keep alive timer

timewait: timewait stage timer

persist: zero window probe timer

unknown: none of the above timers

<expire_time>

how long time the timer will expire

<retrans>

how many times the retransmission occurred

-e, --extended

Show detailed socket information. The output format is:

uid:<uid_number> ino:<inode_number> sk:<cookie>

<uid_number>

the user id the socket belongs to

<inode_number>

the socket's inode number in VFS

<cookie>

an uuid of the socket

-m, --memory

Show socket memory usage. The output format is:

skmem:(r<rmem_alloc>,rb<rcv_buf>,t<wmem_alloc>,tb<snd_buf>,

f<fwd_alloc>,w<wmem_queued>,o<opt_mem>,

bl<back_log>,d<sock_drop>)

<rmem_alloc>

the memory allocated for receiving packet

<rcv_buf>

the total memory can be allocated for receiving packet

<wmem_alloc>

the memory used for sending packet (which has been sent to layer 3)

<snd_buf>

the total memory can be allocated for sending packet

<fwd_alloc>

the memory allocated by the socket as cache, but not used for receiving/sending packet yet. If need memory to send/receive packet, the memory in this cache will be used before allocate additional memory.

<wmem_queued>

The memory allocated for sending packet (which has not been sent to layer 3)

<opt_mem>

The memory used for storing socket option, e.g., the key for TCP MD5 signature

<back_log>

The memory used for the sk backlog queue. On a process context, if the process is receiving packet, and a new packet is received, it will be put into the sk backlog queue, so it can be received by the process immediately

<sock_drop>

the number of packets dropped before they are de-multiplexed into the socket

-p, --processes

Show process using socket.

-T, --threads

Show thread using socket. Implies -p.

-i, --info

Show internal TCP information. Below fields may appear:

ts

show string "ts" if the timestamp option is set

sack

show string "sack" if the sack option is set

ecn

show string "ecn" if the explicit congestion notification option is set

ecnseen

show string "ecnseen" if the saw ecn flag is found in received packets

fastopen

show string "fastopen" if the fastopen option is set

cong_alg

the congestion algorithm name, the default congestion algorithm is "cubic"

wscale:<snd_wscale>:<rcv_wscale>

if window scale option is used, this field shows the send scale factor and receive scale factor

rto:<icsk_rto>

tcp re-transmission timeout value, the unit is millisecond

backoff:<icsk_backoff>

used for exponential backoff re-transmission, the actual re-transmission timeout value is icsk_rto << icsk_backoff

rtt:<rtt>/<rttvar>

rtt is the average round trip time, rttvar is the mean deviation of rtt, their units are millisecond

ato:<ato>

ack timeout, unit is millisecond, used for delay ack mode

mss:<mss>

max segment size

cwnd:<cwnd>

congestion window size

pmtu:<pmtu>

path MTU value

ssthresh:<ssthresh>

tcp congestion window slow start threshold

bytes_acked:<bytes_acked>

bytes acked

bytes_received:<bytes_received>

bytes received

segs_out:<segs_out>

segments sent out

segs_in:<segs_in>

segments received

send <send_bps>bps

egress bps

lastsnd:<lastsnd>

how long time since the last packet sent, the unit is millisecond

lastrcv:<lastrcv>

how long time since the last packet received, the unit is millisecond

lastack:<lastack>

how long time since the last ack received, the unit is millisecond

pacing_rate <pacing_rate>bps/<max_pacing_rate>bps

the pacing rate and max pacing rate

rcv_space:<rcv_space>

a helper variable for TCP internal auto tuning socket receive buffer

tcp-ulp-mptcp flags:[MmBbJjecv] token:<rem_token(rem_id)/loc_token(loc_id)> seq:<sn> sfseq:<ssn> ssnoff:<off> maplen:<maplen>

MPTCP subflow information

--tos

Show ToS and priority information. Below fields may appear:

tos

IPv4 Type-of-Service byte

tclass

IPv6 Traffic Class byte

class_id

Class id set by net_cls cgroup. If class is zero this shows priority set by SO_PRIORITY.

--cgroup

Show cgroup information. Below fields may appear:

cgroup

Cgroup v2 pathname. This pathname is relative to the mount point of the hierarchy.

--tipcinfo

Show internal tipc socket information.

-K, --kill

Attempts to forcibly close sockets. This option displays sockets that are successfully closed and silently skips sockets that the kernel does not support closing. It supports IPv4 and IPv6 sockets only.

-s, --summary

Print summary statistics. This option does not parse socket lists obtaining summary from various sources. It is useful when amount of sockets is so huge that parsing /proc/net/tcp is painful.

-E, --events

Continually display sockets as they are destroyed

-Z, --context

As the -p option but also shows process security context. If the -T option is used, also shows thread security context. For netlink(7) sockets the initiating process context is displayed as follows:

1.

If valid pid show the process context.

2.

If destination is kernel (pid = 0) show kernel initial context.

3.

If a unique identifier has been allocated by the kernel or netlink user, show context as "unavailable". This will generally indicate that a process has more than one netlink socket active.

-z, --contexts

As the -Z option but also shows the socket context. The socket context is taken from the associated inode and is not the actual socket context held by the kernel. Sockets are typically labeled with the context of the creating process, however the context shown will reflect any policy role, type and/or range transition rules applied, and is therefore a useful reference.

-N NSNAME, --net=NSNAME

Switch to the specified network namespace name.

-b, --bpf

Show socket classic BPF filters (only administrators are allowed to get these information).

-4, --ipv4

Display only IP version 4 sockets (alias for -f inet).

-6, --ipv6

Display only IP version 6 sockets (alias for -f inet6).

-0, --packet

Display PACKET sockets (alias for -f link).

-t, --tcp

Display TCP sockets.

-u, --udp

Display UDP sockets.

-d, --dccp

Display DCCP sockets.

-w, --raw

Display RAW sockets.

-x, --unix

Display Unix domain sockets (alias for -f unix).

-S, --sctp

Display SCTP sockets.

--tipc

Display tipc sockets (alias for -f tipc).

--vsock

Display vsock sockets (alias for -f vsock).

--xdp

Display XDP sockets (alias for -f xdp).

-M, --mptcp

Display MPTCP sockets.

--inet-sockopt

Display inet socket options.

-f FAMILY, --family=FAMILY

Display sockets of type FAMILY. Currently the following families are supported: unix, inet, inet6, link, netlink, vsock, tipc, xdp.

-A QUERY, --query=QUERY, --socket=QUERY

List of socket tables to dump, separated by commas. The following identifiers are understood: all, inet, tcp, udp, raw, unix, packet, netlink, unix_dgram, unix_stream, unix_seqpacket, packet_raw, packet_dgram, dccp, sctp, tipc, vsock_stream, vsock_dgram, xdp, mptcp. Any item in the list may optionally be prefixed by an exclamation mark (!) to exclude that socket table from being dumped.

-D FILE, --diag=FILE

Do not display anything, just dump raw information about TCP sockets to FILE after applying filters. If FILE is - stdout is used.

-F FILE, --filter=FILE

Read filter information from FILE. Each line of FILE is interpreted like single command line option. If FILE is - stdin is used.

--bpf-maps

Pretty-print all the BPF socket-local data entries for each socket.

--bpf-map-id=MAP_ID

Pretty-print the BPF socket-local data entries for the requested map ID. Can be used more than once.

FILTER := [ state STATE-FILTER ] [ EXPRESSION ]

Please take a look at the official documentation for details regarding filters.

## STATE-FILTER
STATE-FILTER allows one to construct arbitrary set of states to match. Its syntax is sequence of keywords state and exclude followed by identifier of state.

Available identifiers are:

All standard TCP states: established, syn-sent, syn-recv, fin-wait-1, fin-wait-2, time-wait, closed, close-wait, last-ack, listening and closing.

all - for all the states

connected - all the states except for listening and closed

synchronized - all the connected states except for syn-sent

bucket - states, which are maintained as minisockets, i.e. time-wait and syn-recv

big - opposite to bucket

bound-inactive - bound but otherwise inactive sockets (not listening, connecting, etc.)

## EXPRESSION
EXPRESSION allows filtering based on specific criteria. EXPRESSION consists of a series of predicates combined by boolean operators. The possible operators in increasing order of precedence are or (or | or ||), and (or & or &&), and not (or !). If no operator is between consecutive predicates, an implicit and operator is assumed. Subexpressions can be grouped with "(" and ")".

The following predicates are supported:

{dst|src} [=] HOST

Test if the destination or source matches HOST. See HOST SYNTAX for details.

{dport|sport} [OP] [FAMILY:]:PORT

Compare the destination or source port to PORT. OP can be any of "<", "<=", "=", "!=", ">=" and ">". Following normal arithmetic rules. FAMILY and PORT are as described in HOST SYNTAX below.

dev [=|!=] DEVICE

Match based on the device the connection uses. DEVICE can either be a device name or the index of the interface.

fwmark [=|!=] MASK

Matches based on the fwmark value for the connection. This can either be a specific mark value or a mark value followed by a "/" and a bitmask of which bits to use in the comparison. For example "fwmark = 0x01/0x03" would match if the two least significant bits of the fwmark were 0x01.

cgroup [=|!=] PATH

Match if the connection is part of a cgroup at the given path.

autobound

Match if the port or path of the source address was automatically allocated (rather than explicitly specified).

Most operators have aliases. If no operator is supplied "=" is assumed. Each of the following groups of operators are all equivalent:

- = == eq

- != ne neq

- > gt

- < lt

- >= ge geq

- <= le leq

- ! not

- | || or

- & && and

## HOST SYNTAX
The general host syntax is [FAMILY:]ADDRESS[:PORT].

FAMILY must be one of the families supported by the -f option. If not given it defaults to the family given with the -f option, and if that is also missing, will assume either inet or inet6. Note that all host conditions in the expression should either all be the same family or be only inet and inet6. If there is some other mixture of families, the results will probably be unexpected.

The form of ADDRESS and PORT depends on the family used. "*" can be used as a wildcard for either the address or port. The details for each family are as follows:

unix

ADDRESS is a glob pattern (see fnmatch(3)) that will be matched case-insensitively against the unix socket's address. Both path and abstract names are supported. Unix addresses do not support a port, and "*" cannot be used as a wildcard.

link

ADDRESS is the case-insensitive name of an Ethernet protocol to match. PORT is either a device name or a device index for the desired link device, as seen in the output of ip link.

netlink

ADDRESS is a descriptor of the netlink family. Possible values come from /etc/iproute2/nl_protos. PORT is the port id of the socket, which is usually the same as the owning process id. The value "kernel" can be used to represent the kernel (port id of 0).

vsock

ADDRESS is an integer representing the CID address, and PORT is the port.

inet and inet6

ADDRESS is an ip address (either v4 or v6 depending on the family) or a DNS hostname that resolves to an ip address of the required version. An ipv6 address must be enclosed in "[" and "]" to disambiguate the port separator. The address may additionally have a prefix length given in CIDR notation (a slash followed by the prefix length in bits). PORT is either the numerical socket port, or the service name for the port to match.

## USAGE EXAMPLES
ss -t -a

Display all TCP sockets.

ss -t -a -Z

Display all TCP sockets with process SELinux security contexts.

ss -u -a

Display all UDP sockets.

ss -o state established '( dport = :ssh or sport = :ssh )'

Display all established ssh connections.

ss -x src /tmp/.X11-unix/*

Find all local processes connected to X server.

ss -o state fin-wait-1 '( sport = :http or sport = :https )' dst 193.233.7/24

List all the tcp sockets in state FIN-WAIT-1 for our apache to network 193.233.7/24 and look at their timers.

ss -a -A 'all,!tcp'

List sockets in all states from all socket tables but TCP.

## SEE ALSO
ip(8),

RFC 793 - https://tools.ietf.org/rfc/rfc793.txt (TCP states)

## AUTHOR
ss was written by Alexey Kuznetsov, <kuznet@ms2.inr.ac.ru>.

This manual page was written by Michael Prokop <mika@grml.org> for the Debian project (but may be used by others).

branch: sisyphus


tc

title: tc(8)

text:
# tc(8)

## NAME
tc - show / manipulate traffic control settings

## SYNOPSIS
tc [ OPTIONS ] qdisc [ add | change | replace | link | delete ] dev DEV [ parent qdisc-id | root ] [ handle qdisc-id ] [ ingress_block BLOCK_INDEX ] [ egress_block BLOCK_INDEX ] qdisc [ qdisc specific parameters ]

tc [ OPTIONS ] class [ add | change | replace | delete | show ] dev DEV parent qdisc-id [ classid class-id ] qdisc [ qdisc specific parameters ]

tc [ OPTIONS ] filter [ add | change | replace | delete | get ] dev DEV [ parent qdisc-id | root ] [ handle filter-id ] protocol protocol prio priority filtertype [ filtertype specific parameters ] flowid flow-id

tc [ OPTIONS ] filter [ add | change | replace | delete | get ] block BLOCK_INDEX [ handle filter-id ] protocol protocol prio priority filtertype [ filtertype specific parameters ] flowid flow-id

tc [ OPTIONS ] chain [ add | delete | get ] dev DEV [ parent qdisc-id | root ] filtertype [ filtertype specific parameters ]

tc [ OPTIONS ] chain [ add | delete | get ] block BLOCK_INDEX filtertype [ filtertype specific parameters ]

tc [ OPTIONS ] [ FORMAT ] qdisc { show | list } [ dev DEV ] [ root | ingress | handle QHANDLE | parent CLASSID ] [ invisible ]

tc [ OPTIONS ] [ FORMAT ] class show dev DEV

tc [ OPTIONS ] filter show dev DEV

tc [ OPTIONS ] filter show block BLOCK_INDEX

tc [ OPTIONS ] chain show dev DEV

tc [ OPTIONS ] chain show block BLOCK_INDEX

tc [ OPTIONS ] monitor [ file FILENAME ]

OPTIONS := { [ -force ] -b[atch] [ filename ] | [ -n[etns] name ] | [ -N[umeric] ] | [ -nm | -nam[es] ] | [ { -cf | -c[onf] } [ filename ] ] [ -t[imestamp] ] | [ -t[short] | [ -o[neline] ] | [ -echo ] }

FORMAT := { -s[tatistics] | -d[etails] | -r[aw] | -i[ec] | -g[raph] | -j[son] | -p[retty] | -col[or] }

## DESCRIPTION
Tc is used to configure Traffic Control in the Linux kernel. Traffic Control consists of the following:

SHAPING

When traffic is shaped, its rate of transmission is under control. Shaping may be more than lowering the available bandwidth - it is also used to smooth out bursts in traffic for better network behaviour. Shaping occurs on egress.

SCHEDULING

By scheduling the transmission of packets it is possible to improve interactivity for traffic that needs it while still guaranteeing bandwidth to bulk transfers. Reordering is also called prioritizing, and happens only on egress.

POLICING

Whereas shaping deals with transmission of traffic, policing pertains to traffic arriving. Policing thus occurs on ingress.

DROPPING

Traffic exceeding a set bandwidth may also be dropped forthwith, both on ingress and on egress.

Processing of traffic is controlled by three kinds of objects: qdiscs, classes and filters.

## QDISCS
qdisc is short for 'queueing discipline' and it is elementary to understanding traffic control. Whenever the kernel needs to send a packet to an interface, it is enqueued to the qdisc configured for that interface. Immediately afterwards, the kernel tries to get as many packets as possible from the qdisc, for giving them to the network adaptor driver.

A simple QDISC is the 'pfifo' one, which does no processing at all and is a pure First In, First Out queue. It does however store traffic when the network interface can't handle it momentarily.

## CLASSES
Some qdiscs can contain classes, which contain further qdiscs - traffic may then be enqueued in any of the inner qdiscs, which are within the classes. When the kernel tries to dequeue a packet from such a classful qdisc it can come from any of the classes. A qdisc may for example prioritize certain kinds of traffic by trying to dequeue from certain classes before others.

## FILTERS
A filter is used by a classful qdisc to determine in which class a packet will be enqueued. Whenever traffic arrives at a class with subclasses, it needs to be classified. Various methods may be employed to do so, one of these are the filters. All filters attached to the class are called, until one of them returns with a verdict. If no verdict was made, other criteria may be available. This differs per qdisc.

It is important to notice that filters reside within qdiscs - they are not masters of what happens.

The available filters are:

basic

Filter packets based on an ematch expression. See tc-ematch(8) for details.

bpf

Filter packets using (e)BPF, see tc-bpf(8) for details.

cgroup

Filter packets based on the control group of their process. See tc-cgroup(8) for details.

flow, flower

Flow-based classifiers, filtering packets based on their flow (identified by selectable keys). See tc-flow(8) and tc-flower(8) for details.

fw

Filter based on fwmark. Directly maps fwmark value to traffic class. See tc-fw(8).

route

Filter packets based on routing table. See tc-route(8) for details.

u32

Generic filtering on arbitrary packet data, assisted by syntax to abstract common operations. See tc-u32(8) for details.

matchall

Traffic control filter that matches every packet. See tc-matchall(8) for details.

## QEVENTS
Qdiscs may invoke user-configured actions when certain interesting events take place in the qdisc. Each qevent can either be unused, or can have a block attached to it. To this block are then attached filters using the "tc block BLOCK_IDX" syntax. The block is executed when the qevent associated with the attachment point takes place. For example, packet could be dropped, or delayed, etc., depending on the qdisc and the qevent in question.

For example:

tc qdisc add dev eth0 root handle 1: red limit 500K avpkt 1K \

qevent early_drop block 10

tc filter add block 10 matchall action mirred egress mirror dev eth1

## CLASSLESS QDISCS
The classless qdiscs are:

choke

CHOKe (CHOose and Keep for responsive flows, CHOose and Kill for unresponsive flows) is a classless qdisc designed to both identify and penalize flows that monopolize the queue. CHOKe is a variation of RED, and the configuration is similar to RED.

codel

CoDel (pronounced "coddle") is an adaptive "no-knobs" active queue management algorithm (AQM) scheme that was developed to address the shortcomings of RED and its variants.

[p|b]fifo

Simplest usable qdisc, pure First In, First Out behaviour. Limited in packets or in bytes.

fq

Fair Queue Scheduler realises TCP pacing and scales to millions of concurrent flows per qdisc.

fq_codel

Fair Queuing Controlled Delay is queuing discipline that combines Fair Queuing with the CoDel AQM scheme. FQ_Codel uses a stochastic model to classify incoming packets into different flows and is used to provide a fair share of the bandwidth to all the flows using the queue. Each such flow is managed by the CoDel queuing discipline. Reordering within a flow is avoided since Codel internally uses a FIFO queue.

fq_pie

FQ-PIE (Flow Queuing with Proportional Integral controller Enhanced) is a queuing discipline that combines Flow Queuing with the PIE AQM scheme. FQ-PIE uses a Jenkins hash function to classify incoming packets into different flows and is used to provide a fair share of the bandwidth to all the flows using the qdisc. Each such flow is managed by the PIE algorithm.

gred

Generalized Random Early Detection combines multiple RED queues in order to achieve multiple drop priorities. This is required to realize Assured Forwarding (RFC 2597).

hhf

Heavy-Hitter Filter differentiates between small flows and the opposite, heavy-hitters. The goal is to catch the heavy-hitters and move them to a separate queue with less priority so that bulk traffic does not affect the latency of critical traffic.

ingress

This is a special qdisc as it applies to incoming traffic on an interface, allowing for it to be filtered and policed.

mqprio

The Multiqueue Priority Qdisc is a simple queuing discipline that allows mapping traffic flows to hardware queue ranges using priorities and a configurable priority to traffic class mapping. A traffic class in this context is a set of contiguous qdisc classes which map 1:1 to a set of hardware exposed queues.

multiq

Multiqueue is a qdisc optimized for devices with multiple Tx queues. It has been added for hardware that wishes to avoid head-of-line blocking. It will cycle though the bands and verify that the hardware queue associated with the band is not stopped prior to dequeuing a packet.

netem

Network Emulator is an enhancement of the Linux traffic control facilities that allow one to add delay, packet loss, duplication and more other characteristics to packets outgoing from a selected network interface.

pfifo_fast

Standard qdisc for 'Advanced Router' enabled kernels. Consists of a three-band queue which honors Type of Service flags, as well as the priority that may be assigned to a packet.

pie

Proportional Integral controller-Enhanced (PIE) is a control theoretic active queue management scheme. It is based on the proportional integral controller but aims to control delay.

red

Random Early Detection simulates physical congestion by randomly dropping packets when nearing configured bandwidth allocation. Well suited to very large bandwidth applications.

sfb

Stochastic Fair Blue is a classless qdisc to manage congestion based on packet loss and link utilization history while trying to prevent non-responsive flows (i.e. flows that do not react to congestion marking or dropped packets) from impacting performance of responsive flows. Unlike RED, where the marking probability has to be configured, BLUE tries to determine the ideal marking probability automatically.

sfq

Stochastic Fairness Queueing reorders queued traffic so each 'session' gets to send a packet in turn.

tbf

The Token Bucket Filter is suited for slowing traffic down to a precisely configured rate. Scales well to large bandwidths.

## CONFIGURING CLASSLESS QDISCS
In the absence of classful qdiscs, classless qdiscs can only be attached at the root of a device. Full syntax:

tc qdisc add dev DEV root QDISC QDISC-PARAMETERS

To remove, issue

tc qdisc del dev DEV root

The pfifo_fast qdisc is the automatic default in the absence of a configured qdisc.

## CLASSFUL QDISCS
The classful qdiscs are:

ATM

Map flows to virtual circuits of an underlying asynchronous transfer mode device.

DRR

The Deficit Round Robin Scheduler is a more flexible replacement for Stochastic Fairness Queuing. Unlike SFQ, there are no built-in queues -- you need to add classes and then set up filters to classify packets accordingly. This can be useful e.g. for using RED qdiscs with different settings for particular traffic. There is no default class -- if a packet cannot be classified, it is dropped.

ETS

The ETS qdisc is a queuing discipline that merges functionality of PRIO and DRR qdiscs in one scheduler. ETS makes it easy to configure a set of strict and bandwidth-sharing bands to implement the transmission selection described in 802.1Qaz.

HFSC

Hierarchical Fair Service Curve guarantees precise bandwidth and delay allocation for leaf classes and allocates excess bandwidth fairly. Unlike HTB, it makes use of packet dropping to achieve low delays which interactive sessions benefit from.

HTB

The Hierarchy Token Bucket implements a rich linksharing hierarchy of classes with an emphasis on conforming to existing practices. HTB facilitates guaranteeing bandwidth to classes, while also allowing specification of upper limits to inter-class sharing. It contains shaping elements, based on TBF and can prioritize classes.

PRIO

The PRIO qdisc is a non-shaping container for a configurable number of classes which are dequeued in order. This allows for easy prioritization of traffic, where lower classes are only able to send if higher ones have no packets available. To facilitate configuration, Type Of Service bits are honored by default.

QFQ

Quick Fair Queueing is an O(1) scheduler that provides near-optimal guarantees, and is the first to achieve that goal with a constant cost also with respect to the number of groups and the packet length. The QFQ algorithm has no loops, and uses very simple instructions and data structures that lend themselves very well to a hardware implementation.

## THEORY OF OPERATION
Classes form a tree, where each class has a single parent. A class may have multiple children. Some qdiscs allow for runtime addition of classes (HTB) while others (PRIO) are created with a static number of children.

Qdiscs which allow dynamic addition of classes can have zero or more subclasses to which traffic may be enqueued.

Furthermore, each class contains a leaf qdisc which by default has pfifo behaviour, although another qdisc can be attached in place. This qdisc may again contain classes, but each class can have only one leaf qdisc.

When a packet enters a classful qdisc it can be classified to one of the classes within. Three criteria are available, although not all qdiscs will use all three:

tc filters

If tc filters are attached to a class, they are consulted first for relevant instructions. Filters can match on all fields of a packet header, as well as on the firewall mark applied by iptables.

Type of Service

Some qdiscs have built in rules for classifying packets based on the TOS field.

skb->priority

Userspace programs can encode a class-id in the 'skb->priority' field using the SO_PRIORITY option.

Each node within the tree can have its own filters but higher level filters may also point directly to lower classes.

If classification did not succeed, packets are enqueued to the leaf qdisc attached to that class. Check qdisc specific manpages for details, however.

## NAMING
All qdiscs, classes and filters have IDs, which can either be specified or be automatically assigned.

IDs consist of a major number and a minor number, separated by a colon - major:minor. Both major and minor are hexadecimal numbers and are limited to 16 bits. There are two special values: root is signified by major and minor of all ones, and unspecified is all zeros.

QDISCS

A qdisc, which potentially can have children, gets assigned a major number, called a 'handle', leaving the minor number namespace available for classes. The handle is expressed as '10:'. It is customary to explicitly assign a handle to qdiscs expected to have children.

CLASSES

Classes residing under a qdisc share their qdisc major number, but each have a separate minor number called a 'classid' that has no relation to their parent classes, only to their parent qdisc. The same naming custom as for qdiscs applies.

FILTERS

Filters have a three part ID, which is only needed when using a hashed filter hierarchy.

## PARAMETERS
The following parameters are widely used in TC. For other parameters, see the man pages for individual qdiscs.

RATES

Bandwidths or rates. These parameters accept a floating point number, possibly followed by either a unit (both SI and IEC units supported), or a float followed by a '%' character to specify the rate as a percentage of the device's speed (e.g. 5%, 99.5%). Warning: specifying the rate as a percentage means a fraction of the current speed; if the speed changes, the value will not be recalculated.

bit or a bare number

Bits per second

kbit

Kilobits per second

mbit

Megabits per second

gbit

Gigabits per second

tbit

Terabits per second

bps

Bytes per second

kbps

Kilobytes per second

mbps

Megabytes per second

gbps

Gigabytes per second

tbps

Terabytes per second

To specify in IEC units, replace the SI prefix (k-, m-, g-, t-) with IEC prefix (ki-, mi-, gi- and ti-) respectively.

TC store rates as a 32-bit unsigned integer in bps internally, so we can specify a max rate of 4294967295 bps.

TIMES

Length of time. Can be specified as a floating point number followed by an optional unit:

s, sec or secs

Whole seconds

ms, msec or msecs

Milliseconds

us, usec, usecs or a bare number

Microseconds.

TC defined its own time unit (equal to microsecond) and stores time values as 32-bit unsigned integer, thus we can specify a max time value of 4294967295 usecs.

SIZES

Amounts of data. Can be specified as a floating point number followed by an optional unit:

b or a bare number

Bytes.

kbit

Kilobits

kb or k

Kilobytes

mbit

Megabits

mb or m

Megabytes

gbit

Gigabits

gb or g

Gigabytes

TC stores sizes internally as 32-bit unsigned integer in byte, so we can specify a max size of 4294967295 bytes.

VALUES

Other values without a unit. These parameters are interpreted as decimal by default, but you can indicate TC to interpret them as octal and hexadecimal by adding a '0' or '0x' prefix respectively.

## TC COMMANDS
The following commands are available for qdiscs, classes and filter:

add

Add a qdisc, class or filter to a node. For all entities, a parent must be passed, either by passing its ID or by attaching directly to the root of a device. When creating a qdisc or a filter, it can be named with the handle parameter. A class is named with the classid parameter.

delete

A qdisc can be deleted by specifying its handle, which may also be 'root'. All subclasses and their leaf qdiscs are automatically deleted, as well as any filters attached to them.

change

Some entities can be modified 'in place'. Shares the syntax of 'add', with the exception that the handle cannot be changed and neither can the parent. In other words, change cannot move a node.

replace

Performs a nearly atomic remove/add on an existing node id. If the node does not exist yet it is created.

get

Displays a single filter given the interface DEV, qdisc-id, priority, protocol and filter-id.

show

Displays all filters attached to the given interface. A valid parent ID must be passed.

link

Only available for qdiscs and performs a replace where the node must exist already.

## MONITOR
The tc utility can monitor events generated by the kernel such as adding/deleting qdiscs, filters or actions, or modifying existing ones.

The following command is available for monitor :

file

If the file option is given, the tc does not listen to kernel events, but opens the given file and dumps its contents. The file has to be in binary format and contain netlink messages.

## OPTIONS
-b, -b filename, -batch, -batch filename

read commands from provided file or standard input and invoke them. First failure will cause termination of tc.

-force

don't terminate tc on errors in batch mode. If there were any errors during execution of the commands, the application return code will be non zero.

-o, -oneline

output each record on a single line, replacing line feeds with the '\' character. This is convenient when you want to count records with wc(1) or to grep(1) the output.

-n, -net, -netns <NETNS>

switches tc to the specified network namespace NETNS. Actually it just simplifies executing of: ip netns exec NETNS tc [ OPTIONS ] OBJECT { COMMAND | help }

to

tc -n[etns] NETNS [ OPTIONS ] OBJECT { COMMAND | help }

-N, -Numeric

Print the number of protocol, scope, dsfield, etc directly instead of converting it to human readable name.

-cf, -conf <FILENAME>

specifies path to the config file. This option is used in conjunction with other options (e.g. -nm).

-t, -timestamp

When tc monitor runs, print timestamp before the event message in format:

Timestamp: <Day> <Month> <DD> <hh:mm:ss> <YYYY> <usecs> usec

-ts, -tshort

When tc monitor runs, prints short timestamp before the event message in format:

[<YYYY>-<MM>-<DD>T<hh:mm:ss>.<ms>]

-echo

Request the kernel to send the applied configuration back.

## FORMAT
The show command has additional formatting options:

-s, -stats, -statistics

output more statistics about packet usage.

-d, -details

output more detailed information about rates and cell sizes.

-r, -raw

output raw hex values for handles.

-p, -pretty

for u32 filter, decode offset and mask values to equivalent filter commands based on TCP/IP. In JSON output, add whitespace to improve readability.

-iec

print rates in IEC units (ie. 1K = 1024).

-g, -graph

shows classes as ASCII graph. Prints generic stats info under each class if -s option was specified. Classes can be filtered only by dev option.

-c[color][={always|auto|never}

Configure color output. If parameter is omitted or always, color output is enabled regardless of stdout state. If parameter is auto, stdout is checked to be a terminal before enabling color output. If parameter is never, color output is disabled. If specified multiple times, the last one takes precedence. This flag is ignored if -json is also given.

-j, -json

Display results in JSON format.

-nm, -name

resolve class name from /etc/iproute2/tc_cls file or from file specified by -cf option. This file is just a mapping of classid to class name:

\# Here is comment

1:40 voip # Here is another comment

1:50 web

1:60 ftp

1:2 home

tc will not fail if -nm was specified without -cf option but /etc/iproute2/tc_cls file does not exist, which makes it possible to pass -nm option for creating tc alias.

-br, -brief

Print only essential data needed to identify the filter and action (handle, cookie, etc.) and stats. This option is currently only supported by tc filter show and tc actions ls commands.

## EXAMPLES
tc -g class show dev eth0

Shows classes as ASCII graph on eth0 interface.

tc -g -s class show dev eth0

Shows classes as ASCII graph with stats info under each class.

## HISTORY
tc was written by Alexey N. Kuznetsov and added in Linux 2.2.

## SEE ALSO
tc-basic(8), tc-bfifo(8), tc-bpf(8), tc-cake(8), tc-cgroup(8), tc-choke(8), tc-codel(8), tc-drr(8), tc-ematch(8), tc-ets(8), tc-flow(8), tc-flower(8), tc-fq(8), tc-fq_codel(8), tc-fq_pie(8), tc-fw(8), tc-gact(8), tc-hfsc(7), tc-hfsc(8), tc-htb(8), tc-mqprio(8), tc-pfifo(8), tc-pfifo_fast(8), tc-pie(8), tc-red(8), tc-route(8), tc-sfb(8), tc-sfq(8), tc-stab(8), tc-tbf(8), tc-u32(8)

User documentation at http://lartc.org/, but please direct bugreports and patches to: <netdev@vger.kernel.org>

## AUTHOR
Manpage maintained by bert hubert (ahu@ds9a.nl)


tc-actions

title: tc-actions(8)

summary: independently defined actions in tc

text:
# tc-actions(8)

## NAME
actions - independently defined actions in tc

## SYNOPSIS
tc [ TC_OPTIONS ] actions add | change | replace ACTSPEC

tc [ TC_OPTIONS ] actions get | delete ACTISPEC

tc [ TC_OPTIONS ] actions flush ACTNAMESPEC

tc [ TC_OPTIONS ] actions ls | list ACTNAMESPEC [ ACTFILTER ]

ACTSPEC := action ACTDETAIL [ INDEXSPEC ] [ COOKIESPEC ] [ FLAGS ] [ HWSTATSSPEC ] [ CONTROL ] [ SKIPSPEC ]

ACTISPEC := ACTNAMESPEC INDEXSPEC

ACTNAMESPEC := action ACTNAME

INDEXSPEC := index INDEX

ACTFILTER := since MSTIME

COOKIESPEC := cookie COOKIE

FLAGS := no_percpu

HWSTATSSPEC := hw_stats { immediate | delayed | disabled }

ACTDETAIL := ACTNAME ACTPARAMS

ACTNAME may be any valid action type: gact, mirred, bpf, connmark, csum, police, etc.

MSTIME Time since last update.

CONTROL := { reclassify | pipe | drop | continue | ok }

SKIPSPEC := { skip_sw | skip_hw }

TC_OPTIONS These are the options that are specific to tc and not only the options. Refer to tc(8) for more information.

## DESCRIPTION
The actions object in tc allows a user to define actions independently of a classifier (filter). These actions can then be assigned to one or more filters, with any packets matching the classifier's criteria having that action performed on them.

Each action type (mirred, police, etc.) will have its own table to store all created actions.

## OPERATIONS
add

Create a new action in that action's table.

change

replace

Make modifications to an existing action.

get

Display the action with the specified index value. When combined with the -s option for tc, display the statistics for that action.

delete

Delete the action with the specified index value. If the action is already associated with a classifier, it does not delete the classifier.

ls

list

List all the actions in the specified table. When combined with the -s option for tc, display the statistics for all actions in the specified table. When combined with the option since allows doing a millisecond time-filter since the last time an action was used in the datapath.

flush

Delete all actions stored in the specified table.

## ACTION OPTIONS
Note that these options are available to all action types.

index INDEX

Specify the table index value of an action. INDEX is a 32-bit value that is unique to the specific type of action referenced.

For add, change, and replace operations, the index is optional. When adding a new action, specifying an index value will assign the action to that index unless that index value has already been assigned. Omitting the index value for an add operation will cause the kernel to assign a value to the new action.

For get and delete operations, the index is required to identify the specific action to be displayed or deleted.

cookie COOKIE

In addition to the specific action, mark the matching packet with the value specified by COOKIE. The COOKIE is a 128-bit value that will not be interpreted by the kernel whatsoever. As such, it can be used as a correlating value for maintaining user state. The value to be stored is completely arbitrary and does not require a specific format. It is stored inside the action structure itself.

FLAGS

Action-specific flags. Currently, the only supported flag is no_percpu which indicates that action is expected to have minimal software data-path traffic and doesn't need to allocate stat counters with percpu allocator. This option is intended to be used by hardware-offloaded actions.

hw_stats HW_STATS

Specifies the type of HW stats of new action. If omitted, any stats counter type is going to be used, according to driver and its resources. The HW_STATS indicates the type. Any of the following are valid:

immediate

Means that in dump, user gets the current HW stats state from the device queried at the dump time.

delayed

Means that in dump, user gets HW stats that might be out of date for some time, maybe couple of seconds. This is the case when driver polls stats updates periodically or when it gets async stats update from the device.

disabled

No HW stats are going to be available in dump.

since MSTIME

When dumping large number of actions, a millisecond time-filter can be specified MSTIME. The MSTIME is a millisecond count since last time a packet hit the action. As an example specifying "since 20000" implies to dump all actions that have seen packets in the last 20 seconds. This option is useful when the kernel has a large number of actions and you are only interested in recently used actions.

CONTROL

The CONTROL indicates how tc should proceed after executing the action. Any of the following are valid:

reclassify

Restart the classifiction by jumping back to the first filter attached to the action's parent.

pipe

Continue with the next action. This is the default control.

drop

Drop the packed without running any further actions.

continue

Continue the classification with the next filter.

pass

Return to the calling qdisc for packet processing, and end classification of this packet.

SKIPSPEC

The SKIPSPEC indicates how tc should proceed when executing the action. Any of the following are valid:

skip_sw

Do not process action by software. If hardware has no offload support for this action, operation will fail.

skip_hw

Do not process action by hardware.

## SEE ALSO
tc(8), tc-bpf(8), tc-connmark(8), tc-csum(8), tc-ife(8), tc-mirred(8), tc-nat(8), tc-pedit(8), tc-police(8), tc-simple(8), tc-skbedit(8), tc-skbmod(8), tc-tunnel_key(8), tc-vlan(8), tc-xt(8)


tc-basic

title: tc-basic(8)

summary: basic traffic control filter

text:
# tc-basic(8)

## NAME
basic - basic traffic control filter

## SYNOPSIS

tc filter ... basic [ match EMATCH_TREE ] [ action ACTION_SPEC ] [ classid CLASSID ]

## DESCRIPTION
The basic filter allows one to classify packets using the extended match infrastructure.

## OPTIONS
action ACTION_SPEC

Apply an action from the generic actions framework on matching packets.

classid CLASSID

Push matching packets into the class identified by CLASSID.

match EMATCH_TREE

Match packets using the extended match infrastructure. See tc-ematch(8) for a detailed description of the allowed syntax in EMATCH_TREE.

## SEE ALSO
tc(8), tc-ematch(8)


tc-bfifo

title: tc-bfifo(8)

text:
# tc-bfifo(8)

## NAME
pfifo - Packet limited First In, First Out queue

bfifo - Byte limited First In, First Out queue

## SYNOPSIS
tc qdisc ... add pfifo [ limit packets ]

tc qdisc ... add bfifo [ limit bytes ]

## DESCRIPTION
The pfifo and bfifo qdiscs are unadorned First In, First Out queues. They are the simplest queues possible and therefore have no overhead. pfifo constrains the queue size as measured in packets. bfifo does so as measured in bytes.

Like all non-default qdiscs, they maintain statistics. This might be a reason to prefer pfifo or bfifo over the default.

## ALGORITHM
A list of packets is maintained, when a packet is enqueued it gets inserted at the tail of a list. When a packet needs to be sent out to the network, it is taken from the head of the list.

If the list is too long, no further packets are allowed on. This is called 'tail drop'.

## PARAMETERS
limit

Maximum queue size. Specified in bytes for bfifo, in packets for pfifo. For pfifo, defaults to the interface txqueuelen, as specified with ip(8). The range for this parameter is [0, UINT32_MAX]. For bfifo, it defaults to the txqueuelen multiplied by the interface MTU. The range for this parameter is [0, UINT32_MAX] bytes.

Note: The link layer header was considered when counting packets length.

## OUTPUT
The output of tc -s qdisc ls contains the limit, either in packets or in bytes, and the number of bytes and packets actually sent. An unsent and dropped packet only appears between braces and is not counted as 'Sent'.

In this example, the queue length is 100 packets, 45894 bytes were sent over 681 packets. No packets were dropped, and as the pfifo queue does not slow down packets, there were also no overlimits:

```
# tc -s qdisc ls dev eth0
qdisc pfifo 8001: dev eth0 limit 100p

 Sent 45894 bytes 681 pkts (dropped 0, overlimits 0)
```

If a backlog occurs, this is displayed as well.

## SEE ALSO
tc(8)

## AUTHORS
Alexey N. Kuznetsov, <kuznet@ms2.inr.ac.ru>

This manpage maintained by bert hubert <ahu@ds9a.nl>


tc-bpf

title: tc-bpf(8)

text:
# tc-bpf(8)

## NAME
BPF - BPF programmable classifier and actions for ingress/egress queueing disciplines

## SYNOPSIS

### eBPF classifier (filter) or action:
tc filter ... bpf [ object-file OBJ_FILE ] [ section CLS_NAME ] [ export UDS_FILE ] [ verbose ] [ direct-action | da ] [ skip_hw | skip_sw ] [ police POLICE_SPEC ] [ action ACTION_SPEC ] [ classid CLASSID ]

tc action ... bpf [ object-file OBJ_FILE ] [ section CLS_NAME ] [ export UDS_FILE ] [ verbose ]

### cBPF classifier (filter) or action:
tc filter ... bpf [ bytecode-file BPF_FILE | bytecode BPF_BYTECODE ] [ police POLICE_SPEC ] [ action ACTION_SPEC ] [ classid CLASSID ]

tc action ... bpf [ bytecode-file BPF_FILE | bytecode BPF_BYTECODE ]

## DESCRIPTION
Extended Berkeley Packet Filter ( eBPF ) and classic Berkeley Packet Filter (originally known as BPF, for better distinction referred to as cBPF here) are both available as a fully programmable and highly efficient classifier and actions. They both offer a minimal instruction set for implementing small programs which can safely be loaded into the kernel and thus executed in a tiny virtual machine from kernel space. An in-kernel verifier guarantees that a specified program always terminates and neither crashes nor leaks data from the kernel.

In Linux, it's generally considered that eBPF is the successor of cBPF. The kernel internally transforms cBPF expressions into eBPF expressions and executes the latter. Execution of them can be performed in an interpreter or at setup time, they can be just-in-time compiled (JIT'ed) to run as native machine code.

Currently, the eBPF JIT compiler is available for the following architectures:

- x86_64 (since Linux 3.18)

- arm64 (since Linux 3.18)

- s390 (since Linux 4.1)

- ppc64 (since Linux 4.8)

- sparc64 (since Linux 4.12)

- mips64 (since Linux 4.13)

- arm32 (since Linux 4.14)

- x86_32 (since Linux 4.18)

Whereas the following architectures have cBPF, but did not (yet) switch to eBPF JIT support:

- ppc32

- sparc32

- mips32

eBPF's instruction set has similar underlying principles as the cBPF instruction set, it however is modelled closer to the underlying architecture to better mimic native instruction sets with the aim to achieve a better run-time performance. It is designed to be JIT'ed with a one to one mapping, which can also open up the possibility for compilers to generate optimized eBPF code through an eBPF backend that performs almost as fast as natively compiled code. Given that LLVM provides such an eBPF backend, eBPF programs can therefore easily be programmed in a subset of the C language. Other than that, eBPF infrastructure also comes with a construct called "maps". eBPF maps are key/value stores that are shared between multiple eBPF programs, but also between eBPF programs and user space applications.

For the traffic control subsystem, classifier and actions that can be attached to ingress and egress qdiscs can be written in eBPF or cBPF. The advantage over other classifier and actions is that eBPF/cBPF provides the generic framework, while users can implement their highly specialized use cases efficiently. This means that the classifier or action written that way will not suffer from feature bloat, and can therefore execute its task highly efficient. It allows for non-linear classification and even merging the action part into the classification. Combined with efficient eBPF map data structures, user space can push new policies like classids into the kernel without reloading a classifier, or it can gather statistics that are pushed into one map and use another one for dynamically load balancing traffic based on the determined load, just to provide a few examples.

## PARAMETERS

### object-file
points to an object file that has an executable and linkable format (ELF) and contains eBPF opcodes and eBPF map definitions. The LLVM compiler infrastructure with clang(1) as a C language front end is one project that supports emitting eBPF object files that can be passed to the eBPF classifier (more details in the EXAMPLES section). This option is mandatory when an eBPF classifier or action is to be loaded.

### section
is the name of the ELF section from the object file, where the eBPF classifier or action resides. By default the section name for the classifier is called "classifier", and for the action "action". Given that a single object file can contain multiple classifier and actions, the corresponding section name needs to be specified, if it differs from the defaults.

### export
points to a Unix domain socket file. In case the eBPF object file also contains a section named "maps" with eBPF map specifications, then the map file descriptors can be handed off via the Unix domain socket to an eBPF "agent" herding all descriptors after tc lifetime. This can be some third party application implementing the IPC counterpart for the import, that uses them for calling into bpf(2) system call to read out or update eBPF map data from user space, for example, for monitoring purposes or to push down new policies.

### verbose
if set, it will dump the eBPF verifier output, even if loading the eBPF program was successful. By default, only on error, the verifier log is being emitted to the user.

### direct-action | da
instructs eBPF classifier to not invoke external TC actions, instead use the TC actions return codes (TC_ACT_OK, TC_ACT_SHOT etc.) for classifiers.

### skip_hw | skip_sw
hardware offload control flags. By default TC will try to offload filters to hardware if possible. skip_hw explicitly disables the attempt to offload. skip_sw forces the offload and disables running the eBPF program in the kernel. If hardware offload is not possible and this flag was set kernel will report an error and filter will not be installed at all.

### police
is an optional parameter for an eBPF/cBPF classifier that specifies a police in tc(1) which is attached to the classifier, for example, on an ingress qdisc.

### action
is an optional parameter for an eBPF/cBPF classifier that specifies a subsequent action in tc(1) which is attached to a classifier.

### flowid
provides the default traffic control class identifier for this eBPF/cBPF classifier. The default class identifier can also be overwritten by the return code of the eBPF/cBPF program. A default return code of -1 specifies the here provided default class identifier to be used. A return code of the eBPF/cBPF program of 0 implies that no match took place, and a return code other than these two will override the default classid. This allows for efficient, non-linear classification with only a single eBPF/cBPF program as opposed to having multiple individual programs for various class identifiers which would need to reparse packet contents.

### bytecode
is being used for loading cBPF classifier and actions only. The cBPF bytecode is directly passed as a text string in the form of 's,c t f k,c t f k,c t f k,...' , where s denotes the number of subsequent 4-tuples. One such 4-tuple consists of c t f k decimals, where c represents the cBPF opcode, t the jump true offset target, f the jump false offset target and k the immediate constant/literal. There are various tools that generate code in this loadable format, for example, bpf_asm that ships with the Linux kernel source tree under tools/net/ , so it is certainly not expected to hack this by hand. The bytecode or bytecode-file option is mandatory when a cBPF classifier or action is to be loaded.

### bytecode-file
also being used to load a cBPF classifier or action. It's effectively the same as bytecode only that the cBPF bytecode is not passed directly via command line, but rather resides in a text file.

## EXAMPLES

### eBPF TOOLING
A full blown example including eBPF agent code can be found inside the iproute2 source package under: examples/bpf/

As prerequisites, the kernel needs to have the eBPF system call namely bpf(2) enabled and ships with cls_bpf and act_bpf kernel modules for the traffic control subsystem. To enable eBPF/eBPF JIT support, depending which of the two the given architecture supports:

echo 1 > /proc/sys/net/core/bpf_jit_enable

A given restricted C file can be compiled via LLVM as:

clang -O2 -emit-llvm -c bpf.c -o - | llc -march=bpf -filetype=obj -o bpf.o

The compiler invocation might still simplify in future, so for now, it's quite handy to alias this construct in one way or another, for example:

```
__bcc() {

        clang -O2 -emit-llvm -c $1 -o - | \

        llc -march=bpf -filetype=obj -o "`basename $1 .c`.o"
}
alias bcc=__bcc
```

A minimal, stand-alone unit, which matches on all traffic with the default classid (return code of -1) looks like:

```
#include <linux/bpf.h>
#ifndef __section
# define __section(x)  __attribute__((section(x), used))
#endif
__section("classifier") int cls_main(struct __sk_buff *skb)
{

        return -1;
}
char __license[] __section("license") = "GPL";
```

More examples can be found further below in subsection eBPF PROGRAMMING as focus here will be on tooling.

There can be various other sections, for example, also for actions. Thus, an object file in eBPF can contain multiple entrance points. Always a specific entrance point, however, must be specified when configuring with tc. A license must be part of the restricted C code and the license string syntax is the same as with Linux kernel modules. The kernel reserves its right that some eBPF helper functions can be restricted to GPL compatible licenses only, and thus may reject a program from loading into the kernel when such a license mismatch occurs.

The resulting object file from the compilation can be inspected with the usual set of tools that also operate on normal object files, for example objdump(1) for inspecting ELF section headers:

```
objdump -h bpf.o
[...]
3 classifier    000007f8  0000000000000000  0000000000000000  00000040  2**3

                CONTENTS, ALLOC, LOAD, RELOC, READONLY, CODE
4 action-mark   00000088  0000000000000000  0000000000000000  00000838  2**3

                CONTENTS, ALLOC, LOAD, RELOC, READONLY, CODE
5 action-rand   00000098  0000000000000000  0000000000000000  000008c0  2**3

                CONTENTS, ALLOC, LOAD, RELOC, READONLY, CODE
6 maps          00000030  0000000000000000  0000000000000000  00000958  2**2

                CONTENTS, ALLOC, LOAD, DATA
7 license       00000004  0000000000000000  0000000000000000  00000988  2**0

                CONTENTS, ALLOC, LOAD, DATA
[...]
```

Adding an eBPF classifier from an object file that contains a classifier in the default ELF section is trivial (note that instead of "object-file" also shortcuts such as "obj" can be used):

bcc bpf.c

tc filter add dev em1 parent 1: bpf obj bpf.o flowid 1:1

In case the classifier resides in ELF section "mycls", then that same command needs to be invoked as:

tc filter add dev em1 parent 1: bpf obj bpf.o sec mycls flowid 1:1

Dumping the classifier configuration will tell the location of the classifier, in other words that it's from object file "bpf.o" under section "mycls":

tc filter show dev em1

filter parent 1: protocol all pref 49152 bpf

filter parent 1: protocol all pref 49152 bpf handle 0x1 flowid 1:1 bpf.o:[mycls]

The same program can also be installed on ingress qdisc side as opposed to egress ...

tc qdisc add dev em1 handle ffff: ingress

tc filter add dev em1 parent ffff: bpf obj bpf.o sec mycls flowid ffff:1

... and again dumped from there:

tc filter show dev em1 parent ffff:

filter protocol all pref 49152 bpf

filter protocol all pref 49152 bpf handle 0x1 flowid ffff:1 bpf.o:[mycls]

Attaching a classifier and action on ingress has the restriction that it doesn't have an actual underlying queueing discipline. What ingress can do is to classify, mangle, redirect or drop packets. When queueing is required on ingress side, then ingress must redirect packets to the ifb device, otherwise policing can be used. Moreover, ingress can be used to have an early drop point of unwanted packets before they hit upper layers of the networking stack, perform network accounting with eBPF maps that could be shared with egress, or have an early mangle and/or redirection point to different networking devices.

Multiple eBPF actions and classifier can be placed into a single object file within various sections. In that case, non-default section names must be provided, which is the case for both actions in this example:

tc filter add dev em1 parent 1: bpf obj bpf.o flowid 1:1 \

action bpf obj bpf.o sec action-mark \

action bpf obj bpf.o sec action-rand ok

The advantage of this is that the classifier and the two actions can then share eBPF maps with each other, if implemented in the programs.

In order to access eBPF maps from user space beyond tc(8) setup lifetime, the ownership can be transferred to an eBPF agent via Unix domain sockets. There are two possibilities for implementing this:

1) implementation of an own eBPF agent that takes care of setting up the Unix domain socket and implementing the protocol that tc(8) dictates. A code example of this can be found inside the iproute2 source package under: examples/bpf/

2) use tc exec for transferring the eBPF map file descriptors through a Unix domain socket, and spawning an application such as sh(1) . This approach's advantage is that tc will place the file descriptors into the environment and thus make them available just like stdin, stdout, stderr file descriptors, meaning, in case user applications run from within this fd-owner shell, they can terminate and restart without losing eBPF maps file descriptors. Example invocation with the previous classifier and action mixture:

tc exec bpf imp /tmp/bpf

tc filter add dev em1 parent 1: bpf obj bpf.o exp /tmp/bpf flowid 1:1 \

action bpf obj bpf.o sec action-mark \

action bpf obj bpf.o sec action-rand ok

Assuming that eBPF maps are shared with classifier and actions, it's enough to export them once, for example, from within the classifier or action command. tc will setup all eBPF map file descriptors at the time when the object file is first parsed.

When a shell has been spawned, the environment will have a couple of eBPF related variables. BPF_NUM_MAPS provides the total number of maps that have been transferred over the Unix domain socket. BPF_MAP<X>'s value is the file descriptor number that can be accessed in eBPF agent applications, in other words, it can directly be used as the file descriptor value for the bpf(2) system call to retrieve or alter eBPF map values. <X> denotes the identifier of the eBPF map. It corresponds to the id member of struct bpf_elf_map from the tc eBPF map specification.

The environment in this example looks as follows:

```
sh# env | grep BPF

    BPF_NUM_MAPS=3

    BPF_MAP1=6

    BPF_MAP0=5

    BPF_MAP2=7
sh# ls -la /proc/self/fd

    [...]

    lrwx------. 1 root root 64 Apr 14 16:46 5 -> anon_inode:bpf-map

    lrwx------. 1 root root 64 Apr 14 16:46 6 -> anon_inode:bpf-map

    lrwx------. 1 root root 64 Apr 14 16:46 7 -> anon_inode:bpf-map
sh# my_bpf_agent
```

eBPF agents are very useful in that they can prepopulate eBPF maps from user space, monitor statistics via maps and based on that feedback, for example, rewrite classids in eBPF map values during runtime. Given that eBPF agents are implemented as normal applications, they can also dynamically receive traffic control policies from external controllers and thus push them down into eBPF maps to dynamically adapt to network conditions. Moreover, eBPF maps can also be shared with other eBPF program types (e.g. tracing), thus very powerful combination can therefore be implemented.

### eBPF PROGRAMMING
eBPF classifier and actions are being implemented in restricted C syntax (in future, there could additionally be new language frontends supported).

The header file linux/bpf.h provides eBPF helper functions that can be called from an eBPF program. This man page will only provide two minimal, stand-alone examples, have a look at examples/bpf from the iproute2 source package for a fully fledged flow dissector example to better demonstrate some of the possibilities with eBPF.

Supported 32 bit classifier return codes from the C program and their meanings:

0 , denotes a mismatch

-1 , denotes the default classid configured from the command line

else , everything else will override the default classid to provide a facility for non-linear matching

Supported 32 bit action return codes from the C program and their meanings ( linux/pkt_cls.h ):

TC_ACT_OK (0) , will terminate the packet processing pipeline and allows the packet to proceed

TC_ACT_SHOT (2) , will terminate the packet processing pipeline and drops the packet

TC_ACT_UNSPEC (-1) , will use the default action configured from tc (similarly as returning -1 from a classifier)

TC_ACT_PIPE (3) , will iterate to the next action, if available

TC_ACT_RECLASSIFY (1) , will terminate the packet processing pipeline and start classification from the beginning

else , everything else is an unspecified return code

Both classifier and action return codes are supported in eBPF and cBPF programs.

To demonstrate restricted C syntax, a minimal toy classifier example is provided, which assumes that egress packets, for instance originating from a container, have previously been marked in interval [0, 255]. The program keeps statistics on different marks for user space and maps the classid to the root qdisc with the marking itself as the minor handle:

```
#include <stdint.h>
#include <asm/types.h>
#include <linux/bpf.h>
#include <linux/pkt_sched.h>
#include "helpers.h"
struct tuple {

        long packets;

        long bytes;
};
#define BPF_MAP_ID_STATS        1 /* agent's map identifier */
#define BPF_MAX_MARK            256
struct bpf_elf_map __section("maps") map_stats = {

        .type           =       BPF_MAP_TYPE_ARRAY,

        .id             =       BPF_MAP_ID_STATS,

        .size_key       =       sizeof(uint32_t),

        .size_value     =       sizeof(struct tuple),

        .max_elem       =       BPF_MAX_MARK,

        .pinning        =       PIN_GLOBAL_NS,
};
static inline void cls_update_stats(const struct __sk_buff *skb,

                                    uint32_t mark)
{

        struct tuple *tu;

        tu = bpf_map_lookup_elem(&map_stats, &mark);

        if (likely(tu)) {

                __sync_fetch_and_add(&tu->packets, 1);

                __sync_fetch_and_add(&tu->bytes, skb->len);

        }
}
__section("cls") int cls_main(struct __sk_buff *skb)
{

        uint32_t mark = skb->mark;

        if (unlikely(mark >= BPF_MAX_MARK))

                return 0;

        cls_update_stats(skb, mark);

        return TC_H_MAKE(TC_H_ROOT, mark);
}
char __license[] __section("license") = "GPL";
```

Another small example is a port redirector which demuxes destination port 80 into the interval [8080, 8087] steered by RSS, that can then be attached to ingress qdisc. The exercise of adding the egress counterpart and IPv6 support is left to the reader:

```
#include <asm/types.h>
#include <asm/byteorder.h>
#include <linux/bpf.h>
#include <linux/filter.h>
#include <linux/in.h>
#include <linux/if_ether.h>
#include <linux/ip.h>
#include <linux/tcp.h>
#include "helpers.h"
static inline void set_tcp_dport(struct __sk_buff *skb, int nh_off,

                                 __u16 old_port, __u16 new_port)
{

        bpf_l4_csum_replace(skb, nh_off + offsetof(struct tcphdr, check),

                            old_port, new_port, sizeof(new_port));

        bpf_skb_store_bytes(skb, nh_off + offsetof(struct tcphdr, dest),

                            &new_port, sizeof(new_port), 0);
}
static inline int lb_do_ipv4(struct __sk_buff *skb, int nh_off)
{

        __u16 dport, dport_new = 8080, off;

        __u8 ip_proto, ip_vl;

        ip_proto = load_byte(skb, nh_off +

                             offsetof(struct iphdr, protocol));

        if (ip_proto != IPPROTO_TCP)

                return 0;

        ip_vl = load_byte(skb, nh_off);

        if (likely(ip_vl == 0x45))

                nh_off += sizeof(struct iphdr);

        else

                nh_off += (ip_vl & 0xF) << 2;

        dport = load_half(skb, nh_off + offsetof(struct tcphdr, dest));

        if (dport != 80)

                return 0;

        off = skb->queue_mapping & 7;

        set_tcp_dport(skb, nh_off - BPF_LL_OFF, __constant_htons(80),

                      __cpu_to_be16(dport_new + off));

        return -1;
}
__section("lb") int lb_main(struct __sk_buff *skb)
{

        int ret = 0, nh_off = BPF_LL_OFF + ETH_HLEN;

        if (likely(skb->protocol == __constant_htons(ETH_P_IP)))

                ret = lb_do_ipv4(skb, nh_off);

        return ret;
}
char __license[] __section("license") = "GPL";
```

The related helper header file helpers.h in both examples was:

```
/* Misc helper macros. */
#define __section(x) __attribute__((section(x), used))
#define offsetof(x, y) __builtin_offsetof(x, y)
#define likely(x) __builtin_expect(!!(x), 1)
#define unlikely(x) __builtin_expect(!!(x), 0)
/* Object pinning settings */
#define PIN_NONE       0
#define PIN_OBJECT_NS  1
#define PIN_GLOBAL_NS  2
/* ELF map definition */
struct bpf_elf_map {

    __u32 type;

    __u32 size_key;

    __u32 size_value;

    __u32 max_elem;

    __u32 flags;

    __u32 id;

    __u32 pinning;

    __u32 inner_id;

    __u32 inner_idx;
};
/* Some used BPF function calls. */
static int (*bpf_skb_store_bytes)(void *ctx, int off, void *from,

                                  int len, int flags) =

      (void *) BPF_FUNC_skb_store_bytes;
static int (*bpf_l4_csum_replace)(void *ctx, int off, int from,

                                  int to, int flags) =

      (void *) BPF_FUNC_l4_csum_replace;
static void *(*bpf_map_lookup_elem)(void *map, void *key) =

      (void *) BPF_FUNC_map_lookup_elem;
/* Some used BPF intrinsics. */
unsigned long long load_byte(void *skb, unsigned long long off)

    asm ("llvm.bpf.load.byte");
unsigned long long load_half(void *skb, unsigned long long off)

    asm ("llvm.bpf.load.half");
```

Best practice, we recommend to only have a single eBPF classifier loaded in tc and perform all necessary matching and mangling from there instead of a list of individual classifier and separate actions. Just a single classifier tailored for a given use-case will be most efficient to run.

### eBPF DEBUGGING
Both tc filter and action commands for bpf support an optional verbose parameter that can be used to inspect the eBPF verifier log. It is dumped by default in case of an error.

In case the eBPF/cBPF JIT compiler has been enabled, it can also be instructed to emit a debug output of the resulting opcode image into the kernel log, which can be read via dmesg(1) :

echo 2 > /proc/sys/net/core/bpf_jit_enable

The Linux kernel source tree ships additionally under tools/net/ a small helper called bpf_jit_disasm that reads out the opcode image dump from the kernel log and dumps the resulting disassembly:

bpf_jit_disasm -o

Other than that, the Linux kernel also contains an extensive eBPF/cBPF test suite module called test_bpf . Upon ...

modprobe test_bpf

... it performs a diversity of test cases and dumps the results into the kernel log that can be inspected with dmesg(1) . The results can differ depending on whether the JIT compiler is enabled or not. In case of failed test cases, the module will fail to load. In such cases, we urge you to file a bug report to the related JIT authors, Linux kernel and networking mailing lists.

### cBPF
Although we generally recommend switching to implementing eBPF classifier and actions, for the sake of completeness, a few words on how to program in cBPF will be lost here.

Likewise, the bpf_jit_enable switch can be enabled as mentioned already. Tooling such as bpf_jit_disasm is also independent whether eBPF or cBPF code is being loaded.

Unlike in eBPF, classifier and action are not implemented in restricted C, but rather in a minimal assembler-like language or with the help of other tooling.

The raw interface with tc takes opcodes directly. For example, the most minimal classifier matching on every packet resulting in the default classid of 1:1 looks like:

tc filter add dev em1 parent 1: bpf bytecode '1,6 0 0 4294967295,' flowid 1:1

The first decimal of the bytecode sequence denotes the number of subsequent 4-tuples of cBPF opcodes. As mentioned, such a 4-tuple consists of c t f k decimals, where c represents the cBPF opcode, t the jump true offset target, f the jump false offset target and k the immediate constant/literal. Here, this denotes an unconditional return from the program with immediate value of -1.

Thus, for egress classification, Willem de Bruijn implemented a minimal stand-alone helper tool under the GNU General Public License version 2 for iptables(8) BPF extension, which abuses the libpcap internal classic BPF compiler, his code derived here for usage with tc(8) :

```
#include <pcap.h>
#include <stdio.h>
int main(int argc, char **argv)
{

        struct bpf_program prog;

        struct bpf_insn *ins;

        int i, ret, dlt = DLT_RAW;

        if (argc < 2 || argc > 3)

                return 1;

        if (argc == 3) {

                dlt = pcap_datalink_name_to_val(argv[1]);

                if (dlt == -1)

                        return 1;

        }

        ret = pcap_compile_nopcap(-1, dlt, &prog, argv[argc - 1],

                                  1, PCAP_NETMASK_UNKNOWN);

        if (ret)

                return 1;

        printf("%d,", prog.bf_len);

        ins = prog.bf_insns;

        for (i = 0; i < prog.bf_len - 1; ++ins, ++i)

                printf("%u %u %u %u,", ins->code,

                       ins->jt, ins->jf, ins->k);

        printf("%u %u %u %u",

               ins->code, ins->jt, ins->jf, ins->k);

        pcap_freecode(&prog);

        return 0;
}
```

Given this small helper, any tcpdump(8) filter expression can be abused as a classifier where a match will result in the default classid:

bpftool EN10MB 'tcp[tcpflags] & tcp-syn != 0' > /var/bpf/tcp-syn

tc filter add dev em1 parent 1: bpf bytecode-file /var/bpf/tcp-syn flowid 1:1

Basically, such a minimal generator is equivalent to:

tcpdump -iem1 -ddd 'tcp[tcpflags] & tcp-syn != 0' | tr ' ',' > /var/bpf/tcp-syn

Since libpcap does not support all Linux' specific cBPF extensions in its compiler, the Linux kernel also ships under tools/net/ a minimal BPF assembler called bpf_asm for providing full control. For detailed syntax and semantics on implementing such programs by hand, see references under FURTHER READING .

Trivial toy example in bpf_asm for classifying IPv4/TCP packets, saved in a text file called foobar :

```
ldh [12]
jne #0x800, drop
ldb [23]
jneq #6, drop
ret #-1
drop: ret #0
```

Similarly, such a classifier can be loaded as:

bpf_asm foobar > /var/bpf/tcp-syn

tc filter add dev em1 parent 1: bpf bytecode-file /var/bpf/tcp-syn flowid 1:1

For BPF classifiers, the Linux kernel provides additionally under tools/net/ a small BPF debugger called bpf_dbg , which can be used to test a classifier against pcap files, single-step or add various breakpoints into the classifier program and dump register contents during runtime.

Implementing an action in classic BPF is rather limited in the sense that packet mangling is not supported. Therefore, it's generally recommended to make the switch to eBPF, whenever possible.

## FURTHER READING
Further and more technical details about the BPF architecture can be found in the Linux kernel source tree under Documentation/networking/filter.txt .

Further details on eBPF tc(8) examples can be found in the iproute2 source tree under examples/bpf/ .

## SEE ALSO
tc(8), tc-ematch(8) bpf(2) bpf(4)

## AUTHORS
Manpage written by Daniel Borkmann.

Please report corrections or improvements to the Linux kernel networking mailing list: <netdev@vger.kernel.org>


tc-cake

title: tc-cake(8)

text:
# tc-cake(8)

## NAME
CAKE - Common Applications Kept Enhanced (CAKE)

## SYNOPSIS
tc qdisc ... cake

[ bandwidth RATE | unlimited* | autorate-ingress ]

[ rtt TIME | datacentre | lan | metro | regional | internet* | oceanic | satellite | interplanetary ]

[ besteffort | diffserv8 | diffserv4 | diffserv3* ]

[ flowblind | srchost | dsthost | hosts | flows | dual-srchost | dual-dsthost | triple-isolate* ]

[ nat | nonat* ]

[ wash | nowash* ]

[ split-gso* | no-split-gso ]

[ ack-filter | ack-filter-aggressive | no-ack-filter* ]

[ memlimit LIMIT ]

[ fwmark MASK ]

[ ptm | atm | noatm* ]

[ overhead N | conservative | raw* ]

[ mpu N ]

[ ingress | egress* ]

(* marks defaults)

## DESCRIPTION
CAKE (Common Applications Kept Enhanced) is a shaping-capable queue discipline which uses both AQM and FQ. It combines COBALT, which is an AQM algorithm combining Codel and BLUE, a shaper which operates in deficit mode, and a variant of DRR++ for flow isolation. 8-way set-associative hashing is used to virtually eliminate hash collisions. Priority queuing is available through a simplified diffserv implementation. Overhead compensation for various encapsulation schemes is tightly integrated.

All settings are optional; the default settings are chosen to be sensible in most common deployments. Most people will only need to set the bandwidth parameter to get useful results, but reading the Overhead Compensation and Round Trip Time sections is strongly encouraged.

## SHAPER PARAMETERS
CAKE uses a deficit-mode shaper, which does not exhibit the initial burst typical of token-bucket shapers. It will automatically burst precisely as much as required to maintain the configured throughput. As such, it is very straightforward to configure.

unlimited (default)

No limit on the bandwidth.

bandwidth RATE

Set the shaper bandwidth. See tc(8) or examples below for details of the RATE value.

autorate-ingress

Automatic capacity estimation based on traffic arriving at this qdisc. This is most likely to be useful with cellular links, which tend to change quality randomly. A bandwidth parameter can be used in conjunction to specify an initial estimate. The shaper will periodically be set to a bandwidth slightly below the estimated rate. This estimator cannot estimate the bandwidth of links downstream of itself.

## OVERHEAD COMPENSATION PARAMETERS
The size of each packet on the wire may differ from that seen by Linux. The following parameters allow CAKE to compensate for this difference by internally considering each packet to be bigger than Linux informs it. To assist users who are not expert network engineers, keywords have been provided to represent a number of common link technologies.

### Manual Overhead Specification
overhead BYTES

Adds BYTES to the size of each packet. BYTES may be negative; values between -64 and 256 (inclusive) are accepted.

mpu BYTES

Rounds each packet (including overhead) up to a minimum length BYTES. BYTES may not be negative; values between 0 and 256 (inclusive) are accepted.

atm

Compensates for ATM cell framing, which is normally found on ADSL links. This is performed after the overhead parameter above. ATM uses fixed 53-byte cells, each of which can carry 48 bytes payload.

ptm

Compensates for PTM encoding, which is normally found on VDSL2 links and uses a 64b/65b encoding scheme. It is even more efficient to simply derate the specified shaper bandwidth by a factor of 64/65 or 0.984. See ITU G.992.3 Annex N and IEEE 802.3 Section 61.3 for details.

noatm

Disables ATM and PTM compensation.

### Failsafe Overhead Keywords
These two keywords are provided for quick-and-dirty setup. Use them if you can't be bothered to read the rest of this section.

raw (default)

Turns off all overhead compensation in CAKE. The packet size reported by Linux will be used directly. Other overhead keywords may be added after "raw". The effect of this is to make the overhead compensation operate relative to the reported packet size, not the underlying IP packet size.

conservative

Compensates for more overhead than is likely to occur on any widely-deployed link technology. Equivalent to overhead 48 atm.

### ADSL Overhead Keywords
Most ADSL modems have a way to check which framing scheme is in use. Often this is also specified in the settings document provided by the ISP. The keywords in this section are intended to correspond with these sources of information. All of them implicitly set the atm flag.

pppoa-vcmux

Equivalent to overhead 10 atm

pppoa-llc

Equivalent to overhead 14 atm

pppoe-vcmux

Equivalent to overhead 32 atm

pppoe-llcsnap

Equivalent to overhead 40 atm

bridged-vcmux

Equivalent to overhead 24 atm

bridged-llcsnap

Equivalent to overhead 32 atm

ipoa-vcmux

Equivalent to overhead 8 atm

ipoa-llcsnap

Equivalent to overhead 16 atm

See also the Ethernet Correction Factors section below.

### VDSL2 Overhead Keywords
ATM was dropped from VDSL2 in favour of PTM, which is a much more straightforward framing scheme. Some ISPs retained PPPoE for compatibility with their existing back-end systems.

pppoe-ptm

Equivalent to overhead 30 ptm PPPoE: 2B PPP + 6B PPPoE +

ETHERNET: 6B dest MAC + 6B src MAC + 2B ethertype + 4B Frame Check Sequence +

PTM: 1B Start of Frame (S) + 1B End of Frame (Ck) + 2B TC-CRC (PTM-FCS)

bridged-ptm

Equivalent to overhead 22 ptm ETHERNET: 6B dest MAC + 6B src MAC + 2B ethertype + 4B Frame Check Sequence +

PTM: 1B Start of Frame (S) + 1B End of Frame (Ck) + 2B TC-CRC (PTM-FCS)

See also the Ethernet Correction Factors section below.

### DOCSIS Cable Overhead Keyword
DOCSIS is the universal standard for providing Internet service over cable-TV infrastructure.

In this case, the actual on-wire overhead is less important than the packet size the head-end equipment uses for shaping and metering. This is specified to be an Ethernet frame including the CRC (aka FCS).

docsis

Equivalent to overhead 18 mpu 64 noatm

### Ethernet Overhead Keywords
ethernet

Accounts for Ethernet's preamble, inter-frame gap, and Frame Check Sequence. Use this keyword when the bottleneck being shaped for is an actual Ethernet cable. Equivalent to overhead 38 mpu 84 noatm

ether-vlan

Adds 4 bytes to the overhead compensation, accounting for an IEEE 802.1Q VLAN header appended to the Ethernet frame header. NB: Some ISPs use one or even two of these within PPPoE; this keyword may be repeated as necessary to express this.

## ROUND TRIP TIME PARAMETERS
Active Queue Management (AQM) consists of embedding congestion signals in the packet flow, which receivers use to instruct senders to slow down when the queue is persistently occupied. CAKE uses ECN signalling when available, and packet drops otherwise, according to a combination of the Codel and BLUE AQM algorithms called COBALT.

Very short latencies require a very rapid AQM response to adequately control latency. However, such a rapid response tends to impair throughput when the actual RTT is relatively long. CAKE allows specifying the RTT it assumes for tuning various parameters. Actual RTTs within an order of magnitude of this will generally work well for both throughput and latency management.

At the 'lan' setting and below, the time constants are similar in magnitude to the jitter in the Linux kernel itself, so congestion might be signalled prematurely. The flows will then become sparse and total throughput reduced, leaving little or no back-pressure for the fairness logic to work against. Use the "metro" setting for local lans unless you have a custom kernel.

rtt TIME

Manually specify an RTT.

datacentre

For extremely high-performance 10GigE+ networks only.

Equivalent to rtt 100us.

lan

For pure Ethernet (not Wi-Fi) networks, at home or in the office. Don't use this when shaping for an Internet access link.

Equivalent to rtt 1ms.

metro

For traffic mostly within a single city.

Equivalent to rtt 10ms.

regional

For traffic mostly within a European-sized country.

Equivalent to rtt 30ms.

internet (default)

This is suitable for most Internet traffic.

Equivalent to rtt 100ms.

oceanic

For Internet traffic with generally above-average latency, such as that suffered by Australasian residents.

Equivalent to rtt 300ms.

satellite

For traffic via geostationary satellites.

Equivalent to rtt 1000ms.

interplanetary

So named because Jupiter is about 1 light-hour from Earth. Use this to (almost) completely disable AQM actions.

Equivalent to rtt 3600s.

## FLOW ISOLATION PARAMETERS
With flow isolation enabled, CAKE places packets from different flows into different queues, each of which carries its own AQM state. Packets from each queue are then delivered fairly, according to a DRR++ algorithm which minimizes latency for "sparse" flows. CAKE uses a set-associative hashing algorithm to minimize flow collisions.

These keywords specify whether fairness based on source address, destination address, individual flows, or any combination of those is desired.

flowblind

Disables flow isolation; all traffic passes through a single queue for each tin.

srchost

Flows are defined only by source address. Could be useful on the egress path of an ISP backhaul.

dsthost

Flows are defined only by destination address. Could be useful on the ingress path of an ISP backhaul.

hosts

Flows are defined by source-destination host pairs. This is host isolation, rather than flow isolation.

flows

Flows are defined by the entire 5-tuple of source address, destination address, transport protocol, source port and destination port. This is the type of flow isolation performed by SFQ and fq_codel.

dual-srchost

Flows are defined by the 5-tuple, and fairness is applied first over source addresses, then over individual flows. Good for use on egress traffic from a LAN to the internet, where it'll prevent any one LAN host from monopolising the uplink, regardless of the number of flows they use.

dual-dsthost

Flows are defined by the 5-tuple, and fairness is applied first over destination addresses, then over individual flows. Good for use on ingress traffic to a LAN from the internet, where it'll prevent any one LAN host from monopolising the downlink, regardless of the number of flows they use.

triple-isolate (default)

Flows are defined by the 5-tuple, and fairness is applied over source *and* destination addresses intelligently (ie. not merely by host-pairs), and also over individual flows. Use this if you're not certain whether to use dual-srchost or dual-dsthost; it'll do both jobs at once, preventing any one host on *either* side of the link from monopolising it with a large number of flows.

nat

Instructs Cake to perform a NAT lookup before applying flow-isolation rules, to determine the true addresses and port numbers of the packet, to improve fairness between hosts "inside" the NAT. This has no practical effect in "flowblind" or "flows" modes, or if NAT is performed on a different host.

nonat (default)

Cake will not perform a NAT lookup. Flow isolation will be performed using the addresses and port numbers directly visible to the interface Cake is attached to.

## PRIORITY QUEUE PARAMETERS
CAKE can divide traffic into "tins" based on the Diffserv field. Each tin has its own independent set of flow-isolation queues, and is serviced based on a WRR algorithm. To avoid perverse Diffserv marking incentives, tin weights have a "priority sharing" value when bandwidth used by that tin is below a threshold, and a lower "bandwidth sharing" value when above. Bandwidth is compared against the threshold using the same algorithm as the deficit-mode shaper.

Detailed customisation of tin parameters is not provided. The following presets perform all necessary tuning, relative to the current shaper bandwidth and RTT settings.

besteffort

Disables priority queuing by placing all traffic in one tin.

precedence

Enables legacy interpretation of TOS "Precedence" field. Use of this preset on the modern Internet is firmly discouraged.

diffserv4

Provides a general-purpose Diffserv implementation with four tins: • Bulk (CS1, LE in kernel v5.9+), 6.25% threshold, generally low priority.

• Best Effort (general), 100% threshold.

• Video (AF4x, AF3x, CS3, AF2x, CS2, TOS4, TOS1), 50% threshold.

• Voice (CS7, CS6, EF, VA, CS5, CS4), 25% threshold.

diffserv3 (default)

Provides a simple, general-purpose Diffserv implementation with three tins: • Bulk (CS1, LE in kernel v5.9+), 6.25% threshold, generally low priority.

• Best Effort (general), 100% threshold.

• Voice (CS7, CS6, EF, VA, TOS4), 25% threshold, reduced Codel interval.

fwmark MASK

This options turns on fwmark-based overriding of CAKE's tin selection. If set, the option specifies a bitmask that will be applied to the fwmark associated with each packet. If the result of this masking is non-zero, the result will be right-shifted by the number of least-significant unset bits in the mask value, and the result will be used as a the tin number for that packet. This can be used to set policies in a firewall script that will override CAKE's built-in tin selection.

## OTHER PARAMETERS
ingress

Indicates that CAKE is running in ingress mode (i.e. running on the downlink of a connection). This changes the shaper to also count dropped packets as data transferred, as these will have already traversed the link before CAKE can choose what to do with them. In addition, the AQM will be tuned to always keep at least two packets queued per flow. The reason for this is that retransmits are more expensive in ingress mode, since dropped packets have to traverse the link again; thus, keeping a minimum number of packets queued will improve throughput in cases where the number of active flows are so large that they saturate the link even at their minimum window size.

memlimit LIMIT

Limit the memory consumed by Cake to LIMIT bytes. Note that this does not translate directly to queue size (so do not size this based on bandwidth delay product considerations, but rather on worst case acceptable memory consumption), as there is some overhead in the data structures containing the packets, especially for small packets. By default, the limit is calculated based on the bandwidth and RTT settings.

wash

Traffic entering your diffserv domain is frequently mis-marked in transit from the perspective of your network, and traffic exiting yours may be mis-marked from the perspective of the transiting provider. Apply the wash option to clear all extra diffserv (but not ECN bits), after priority queuing has taken place.

If you are shaping inbound, and cannot trust the diffserv markings (as is the case for Comcast Cable, among others), it is best to use a single queue "besteffort" mode with wash.

split-gso

This option controls whether CAKE will split General Segmentation Offload (GSO) super-packets into their on-the-wire components and dequeue them individually. Super-packets are created by the networking stack to improve efficiency. However, because they are larger they take longer to dequeue, which translates to higher latency for competing flows, especially at lower bandwidths. CAKE defaults to splitting GSO packets to achieve the lowest possible latency. At link speeds higher than 10 Gbps, setting the no-split-gso parameter can increase the maximum achievable throughput by retaining the full GSO packets.

## OVERRIDING CLASSIFICATION WITH TC FILTERS
CAKE supports overriding of its internal classification of packets through the tc filter mechanism. Packets can be assigned to different priority tins by setting the priority field on the skb, and the flow hashing can be overridden by setting the classid parameter.

### Tin override
To assign a priority tin, the major number of the priority field needs to match the qdisc handle of the cake instance; if it does, the minor number will be interpreted as the tin index. For example, to classify all ICMP packets as 'bulk', the following filter can be used:

```
# tc qdisc replace dev eth0 handle 1: root cake diffserv3
# tc filter add dev eth0 parent 1: protocol ip prio 1 \

  u32 match icmp type 0 0 action skbedit priority 1:1
```

### Flow hash override
To override flow hashing, the classid can be set. CAKE will interpret the major number of the classid as the host hash used in host isolation mode, and the minor number as the flow hash used for flow-based queueing. One or both of those can be set, and will be used if the relevant flow isolation parameter is set (i.e., the major number will be ignored if CAKE is not configured in hosts mode, and the minor number will be ignored if CAKE is not configured in flows mode).

This example will assign all ICMP packets to the first queue:

```
# tc qdisc replace dev eth0 handle 1: root cake
# tc filter add dev eth0 parent 1: protocol ip prio 1 \

  u32 match icmp type 0 0 classid 0:1
```

If only one of the host and flow overrides is set, CAKE will compute the other hash from the packet as normal. Note, however, that the host isolation mode works by assigning a host ID to the flow queue; so if overriding both host and flow, the same flow cannot have more than one host assigned. In addition, it is not possible to assign different source and destination host IDs through the override mechanism; if a host ID is assigned, it will be used as both source and destination host.

## EXAMPLES

```
# tc qdisc delete root dev eth0
# tc qdisc add root dev eth0 cake bandwidth 100Mbit ethernet
# tc -s qdisc show dev eth0
qdisc cake 1: root refcnt 2 bandwidth 100Mbit diffserv3 triple-isolate rtt 100.0ms noatm overhead 38 mpu 84

 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)

 backlog 0b 0p requeues 0

 memory used: 0b of 5000000b

 capacity estimate: 100Mbit

 min/max network layer size:        65535 /       0

 min/max overhead-adjusted size:    65535 /       0

 average network hdr offset:            0

                   Bulk  Best Effort        Voice

  thresh       6250Kbit      100Mbit       25Mbit

  target          5.0ms        5.0ms        5.0ms

  interval      100.0ms      100.0ms      100.0ms

  pk_delay          0us          0us          0us

  av_delay          0us          0us          0us

  sp_delay          0us          0us          0us

  pkts                0            0            0

  bytes               0            0            0

  way_inds            0            0            0

  way_miss            0            0            0

  way_cols            0            0            0

  drops               0            0            0

  marks               0            0            0

  ack_drop            0            0            0

  sp_flows            0            0            0

  bk_flows            0            0            0

  un_flows            0            0            0

  max_len             0            0            0

  quantum           300         1514          762
```

### After some use:

```
# tc -s qdisc show dev eth0
qdisc cake 1: root refcnt 2 bandwidth 100Mbit diffserv3 triple-isolate rtt 100.0ms noatm overhead 38 mpu 84

 Sent 44709231 bytes 31931 pkt (dropped 45, overlimits 93782 requeues 0)

 backlog 33308b 22p requeues 0

 memory used: 292352b of 5000000b

 capacity estimate: 100Mbit

 min/max network layer size:           28 /    1500

 min/max overhead-adjusted size:       84 /    1538

 average network hdr offset:           14

                   Bulk  Best Effort        Voice

  thresh       6250Kbit      100Mbit       25Mbit

  target          5.0ms        5.0ms        5.0ms

  interval      100.0ms      100.0ms      100.0ms

  pk_delay        8.7ms        6.9ms        5.0ms

  av_delay        4.9ms        5.3ms        3.8ms

  sp_delay        727us        1.4ms        511us

  pkts             2590        21271         8137

  bytes         3081804     30302659     11426206

  way_inds            0           46            0

  way_miss            3           17            4

  way_cols            0            0            0

  drops              20           15           10

  marks               0            0            0

  ack_drop            0            0            0

  sp_flows            2            4            1

  bk_flows            1            2            1

  un_flows            0            0            0

  max_len          1514         1514         1514

  quantum           300         1514          762
```

## SEE ALSO
tc(8), tc-codel(8), tc-fq_codel(8), tc-htb(8)

## AUTHORS
Cake's principal author is Jonathan Morton, with contributions from Tony Ambardar, Kevin Darbyshire-Bryant, Toke Høiland-Jørgensen, Sebastian Moeller, Ryan Mounce, Dean Scarff, Nils Andreas Svee, and Dave Täht.

This manual page was written by Loganaden Velvindron. Please report corrections to the Linux Networking mailing list <netdev@vger.kernel.org>.


tc-cbs

title: tc-cbs(8)

summary: Credit Based Shaper (CBS) Qdisc

text:
# tc-cbs(8)

## NAME
CBS - Credit Based Shaper (CBS) Qdisc

## SYNOPSIS
tc qdisc ... dev dev parent classid [ handle major: ] cbs idleslope idleslope sendslope sendslope hicredit hicredit locredit locredit [ offload 0|1 ]

## DESCRIPTION
The CBS (Credit Based Shaper) qdisc implements the shaping algorithm defined by the IEEE 802.1Q-2014 Section 8.6.8.2, which applies a well defined rate limiting method to the traffic.

This queueing discipline is intended to be used by TSN (Time Sensitive Networking) applications, the CBS parameters are derived directly by what is described by the Annex L of the IEEE 802.1Q-2014 Specification. The algorithm and how it affects the latency are detailed there.

CBS is meant to be installed under another qdisc that maps packet flows to traffic classes, one example is mqprio(8).

## PARAMETERS
idleslope

Idleslope is the rate of credits that is accumulated (in kilobits per second) when there is at least one packet waiting for transmission. Packets are transmitted when the current value of credits is equal or greater than zero. When there is no packet to be transmitted the amount of credits is set to zero. This is the main tunable of the CBS algorithm and represents the bandwidth that will be consumed. Note that when calculating idleslope, the entire packet size must be considered, including headers from all layers (i.e. MAC framing and any overhead from the physical layer), as described by IEEE 802.1Q-2014 section 34.4. As an example, for an ethernet frame carrying 284 bytes of payload, and with no VLAN tags, you must add 14 bytes for the Ethernet headers, 4 bytes for the Frame check sequence (CRC), and 20 bytes for the L1 overhead: 12 bytes of interpacket gap, 7 bytes of preamble and 1 byte of start of frame delimiter. That results in 322 bytes for the total packet size, which is then used for calculating the idleslope.

sendslope

Sendslope is the rate of credits that is depleted (it should be a negative number of kilobits per second) when a transmission is occurring. It can be calculated as follows, (IEEE 802.1Q-2014 Section 8.6.8.2 item g): sendslope = idleslope - port_transmit_rate

hicredit

Hicredit defines the maximum amount of credits (in bytes) that can be accumulated. Hicredit depends on the characteristics of interfering traffic, 'max_interference_size' is the maximum size of any burst of traffic that can delay the transmission of a frame that is available for transmission for this traffic class, (IEEE 802.1Q-2014 Annex L, Equation L-3): hicredit = max_interference_size * (idleslope / port_transmit_rate)

locredit

Locredit is the minimum amount of credits that can be reached. It is a function of the traffic flowing through this qdisc (IEEE 802.1Q-2014 Annex L, Equation L-2): locredit = max_frame_size * (sendslope / port_transmit_rate)

offload

When offload is 1, cbs(8) will try to configure the network interface so the CBS algorithm runs in the controller. The default is 0.

## EXAMPLES
CBS is used to enforce a Quality of Service by limiting the data rate of a traffic class, to separate packets into traffic classes the user may choose mqprio(8), and configure it like this:

```
# tc qdisc add dev eth0 handle 100: parent root mqprio num_tc 3 \
	map 2 2 1 0 2 2 2 2 2 2 2 2 2 2 2 2 \
	queues 1@0 1@1 2@2 \
	hw 0
```

To replace the current queuing disciple by CBS in the current queueing discipline connected to traffic class number 0, issue:

```
# tc qdisc replace dev eth0 parent 100:4 cbs \
	locredit -1470 hicredit 30 sendslope -980000 idleslope 20000
```

These values are obtained from the following parameters, idleslope is 20mbit/s, the transmission rate is 1Gbit/s and the maximum interfering frame size is 1500 bytes.

## AUTHORS
Vinicius Costa Gomes <vinicius.gomes@intel.com>