bcc-softirqslower(8)
| SOFTIRQSLOWER(8) | Linux Performance Analysis | SOFTIRQSLOWER(8) |
NAME
softirqslower - Trace softirq handlers with latency exceeding a threshold
SYNOPSIS
softirqslower.py [min_us]
softirqslower.py [-c CPU] [min_us]
DESCRIPTION
The softirqslower tool traces softirq events that exceed a specified latency threshold. This tool helps diagnose interrupt handling bottlenecks by measuring two critical latency dimensions:
- irq(hard) to softirq: Time delay between hardware interrupt completion and softirq execution start
- softirq runtime: Actual execution duration of softirq handlers
Developed as part of the BCC (BPF Compiler Collection) tools, it's particularly useful for network stack optimization, scheduler tuning, and real-time systems analysis.
OPTIONS
- min_us
- Minimum latency threshold to trace (in microseconds). Default: 10,000 μs (10 ms)
- -c, --cpu CPU
- Trace only events on the specified CPU core
OUTPUT FORMAT
The tool outputs events in the following table format:
TIME STAGE SOFTIRQ LAT(us) CPU COMM
With fields defined as:
- TIME
- Event timestamp (HH:MM:SS format)
- STAGE
- Latency measurement type:
- irq(hard) to softirq
- Time from hardware interrupt completion to softirq dispatch
- softirq runtime
- SoftIRQ handler execution duration
- SOFTIRQ
- SoftIRQ category (case-sensitive). Common values:
- NET_RX
- Network reception processing
- NET_TX
- Network transmission processing
- TIMER
- Timer callbacks
- SCHED
- Scheduler operations
- RCU
- Read-Copy-Update synchronization
- TASKLET
- Deferred task execution
- HRTIMER
- High-resolution timers
- BLOCK
- Block device operations
- LAT(us)
- Measured latency in microseconds
- CPU
- CPU core where softirq was handled (0-based numbering)
- COMM
- Process context handling the softirq:
- swapper/N
- Idle thread for CPU N
- ksoftirqd/N
- Softirq daemon for CPU N
- <process>
- User-space process
EXAMPLES
- Trace softirqs exceeding 10μs latency:
- softirqslower 10
- Monitor only CPU core 1:
- softirqslower -c 1
- Trace network-related softirq delays (>50μs):
- softirqslower 50 | grep -E 'net_rx|net_tx'
- Capture RCU delays longer than 100μs to file:
- softirqslower 100 | grep rcu > rcu_latency.log
SIGNALS
Ctrl+C Stop tracing and exit gracefully
AUTHOR
Chenyue Zhou
SEE ALSO
runqslower(8), hardirqs(8), softirqs(8), trace(8), funclatency(8),
REPORTING BUGS
BCC Tools Issue Tracker: https://github.com/iovisor/bcc/issues
| 2025-07-08 | BCC Tools |