vzstat(8)

System Manager's Manual

vzstat(8)

NAME

vzstat - top-like utility for gathering statistics

SYNOPSIS

vzstat [-l] [-d X] [-p CTID] [-b|-v] [-t] [-m] [-a] [-o|-O|-i|-n] [-c<B|K|M|G>]

DESCRIPTION

The vzstat utility is a top-like utility for real-time monitoring in OpenVZ. It displays the current system state and load, lists most resource - consuming Containers, and can provide an interactive interface for displaying parameters. It also can sort Containers by a number of parameters.

COMMAND-LINE OPTIONS

-l

Batch mode. Prints the information once and exits immediately if no delay time (-d option) is specified. Otherwise, continues looping with the specified delay between the cycles.

-d delay

Specifies the delay time between screen updates. In the screen view mode, you can change this with the t interactive command. The default delay time is 1 sec.

-p CTID

Monitor only Containers with the specified IDs. This flag can be given up to twenty times, in form -p CTID1 -p CTID2 .... This option is not available interactively.

-b

"Brief" mode. Minimal detail level. Shows only one summary line about each monitoring subsystem. By default, the "standard" detail level is used. Valid levels are "brief", "standard" and "verbose". Can be set on the fly using the b interactive command. See also the -v command-line option and the s and v interactive commands.

-t

Text mode. Prints the information tersely, in a form suitable for parsing by other programs. The output data is neither aligned nor colored and numbers are not displayed in a human readable format. If no delay time (-d option) is specified, prints the information once and exits immediately. Otherwise, continues looping with the specified delay time between the cycles.

-v

"Verbose" mode. Provides maximum details about all monitored subsystems. Can be set on the fly using v interactive command. See also the -b command-line option and the b and s interactive commands.

-m

Display disc statistics for all fs types (by default only for ext2, ext3, ext4, reiserfs).

-o format

Set user-defined format. format is a single argument in the form of comma-separated list, which offers a way to specify individual output columns. The recognized keywords are described in the OUTPUT section below (see CODE in per Container info). Headers may be renamed (vzstat -o st,vm=VM) as desired or may be empty (vzstat -o st=,ior=). The column width can also be specified after the colon (-o name:12,host=HOST:20).

-O format

is like -o, but preloaded with some default columns. Identical to -o id,st,format,ip

-i

Show IO statistics. Identical to -o id,st,ior,iow,iort,iowt,ip

IO statistics display may be toggled by the I interactive command.

-n

Show network statistics. Identical to -o id,st,rx,tx,rxt,txt,rxpt,txpt,ip

Network statistics display may be toggled by the X interactive command.

-s key

Sort per-Container statistics by key. The recognized keywords are described in the OUTPUT section below (see CODE in per Container info). Sort keywords are identical to keywords for the -o option.

-a

Display disk input and output (I/O) statistics for Containers.

-c <B|K|M|G>

Scale output to the specified units.

OUTPUT

vzstat displays a variety of information about the OpenVZ system state. The display is updated every 1 second by default, but you can change that with the -d command-line option or the t interactive command.

uptime: This line displays the time the system has been up, and the three load averages for the system. The load averages are the average number of process ready to run during the last 1, 5 and 15 minutes. This line is just like the output of uptime(1). For example: "1:22am, up 1:31, 2 users, load average: 0.00, 0.06, 0.33". The uptime display may be toggled by the interactive l command.
Containers and processes: The total number of Containers and processes running at the time of the last update. This is also broken down into the number of tasks which are running, sleeping, uninterruptible, zombie, stopped and dead. For example: "CTNum 102, procs 467: R 12, S 455, D 0, Z 0, T 0, X 1". The processes and states display may be toggled by the p interactive command.
CPU states: Shows the percentage of CPU time used by all Containers (except for CT0) and by CT0, CPU time usage in user mode, system mode, idle and max/avg scheduling latency in ms. Scheduling latency is time spent by the processes in the system awaiting for scheduling. For example: "CPU [ OK ]: CTs 43%, CT0 12%, user 41%, sys 13%, idle 45%, lat(ms) 3/2". The CPU usage display may be toggled by the c interactive command.
Mem: Statistics about memory usage, including the total available memory, free memory and max/avg memory allocation latency. Free memory is displayed both for low and high memory. Low memory is the sum of DMA and Normal zones memory and high memory is the High zone memory. Allocation latency is the time required to allocate memory inside the kernel in milliseconds. A long allocation latency can be a sign of node's overload. For example: "Mem [ OK ]: total 755MB, free 671MB/0MB (low/high), lat(ms) 10/7". That means that the maximum allocation latency is 10 and the average is 7. The display of memory information may be toggled by the m, M interactive commands.
Memory zones information: Information about memory zone state. This information includes: total memory zone size in MB, size of active and inactive lists, free memory and zone limits. For example: "ZONE1 (Normal): size 752MB, act 29MB, inact 31MB, free 658MB (0/1/2)". The display of memory information may be toggled by the m, M interactive commands.
Memory zones fragmentation: Information about memory zones fragmentation. This information describes how much system memory is fragmented and which is the biggest block size possible to allocate atomically. For example: "fragm 2*1 3*2 15*4 22*8 25*16 12*32 4*64 0*128 1*256 326*512". The first number before * is the number of blocks and the second one is the block size in pages. The display of memory information may be toggled by the m, M interactive commands.
Memory allocation latency: Memory allocation latency is an average time spent in kernel memory allocator for different memory type requests. Memory type is coded as XY, where X is A for GFP_ATOMIC, K for GFP_KERNEL and U for GFP_USER and Y denotes allocation request order, i.e. Y=0 for order=0 and 1 for order=1. For example: "Mem lat (ms): A0 0, K0 0, U0 1, K1 3, U1 2". The display of memory information may be toggled by the m, M interactive commands.
Slab cache information: Slab cache information includes the total slab cache size/real cache size, inode cache size, dentry cache size, buffer heads cache size, and page beancounters cache size. Real cache size is the size the cache can be shrunk to; it's always less than the total cache size. For example: "Slab pages: 13MB/13MB (ino 8MB, de 1MB, bh 1MB, pb 0MB)". The display of memory information may be toggled by the m, M interactive commands.
Swap: Statistics on swap space, including total swap space, available swap space and swap in/swap out in MB/s. For example: "Swap [ OK ]: tot 1004MB, free 1004MB, in 0.000MB/s, out 0.000MB/s". The display of swap information may be toggled by the w, W interactive commands.
Swap latency: Swap operations latency. This includes swap in count, swap in max/avg latency in ms, swap out count, swap out max/avg latency in ms and max/avg cpu time spent in swap out. For example: "Swap lat: si 0, 0/0 ms, so 0, 0/0 ms, 0/0 cpu ms". The display of swap information may be toggled by the w, W interactive commands.
Swap cache: Swap cache information includes: number of adds to swap cache, dels from swap cache and finds. For example: "Swap cache: add 0, del 0, find 0/0". The display of swap information may be toggled by the w, W interactive commands.
Network information: Network statistics summary. Total incoming traffic in MB/s and incoming packets/s, outcoming traffic in MB/s and packets/s. For example: "Net [ OK ]: tot in 1.020MB/s 267pkt/s, out 0.001MB/s 1pkt/s". The display of network information may be toggled by the n, N interactive commands.
Network interface information: Network interface statistics summary. Total incoming traffic in MB/s and incoming packets/s, outcoming traffic in MB/s and packets/s. For example: "eth0: in 0.000MB/s 3pkt/s, out 0.001MB/s 1pkt/s". The display of network information may be toggled by the n, N interactive commands. Note: veth interfaces statistics is shown only in text mode, i.e. when the -l or -t option is specified.
Disks statistics: Summary disks statistics. Reads in MB/s and writes in MB/s. For example: "Disks [ OK ]: in 0.000MB/s, out 0.000MB/s". The display of disk information may be toggled by the d, D interactive commands.
Mounted disks statistics: Information on mounted disks, such as mount points, free space, and the number of free inodes left on the device. For example: "root(/) free: 3489MB(46%), 511077ino(52%)". The display of disk information may be toggled by the d, D interactive commands.
Disk I/O statistics: Show disk input and output statistics for Containers. The following information is shown: * IOUSED%: the percentage of time the disks are used by the Container. * IOWAIT%: the percentage of time when at least one I/O transaction in the Container is waiting for being served. * IOSPEED: the current speed of disk I/O operations in the Container and the I/O limit set for this Container, if any. The value can be displayed in bytes, kilobytes, megabytes, or gigabytes per second, depending on the units you used to set the I/O limit. The display of disk information may be toggled by the A interactive command.
per-Container info: Per-Container info is a list of Containers with some information about them like IDs, memory usage, processes, CPU usage, socket usage, fail counters, IP addresses. The columns to be shown can be specified using the -o option (see above). The following columns can be displayed for Containers:

CODE

HEADER DESCRIPTION

id

CTID Container ID.

st

ST Container status. OK if Container latency and fail counters are good; otherwise, red "!!". Your can sort Containers by their statuses, displaying Containers with the "failed" status first.

vm

%VM Virtual memory usage, i.e. UBC privvmpages parameter (in percentage from total memory). The first number is how much privvmpages are being held (i.e. actual usage), the second is privvmpages barrier. For example: 5%/50% means that Container in question holds 5% of pages (of total memory) for privvmpages and can use even more, up to 50% of total memory pages.

When a Container is running in SLM mode, privvmpages barrier has no fixed value, because it's calculated dynamically in the kernel, and this value depends on the current memory usage, Container activity, SLM settings and UBC limits for this Container. In SLM mode, privvmpages barrier for a Container is usually a little bit bigger than the usage, so this real value is not very useful and marked by '-'.

sw

%SW Virtual swap usage, i.e. UBC swappages parameter. The first parameter is percent of virtual swap usage, the second is the size of the virtual swap configured.

km

%KM Kernel memory usage, i.e. the UBC kmemsize parameter (in percentage from normal zone size). The first parameter is how much kmemsize is being used, the seconds is kmemsize barrier. For example: 3%/7% means that the Container in question holds 3% of the normal zone for kernel memory and can use even more, up to 7% of the normal zone.

When SLM module is loaded, kmemsize barrier has no fixed value, and like privvmpages barrier it's marked by '-'.

proc

PROC Process information, in the format running tasks/total tasks/barrier. For example: 3/20/100 means that 3 processes are running, there are 20 processes in total, and at most 100 processes can be spawned in the Container in total. You can sort Containers by the number of total or running tasks.

cpu

CPU CPU usage. The actual CPU usage and the requested CPU usage are printed here. The first number is the actual CPU usage by a Container. When running on more than 1 CPU (i.e. on SMP host), all the processors are considered as 100%. The seconds number is the guaranteed CPU usage according to the assigned cpuunits weight. For example: 3%/5%, 99%/5%. Note that the actual CPU usage can be higher than the guaranteed one.

sock

SOCK Sockets usage, i.e. the sum of UBC numtcpsock and numothersock parameters. The first parameter is how much sockets are opened, the seconds is the sockets barrier. For example: 12/100 means that the Container in question holds 12 sockets and can open more sockets, but it is limited to 100 sockets.

fcnt

FCNT Fail counters, i.e. the sum of UBC fail counters for all resources. The FCNT parameter is printed differently in average and standart modes. In standart mode fail counters delta is printed, i.e. the increase of failcounter sum since the last screen update. If average mode is on, the absolute fail counters sum is printed.

mlat

MLAT Maximum scheduling latency for a Container, in ms. This parameter shows the maximum process scheduling latency inside the Container, i.e. how long (maximum) a process inside Container awaits for the CPU.

name

NAME Container name.

io

IO I/O rate and I/O limit in MBytes per second.

iops

IOPS The current rate of I/O operations in a Container and the limit set for the Container, calculated per second.

iow

IOW total number of bytes that were written to all block devices (IDE, SCSI, NFS ...) in MBytes per second.

ior

IOR total number of bytes that were read from all block devices (IDE, SCSI, NFS ...) in MBytes per second.

iowt

IOWT total number of bytes that were written to all block devices (IDE, SCSI, NFS ...) in MBytes.

iort

IORT total number of bytes that were read from all block devices (IDE, SCSI, NFS ...) in MBytes.

ios

IOS sync rate (MB/sec)

iod

IOD dirty pages rate (MB/sec). This value shows how many memory pages are dirty. Dirty pages are those pages that have data not yet flushed to the disk.

cpu_iowait

CPU IOWAIT current CPU IO wait in percent

cpu_iowait_acc

ACC IOWAIT accumulated CPU IO wait in percent

swapin

SWAPIN Number of I/O operations with swap in a Container, per second.

Per-Container network related codes.

Total network incoming/outgoing traffic statistics include information about all defined network classes. The network classes are described in the /etc/vz/conf/networks_classes file. See networks_classes (5) man page for more details.

rx: RX received bytes rate (MB/sec).
tx: TX transmitted bytes rate (MB/sec).
rxt: RXT received bytes total (MB).
txt: TXT transmitted bytes total (MB).
rxp: RXP received packets rate (packets/sec).
txp: TXP transmitted packets rate (packets/sec).
rxpt: RXPT received packets total.
txpt: TXPT transmitted packets total.

INTERACTIVE KEYS

Several single-keys commands are recognized while vzstat is running. Interactive commands are not available if vzstat is executed with the -t or -l command-line option.

h, ?: Print help usage.
space: Update the display immediately.
q: Quit.
t: Change the delay between screen updates. You will be prompted to enter a new delay time, in seconds. Entering 0 causes continuous updates. See also the -d command-line parameter.
r: Redraw full screen from scratch.
b: Set "brief" detail level. See also the -b command-line parameter.
s: Set"normal" detail level.
v: Set"verbose" detail level. See also the -v command-line parameter.
a: "Averaged" mode. The average (for a minute) for monitoring parameters is shown. This includes:

: Number of uninterruptable processes
Scheduling max latency
Memory allocation max latency
Size of free/active/inactive memory
Swap-in latency
UBC fail counters absolute values

i: Toggle display of idle Containers.
l: Toggle display of load average.
p: Toggle display of processes statistics.
c: Toggle display of CPU usage statistics.
e: Toggle display of hostname/ip information.
m, M: Toggle/expand the display of memory information. Each subsystem, including memory, network, and disk, has a number of verbosity levels. On the minimal level, no information is displayed. Interactive lowercase keys decrease the verbosity level, the same keys in uppercase increase it.
n, N: Toggle/expand display of network statistics.
d, D: Toggle/expand display of disk usage and activity information.
w, W: Toggle/expand display of swap information.
I: Switch to IO accounting column set.
X: Switch to network statistics column set.
A: Switch to IO statistics column set.
0: Toggle display of CT0 statistics.
o: Sort key. One of the sort option keys should follow it:

n: Sort by CTID
t: Sort by Container status
v: Sort by virtual memory usage
k: Sort by kernel memory usage
r: Sort by the number of running processes
p: Sort by the total number of processes
c: Sort by CPU usage
s: Sort by number of open sockets
f: Sort by UBC failure counters
m: Sort by memory latency

VZSTAT CONFIGURATION FILES

The vzstat configuration file is located in /etc/vz/vzstat.conf. The parameter values in this file are highlighting thresholds for the corresponding monitoring parameters. If the current value exceeds the warning level, the parameter is shown in yellow. If the value exceeds the error level, the parameter is shown in red. Some parameters, however, have one threshold only. All the parameters are numerical and specified in the form PARAMETER=NNN.

LOAD_AVG: Load average. Warning level. If exceeded, the parameter is shown is yellow.
PROC_RUN: Number of running processes. Warning level. If exceeded, the parameter is shown is yellow.
PROC_UNINT: Number of uninterruptable processes (in ´D´ state). Warning level. If exceeded, the parameter is shown is yellow.
CPU_IDLE: CPU idle time, in percent. Warning level. If exceeded, the parameter is shown is yellow.
CPU_SYS: CPU system time, in percent. Warning level. If exceeded, the parameter is shown is yellow.
CPU_LAT_MAX_WARN: Scheduling latency, in milliseconds (maximum over 5 sec period). Warning level. If exceeded, the parameter is shown is yellow.
CPU_LAT_AVG_ERR: Scheduling latency, in milliseconds (maximum over 5 sec period). Error level. If exceeded, the parameter is shown in red.
CPU_LAT_AVG_WARN: Scheduling latency, in milliseconds (5 sec average). Warning level. If exceeded, the parameter is shown is yellow.
CPU_LAT_MAX_ERR: Scheduling latency, in milliseconds (5 sec average). Error level. If exceeded, the parameter is shown in red.
MEM_LAT_MAX_WARN: Memory allocation latency, in milliseconds (maximum over 5 sec period). Warning level. If exceeded, the parameter is shown is yellow.
MEM_LAT_MAX_ERR: Memory allocation latency, in milliseconds (maximum over 5 sec period). Error level. If exceeded, the parameter is shown in red.
MEM_LAT_AVG_WARN: Memory allocation latency, in milliseconds (5 sec average). Warning level. If exceeded, the parameter is shown is yellow.
MEM_LAT_AVG_ERR: Memory allocation latency, in milliseconds (5 sec average). Error level. If exceeded, the parameter is shown in red.
MEM_ZONE_ACT_INACT_FREE_WARN: Size of available memory (free + active + inactive pages), in percent. Warning level. If the current value is below this level, the parameter is shown is yellow.
MEM_ZONE_ACT_INACT_FREE_ERR: Size of available memory (free + active + inactive pages), in percent. Error level. If the current value is below this level, the parameter is shown in red.
MEM_ZONE_ACT_INACT_FREE_ABS_WARN: Size of available memory (free + active + inactive pages), in MB. Warning level. If the current value is below this level, the parameter is shown is yellow.
MEM_ZONE_ACT_INACT_FREE_ABS_ERR: Size of available memory (free + active + inactive pages), in MB. Error level. If the current value is below this level, the parameter is shown in red.
MEM_ZONE_ORDER_GT_0: Number of pages which are gathered in blocks with order > 0. For example, if current memory distribution looks like: 3*1 1*2 3*4 5*8 .... Then number of pages with order>0 is 1*2 + 3*4 + 5*8 + ... This sum is printed yellow if it is too low.
SWAP_FREE_WARN: Free swap space, in percent. Warning level. If exceeded, the parameter is shown is yellow.
SWAP_FREE_ERR: Free swap space, in percent. Error level. If the current value is below this level, the parameter is shown in red.
SWAP_IN_WARN: Swap-in activity, in Mb/sec. Warning level. If exceeded, the parameter is shown is yellow.
SWAP_IN_ERR: Swap-in activity, in Mb/sec. Error level. If exceeded, the parameter is shown in red.
SWAP_OUT_WARN: Swap-out activity, in Mb/sec. Warning level. If exceeded, the parameter is shown is yellow.
SWAP_OUT_ERR: Swap-out activity, in Mb/sec. Error level. If exceeded, the parameter is shown in red.
SWAP_LAT_MAX_WARN: Swap-in latency, in milliseconds (maximum over 5 sec period). Warning level. If exceeded, the parameter is shown is yellow.
SWAP_LAT_MAX_ERR: Swap-in latency, in milliseconds (maximum over 5 sec period). Error level. If exceeded, the parameter is shown in red.
SWAP_LAT_AVG_WARN: Swap-in latency, in milliseconds (5 sec average). Warning level. If exceeded, the parameter is shown is yellow.
SWAP_LAT_AVG_ERR: Swap-in latency, in milliseconds (5 sec average). Error level. If exceeded, the parameter is shown in red.
DISK_FREE_INODES_WARN: Free inodes on disk, in percent. Warning level. If the current value is below this level, the parameter is shown is yellow.
DISK_FREE_INODES_ERR: Free inodes on disk, in percent. Error level. If the current value is below this level, the parameter is shown in red.
DISK_FREE_SPACE_WARN: Free disk space, in percent. Warning level. If the current value is below this level, the parameter is shown is yellow.
DISK_FREE_SPACE_ERR: Free disk space, in percent. Error level. If the current value is below this level, the parameter is shown in red.
DISK_IO_LOAD_WARN: Disk IO load, in percent. Warning level. If exceeded, the parameter is shown is yellow.
DISK_IO_LOAD_ERR: Disk IO load, in percent. Error level. If exceeded, the parameter is shown in red.
CT_FAILCNT_DELTA: Number of failed UBC resource allocations for a Container between vzstat screen updates (any resource type is taken into account). Error level. If exceeded, the parameter is shown in red.

NOTES

vzstat requires superuser privileges to run.

COPYRIGHT

July 2015

OpenVZ

vzstat(8)

NAME

SYNOPSIS

DESCRIPTION

COMMAND-LINE OPTIONS

OUTPUT

INTERACTIVE KEYS

VZSTAT CONFIGURATION FILES

NOTES

SEE ALSO

COPYRIGHT