lvmthin

title: lvmthin(7)

summary: LVM thin provisioning

text:
# lvmthin(7)

## NAME
lvmthin — LVM thin provisioning

## DESCRIPTION
Blocks in a standard lvm(8) Logical Volume (LV) are allocated when the LV is created, but blocks in a thin provisioned LV are allocated as they are written. Because of this, a thin provisioned LV has a virtual size that can be much larger than the available physical storage. The amount of physical storage provided for thin provisioned LVs can be increased later as the need arises.

Blocks in a standard LV are allocated (during creation) from the Volume Group (VG), but blocks in a thin LV are allocated (during use) from a "thin pool". The thin pool contains blocks of physical storage, and thin LV blocks reference blocks in the thin pool.

A special "thin pool LV" must be created before thin LVs can be created within it. A thin pool LV is created by combining two standard LVs: a data LV that will hold blocks for thin LVs, and a metadata LV that will hold metadata. Thin pool metadata is created and used by the "dm_thin_pool" kernel module to track the data blocks used by thin LVs.

Snapshots of thin LVs are efficient because the data blocks common to a thin LV and any of its snapshots are shared. Snapshots may be taken of thin LVs or of other thin snapshots. Blocks common to recursive snapshots are also shared in the thin pool. There is no limit to or degradation from sequences of snapshots.

As thin LVs or snapshot LVs are written to, they consume data blocks in the thin pool. As free data blocks in the pool decrease, more physical space may need to be added to the pool. This is done by extending the thin pool with additional physical space from the VG. Removing thin LVs or snapshots from the thin pool can also make more space available. However, removing thin LVs is not always an effective way of freeing space in a thin pool because blocks may be shared by snapshots, and free blocks may be too fragmented to make available.

On-demand block allocation can cause thin LV blocks to be fragmented in the thin pool, which can cause reduced performance compared to standard fully provisioned LV.

## DEFINITIONS
Thin LV

A thin LV is an LVM logical volume for which storage is allocated on demand. As a thin LV is written, blocks are allocated from a thin pool to hold the data. A thin LV has a virtual size that can be larger than the physical space in the thin pool.

Thin Pool

A thin pool is a special LV containing physical extents from which thin LVs are allocated. The thin pool LV is not used as a block device, but the thin pool name is referenced when creating thin LVs. The thin pool LV must be extended with additional physical extents before it runs out of space. A thin pool has two hidden component LVs: one for holding thin data and another for holding thin metadata.

Thin Pool Data LV

A component of a thin pool that holds thin LV data. The data LV is a hidden LV with a _tdata suffix, and is not used directly. The physical size of the data LV is displayed as the thin pool size.

Thin Pool Metadata LV

A component of a thin pool that holds metadata for the "dm_thin_pool" kernel module. "dm_thin_pool" generates and uses this metadata to track data blocks used by thin LVs. The metadata LV is a hidden LV with a _tmeta suffix, and is not used directly.

Thin Snapshot

A thin snapshot is a thin LV that is created in reference to an existing thin LV or other thin snapshot. The thin snapshot initially refers to the same blocks as the existing thin LV. It acts as a point in time copy of the thin LV it referenced.

External Origin

A read-only LV that is used as a snapshot origin for thin LVs. Unwritten portions of the thin LVs are read from the external origin.

## USAGE

### Thin Pool Creation
A thin pool can be created with the lvcreate command. The data and metadata component LVs are each allocated from the VG, and combined into a thin pool. The lvcreate -L|--size will be the size of the thin pool data LV, and the size of the metadata LV will be calculated automatically (or, can be optionally specified with --poolmetadatasize.)

```
# lvcreate --type thin-pool -n ThinPool -L Size VG
```

### Thin Pool Conversion
For a customized thin pool layout, data and metadata LVs can be created separately, and then combined into a thin pool with lvconvert. This allows specific LV types, or specific devices, to be used for data/metadata LVs. Combining the data and metadata LVs into a thin pool erases the content of both LVs. The resulting thin pool takes the name and size of the data LV. (If a metadata LV is not specified, lvconvert will automatically create one to use in the thin pool.)

```
# lvcreate -n DataLV -L Size VG DataDevices
# lvcreate -n MetadataLV -L MetadataSize VG MetadataDevices
# lvconvert --type thin-pool --poolmetadata MetadataLV VG/DataLV
```

(DataLV would now be referred to as ThinPool, and can be used for creating thin LVs.)

### Thin LV Creation
Thin LVs are created in a thin pool, and are created with a virtual size using the option -V|--virtualsize. The virtual size may be larger than the physical space available in the thin pool.

```
# lvcreate --type thin -n ThinLV -V VirtualSize --thinpool ThinPool VG
```

### Thin Snapshot Creation
Snapshots of thin LVs are thin LVs themselves, but the snapshot LV initially refers to the same blocks as the origin thin LV. The origin thin LV and its snapshot thin LVs will diverge as either are written. The origin thin LV can be removed without affecting snapshots that reference it. Snapshots can be taken of thin LVs that were themselves created as snapshots. (A size option must not be used when creating a thin snapshot, otherwise a COW snapshot will be created.)

```
# lvcreate --snapshot -n SnapLV VG/ThinLV
```

### Thin Pool Data Percent and Metadata Percent
For active thin pool LVs, the 'lvs' command displays "Data%" (-o data_percent) and "Meta%" (-o metadata_percent). Data percent is the percent of space in the data LV that is currently used by thin LVs. Metadata percent is the percent of space in the metadata LV that is currently used by the "dm_thin_pool" module. The thin pool should be extended before either of these values reach 100%.

```
# lvs -o data_percent VG/ThinPool
# lvs -o metadata_percent VG/ThinPool
```

### Thin Pool Extension
When lvextend is run on a thin pool, it will extend the internal data LV by the specified amount, and the internal metadata LV will also be extended, if needed, relative to the new data size.

```
# lvextend --size Size VG/ThinPool
```

A new metadata size can be requested when extending the thin pool data.

```
# lvextend --size Size --poolmetadatasize MetadataSize VG/ThinPool
```

The metadata size can be extended without extending the data size.

```
# lvextend --poolmetadatasize MetadataSize VG/ThinPool
```

The internal data or metadata LV can be extended by name.

```
# lvextend -L Size VG/ThinPool_tdata
# lvextend -L MetadataSize VG/ThinPool_tmeta
```

### Thin Pool Automatic Extension
It is important to extend a thin pool before it runs out of space, otherwise it may be damaged, and difficult or impossible to repair. LVM can be configured so that dmeventd(8) automatically extends thin pools when they run low on space. Free extents must be available in the VG to use for extending the thin pools.

dmeventd(8) is usually started by the lvm2-monitor service. dmeventd receives notifications from the kernel indicating when thin pool data or metadata are becoming full. In response, dmeventd runs the command "lvextend --use-policies VG/ThinPool", which compares the current usage of data and metadata with the autoextend threshold. The data LV and/or metadata LV may be extended in response. System messages will show when these extensions have happened.

To enable thin pool automatic extension, set lvm.conf(5):

activation/thin_pool_autoextend_threshold

Extend the thin pool when the current usage reaches this percentage. The chosen value should depend on the rate at which new data may be written. If space is consumed more quickly, then a lower threshold will provide dmeventd and lvextend more time to react and extend the pool. The minimum is 50. Setting to 100 disables autoextend.

activation/thin_pool_autoextend_percent

A thin pool is extended by this percent of its current size.

The thin pool itself must be monitored by dmeventd to be automatically extended. When activating a thin pool, lvm normally requests monitoring by dmeventd. To verify this, run:

```
# lvs -o+seg_monitor VG/ThinPool
```

To begin monitoring a thin pool in dmeventd:

```
# lvchange --monitor y VG/ThinPool
```

### Thin LV Activation
A thin LV that is created as a snapshot is given the "skip activation" property. It is reported with lvs -o skip_activation, or 'k' in the tenth lv_attr. This property causes vgchange -ay and lvchange -ay commands to skip activating the thin LV unless the -K|--ignoreactivationskip option is also set.

```
# lvchange -ay -K VG/SnapLV
```

The skip activation property on a thin LV can be cleared, so that -K is not required to activate it (or enabled so -K is required.)

```
# lvchange --setactivationskip y VG/SnapLV
# lvchange --setactivationskip n VG/SnapLV
```

To configure the "skip activation" setting that lvcreate applies to new snapshots, set lvm.conf(5) activation/auto_set_activation_skip

### Thick LV to Thin LV Conversion
A thick LV (e.g. linear, striped) can be converted to a thin LV in a new thin pool. The new thin pool is created using the existing thick LV as thin pool data. New thin pool metadata is generated and written to a new metadata LV. The new thin LV references the original thick data now located in the thin pool data LV. Note: This conversion cannot be reversed; the thin volume cannot be reverted back to the thick LV.

```
# lvconvert --type thin VG/ThickLV
```

(ThickLV would now be referred to as ThinLV, and a new thin pool will exist named ThinLV_tpool0.)

After the conversion, the resulting thin LV and thin pool will look somewhat different from ordinary thin LVs/pools: the new thin LV will be fully provisioned in the thin pool, and the thin pool data usage will be 100%. The thin pool will require extension before new thin LVs or snapshots are used.

### Thin Pool on LVM RAID
Thin pool data or metadata component LVs can use LVM RAID by first creating RAID LVs for data and/or metadata component LVs, and then converting these RAID LVs into a thin pool.

```
# lvcreate --type raidN -n DataLV -L Size VG DataDevices
# lvcreate --type raidN -n MetadataLV -L MetadataSize VG MetadataDevices
# lvconvert --type thin-pool --poolmetadata MetadataLV VG/DataLV
```

(DataLV would now be referred to as ThinPool, and can be used for creating thin LVs.)

To use MD RAID instead of LVM RAID, create linear data/metadata LVs on MD devices, and refer to the MD devices for DataDevices/MetadataDevices.

### Thin Pool on LVM VDO
Thin pool data can be compressed and deduplicated using VDO. Data for all thin LVs in the thin pool will be compressed and deduplicated using the dm-vdo module.

```
# lvcreate --type thin-pool -n ThinPool -L Size --pooldatavdo y VG
```

Or, convert an existing LV (e.g. linear, striped) into a thin-pool that uses VDO compression/deduplication for thin data. Existing content on the LV will be erased.

```
# lvconvert --type thin-pool --pooldatavdo y VG/LV
```

(LV would now be referred to as ThinPool, and can be used for creating thin LVs.)

### Thin Pool and Thin LV Combined Creation
One command can be used to create a new thin pool with a new thin LV.

```
# lvcreate --type thin -n ThinLV -V VirtualSize \
        --thinpool ThinPool -L ThinPoolSize VG
```

First, a new thin pool is created:

Thin Pool name is from --thinpool ThinPool

Thin Pool size is from -L|--size ThinPoolSize

Second, a new thin LV is created:

Thin LV name is from -n|--name ThinLV

Thin LV size is from -V|--virtualsize VirtualSize

Other thin LVs can then be created in the thin pool using standard lvcreate commands for thin LVs.

### Thin Snapshot Creation of an External Origin
Thin snapshots are typically taken of other thin LVs within the same thin pool. But, it is also possible to create a thin snapshot of an external LV (e.g. linear, striped, thin LV in another thin pool.) The external LV must be read-only (lvchange --permission r) and inactive to be used as a thin external origin. Writes to the thin snapshot LV are stored in its thin pool, and unwritten parts are read from the external origin. One external origin LV can be used for multiple thin snapshots.

```
# lvcreate --snapshot -n SnapLV --thinpool ThinPool VG/ExternalOrigin
```

### Thin Snapshot and External Origin Conversion
In this case, an existing, non-thin LV is converted to a read-only external origin, and a new thin LV is created as a snapshot of that external origin. The new thin LV is given the name of the existing LV, and the existing LV is given a new name from --originname.

Unwritten portions of the new thin LV are read from the external origin. If the thin LV is removed, the external origin LV can be used again in read/write mode. Thus, the thin LV can be seen as a snapshot of the original volume.

```
# lvconvert --type thin --thinpool ThinPool --originname ExtOrigin VG/LV
```

The existing LV argument is renamed ExtOrigin, and the new thin LV has the name of the existing LV.

### Thin Snapshot Merge
A thin snapshot can be merged into its origin thin LV. The result of a snapshot merge is that the origin thin LV takes the content of the snapshot LV, and the snapshot LV is removed. Any content that was unique to the origin thin LV is lost after the merge.

Because a merge changes the content of an LV, it cannot be done while the LVs are open, e.g. mounted. If a merge is initiated while the LVs are open, the effect of the merge is delayed until the origin thin LV is next activated.

```
# lvconvert --merge VG/SnapLV
```

## EXAMPLES

### Thin Pool Creation

```
# lvcreate --type thin-pool -n pool0 -L 500M vg
# lvs -a vg
  LV              VG Attr       LSize   Data%  Meta%
  [lvol0_pmspare] vg ewi-------   4.00m
  pool0           vg twi-a-tz-- 500.00m 0.00   10.84
  [pool0_tdata]   vg Twi-ao---- 500.00m
  [pool0_tmeta]   vg ewi-ao----   4.00m
```

### Thin Pool Conversion

```
# lvcreate -n pool0 -L 500M vg
# lvcreate -n pool0_meta -L 100M vg
# lvconvert --type thin-pool --poolmetadata pool0_meta vg/pool0
# lvs -a vg
  LV              VG Attr       LSize   Data%  Meta%
  [lvol0_pmspare] vg ewi------- 100.00m
  pool0           vg twi-a-tz-- 500.00m 0.00   10.04
  [pool0_tdata]   vg Twi-ao---- 500.00m
  [pool0_tmeta]   vg ewi-ao---- 100.00m
```

### Thin LV Creation

```
# lvcreate --type thin-pool -n pool0 -L 500M vg
# lvcreate --type thin -n vol -V 1G --thinpool pool0 vg
# lvs -a vg
  LV              VG Attr       LSize   Pool  Data%  Meta%
  [lvol0_pmspare] vg ewi-------   4.00m
  pool0           vg twi-aotz-- 500.00m       0.00   10.94
  [pool0_tdata]   vg Twi-ao---- 500.00m
  [pool0_tmeta]   vg ewi-ao----   4.00m
  vol             vg Vwi-a-tz--   1.00g pool0 0.00
```

### Thin Snapshot Creation

```
# lvcreate --type thin-pool -n pool0 -L 500M vg
# lvcreate --type thin -n vol -V 1G --thinpool pool0 vg
# lvcreate --snapshot -n snap1 vg/vol
# lvcreate --snapshot -n snap2 vg/snap1
# lvs -a vg
  LV              VG Attr       LSize   Pool  Origin Data%  Meta%
  [lvol0_pmspare] vg ewi-------   4.00m
  pool0           vg twi-aotz-- 500.00m              0.00   10.94
  [pool0_tdata]   vg Twi-ao---- 500.00m
  [pool0_tmeta]   vg ewi-ao----   4.00m
  snap1           vg Vwi---tz-k   1.00g pool0 vol
  snap2           vg Vwi---tz-k   1.00g pool0 snap1
  vol             vg Vwi-a-tz--   1.00g pool0        0.00
```

### Thin Pool Extension

```
# lvcreate --type thin-pool -n pool0 -L 500M vg
# lvextend -L+100M vg/pool0
# lvs -a vg
  LV              VG Attr       LSize   Data%  Meta%
  [lvol0_pmspare] vg ewi-------   4.00m
  pool0           vg twi-a-tz-- 600.00m 0.00   10.84
  [pool0_tdata]   vg Twi-ao---- 600.00m
  [pool0_tmeta]   vg ewi-ao----   4.00m
# lvextend -L+100M --poolmetadatasize 8M vg/pool0
# lvs -a vg
  LV              VG Attr       LSize   Data%  Meta%
  [lvol0_pmspare] vg ewi-------   8.00m
  pool0           vg twi-a-tz-- 700.00m 0.00   10.40
  [pool0_tdata]   vg Twi-ao---- 700.00m
  [pool0_tmeta]   vg ewi-ao----   8.00m
```

### Thick LV to Thin LV Conversion

```
# lvcreate -n vol -L500M vg
# lvconvert --type thin vg/vol
# lvs -a vg
  LV                 VG Attr       LSize   Pool       Data%  Meta%
  [lvol0_pmspare]    vg ewi-------   4.00m
  vol                vg Vwi-a-tz-- 500.00m vol_tpool0 100.00
  vol_tpool0         vg twi-aotz-- 500.00m            100.00 14.06
  [vol_tpool0_tdata] vg Twi-ao---- 500.00m
  [vol_tpool0_tmeta] vg ewi-ao----   4.00m
# lvextend -L1G vg/vol
# lvs -a vg
  LV                 VG Attr       LSize    Pool       Data%  Meta%
  [lvol0_pmspare]    vg ewi-------    4.00m
  vol                vg Vwi-a-tz--    1.00g vol_tpool0 48.83
  vol_tpool0         vg twi-aotz-- 1000.00m            50.00  14.06
  [vol_tpool0_tdata] vg Twi-ao---- 1000.00m
  [vol_tpool0_tmeta] vg ewi-ao----    4.00m
```

(Extending the virtual size of the thin LV triggered autoextend of the thin pool.)

### Thin Pool on LVM RAID

```
# lvcreate --type raid1 -n pool0 -m1 -L500M vg
# lvcreate --type raid1 -n pool0_meta -m1 -L8M vg
# lvs -a vg
  LV                    VG Attr       LSize   Cpy%Sync
  pool0                 vg rwi-a-r--- 500.00m 100.00
  pool0_meta            vg rwi-a-r---   8.00m 100.00
  [pool0_meta_rimage_0] vg iwi-aor---   8.00m
  [pool0_meta_rimage_1] vg iwi-aor---   8.00m
  [pool0_meta_rmeta_0]  vg ewi-aor---   4.00m
  [pool0_meta_rmeta_1]  vg ewi-aor---   4.00m
  [pool0_rimage_0]      vg iwi-aor--- 500.00m
  [pool0_rimage_1]      vg iwi-aor--- 500.00m
  [pool0_rmeta_0]       vg ewi-aor---   4.00m
  [pool0_rmeta_1]       vg ewi-aor---   4.00m
# lvconvert --type thin-pool --poolmetadata pool0_meta vg/pool0
# lvs -a vg
  LV                     VG Attr       LSize   Data%  Meta%  Cpy%Sync
  [lvol0_pmspare]        vg ewi-------   8.00m
  pool0                  vg twi-a-tz-- 500.00m 0.00   10.40
  [pool0_tdata]          vg rwi-aor--- 500.00m               100.00
  [pool0_tdata_rimage_0] vg iwi-aor--- 500.00m
  [pool0_tdata_rimage_1] vg iwi-aor--- 500.00m
  [pool0_tdata_rmeta_0]  vg ewi-aor---   4.00m
  [pool0_tdata_rmeta_1]  vg ewi-aor---   4.00m
  [pool0_tmeta]          vg ewi-aor---   8.00m               100.00
  [pool0_tmeta_rimage_0] vg iwi-aor---   8.00m
  [pool0_tmeta_rimage_1] vg iwi-aor---   8.00m
  [pool0_tmeta_rmeta_0]  vg ewi-aor---   4.00m
  [pool0_tmeta_rmeta_1]  vg ewi-aor---   4.00m
```

### Thin Pool on LVM VDO Creation

```
# lvcreate --type thin-pool -n pool0 -L5G --pooldatavdo y vg
# lvs -a vg
  LV                   VG Attr       LSize Pool         Data%  Meta%
  [lvol0_pmspare]      vg ewi------- 8.00m
  pool0                vg twi-a-tz-- 5.00g              0.00   10.64
  [pool0_tdata]        vg vwi-aov--- 5.00g pool0_vpool0 0.00
  [pool0_tmeta]        vg ewi-ao---- 8.00m
  pool0_vpool0         vg dwi------- 5.00g              60.03
  [pool0_vpool0_vdata] vg Dwi-ao---- 5.00g
```

### Thin Pool on LVM VDO Conversion

```
# lvcreate -n pool0 -L5G vg
# lvconvert --type thin-pool --pooldatavdo y vg/pool0
# lvs -a vg
  LV                   VG Attr       LSize Pool         Data%  Meta%
  [lvol0_pmspare]      vg ewi------- 8.00m
  pool0                vg twi-a-tz-- 5.00g              0.00   10.64
  [pool0_tdata]        vg vwi-aov--- 5.00g pool0_vpool0 0.00
  [pool0_tmeta]        vg ewi-ao---- 8.00m
  pool0_vpool0         vg dwi------- 5.00g              60.03
  [pool0_vpool0_vdata] vg Dwi-ao---- 5.00g
```

### Thin Snapshot Creation of an External Origin

```
# lvcreate -n vol -L 500M vg
# lvchange --permission r vg/vol
# lvchange -an vg/vol
# lvcreate --type thin-pool -n pool0 -L 500M vg
# lvcreate --snapshot -n snap --thinpool pool0 vg/vol
# lvs -a vg
  LV              VG Attr       LSize   Pool  Origin Data%  Meta%
  [lvol0_pmspare] vg ewi-------   4.00m
  pool0           vg twi-aotz-- 500.00m              0.00   10.94
  [pool0_tdata]   vg Twi-ao---- 500.00m
  [pool0_tmeta]   vg ewi-ao----   4.00m
  snap            vg Vwi-a-tz-- 500.00m pool0 vol    0.00
  vol             vg ori------- 500.00m
```

### Thin Pool and Thin LV Combined Creation

```
# lvcreate --type thin -n vol -V 1G --thinpool pool0 -L500M vg
# lvs -a vg
  LV              VG Attr       LSize   Pool  Data%  Meta%
  [lvol0_pmspare] vg ewi-------   4.00m
  pool0           vg twi-aotz-- 500.00m       0.00   10.94
  [pool0_tdata]   vg Twi-ao---- 500.00m
  [pool0_tmeta]   vg ewi-ao----   4.00m
  vol             vg Vwi-a-tz--   1.00g pool0 0.00
```

### Thin Snapshot Merge

```
# lvcreate --type thin-pool -n pool0 -L500M vg
# lvcreate --type thin -n vol -V 1G --thinpool pool0 vg
# lvcreate --snapshot -n snap vg/vol
# lvs -a vg
  LV              VG Attr       LSize   Pool  Origin Data%  Meta%
  [lvol0_pmspare] vg ewi-------   4.00m
  pool0           vg twi-aotz-- 500.00m              0.00   10.94
  [pool0_tdata]   vg Twi-ao---- 500.00m
  [pool0_tmeta]   vg ewi-ao----   4.00m
  snap            vg Vwi---tz-k   1.00g pool0 vol
  vol             vg Vwi-a-tz--   1.00g pool0        0.00
# lvconvert --merge vg/snap
# lvs -a vg
  LV              VG Attr       LSize   Pool  Data%  Meta%
  [lvol0_pmspare] vg ewi-------   4.00m
  pool0           vg twi-aotz-- 500.00m       0.00   10.94
  [pool0_tdata]   vg Twi-ao---- 500.00m
  [pool0_tmeta]   vg ewi-ao----   4.00m
  vol             vg Vwi-a-tz--   1.00g pool0 0.00
```

### Thin Snapshot Merge Delayed

```
# lvcreate --type thin-pool -n pool0 -L500M vg
# lvcreate --type thin -n vol -V 1G --thinpool pool0 vg
# mkfs.xfs /dev/vg/vol
# mount /dev/vg/vol /mnt
# touch /mnt/file1 /mnt/file2 /mnt/file3
# lvcreate --snapshot -n snap vg/vol
# mount /dev/vg/snap /snap -o nouuid
# touch /snap/file4 /snap/file5 /snap/file6
# ls /snap
file1  file2  file3  file4  file5  file6
# ls /mnt
file1  file2  file3
# lvconvert --merge vg/snap
  Logical volume vg/snap contains a filesystem in use.
  Delaying merge since snapshot is open.
  Merging of thin snapshot vg/snap will occur on next activation of vg/vol.
# umount /snap
# umount /mnt
# lvchange -an vg/vol
# lvs -a vg
  LV              VG Attr       LSize   Pool  Origin Data%  Meta%
  [lvol0_pmspare] vg ewi-------   4.00m
  pool0           vg twi-aotz-- 500.00m              13.36  11.62
  [pool0_tdata]   vg Twi-ao---- 500.00m
  [pool0_tmeta]   vg ewi-ao----   4.00m
  [snap]          vg Swi---tz-k   1.00g pool0 vol
  vol             vg Owi---tz--   1.00g pool0
# lvchange -ay vg/vol
# lvs -a vg
  LV              VG Attr       LSize   Pool  Data%  Meta%
  [lvol0_pmspare] vg ewi-------   4.00m
  pool0           vg twi-aotz-- 500.00m       12.94  11.43
  [pool0_tdata]   vg Twi-ao---- 500.00m
  [pool0_tmeta]   vg ewi-ao----   4.00m
  vol             vg Vwi-a-tz--   1.00g pool0 6.32
# mount /dev/vg/vol /mnt
# ls /mnt
file1  file2  file3  file4  file5  file6
```

## SPECIAL TOPICS

### Physical Devices for Thin Pool Data and Metadata
Placing the thin pool data LV and metadata LV on separate physical devices will improve performance. Faster, redundant devices for metadata is also recommended. To best customize the data and metadata LVs, create them separately and then combine them into a thin pool with lvconvert.

To configure lvcreate behavior to place thin pool data and metadata on separate devices, set lvm.conf(5) allocation/thin_pool_metadata_require_separate_pvs.

### Spare Metadata LV
The first time a thin pool LV is created, lvm will create a spare metadata LV in the VG. This behavior can be controlled with the option --poolmetadataspare y|n. To create the pmspare ("pool metadata spare") LV, lvm first creates an LV with a default name, e.g. lvol0, and then converts this LV to a hidden LV with the _pmspare suffix, e.g. lvol0_pmspare.

One pmspare LV is kept in a VG to be used for any thin pool.

The pmspare LV cannot be created explicitly, but may be removed explicitly.

The "Thin Pool Metadata check and repair" section describes the use of the pmspare LV.

### Thin Pool Metadata check and repair
If thin pool metadata is damaged, it may be repairable. Checking and repairing thin pool metadata is analogous to running fsck/repair on a file system. Thin pool metadata is compact, so even small areas of damage or corruption can result in significant data loss. Resilient storage for thin pool metadata can have extra value.

When a thin pool LV is activated, lvm runs the thin_check(8) command to check the correctness of the metadata on the pool metadata LV. To configure thin_check use, location or options used by lvm, set lvm.conf(5) global/thin_check_executable

The location of the program. Setting to an empty string ("") disables running thin_check by lvm. This is not recommended.

lvm.conf(5) global/thin_check_options

Controls the command options that lvm will use when running thin_check.

If thin_check finds a problem with the metadata, the thin pool LV is not activated, and the thin pool metadata needs to be repaired.

Simple repair commands are not always successful. Advanced repair may require editing thin pool metadata and lvm metadata. Newer versions of the kernel and lvm tools may be more successful at repair. Report the details of damaged thin metadata to get the best advice on recovery.

Command to repair a thin pool:

```
# lvconvert --repair VG/ThinPool
```

Repair performs the following steps:

1.

Creates a new, repaired copy of the metadata.

lvconvert runs the thin_repair(8) command to read damaged metadata from the existing pool metadata LV, and writes a new repaired copy to the VG's pmspare LV.

2.

Replaces the thin pool metadata LV.

If step 1 is successful, the thin pool metadata LV is replaced with the pmspare LV containing the corrected metadata. The previous thin pool metadata LV, containing the damaged metadata, becomes visible with the new name ThinPool_metaN (where N is 0,1,...).

If the repair works, the thin pool LV and its thin LVs can be activated. The user should verify that each thin LV in the thin pool can be successfully activated, and then verify the integrity of the file system on each thin LV (e.g. using fsck or other tools.) Once the thin pool is considered fully recovered, the ThinPool_metaN LV containing the original, damaged metadata can be manually removed to recovery the space.

If the repair fails, the original, unmodified ThinPool_metaN LV should be preserved for support, or more advanced recovery methods. Data from thin LVs may ultimately be unrecoverable.

If metadata is manually restored with thin_repair directly, the pool metadata LV can be manually swapped with another LV containing new metadata:

```
# lvconvert --thinpool VG/ThinPool --poolmetadata VG/NewMetadataLV
```

### Removing thin pool LVs, thin LVs and snapshots
Removing a thin LV and its related snapshots returns the blocks they used to the thin pool. These blocks will be reused for other thin LVs and snapshots.

Removing a thin pool LV removes both the data LV and metadata LV and returns the space to the VG.

lvremove of thin pool LVs, thin LVs and snapshots cannot be reversed with vgcfgrestore.

vgcfgbackup does not back up thin pool metadata.

### Using fstrim to increase free space in a thin pool
Removing files in a file system on a thin LV does not generally return free space to the thin pool, because file systems are not usually mounted with the discard mount option (due to the performance penalty.)

Manually running the fstrim command can return space from a thin LV back to the thin pool that had been used by removed files. This is only effective for entire thin pool chunks that have become unused (unused file system areas may not cover an entire chunk.) Thin snapshots also keep thin pool chunks from being freed. fstrim uses discards and will have no effect if the thin pool is configured to ignore discards.

Example

A thin pool has 10G of physical data space, and a thin LV has a virtual size of 100G. Writing a 1G file to the file system reduces the free space in the thin pool by 10% and increases the virtual usage of the file system by 1%. Removing the 1G file restores the virtual 1% to the file system, but does not restore the physical 10% to the thin pool. The fstrim command restores the physical space to the thin pool.

```
# lvs -a -oname,attr,size,pool_lv,origin,data_percent,metadata_percent vg
  LV            Attr       LSize   Pool  Origin Data%  Meta%
  pool0         twi-a-tz--  10.00g              47.01  21.03
  thin1         Vwi-aotz-- 100.00g pool0         2.70
```

```
# df -h /mnt/X
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg-thin1   99G  1.1G   93G   2% /mnt/X
```

```
# dd if=/dev/zero of=/mnt/X/1Gfile bs=4096 count=262144; sync
```

```
# lvs
  pool0         vg   twi-a-tz--  10.00g         57.01  25.26
  thin1         vg   Vwi-aotz-- 100.00g pool0    3.70
```

```
# df -h /mnt/X
/dev/mapper/vg-thin1   99G  2.1G   92G   3% /mnt/X
```

```
# rm /mnt/X/1Gfile
```

```
# lvs
  pool0         vg   twi-a-tz--  10.00g         57.01  25.26
  thin1         vg   Vwi-aotz-- 100.00g pool0    3.70
```

```
# df -h /mnt/X
/dev/mapper/vg-thin1   99G  1.1G   93G   2% /mnt/X
```

```
# fstrim -v /mnt/X
```

```
# lvs
  pool0         vg   twi-a-tz--  10.00g         47.01  21.03
  thin1         vg   Vwi-aotz-- 100.00g pool0    2.70
```

### Thin Pool Data Exhaustion
When properly managed, thin pool data space should be extended before it is all used (see sections on extending a thin pool automatically and manually.)

However, if a thin pool does run out of space, the behavior of the full thin pool can be configured with the "when full" property, reported with lvs -o whenfull. The "when full" property can be set to "error" or "queue". When set to "error", a full thin pool will immediately return errors for writes. When set to "queue", writes are queued for a period of time.

Display the current "when full" setting:

```
# lvs -o whenfull VG/ThinPool
```

Set the "when full" property to "error":

```
# lvchange --errorwhenfull y VG/ThinPool
```

Set the "when full" property to "queue":

```
# lvchange --errorwhenfull n VG/ThinPool
```

To configure the value that will be assigned to new thin pools, set lvm.conf(5) activation/error_when_full

The whenfull setting does not effect the monitoring and autoextend settings, and the monitoring/autoextend settings do not effect the whenfull setting. It is only when monitoring/autoextend are not effective that the thin pool becomes full and the whenfull setting is applied.

— queue when full —

The default is to queue writes for a period of time when the thin pool becomes full. Writes to thin LVs are accepted and queued, with the expectation that pool data space will be extended soon. Once data space is extended, the queued writes will be processed, and the thin pool will return to normal operation.

While waiting to be extended, the thin pool will queue writes for up to 60 seconds (the default). If data space has not been extended after this time, the queued writes will return an error to the caller, e.g. the file system. This can result in file system damage that requires repair. When a thin pool returns errors for writes to a thin LV, any file system is subject to losing unsynced user data.

The 60 second timeout can be changed or disabled with the "dm_thin_pool" kernel module option no_space_timeout. This option sets the number of seconds that thin pools will queue writes. If set to 0, writes will not time out. Disabling timeouts can result in the system running out of resources, memory exhaustion, hung tasks, and deadlocks. (The timeout applies to all thin pools on the system.)

— error when full —

Writes to thin LVs immediately return an error, and no writes are queued. This can result in file system damage that requires repair.

— data percent —

When data space is exhausted, the lvs command displays 100 under Data% and letter D in 9th. column for the thin pool LV:

```
# lvs -o name,data_percent,attr  vg/pool0
  LV     Data%  Attr
  pool0  100.00 twi-aot-D-
```

— causes —

A thin pool may run out of data space for any of the following reasons:

•

Automatic extension of the thin pool is disabled, and the thin pool is not manually extended.

(Disabling automatic extension is not recommended.)

•

The dmeventd(8) daemon is not running and the thin pool is not manually extended.

(Disabling dmeventd is not recommended.)

•

Automatic extension of the thin pool is too slow given the rate of writes to thin LVs in the pool. (This can be addressed by tuning the activation/thin_pool_autoextend_threshold and activation/thin_pool_autoextend_percent.)

•

The VG does not have enough free blocks to extend the thin pool.

### Thin Pool Metadata Exhaustion
If thin pool metadata space is exhausted (or a thin pool metadata operation fails), errors will be returned for IO operations on thin LVs.

When metadata space is exhausted, the lvs command displays 100 under Meta% for the thin pool LV:

```
# lvs -o name,metadata_percent vg/pool0
  LV    Meta%
  pool0 100.00
```

The same reasons for thin pool data space exhaustion apply to thin pool metadata space.

Metadata space exhaustion can lead to inconsistent thin pool metadata and inconsistent file systems, so the response requires offline checking and repair.

1.

Deactivate the thin pool LV, or reboot the system if this is not possible.

2.

Repair thin pool with lvconvert --repair.

See "Thin Pool Metadata check and repair".

3.

Extend pool metadata space with lvextend --poolmetadatasize.

See "Thin Pool Extension".

4.

Check and repair file system.

### Custom Thin Pool Configuration
It can be useful for different thin pools to have different thin pool settings like autoextend thresholds and percents. To change lvm.conf(5) values on a per-VG or per-LV basis, attach a "profile" to the VG or LV. A profile is a collection of config settings, saved in a local text file (using the lvm.conf(5) format). lvm looks for profiles in the profile_dir directory, e.g. /etc/lvm/profile/. Once attached to a VG or LV, lvm will process the VG or LV using the settings from the attached profile. A profile is named and referenced by its file name.

To use a profile to customize the lvextend settings for an LV:

•

Create a file containing settings, saved in profile_dir.

For the profile_dir location, run:

```
# lvmconfig config/profile_dir

```

•

Attach the profile to an LV, using the command:

```
# lvchange --metadataprofile ProfileName VG/ThinPool

```

•

Extend the LV using the profile settings:

```
# lvextend --use-policies VG/ThinPool

```

Example

```
# lvmconfig config/profile_dir
profile_dir="/etc/lvm/profile"
```

```
# cat /etc/lvm/profile/pool0extend.profile
activation {
        thin_pool_autoextend_threshold=50
        thin_pool_autoextend_percent=10
}
```

```
# lvchange --metadataprofile pool0extend vg/pool0
```

```
# lvextend --use-policies vg/pool0
```

Notes

•

A profile is attached to a VG or LV by name, where the name references a local file in profile_dir. If the VG is moved to another machine, the file with the profile also needs to be moved.

•

Only certain settings can be used in a VG or LV profile, see:

```
# lvmconfig --type profilable-metadata

```

•

An LV without a profile of its own will inherit the VG profile.

•

Remove a profile from an LV using the command:

```
# lvchange --detachprofile VG/ThinPool

```

•

Commands can also have profiles applied to them. The settings that can be applied to a command are different than the settings that can be applied to a VG or LV. See lvmconfig --type profilable-command. To apply a profile to a command, write a profile, save it in the profile directory, and run the command using the option: --commandprofile ProfileName.

### Zeroing
The "zero" property of a thin pool determines if chunks are overwritten with zeros when they are provisioned for a thin LV. The current setting is reported with lvs -o zero (displaying "zero" or "1" when zeroing is enabled), or 'z' in the eighth lv_attr. The option -Z|--zero is used to specify the zeroing mode.

Create a thin pool with zeroing mode:

```
# lvcreate --type thin-pool -n ThinPool -L Size -Z y VG
```

Disable the zeroing mode of an existing thin pool:

```
# lvchange -Z n VG/ThinPool
```

If zeroing mode is changed from "n" to "y", previously provisioned blocks are not zeroed.

Provisioning of large zeroed chunks reduces performance.

To configure the zeroing mode used for new thin pools when not specified on the command line, set lvm.conf(5) allocation/thin_pool_zero

### Discard
The "discards" property of a thin pool determines how discard requests are handled. The current setting is reported with lvs -o discards. The option --discards is used to specify the discards mode.

Possible discard modes:

ignore

Ignore any discards that are received.

nopassdown

Process any discards in the thin pool itself, and allow the newly unused chunks to be used for new data.

passdown

Process discards in the thin pool (as with nopassdown), and pass the discards down the underlying device. This is the default mode.

Create a thin pool with a specific discards mode:

```
# lvcreate --type thin-pool -n ThinPool -L Size \
	--discards nopassdown VG
```

Change the discards mode of an existing thin pool:

```
# lvchange --discards nopassdown VG/ThinPool
```

To configure the discards mode used for new thin pools when not specified on the command line, set lvm.conf(5) allocation/thin_pool_discards

Discards can have an adverse impact on performance, see the fstrim section for more information.

### Chunk size
A thin pool allocates physical storage for thin LVs in units of "chunks". The current chunk size of a thin pool is reported with lvs -o chunksize. The option --chunksize is used to specify the value for a new thin pool (default units are KiB.) The value must be a multiple of 64 KiB, between 64 KiB and 1 GiB.

When a thin pool is used primarily for the thin provisioning feature, a larger value is optimal. To optimize for many snapshots, a smaller value reduces copying time and consumes less space.

To configure the chunk size used for new thin pools when not specified on the command line, set lvm.conf(5) allocation/thin_pool_chunk_size

The default value is shown by:

```
# lvmconfig --type default allocation/thin_pool_chunk_size
```

### Thin Pool Metadata Size
The amount of thin pool metadata depends on how many blocks are shared between thin LVs (i.e. through snapshots). A thin pool with many snapshots may need a larger metadata LV. Thin pool metadata LV sizes can be from 2 MiB to approximately 16 GiB.

When an LVM command automatically creates a thin pool metadata LV, the size is specified with the --poolmetadatasize option. When this option is not given, LVM automatically chooses a size based on the data size and chunk size.

It can be hard to predict the amount of metadata space that will be needed, so it is recommended to start with a size of 1 GiB which should be enough for all practical purposes. A thin pool metadata LV can later be manually or automatically extended if needed.

(For purposes of backward compatibility, lvm.conf(5) allocation/thin_pool_crop_metadata controls cropping the metadata LV size to 15.81 GiB to be backward compatible with older versions of lvm. With cropping, there can be problems with volumes above this size when used with thin tools, i.e. thin_repair. Cropping should be enabled only when compatibility is required.)

### XFS on snapshots
Mounting an XFS file system on a new snapshot LV requires attention to the file system's log state and uuid. On the snapshot LV, the xfs log will contain a dummy transaction, and the xfs uuid will match the uuid from the file system on the origin LV.

If the snapshot LV is writable, mounting will recover the log to clear the dummy transaction, but will require skipping the uuid check:

```
# mount /dev/VG/SnapLV /mnt -o nouuid
```

After the first mount with the above approach, the UUID can subsequently be changed using:

```
# xfs_admin -U generate /dev/VG/SnapLV
# mount /dev/VG/SnapLV /mnt
```

Once the UUID has been changed, the mount command will no longer require the nouuid option.

If the snapshot LV is readonly, the log recovery and uuid check need to be skipped while mounting readonly:

```
# mount /dev/VG/SnapLV /mnt -o ro,nouuid,norecovery
```

## SEE ALSO
lvm(8), lvm.conf(5), lvmconfig(8), lvcreate(8), lvconvert(8), lvchange(8), lvextend(8), lvremove(8), lvs(8), vgcfgbackup(8), vgcfgrestore(8),

dmeventd(8), thin_check(8), thin_dump(8), thin_repair(8), thin_restore(8),

vdoformat(8), vdostats(8),

insmod(8), modprobe(8), mount(8), systemd(1), xfs_admin(8)


lvmvdo

title: lvmvdo(7)

text:
# lvmvdo(7)

## NAME
lvmvdo — Support for Virtual Data Optimizer in LVM

## DESCRIPTION
VDO is software that provides inline block-level deduplication, compression, and thin provisioning capabilities for primary storage.

Deduplication is a technique for reducing the consumption of storage resources by eliminating multiple copies of duplicate blocks. Compression takes the individual unique blocks and shrinks them. These reduced blocks are then efficiently packed together into physical blocks. Thin provisioning manages the mapping from logical blocks presented by VDO to where the data has actually been physically stored, and also eliminates any blocks of all zeroes.

With deduplication, instead of writing the same data more than once, VDO detects and records each duplicate block as a reference to the original block. VDO maintains a mapping from Logical Block Addresses (LBA) (used by the storage layer above VDO) to physical block addresses (used by the storage layer under VDO). After deduplication, multiple logical block addresses may be mapped to the same physical block address; these are called shared blocks and are reference-counted by the software.

With compression, VDO compresses multiple blocks (or shared blocks) with the fast LZ4 algorithm, and bins them together where possible so that multiple compressed blocks fit within a 4 KB block on the underlying storage. Mapping from LBA is to a physical block address and index within it for the desired compressed data. All compressed blocks are individually reference counted for correctness.

Block sharing and block compression are invisible to applications using the storage, which read and write blocks as they would if VDO were not present. When a shared block is overwritten, a new physical block is allocated for storing the new block data to ensure that other logical block addresses that are mapped to the shared physical block are not modified.

To use VDO with lvm(8), you must install the standard VDO user-space tools vdoformat(8) and kernel module "dm_vdo" (For older kernels <6.9 the out of tree kernel VDO module "kvdo" is necessary).

The kernel module implements fine-grained storage virtualization, thin provisioning, block sharing, compression and memory-efficient duplicate identification. The user-space tools include vdostats(8) for extracting statistics from VDO volumes.

## VDO TERMS
VDODataLV

VDO data LV is a large hidden LV with the _vdata suffix. It is created in a VG used by the VDO kernel target to store all data and metadata blocks.

VDOPoolLV

VDO pool LV is a pool for virtual VDOLV(s), which are the size of used VDODataLV. Only a single VDOLV is currently supported.

VDOLV

VDO LV is a virtual LV created from VDOPoolLV. Appears blank after creation.

## VDO USAGE
The primary methods for using VDO with lvm2:

### 1. Create a VDOPoolLV and a VDOLV
Create a VDOPoolLV that will hold VDO data, and a virtual size VDOLV that the user can use. If you do not specify the virtual size, then the VDOLV is created with the maximum size that always fits into data volume even if no deduplication or compression can happen (i.e. it can hold the incompressible content of /dev/urandom). If you do not specify the name of VDOPoolLV, it is taken from the sequence of vpool0, vpool1, ...

Note: The performance of TRIM/Discard operations is slow for large volumes of VDO type. Please try to avoid sending discard requests unless necessary because it might take considerable amount of time to finish the discard operation.

lvcreate --type vdo -n VDOLV -L DataSize -V LargeVirtualSize VG/VDOPoolLV

lvcreate --vdo -L DataSize VG

Example

```
# lvcreate --type vdo -n vdo0 -L 10G -V 100G vg/vdopool0
# mkfs.ext4 -E nodiscard /dev/vg/vdo0
```

### 2. Convert an existing LV into VDOPoolLV
Convert an already created or existing LV into a VDOPoolLV, which is a volume that can hold data and metadata. You will be prompted to confirm such conversion because it IRREVERSIBLY DESTROYS the content of such volume and the volume is immediately formatted by vdoformat(8) as a VDO pool data volume. You can specify the virtual size of the VDOLV associated with this VDOPoolLV. If you do not specify the virtual size, it will be set to the maximum size that can keep 100% incompressible data in it.

lvconvert --type vdo-pool -n VDOLV -V VirtualSize VG/VDOPoolLV

lvconvert --vdopool VG/VDOPoolLV

Example

```
# lvconvert --type vdo-pool -n vdo0 -V10G vg/ExistingLV
```

### 3. Change the compression and deduplication of a VDOPoolLV
Disable or enable the compression and deduplication for VDOPoolLV (the volume that maintains all VDO LV(s) associated with it).

lvchange --compression y|n --deduplication y|n VG/VDOPoolLV

Example

```
# lvchange --compression n  vg/vdopool0
# lvchange --deduplication y vg/vdopool1
```

### 4. Change the default settings used for creating a VDOPoolLV
VDO allows you to set a large variety of options. Lots of these settings can be specified in lvm.conf or profile settings. You can prepare a number of different profiles in the /etc/lvm/profile directory and just specify the profile file name. Check the output of lvmconfig --type default --withcomments for a detailed description of all individual VDO settings.

Example

```
# cat <<EOF > /etc/lvm/profile/vdo_create.profile
allocation {
```

```
vdo_use_compression=1
vdo_use_deduplication=1
vdo_minimum_io_size=4096
vdo_block_map_cache_size_mb=128
vdo_block_map_period=16380
vdo_use_sparse_index=0
vdo_index_memory_size_mb=256
vdo_slab_size_mb=2048
vdo_ack_threads=1
vdo_bio_threads=1
vdo_bio_rotation=64
vdo_cpu_threads=2
vdo_hash_zone_threads=1
vdo_logical_threads=1
vdo_physical_threads=1
vdo_max_discard=1
```

```
}
EOF
```

```
# lvcreate --vdo -L10G --metadataprofile vdo_create vg/vdopool0
# lvcreate --vdo -L10G --config 'allocation/vdo_cpu_threads=4' vg/vdopool1
```

### 5. Set or change VDO settings with option --vdosettings
Use the form 'option=value' or 'option1=value option2=value', or repeat --vdosettings for each option being set. Options are listed in the Example section above, for the full description see lvm.conf(5). Options can omit 'vdo_' and 'vdo_use_' prefixes and all its underscores. So i.e. vdo_use_deduplication=1 and deduplication=1 are equivalent. To change the option for an already existing VDOPoolLV use lvchange(8) command. However not all option can be changed. Only compression and deduplication options can be also changed for an active VDO LV. Lowest priority options are specified with configuration file, then with --vdosettings and highest are explicit option --compression and --deduplication.

Example

```
# lvcreate --vdo -L10G --vdosettings 'ack_threads=1 hash_zone_threads=2' vg/vdopool0
# lvchange --vdosettings 'bio_threads=2 deduplication=1' vg/vdopool0
```

### 6. Checking the usage of VDOPoolLV
To quickly check how much data on a VDOPoolLV is already consumed, use lvs(8). The Data% field reports how much data is occupied in the content of the virtual data for the VDOLV and how much space is already consumed with all the data and metadata blocks in the VDOPoolLV. For a detailed description, use the vdostats(8) command.

Note: vdostats(8) currently understands only /dev/mapper device names.

Example

```
# lvcreate --type vdo -L10G -V20G -n vdo0 vg/vdopool0
# mkfs.ext4 -E nodiscard /dev/vg/vdo0
# lvs -a vg
```

```
  LV               VG Attr       LSize  Pool     Origin Data%
  vdo0             vg vwi-a-v--- 20.00g vdopool0        0.01
  vdopool0         vg dwi-ao---- 10.00g                 30.16
  [vdopool0_vdata] vg Dwi-ao---- 10.00g
```

```
# vdostats --all /dev/mapper/vg-vdopool0-vpool
/dev/mapper/vg-vdopool0 :
  version                             : 30
  release version                     : 133524
  data blocks used                    : 79
  ...
```

### 7. Extending the VDOPoolLV size
You can add more space to hold VDO data and metadata by extending the VDODataLV using the commands lvresize(8) and lvextend(8). The extension needs to add at least one new VDO slab. You can configure the slab size with the allocation/vdo_slab_size_mb setting.

You can also enable automatic size extension of a monitored VDOPoolLV with the activation/vdo_pool_autoextend_percent and activation/vdo_pool_autoextend_threshold settings.

Note: You cannot reduce the size of a VDOPoolLV.

lvextend -L+AddingSize VG/VDOPoolLV

Example

```
# lvextend -L+50G vg/vdopool0
# lvresize -L300G vg/vdopool1
```

### 8. Extending or reducing the VDOLV size
You can extend or reduce a virtual VDO LV as a standard LV with the lvresize(8), lvextend(8), and lvreduce(8) commands.

Note: The reduction needs to process TRIM for reduced disk area to unmap used data blocks from the VDOPoolLV, which might take a long time.

lvextend -L+AddingSize VG/VDOLV

lvreduce -L-ReducingSize VG/VDOLV

Example

```
# lvextend -L+50G vg/vdo0
# lvreduce -L-50G vg/vdo1
# lvresize -L200G vg/vdo2
```

### 9. Component activation of a VDODataLV
You can activate a VDODataLV separately as a component LV for examination purposes. The activation of the VDODataLV activates the data LV in read-only mode, and the data LV cannot be modified. If the VDODataLV is active as a component, any upper LV using this volume CANNOT be activated. You have to deactivate the VDODataLV first to continue to use the VDOPoolLV.

Example

```
# lvchange -ay vg/vpool0_vdata
# lvchange -an vg/vpool0_vdata
```

## VDO TOPICS

### 1. Stacking VDO
You can convert or stack a VDOPooLV with these currently supported volume types: linear, stripe, raid and cache with cachepool.

### 1. Using multiple volumes using same VDOPoolLV
You can convert existing VDO LV into a thin volume. After this conversion you can create a thin snapshot or you can add more thin volumes with thin-pool named after original LV name LV_tpool0. See lvmthin(7) for more details.

Example

```
# lvcreate --type vdo -L 5G -V 10G -n vdo1 vg/vdopool
# lvconvert --type thin vg/vdo1
# lvcreate -V20 vg/vdo1_tpool0
```

### 2. VDOPoolLV on top of raid
Using a raid type LV for a VDODataLV.

Example

```
# lvcreate --type raid1 -L 5G -n vdopool vg
# lvconvert --type vdo-pool -V 10G vg/vdopool
```

### 3. Caching a VDOPoolLV
VDOPoolLV (accepts also VDODataLV volume name) caching provides a mechanism to accelerate reads and writes of already compressed and deduplicated data blocks together with VDO metadata.

Example

```
# lvcreate --type vdo -L 5G -V 10G -n vdo1 vg/vdopool
# lvcreate --type cache-pool -L 1G -n cachepool vg
# lvconvert --cache --cachepool vg/cachepool vg/vdopool
# lvconvert --uncache vg/vdopool
```

### 4. Caching a VDOLV
VDO LV cache allow you to 'cache' a device for better performance before it hits the processing of the VDO Pool LV layer.

Example

```
# lvcreate --type vdo -L 5G -V 10G -n vdo1 vg/vdopool
# lvcreate --type cache-pool -L 1G -n cachepool vg
# lvconvert --cache --cachepool vg/cachepool vg/vdo1
# lvconvert --uncache vg/vdo1
```

### 5. Usage of Discard/TRIM with a VDOLV
You can discard data on a VDO LV and reduce used blocks on a VDOPoolLV. However, the current performance of discard operations is still not optimal and takes a considerable amount of time and CPU. Unless you really need it, you should avoid using discard.

When a block device is going to be rewritten, its blocks will be automatically reused for new data. Discard is useful in situations when user knows that the given portion of a VDO LV is not going to be used and the discarded space can be used for block provisioning in other regions of the VDO LV. For the same reason, you should avoid using mkfs with discard for a freshly created VDO LV to save a lot of time that this operation would take otherwise as device is already expected to be empty.

### 6. Memory usage
The VDO target requires 38 MiB of RAM and several variable amounts:

- 1.15 MiB of RAM for each 1 MiB of configured block map cache size. The block map cache requires a minimum of 150 MiB RAM.

- 1.6 MiB of RAM for each 1 TiB of logical space.

- 268 MiB of RAM for each 1 TiB of physical storage managed by the volume.

UDS requires a minimum of 250 MiB of RAM, which is also the default amount that deduplication uses.

The memory required for the UDS index is determined by the index type and the required size of the deduplication window and is controlled by the allocation/vdo_use_sparse_index setting.

With enabled UDS sparse indexing, it relies on the temporal locality of data and attempts to retain only the most relevant index entries in memory and can maintain a deduplication window that is ten times larger than with dense while using the same amount of memory.

Although the sparse index provides the greatest coverage, the dense index provides more deduplication advice. For most workloads, given the same amount of memory, the difference in deduplication rates between dense and sparse indexes is negligible.

A dense index with 1 GiB of RAM maintains a 1 TiB deduplication window, while a sparse index with 1 GiB of RAM maintains a 10 TiB deduplication window. In general, 1 GiB is sufficient for 4 TiB of physical space with a dense index and 40 TiB with a sparse index.

### 7. Storage space requirements
You can configure a VDOPoolLV to use up to 256 TiB of physical storage. Only a certain part of the physical storage is usable to store data. This section provides the calculations to determine the usable size of a VDO-managed volume.

The VDO target requires storage for two types of VDO metadata and for the UDS index:

- The first type of VDO metadata uses approximately 1 MiB for each 4 GiB of physical storage plus an additional 1 MiB per slab.

- The second type of VDO metadata consumes approximately 1.25 MiB for each 1 GiB of logical storage, rounded up to the nearest slab.

- The amount of storage required for the UDS index depends on the type of index and the amount of RAM allocated to the index. For each 1 GiB of RAM, a dense UDS index uses 17 GiB of storage and a sparse UDS index will use 170 GiB of storage.

## SEE ALSO
lvm(8), lvm.conf(5), lvmconfig(8), lvcreate(8), lvconvert(8), lvchange(8), lvextend(8), lvreduce(8), lvresize(8), lvremove(8), lvs(8),

lvmthin(7), vdoformat(8), vdostats(8),

mkfs(8)

branch: sisyphus


lxc

package: lxc-core

title: lxc(7)

summary: linux containers

text:
# lxc(7)

## NAME
lxc - linux containers

## OVERVIEW
The container technology is actively being pushed into the mainstream Linux kernel. It provides resource management through control groups and resource isolation via namespaces.

lxc, aims to use these new functionalities to provide a userspace container object which provides full resource isolation and resource control for an applications or a full system.

lxc is small enough to easily manage a container with simple command lines and complete enough to be used for other purposes.

## REQUIREMENTS
The kernel version >= 3.10 shipped with the distros, will work with lxc, this one will have less functionalities but enough to be interesting.

lxc relies on a set of functionalities provided by the kernel. The helper script lxc-checkconfig will give you information about your kernel configuration, required, and missing features.

## FUNCTIONAL SPECIFICATION
A container is an object isolating some resources of the host, for the application or system running in it.

The application / system will be launched inside a container specified by a configuration that is either initially created or passed as a parameter of the commands.

How to run an application in a container

Before running an application, you should know what are the resources you want to isolate. The default configuration is to isolate PIDs, the sysv IPC and mount points. If you want to run a simple shell inside a container, a basic configuration is needed, especially if you want to share the rootfs. If you want to run an application like sshd, you should provide a new network stack and a new hostname. If you want to avoid conflicts with some files eg. /var/run/httpd.pid, you should remount /var/run with an empty directory. If you want to avoid the conflicts in all the cases, you can specify a rootfs for the container. The rootfs can be a directory tree, previously bind mounted with the initial rootfs, so you can still use your distro but with your own /etc and /home

Here is an example of directory tree for sshd:

```

[root@lxc sshd]$ tree -d rootfs

rootfs
|-- bin
|-- dev
|   |-- pts
|   `-- shm
|       `-- network
|-- etc
|   `-- ssh
|-- lib
|-- proc
|-- root
|-- sbin
|-- sys
|-- usr
`-- var

    |-- empty

    |   `-- sshd

    |-- lib

    |   `-- empty

    |       `-- sshd

    `-- run

        `-- sshd

```

and the mount points file associated with it:

```
	[root@lxc sshd]$ cat fstab
	/lib /home/root/sshd/rootfs/lib none ro,bind 0 0
	/bin /home/root/sshd/rootfs/bin none ro,bind 0 0
	/usr /home/root/sshd/rootfs/usr none ro,bind 0 0
	/sbin /home/root/sshd/rootfs/sbin none ro,bind 0 0

```

How to run a system in a container

Running a system inside a container is paradoxically easier than running an application. Why? Because you don't have to care about the resources to be isolated, everything needs to be isolated, the other resources are specified as being isolated but without configuration because the container will set them up. eg. the ipv4 address will be setup by the system container init scripts. Here is an example of the mount points file:

```
	[root@lxc debian]$ cat fstab
	/dev	/home/root/debian/rootfs/dev none bind 0 0
	/dev/pts /home/root/debian/rootfs/dev/pts  none bind 0 0

```

### CONTAINER LIFE CYCLE
When the container is created, it contains the configuration information. When a process is launched, the container will be starting and running. When the last process running inside the container exits, the container is stopped.

In case of failure when the container is initialized, it will pass through the aborting state.

```
   ---------

  | STOPPED |<---------------

   ---------                 |

       |                     |

     start                   |

       |                     |

       V                     |

   ----------                |

  | STARTING |--error-       |

   ----------         |      |

       |              |      |

       V              V      |

   ---------    ----------   |

  | RUNNING |  | ABORTING |  |

   ---------    ----------   |

       |              |      |

  no process          |      |

       |              |      |

       V              |      |

   ----------         |      |

  | STOPPING |<-------       |

   ----------                |

       |                     |

        ---------------------

```

### CONFIGURATION
The container is configured through a configuration file, the format of the configuration file is described in lxc.conf(5)

### CREATING / DESTROYING CONTAINERS
A persistent container object can be created via the lxc-create command. It takes a container name as parameter and optional configuration file and template. The name is used by the different commands to refer to this container. The lxc-destroy command will destroy the container object.

```
	  lxc-create -n foo
	  lxc-destroy -n foo

```

### VOLATILE CONTAINER
It is not mandatory to create a container object before starting it. The container can be directly started with a configuration file as parameter.

### STARTING / STOPPING CONTAINER
When the container has been created, it is ready to run an application / system. This is the purpose of the lxc-execute and lxc-start commands. If the container was not created before starting the application, the container will use the configuration file passed as parameter to the command, and if there is no such parameter either, then it will use a default isolation. If the application ended, the container will be stopped, but if needed the lxc-stop command can be used to stop the container.

Running an application inside a container is not exactly the same thing as running a system. For this reason, there are two different commands to run an application into a container:

```
	  lxc-execute -n foo [-f config] /bin/bash
	  lxc-start -n foo [-f config] [/bin/bash]

```

The lxc-execute command will run the specified command into a container via an intermediate process, lxc-init. This lxc-init after launching the specified command, will wait for its end and all other reparented processes. (to support daemons in the container). In other words, in the container, lxc-init has PID 1 and the first process of the application has PID 2.

The lxc-start command will directly run the specified command in the container. The PID of the first process is 1. If no command is specified lxc-start will run the command defined in lxc.init.cmd or if not set, /sbin/init .

To summarize, lxc-execute is for running an application and lxc-start is better suited for running a system.

If the application is no longer responding, is inaccessible or is not able to finish by itself, a wild lxc-stop command will kill all the processes in the container without pity.

```
	  lxc-stop -n foo -k

```

### CONNECT TO AN AVAILABLE TTY
If the container is configured with ttys, it is possible to access it through them. It is up to the container to provide a set of available ttys to be used by the following command. When the tty is lost, it is possible to reconnect to it without login again.

```
	  lxc-console -n foo -t 3

```

### FREEZE / UNFREEZE CONTAINER
Sometime, it is useful to stop all the processes belonging to a container, eg. for job scheduling. The commands:

```
	  lxc-freeze -n foo

```

will put all the processes in an uninteruptible state and

```
	  lxc-unfreeze -n foo

```

will resume them.

This feature is enabled if the freezer cgroup v1 controller is enabled in the kernel.

### GETTING INFORMATION ABOUT CONTAINER
When there are a lot of containers, it is hard to follow what has been created or destroyed, what is running or what are the PIDs running in a specific container. For this reason, the following commands may be useful:

```
	  lxc-ls -f
	  lxc-info -n foo

```

lxc-ls lists containers.

lxc-info gives information for a specific container.

Here is an example on how the combination of these commands allows one to list all the containers and retrieve their state.

```
	  for i in $(lxc-ls -1); do
	    lxc-info -n $i
	  done

```

### MONITORING CONTAINER
It is sometime useful to track the states of a container, for example to monitor it or just to wait for a specific state in a script.

lxc-monitor command will monitor one or several containers. The parameter of this command accepts a regular expression for example:

```
	  lxc-monitor -n "foo|bar"

```

will monitor the states of containers named 'foo' and 'bar', and:

```
	  lxc-monitor -n ".*"

```

will monitor all the containers.

For a container 'foo' starting, doing some work and exiting, the output will be in the form:

```
	  'foo' changed state to [STARTING]
	  'foo' changed state to [RUNNING]
	  'foo' changed state to [STOPPING]
	  'foo' changed state to [STOPPED]

```

lxc-wait command will wait for a specific state change and exit. This is useful for scripting to synchronize the launch of a container or the end. The parameter is an ORed combination of different states. The following example shows how to wait for a container if it successfully started as a daemon.

```
	  # launch lxc-wait in background
	  lxc-wait -n foo -s STOPPED &
	  LXC_WAIT_PID=$!
	  # this command goes in background
	  lxc-execute -n foo mydaemon &
	  # block until the lxc-wait exits
	  # and lxc-wait exits when the container
	  # is STOPPED
	  wait $LXC_WAIT_PID
	  echo "'foo' is finished"

```

### CGROUP SETTINGS FOR CONTAINERS
The container is tied with the control groups, when a container is started a control group is created and associated with it. The control group properties can be read and modified when the container is running by using the lxc-cgroup command.

lxc-cgroup command is used to set or get a control group subsystem which is associated with a container. The subsystem name is handled by the user, the command won't do any syntax checking on the subsystem name, if the subsystem name does not exists, the command will fail.

```
	  lxc-cgroup -n foo cpuset.cpus

```

will display the content of this subsystem.

```
	  lxc-cgroup -n foo cpu.shares 512

```

will set the subsystem to the specified value.

## SEE ALSO
lxc(7), lxc-create(1), lxc-copy(1), lxc-destroy(1), lxc-start(1), lxc-stop(1), lxc-execute(1), lxc-console(1), lxc-monitor(1), lxc-wait(1), lxc-cgroup(1), lxc-ls(1), lxc-info(1), lxc-freeze(1), lxc-unfreeze(1), lxc-attach(1), lxc.conf(5)


package: man-pages

title: address_families(7)

summary: socket address families (domains)

text:
# address_families(7)

## NAME
address_families - socket address families (domains)

## SYNOPSIS

```
#include <sys/types.h>          /* See NOTES */
#include <sys/socket.h>
```

```
int socket(int domain, int type, int protocol);
```

## DESCRIPTION
The domain argument of the socket(2) specifies a communication domain; this selects the protocol family which will be used for communication. These families are defined in <sys/socket.h>. The formats currently understood by the Linux kernel include:

AF_UNIX, AF_LOCAL

Local communication. For further information, see unix(7).

AF_INET

IPv4 Internet protocols. For further information, see ip(7).

AF_AX25

Amateur radio AX.25 protocol. For further information, see ax25(4).

AF_IPX

IPX - Novell protocols.

AF_APPLETALK

AppleTalk For further information, see ddp(7).

AF_NETROM

AX.25 packet layer protocol. For further information, see netrom(4), The Packet Radio Protocols and Linux and the AX.25, NET/ROM, and ROSE network programming chapters of the Linux Amateur Radio AX.25 HOWTO.

AF_BRIDGE

Can't be used for creating sockets; mostly used for bridge links in rtnetlink(7) protocol commands.

AF_ATMPVC

Access to raw ATM Permanent Virtual Circuits (PVCs). For further information, see the ATM on Linux HOWTO.

AF_X25

ITU-T X.25 / ISO-8208 protocol. For further information, see x25(7).

AF_INET6

IPv6 Internet protocols. For further information, see ipv6(7).

AF_ROSE

RATS (Radio Amateur Telecommunications Society). Open Systems environment (ROSE) AX.25 packet layer protocol. For further information, see the resources listed for AF_NETROM.

AF_DECnet

DECet protocol sockets. See Documentation/networking/decnet.txt in the Linux kernel source tree for details.

AF_NETBEUI

Reserved for "802.2LLC project"; never used.

AF_SECURITY

This was a short-lived (between Linux 2.1.30 and 2.1.99pre2) protocol family for firewall upcalls.

AF_KEY

Key management protocol, originally developed for usage with IPsec (since Linux 2.1.38). This has no relation to keyctl(2) and the in-kernel key storage facility. See RFC 2367 PF_KEY Key Management API, Version 2 for details.

AF_NETLINK

Kernel user interface device. For further information, see netlink(7).

AF_PACKET

Low-level packet interface. For further information, see packet(7).

AF_ECONET

Acorn Econet protocol (removed in Linux 3.5). See the Econet documentation for details.

AF_ATMSVC

Access to ATM Switched Virtual Circuits (SVCs) See the ATM on Linux HOWTO for details.

AF_RDS

Reliable Datagram Sockets (RDS) protocol (since Linux 2.6.30). RDS over RDMA has no relation to AF_SMC or AF_XDP. For further information, see rds(7), rds-rdma(7), and Documentation/networking/rds.txt in the Linux kernel source tree.

AF_IRDA

Socket interface over IrDA (moved to staging in Linux 4.14, removed in Linux 4.17). For further information, see irda(7).

AF_PPPOX

Generic PPP transport layer, for setting up L2 tunnels (L2TP and PPPoE). See Documentation/networking/l2tp.txt in the Linux kernel source tree for details.

AF_WANPIPE

Legacy protocol for wide area network (WAN) connectivity that was used by Sangoma WAN cards (called "WANPIPE"); removed in Linux 2.6.21.

AF_LLC

Logical link control (IEEE 802.2 LLC) protocol, upper part of data link layer of ISO/OSI networking protocol stack (since Linux 2.4); has no relation to AF_PACKET. See chapter 13.5.3. Logical Link Control in Understanding Linux Kernel Internals (O'Reilly Media, 2006) and IEEE Standards for Local Area Networks: Logical Link Control (The Institute of Electronics and Electronics Engineers, Inc., New York, New York, 1985) for details. See also some historical notes regarding its development.

AF_IB

InfiniBand native addressing (since Linux 3.11).

AF_MPLS

Multiprotocol Label Switching (since Linux 4.1); mostly used for configuring MPLS routing via netlink(7), as it doesn't expose ability to create sockets to user space.

AF_CAN

Controller Area Network automotive bus protocol (since Linux 2.6.25). See Documentation/networking/can.rst in the Linux kernel source tree for details.

AF_TIPC

TIPC, "cluster domain sockets" protocol (since Linux 2.6.16). See TIPC Programmer's Guide and the protocol description for details.

AF_BLUETOOTH

Bluetooth low-level socket protocol (since Linux 3.11). See Bluetooth Management API overview and An Introduction to Bluetooth Programming by Albert Huang for details.

AF_IUCV

IUCV (inter-user communication vehicle) z/VM protocol for hypervisor-guest interaction (since Linux 2.6.21); has no relation to AF_VSOCK and/or AF_SMC See IUCV protocol overview for details.

AF_RXRPC

Rx, Andrew File System remote procedure call protocol (since Linux 2.6.22). See Documentation/networking/rxrpc.txt in the Linux kernel source tree for details.

AF_ISDN

New "modular ISDN" driver interface protocol (since Linux 2.6.27). See the mISDN wiki for details.

AF_PHONET

Nokia cellular modem IPC/RPC interface (since Linux 2.6.31). See Documentation/networking/phonet.txt in the Linux kernel source tree for details.

AF_IEEE802154

IEEE 802.15.4 WPAN (wireless personal area network) raw packet protocol (since Linux 2.6.31). See Documentation/networking/ieee802154.txt in the Linux kernel source tree for details.

AF_CAIF

Ericsson's Communication CPU to Application CPU interface (CAIF) protocol (since Linux 2.6.36). See Documentation/networking/caif/Linux-CAIF.txt in the Linux kernel source tree for details.

AF_ALG

Interface to kernel crypto API (since Linux 2.6.38). See Documentation/crypto/userspace-if.rst in the Linux kernel source tree for details.

AF_VSOCK

VMWare VSockets protocol for hypervisor-guest interaction (since Linux 3.9); has no relation to AF_IUCV and AF_SMC. For further information, see vsock(7).

AF_KCM

KCM (kernel connection multiplexer) interface (since Linux 4.6). See Documentation/networking/kcm.txt in the Linux kernel source tree for details.

AF_QIPCRTR

Qualcomm IPC router interface protocol (since Linux 4.7).

AF_SMC

SMC-R (shared memory communications over RDMA) protocol (since Linux 4.11), and SMC-D (shared memory communications, direct memory access) protocol for intra-node z/VM quest interaction (since Linux 4.19); has no relation to AF_RDS, AF_IUCV or AF_VSOCK. See RFC 7609 IBM's Shared Memory Communications over RDMA (SMC-R) Protocol for details regarding SMC-R. See SMC-D Reference Information for details regarding SMC-D.

AF_XDP

XDP (express data path) interface (since Linux 4.18). See Documentation/networking/af_xdp.rst in the Linux kernel source tree for details.

## SEE ALSO
socket(2), socket(7)


aio

title: aio(7)

summary: POSIX asynchronous I/O overview

text:
# aio(7)

## NAME
aio - POSIX asynchronous I/O overview

## DESCRIPTION
The POSIX asynchronous I/O (AIO) interface allows applications to initiate one or more I/O operations that are performed asynchronously (i.e., in the background). The application can elect to be notified of completion of the I/O operation in a variety of ways: by delivery of a signal, by instantiation of a thread, or no notification at all.

The POSIX AIO interface consists of the following functions:

aio_read(3)

Enqueue a read request. This is the asynchronous analog of read(2).

aio_write(3)

Enqueue a write request. This is the asynchronous analog of write(2).

aio_fsync(3)

Enqueue a sync request for the I/O operations on a file descriptor. This is the asynchronous analog of fsync(2) and fdatasync(2).

aio_error(3)

Obtain the error status of an enqueued I/O request.

aio_return(3)

Obtain the return status of a completed I/O request.

aio_suspend(3)

Suspend the caller until one or more of a specified set of I/O requests completes.

aio_cancel(3)

Attempt to cancel outstanding I/O requests on a specified file descriptor.

lio_listio(3)

Enqueue multiple I/O requests using a single function call.

The aiocb ("asynchronous I/O control block") structure defines parameters that control an I/O operation. An argument of this type is employed with all of the functions listed above. This structure has the following form:

```
#include <aiocb.h>
struct aiocb {

    /* The order of these fields is implementation-dependent */

    int             aio_fildes;     /* File descriptor */

    off_t           aio_offset;     /* File offset */

    volatile void  *aio_buf;        /* Location of buffer */

    size_t          aio_nbytes;     /* Length of transfer */

    int             aio_reqprio;    /* Request priority */

    struct sigevent aio_sigevent;   /* Notification method */

    int             aio_lio_opcode; /* Operation to be performed;

                                       lio_listio() only */

    /* Various implementation-internal fields not shown */
};
/* Operation codes for 'aio_lio_opcode': */
enum { LIO_READ, LIO_WRITE, LIO_NOP };
```

The fields of this structure are as follows:

aio_fildes

The file descriptor on which the I/O operation is to be performed.

aio_offset

This is the file offset at which the I/O operation is to be performed.

aio_buf

This is the buffer used to transfer data for a read or write operation.

aio_nbytes

This is the size of the buffer pointed to by aio_buf.

aio_reqprio

This field specifies a value that is subtracted from the calling thread's real-time priority in order to determine the priority for execution of this I/O request (see pthread_setschedparam(3)). The specified value must be between 0 and the value returned by sysconf(_SC_AIO_PRIO_DELTA_MAX). This field is ignored for file synchronization operations.

aio_sigevent

This field is a structure that specifies how the caller is to be notified when the asynchronous I/O operation completes. Possible values for aio_sigevent.sigev_notify are SIGEV_NONE, SIGEV_SIGNAL, and SIGEV_THREAD. See sigevent(7) for further details.

aio_lio_opcode

The type of operation to be performed; used only for lio_listio(3).

In addition to the standard functions listed above, the GNU C library provides the following extension to the POSIX AIO API:

aio_init(3)

Set parameters for tuning the behavior of the glibc POSIX AIO implementation.

## ERRORS
EINVAL

The aio_reqprio field of the aiocb structure was less than 0, or was greater than the limit returned by the call sysconf(_SC_AIO_PRIO_DELTA_MAX).

## STANDARDS
POSIX.1-2008.

## HISTORY
POSIX.1-2001. glibc 2.1.

## NOTES
It is a good idea to zero out the control block buffer before use (see memset(3)). The control block buffer and the buffer pointed to by aio_buf must not be changed while the I/O operation is in progress. These buffers must remain valid until the I/O operation completes.

Simultaneous asynchronous read or write operations using the same aiocb structure yield undefined results.

The current Linux POSIX AIO implementation is provided in user space by glibc. This has a number of limitations, most notably that maintaining multiple threads to perform I/O operations is expensive and scales poorly. Work has been in progress for some time on a kernel state-machine-based implementation of asynchronous I/O (see io_submit(2), io_setup(2), io_cancel(2), io_destroy(2), io_getevents(2)), but this implementation hasn't yet matured to the point where the POSIX AIO implementation can be completely reimplemented using the kernel system calls.

## EXAMPLES
The program below opens each of the files named in its command-line arguments and queues a request on the resulting file descriptor using aio_read(3). The program then loops, periodically monitoring each of the I/O operations that is still in progress using aio_error(3). Each of the I/O requests is set up to provide notification by delivery of a signal. After all I/O requests have completed, the program retrieves their status using aio_return(3).

The SIGQUIT signal (generated by typing control-\) causes the program to request cancelation of each of the outstanding requests using aio_cancel(3).

Here is an example of what we might see when running this program. In this example, the program queues two requests to standard input, and these are satisfied by two lines of input containing "abc" and "x".

```
$ ./a.out /dev/stdin /dev/stdin
opened /dev/stdin on descriptor 3
opened /dev/stdin on descriptor 4
aio_error():

    for request 0 (descriptor 3): In progress

    for request 1 (descriptor 4): In progress
abc
I/O completion signal received
aio_error():

    for request 0 (descriptor 3): I/O succeeded

    for request 1 (descriptor 4): In progress
aio_error():

    for request 1 (descriptor 4): In progress
x
I/O completion signal received
aio_error():

    for request 1 (descriptor 4): I/O succeeded
All I/O requests completed
aio_return():

    for request 0 (descriptor 3): 4

    for request 1 (descriptor 4): 2
```

### Program source

```
#include <fcntl.h>
#include <stdlib.h>
#include <unistd.h>
#include <stdio.h>
#include <errno.h>
#include <aio.h>
#include <signal.h>
#define BUF_SIZE 20     /* Size of buffers for read operations */
#define errExit(msg) do { perror(msg); exit(EXIT_FAILURE); } while (0)
struct ioRequest {      /* Application-defined structure for tracking

                           I/O requests */

    int           reqNum;

    int           status;

    struct aiocb *aiocbp;
};
static volatile sig_atomic_t gotSIGQUIT = 0;

                        /* On delivery of SIGQUIT, we attempt to

                           cancel all outstanding I/O requests */
static void             /* Handler for SIGQUIT */
quitHandler(int sig)
{

    gotSIGQUIT = 1;
}
#define IO_SIGNAL SIGUSR1   /* Signal used to notify I/O completion */
static void                 /* Handler for I/O completion signal */
aioSigHandler(int sig, siginfo_t *si, void *ucontext)
{

    if (si->si_code == SI_ASYNCIO) {

        write(STDOUT_FILENO, "I/O completion signal received\n", 31);

        /* The corresponding ioRequest structure would be available as

               struct ioRequest *ioReq = si->si_value.sival_ptr;

           and the file descriptor would then be available via

               ioReq->aiocbp->aio_fildes */

    }
}
int
main(int argc, char *argv[])
{

    struct sigaction sa;

    int s;

    int numReqs;        /* Total number of queued I/O requests */

    int openReqs;       /* Number of I/O requests still in progress */

    if (argc < 2) {

        fprintf(stderr, "Usage: %s <pathname> <pathname>...\n",

                argv[0]);

        exit(EXIT_FAILURE);

    }

    numReqs = argc - 1;

    /* Allocate our arrays. */

    struct ioRequest *ioList = calloc(numReqs, sizeof(*ioList));

    if (ioList == NULL)

        errExit("calloc");

    struct aiocb *aiocbList = calloc(numReqs, sizeof(*aiocbList));

    if (aiocbList == NULL)

        errExit("calloc");

    /* Establish handlers for SIGQUIT and the I/O completion signal. */

    sa.sa_flags = SA_RESTART;

    sigemptyset(&sa.sa_mask);

    sa.sa_handler = quitHandler;

    if (sigaction(SIGQUIT, &sa, NULL) == -1)

        errExit("sigaction");

    sa.sa_flags = SA_RESTART | SA_SIGINFO;

    sa.sa_sigaction = aioSigHandler;

    if (sigaction(IO_SIGNAL, &sa, NULL) == -1)

        errExit("sigaction");

    /* Open each file specified on the command line, and queue

       a read request on the resulting file descriptor. */

    for (size_t j = 0; j < numReqs; j++) {

        ioList[j].reqNum = j;

        ioList[j].status = EINPROGRESS;

        ioList[j].aiocbp = &aiocbList[j];

        ioList[j].aiocbp->aio_fildes = open(argv[j + 1], O_RDONLY);

        if (ioList[j].aiocbp->aio_fildes == -1)

            errExit("open");

        printf("opened %s on descriptor %d\n", argv[j + 1],

                ioList[j].aiocbp->aio_fildes);

        ioList[j].aiocbp->aio_buf = malloc(BUF_SIZE);

        if (ioList[j].aiocbp->aio_buf == NULL)

            errExit("malloc");

        ioList[j].aiocbp->aio_nbytes = BUF_SIZE;

        ioList[j].aiocbp->aio_reqprio = 0;

        ioList[j].aiocbp->aio_offset = 0;

        ioList[j].aiocbp->aio_sigevent.sigev_notify = SIGEV_SIGNAL;

        ioList[j].aiocbp->aio_sigevent.sigev_signo = IO_SIGNAL;

        ioList[j].aiocbp->aio_sigevent.sigev_value.sival_ptr =

                                &ioList[j];

        s = aio_read(ioList[j].aiocbp);

        if (s == -1)

            errExit("aio_read");

    }

    openReqs = numReqs;

    /* Loop, monitoring status of I/O requests. */

    while (openReqs > 0) {

        sleep(3);       /* Delay between each monitoring step */

        if (gotSIGQUIT) {

            /* On receipt of SIGQUIT, attempt to cancel each of the

               outstanding I/O requests, and display status returned

               from the cancelation requests. */

            printf("got SIGQUIT; canceling I/O requests: \n");

            for (size_t j = 0; j < numReqs; j++) {

                if (ioList[j].status == EINPROGRESS) {

                    printf("    Request %zu on descriptor %d:", j,

                            ioList[j].aiocbp->aio_fildes);

                    s = aio_cancel(ioList[j].aiocbp->aio_fildes,

                            ioList[j].aiocbp);

                    if (s == AIO_CANCELED)

                        printf("I/O canceled\n");

                    else if (s == AIO_NOTCANCELED)

                        printf("I/O not canceled\n");

                    else if (s == AIO_ALLDONE)

                        printf("I/O all done\n");

                    else

                        perror("aio_cancel");

                }

            }

            gotSIGQUIT = 0;

        }

        /* Check the status of each I/O request that is still

           in progress. */

        printf("aio_error():\n");

        for (size_t j = 0; j < numReqs; j++) {

            if (ioList[j].status == EINPROGRESS) {

                printf("    for request %zu (descriptor %d): ",

                        j, ioList[j].aiocbp->aio_fildes);

                ioList[j].status = aio_error(ioList[j].aiocbp);

                switch (ioList[j].status) {

                case 0:

                    printf("I/O succeeded\n");

                    break;

                case EINPROGRESS:

                    printf("In progress\n");

                    break;

                case ECANCELED:

                    printf("Canceled\n");

                    break;

                default:

                    perror("aio_error");

                    break;

                }

                if (ioList[j].status != EINPROGRESS)

                    openReqs--;

            }

        }

    }

    printf("All I/O requests completed\n");

    /* Check status return of all I/O requests. */

    printf("aio_return():\n");

    for (size_t j = 0; j < numReqs; j++) {

        ssize_t s;

        s = aio_return(ioList[j].aiocbp);

        printf("    for request %zu (descriptor %d): %zd\n",

                j, ioList[j].aiocbp->aio_fildes, s);

    }

    exit(EXIT_SUCCESS);
}
```

## SEE ALSO
io_cancel(2), io_destroy(2), io_getevents(2), io_setup(2), io_submit(2), aio_cancel(3), aio_error(3), aio_init(3), aio_read(3), aio_return(3), aio_write(3), lio_listio(3)

"Asynchronous I/O Support in Linux 2.5", Bhattacharya, Pratt, Pulavarty, and Morgan, Proceedings of the Linux Symposium, 2003, https://www.kernel.org/doc/ols/2003/ols2003-pages-351-366.pdf


title: armscii-8(7)

text:
# armscii-8(7)

## NAME
armscii-8 - Armenian character set encoded in octal, decimal, and hexadecimal

## DESCRIPTION
The Armenian Standard Code for Information Interchange, 8-bit coded character set.

### ArmSCII-8 characters
The following table displays the characters in ArmSCII-8 that are printable and unlisted in the ascii(7) manual page.

Oct

Dec

Hex

Char

Description

240

160

A0

NO-BREAK SPACE

242

162

A2

և

ARMENIAN SMALL LIGATURE ECH YIWN

243

163

A3

։

ARMENIAN FULL STOP

244

164

A4

)

RIGHT PARENTHESIS

245

165

A5

(

LEFT PARENTHESIS

246

166

A6

»

RIGHT-POINTING DOUBLE ANGLE QUOTATION MARK

247

167

A7

«

LEFT-POINTING DOUBLE ANGLE QUOTATION MARK

250

168

A8

—

EM DASH

251

169

A9

.

FULL STOP

252

170

AA

՝

ARMENIAN COMMA

253

171

AB

,

COMMA

254

172

AC

-

HYPHEN-MINUS

255

173

AD

֊

ARMENIAN HYPHEN

256

174

AE

…

HORIZONTAL ELLIPSIS

257

175

AF

՜

ARMENIAN EXCLAMATION MARK

260

176

B0

՛

ARMENIAN EMPHASIS MARK

261

177

B1

՞

ARMENIAN QUESTION MARK

262

178

B2

Ա

ARMENIAN CAPITAL LETTER AYB

263

179

B3

ա

ARMENIAN SMALL LETTER AYB

264

180

B4

Բ

ARMENIAN CAPITAL LETTER BEN

265

181

B5

բ

ARMENIAN SMALL LETTER BEN

266

182

B6

Գ

ARMENIAN CAPITAL LETTER GIM

267

183

B7

գ

ARMENIAN SMALL LETTER GIM

270

184

B8

Դ

ARMENIAN CAPITAL LETTER DA

271

185

B9

դ

ARMENIAN SMALL LETTER DA

272

186

BA

Ե

ARMENIAN CAPITAL LETTER ECH

273

187

BB

ե

ARMENIAN SMALL LETTER ECH

274

188

BC

Զ

ARMENIAN CAPITAL LETTER ZA

275

189

BD

զ

ARMENIAN SMALL LETTER ZA

276

190

BE

Է

ARMENIAN CAPITAL LETTER EH

277

191

BF

է

ARMENIAN SMALL LETTER EH

300

192

C0

Ը

ARMENIAN CAPITAL LETTER ET

301

193

C1

ը

ARMENIAN SMALL LETTER ET

302

194

C2

Թ

ARMENIAN CAPITAL LETTER TO

303

195

C3

թ

ARMENIAN SMALL LETTER TO

304

196

C4

Ժ

ARMENIAN CAPITAL LETTER ZHE

305

197

C5

ժ

ARMENIAN SMALL LETTER ZHE

306

198

C6

Ի

ARMENIAN CAPITAL LETTER INI

307

199

C7

ի

ARMENIAN SMALL LETTER INI

310

200

C8

Լ

ARMENIAN CAPITAL LETTER LIWN

311

201

C9

լ

ARMENIAN SMALL LETTER LIWN

312

202

CA

Խ

ARMENIAN CAPITAL LETTER XEH

313

203

CB

խ

ARMENIAN SMALL LETTER XEH

314

204

CC

Ծ

ARMENIAN CAPITAL LETTER CA

315

205

CD

ծ

ARMENIAN SMALL LETTER CA

316

206

CE

Կ

ARMENIAN CAPITAL LETTER KEN

317

207

CF

կ

ARMENIAN SMALL LETTER KEN

320

208

D0

Հ

ARMENIAN CAPITAL LETTER HO

321

209

D1

հ

ARMENIAN SMALL LETTER HO

322

210

D2

Ձ

ARMENIAN CAPITAL LETTER JA

323

211

D3

ձ

ARMENIAN SMALL LETTER JA

324

212

D4

Ղ

ARMENIAN CAPITAL LETTER GHAD

325

213

D5

ղ

ARMENIAN SMALL LETTER GHAD

326

214

D6

Ճ

ARMENIAN CAPITAL LETTER CHEH

327

215

D7

ճ

ARMENIAN SMALL LETTER CHEH

330

216

D8

Մ

ARMENIAN CAPITAL LETTER MEN

331

217

D9

մ

ARMENIAN SMALL LETTER MEN

332

218

DA

Յ

ARMENIAN CAPITAL LETTER YI

333

219

DB

յ

ARMENIAN SMALL LETTER YI

334

220

DC

Ն

ARMENIAN CAPITAL LETTER NOW

335

221

DD

ն

ARMENIAN SMALL LETTER NOW

336

222

DE

Շ

ARMENIAN CAPITAL LETTER SHA

337

223

DF

շ

ARMENIAN SMALL LETTER SHA

340

224

E0

Ո

ARMENIAN CAPITAL LETTER VO

341

225

E1

ո

ARMENIAN SMALL LETTER VO

342

226

E2

Չ

ARMENIAN CAPITAL LETTER CHA

343

227

E3

չ

ARMENIAN SMALL LETTER CHA

344

228

E4

Պ

ARMENIAN CAPITAL LETTER PEH

345

229

E5

պ

ARMENIAN SMALL LETTER PEH

346

230

E6

Ջ

ARMENIAN CAPITAL LETTER JHEH

347

231

E7

ջ

ARMENIAN SMALL LETTER JHEH

350

232

E8

Ռ

ARMENIAN CAPITAL LETTER RA

351

233

E9

ռ

ARMENIAN SMALL LETTER RA

352

234

EA

Ս

ARMENIAN CAPITAL LETTER SEH

353

235

EB

ս

ARMENIAN SMALL LETTER SEH

354

236

EC

Վ

ARMENIAN CAPITAL LETTER VEW

355

237

ED

վ

ARMENIAN SMALL LETTER VEW

356

238

EE

Տ

ARMENIAN CAPITAL LETTER TIWN

357

239

EF

տ

ARMENIAN SMALL LETTER TIWN

360

240

F0

Ր

ARMENIAN CAPITAL LETTER REH

361

241

F1

ր

ARMENIAN SMALL LETTER REH

362

242

F2

Ց

ARMENIAN CAPITAL LETTER CO

363

243

F3

ց

ARMENIAN SMALL LETTER CO

364

244

F4

Ւ

ARMENIAN CAPITAL LETTER YIWN

365

245

F5

ւ

ARMENIAN SMALL LETTER YIWN

366

246

F6

Փ

ARMENIAN CAPITAL LETTER PIWR

367

247

F7

փ

ARMENIAN SMALL LETTER PIWR

370

248

F8

Ք

ARMENIAN CAPITAL LETTER KEH

371

249

F9

ք

ARMENIAN SMALL LETTER KEH

372

250

FA

Օ

ARMENIAN CAPITAL LETTER OH

373

251

FB

օ

ARMENIAN SMALL LETTER OH

374

252

FC

Ֆ

ARMENIAN CAPITAL LETTER FEH

375

253

FD

ֆ

ARMENIAN SMALL LETTER FEH

376

254

FE

՚

ARMENIAN APOSTROPHE

## SEE ALSO
ascii(7), charsets(7), utf-8(7)


arp

title: arp(7)

summary: Linux ARP kernel module.

text:
# arp(7)

## NAME
arp - Linux ARP kernel module.

## DESCRIPTION
This kernel protocol module implements the Address Resolution Protocol defined in RFC 826. It is used to convert between Layer2 hardware addresses and IPv4 protocol addresses on directly connected networks. The user normally doesn't interact directly with this module except to configure it; instead it provides a service for other protocols in the kernel.

A user process can receive ARP packets by using packet(7) sockets. There is also a mechanism for managing the ARP cache in user-space by using netlink(7) sockets. The ARP table can also be controlled via ioctl(2) on any AF_INET socket.

The ARP module maintains a cache of mappings between hardware addresses and protocol addresses. The cache has a limited size so old and less frequently used entries are garbage-collected. Entries which are marked as permanent are never deleted by the garbage-collector. The cache can be directly manipulated by the use of ioctls and its behavior can be tuned by the /proc interfaces described below.

When there is no positive feedback for an existing mapping after some time (see the /proc interfaces below), a neighbor cache entry is considered stale. Positive feedback can be gotten from a higher layer; for example from a successful TCP ACK. Other protocols can signal forward progress using the MSG_CONFIRM flag to sendmsg(2). When there is no forward progress, ARP tries to reprobe. It first tries to ask a local arp daemon app_solicit times for an updated MAC address. If that fails and an old MAC address is known, a unicast probe is sent ucast_solicit times. If that fails too, it will broadcast a new ARP request to the network. Requests are sent only when there is data queued for sending.

Linux will automatically add a nonpermanent proxy arp entry when it receives a request for an address it forwards to and proxy arp is enabled on the receiving interface. When there is a reject route for the target, no proxy arp entry is added.

### Ioctls
Three ioctls are available on all AF_INET sockets. They take a pointer to a struct arpreq as their argument.

```
struct arpreq {

    struct sockaddr arp_pa;      /* protocol address */

    struct sockaddr arp_ha;      /* hardware address */

    int             arp_flags;   /* flags */

    struct sockaddr arp_netmask; /* netmask of protocol address */

    char            arp_dev[16];
};
```

SIOCSARP, SIOCDARP and SIOCGARP respectively set, delete, and get an ARP mapping. Setting and deleting ARP maps are privileged operations and may be performed only by a process with the CAP_NET_ADMIN capability or an effective UID of 0.

arp_pa must be an AF_INET address and arp_ha must have the same type as the device which is specified in arp_dev. arp_dev is a zero-terminated string which names a device.

arp_flags

flag

meaning

ATF_COM

Lookup complete

ATF_PERM

Permanent entry

ATF_PUBL

Publish entry

ATF_USETRAILERS

Trailers requested

ATF_NETMASK

Use a netmask

ATF_DONTPUB

Don't answer

If the ATF_NETMASK flag is set, then arp_netmask should be valid. Linux 2.2 does not support proxy network ARP entries, so this should be set to 0xffffffff, or 0 to remove an existing proxy arp entry. ATF_USETRAILERS is obsolete and should not be used.

### /proc interfaces
ARP supports a range of /proc interfaces to configure parameters on a global or per-interface basis. The interfaces can be accessed by reading or writing the /proc/sys/net/ipv4/neigh/*/* files. Each interface in the system has its own directory in /proc/sys/net/ipv4/neigh/. The setting in the "default" directory is used for all newly created devices. Unless otherwise specified, time-related interfaces are specified in seconds.

anycast_delay (since Linux 2.2)

The maximum number of jiffies to delay before replying to a IPv6 neighbor solicitation message. Anycast support is not yet implemented. Defaults to 1 second.

app_solicit (since Linux 2.2)

The maximum number of probes to send to the user space ARP daemon via netlink before dropping back to multicast probes (see mcast_solicit). Defaults to 0.

base_reachable_time (since Linux 2.2)

Once a neighbor has been found, the entry is considered to be valid for at least a random value between base_reachable_time/2 and 3*base_reachable_time/2. An entry's validity will be extended if it receives positive feedback from higher level protocols. Defaults to 30 seconds. This file is now obsolete in favor of base_reachable_time_ms.

base_reachable_time_ms (since Linux 2.6.12)

As for base_reachable_time, but measures time in milliseconds. Defaults to 30000 milliseconds.

delay_first_probe_time (since Linux 2.2)

Delay before first probe after it has been decided that a neighbor is stale. Defaults to 5 seconds.

gc_interval (since Linux 2.2)

How frequently the garbage collector for neighbor entries should attempt to run. Defaults to 30 seconds.

gc_stale_time (since Linux 2.2)

Determines how often to check for stale neighbor entries. When a neighbor entry is considered stale, it is resolved again before sending data to it. Defaults to 60 seconds.

gc_thresh1 (since Linux 2.2)

The minimum number of entries to keep in the ARP cache. The garbage collector will not run if there are fewer than this number of entries in the cache. Defaults to 128.

gc_thresh2 (since Linux 2.2)

The soft maximum number of entries to keep in the ARP cache. The garbage collector will allow the number of entries to exceed this for 5 seconds before collection will be performed. Defaults to 512.

gc_thresh3 (since Linux 2.2)

The hard maximum number of entries to keep in the ARP cache. The garbage collector will always run if there are more than this number of entries in the cache. Defaults to 1024.

locktime (since Linux 2.2)

The minimum number of jiffies to keep an ARP entry in the cache. This prevents ARP cache thrashing if there is more than one potential mapping (generally due to network misconfiguration). Defaults to 1 second.

mcast_solicit (since Linux 2.2)

The maximum number of attempts to resolve an address by multicast/broadcast before marking the entry as unreachable. Defaults to 3.

proxy_delay (since Linux 2.2)

When an ARP request for a known proxy-ARP address is received, delay up to proxy_delay jiffies before replying. This is used to prevent network flooding in some cases. Defaults to 0.8 seconds.

proxy_qlen (since Linux 2.2)

The maximum number of packets which may be queued to proxy-ARP addresses. Defaults to 64.

retrans_time (since Linux 2.2)

The number of jiffies to delay before retransmitting a request. Defaults to 1 second. This file is now obsolete in favor of retrans_time_ms.

retrans_time_ms (since Linux 2.6.12)

The number of milliseconds to delay before retransmitting a request. Defaults to 1000 milliseconds.

ucast_solicit (since Linux 2.2)

The maximum number of attempts to send unicast probes before asking the ARP daemon (see app_solicit). Defaults to 3.

unres_qlen (since Linux 2.2)

The maximum number of packets which may be queued for each unresolved address by other network layers. Defaults to 3.

## VERSIONS
The struct arpreq changed in Linux 2.0 to include the arp_dev member and the ioctl numbers changed at the same time. Support for the old ioctls was dropped in Linux 2.2.

Support for proxy arp entries for networks (netmask not equal 0xffffffff) was dropped in Linux 2.2. It is replaced by automatic proxy arp setup by the kernel for all reachable hosts on other interfaces (when forwarding and proxy arp is enabled for the interface).

The neigh/* interfaces did not exist before Linux 2.2.

## BUGS
Some timer settings are specified in jiffies, which is architecture- and kernel version-dependent; see time(7).

There is no way to signal positive feedback from user space. This means connection-oriented protocols implemented in user space will generate excessive ARP traffic, because ndisc will regularly reprobe the MAC address. The same problem applies for some kernel protocols (e.g., NFS over UDP).

This man page mashes together functionality that is IPv4-specific with functionality that is shared between IPv4 and IPv6.

## SEE ALSO
capabilities(7), ip(7), arpd(8)

RFC 826 for a description of ARP. RFC 2461 for a description of IPv6 neighbor discovery and the base algorithms used. Linux 2.2+ IPv4 ARP uses the IPv6 algorithms when applicable.