Thin Provisioning

A number of storages, and the QEMU image format qcow2, support thin provisioning. With thin provisioning activated, only the blocks that the guest system actually use will be written to the storage.

Say for instance you create a VM with a 32GB hard disk, and after installing the guest system OS, the root file system of the VM contains 3 GB of data. In that case only 3GB are written to the storage, even if the guest VM sees a 32GB hard drive. In this way thin provisioning allows you to create disk images which are larger than the currently available storage blocks. You can create large disk images for your VMs, and when the need arises, add more disks to your storage without resizing the VMs' file systems.

All storage types which have the “Snapshots” feature also support thin provisioning.

Storage Pools

Each storage pool has a <type>, and is uniquely identified by its <STORAGE_ID>. A pool configuration looks like this:

<type>: <STORAGE_ID>


        <property> <value>


        <property> <value>


        <property>


        ...

The <type>: <STORAGE_ID> line starts the pool definition, which is then followed by a list of properties. Most properties require a value. Some have reasonable defaults, in which case you can omit the value.

To be more specific, take a look at the default storage configuration after installation. It contains one special local storage pool named local, which refers to the directory /var/lib/vz and is always available. The Proxmox VE installer creates additional storage entries depending on the storage type chosen at installation time.

Default storage configuration (/etc/pve/storage.cfg).

dir: local


        path /var/lib/vz


        content iso,vztmpl,backup
# default image store on LVM based installation
lvmthin: local-lvm


        thinpool data


        vgname pve


        content rootdir,images
# default image store on ZFS based installation
zfspool: local-zfs


        pool rpool/data


        sparse


        content images,rootdir

Common Storage Properties

A few storage properties are common among different storage types.

nodes

content

shared

disable

maxfiles

prune-backups

format

preallocation

Volume Ownership

There exists an ownership relation for image type volumes. Each such volume is owned by a VM or Container. For example volume local:230/example-image.raw is owned by VM 230. Most storage backends encodes this ownership information into the volume name.

When you remove a VM or Container, the system also removes all associated volumes which are owned by that VM or Container.

Examples

Add storage pools

pvesm add <TYPE> <STORAGE_ID> <OPTIONS>
pvesm add dir <STORAGE_ID> --path <PATH>
pvesm add nfs <STORAGE_ID> --path <PATH> --server <SERVER> --export <EXPORT>
pvesm add lvm <STORAGE_ID> --vgname <VGNAME>
pvesm add iscsi <STORAGE_ID> --portal <HOST[:PORT]> --target <TARGET>

Disable storage pools

pvesm set <STORAGE_ID> --disable 1

Enable storage pools

pvesm set <STORAGE_ID> --disable 0

Change/set storage options

pvesm set <STORAGE_ID> <OPTIONS>
pvesm set <STORAGE_ID> --shared 1
pvesm set local --format qcow2
pvesm set <STORAGE_ID> --content iso

Remove storage pools. This does not delete any data, and does not disconnect or unmount anything. It just removes the storage configuration.

pvesm remove <STORAGE_ID>

Allocate volumes

pvesm alloc <STORAGE_ID> <VMID> <name> <size> [--format <raw|qcow2>]

Allocate a 4G volume in local storage. The name is auto-generated if you pass an empty string as <name>

pvesm alloc local <VMID> '' 4G

Free volumes

pvesm free <VOLUME_ID>

List storage status

pvesm status

List storage contents

pvesm list <STORAGE_ID> [--vmid <VMID>]

List volumes allocated by VMID

pvesm list <STORAGE_ID> --vmid <VMID>

List iso images

pvesm list <STORAGE_ID> --iso

List container templates

pvesm list <STORAGE_ID> --vztmpl

Show file system path for a volume

pvesm path <VOLUME_ID>

Exporting the volume local:103/vm-103-disk-0.qcow2 to the file target. This is mostly used internally with pvesm import. The stream format qcow2+size is different to the qcow2 format. Consequently, the exported file cannot simply be attached to a VM. This also holds for the other formats.

pvesm export local:103/vm-103-disk-0.qcow2 qcow2+size target --with-snapshots 1

Configuration

This backend supports all common storage properties, and adds two additional properties. The path property is used to specify the directory. This needs to be an absolute file system path.

The optional content-dirs property allows for the default layout to be changed. It consists of a comma-separated list of identifiers in the following format:

vtype=path

Where vtype is one of the allowed content types for the storage, and path is a path relative to the mountpoint of the storage.

Configuration Example (/etc/pve/storage.cfg).

dir: backup


        path /mnt/backup


        content backup


        prune-backups keep-last=7


        max-protected-backups 3


        content-dirs backup=custom/backup/dir

The above configuration defines a storage pool called backup. That pool can be used to store up to 7 regular backups (keep-last=7) and 3 protected backups per VM. The real path for the backup files is /mnt/backup/custom/backup/dir/....

File naming conventions

This backend uses a well defined naming scheme for VM images:

vm-<VMID>-<NAME>.<FORMAT>

<VMID>

<NAME>

<FORMAT>

When you create a VM template, all VM images are renamed to indicate that they are now read-only, and can be used as a base image for clones:

base-<VMID>-<NAME>.<FORMAT>

Storage Features

As mentioned above, most file systems do not support snapshots out of the box. To workaround that problem, this backend is able to use qcow2 internal snapshot capabilities.

Same applies to clones. The backend uses the qcow2 base image feature to create clones.

Table 3. Storage features for backend dir

Examples

Please use the following command to allocate a 4GB image on storage local:

# pvesm alloc local 100 vm-100-disk10.raw 4G
Formatting '/var/lib/vz/images/100/vm-100-disk10.raw', fmt=raw size=4294967296
successfully created 'local:100/vm-100-disk10.raw'

The real file system path is shown with:

# pvesm path local:100/vm-100-disk10.raw
/var/lib/vz/images/100/vm-100-disk10.raw

And you can remove the image with:

# pvesm free local:100/vm-100-disk10.raw

Configuration

The backend supports all common storage properties, except the shared flag, which is always set. Additionally, the following properties are used to configure the NFS server:

server

export

You can also set NFS mount options:

path

content-dirs

options

Configuration Example (/etc/pve/storage.cfg).

nfs: iso-templates


        path /mnt/pve/iso-templates


        server 10.0.0.10


        export /space/iso-templates


        options vers=3,soft


        content iso,vztmpl

Storage Features

NFS does not support snapshots, but the backend uses qcow2 features to implement snapshots and cloning.

Table 4. Storage features for backend nfs

Examples

You can get a list of exported NFS shares with:

# pvesm nfsscan <server>

Configuration

The backend supports all common storage properties, except the shared flag, which is always set. Additionally, the following CIFS special properties are available:

server

share

username

password

domain

smbversion

path

content-dirs

subdir

Configuration Example (/etc/pve/storage.cfg).

cifs: backup


        path /mnt/pve/backup


        server 10.0.0.11


        share VMData


        content backup


        username anna


        smbversion 3


        subdir /data

Storage Features

CIFS does not support snapshots on a storage level. But you may use qcow2 backing files if you still want to have snapshots and cloning features available.

Table 5. Storage features for backend cifs

Examples

You can get a list of exported CIFS shares with:

# pvesm scan cifs <server> [--username <username>] [--password]

Then you could add this share as a storage to the whole Proxmox VE cluster with:

# pvesm add cifs <storagename> --server <server> --share <share> [--username <username>] [--password]

Configuration

The backend supports all common storage properties, except the shared flag, which is always set. Additionally, the following special properties to Proxmox Backup Server are available:

server

username

password

datastore

fingerprint

encryption-key

master-pubkey

Configuration Example (/etc/pve/storage.cfg).

pbs: backup


        datastore main


        server enya.basealt.ru


        content backup


        fingerprint 09:54:ef:..snip..:88:af:47:fe:4c:3b:cf:8b:26:88:0b:4e:3c:b2


        prune-backups keep-all=1


        username archiver@pbs

Storage Features

Proxmox Backup Server only supports backups, they can be block-level or file-level based. Proxmox VE uses block-level for virtual machines and file-level for container.

Table 6. Storage features for backend pbs

Encryption

Optionally, you can configure client-side encryption with AES-256 in GCM mode. Encryption can be configured either via the web interface, or on the CLI with the encryption-key option (see above). The key will be saved in the file /etc/pve/priv/storage/<STORAGE-ID>.enc, which is only accessible by the root user.

It is recommended that you keep your key safe, but easily accessible, in order for quick disaster recovery. For this reason, the best place to store it is in your password manager, where it is immediately recoverable. As a backup to this, you should also save the key to a USB drive and store that in a secure place. This way, it is detached from any system, but is still easy to recover from, in case of emergency. Finally, in preparation for the worst case scenario, you should also consider keeping a paper copy of your key locked away in a safe place. The paperkey subcommand can be used to create a QR encoded version of your key. The following command sends the output of the paperkey command to a text file, for easy printing.

# proxmox-backup-client key paperkey /etc/pve/priv/storage/<STORAGE-ID>.enc --output-format text > qrkey.txt

Additionally, it is possible to use a single RSA master key pair for key recovery purposes: configure all clients doing encrypted backups to use a single public master key, and all subsequent encrypted backups will contain a RSA-encrypted copy of the used AES encryption key. The corresponding private master key allows recovering the AES key and decrypting the backup even if the client system is no longer available.

Because the encryption is managed on the client side, you can use the same datastore on the server for unencrypted backups and encrypted backups, even if they are encrypted with different keys. However, deduplication between backups with different keys is not possible, so it is often better to create separate datastores.

Example: Add Storage over CLI

Then you could add this share as a storage to the whole Proxmox VE cluster with:

# pvesm add pbs <id> --server <server> --datastore <datastore> --username <username> --fingerprint 00:B4:... --password

Configuration

The backend supports all common storage properties, and adds the following GlusterFS specific options:

server

server2

volume

transport

Configuration Example (/etc/pve/storage.cfg).

glusterfs: Gluster


        server 10.2.3.4


        server2 10.2.3.5


        volume glustervol


        content images,iso

File naming conventions

The directory layout and the file naming conventions are inherited from the dir backend.

Storage Features

The storage provides a file level interface, but no native snapshot/clone implementation.

Table 7. Storage features for backend glusterfs

Configuration

The backend supports the common storage properties content, nodes, disable, and the following ZFS specific properties:

pool

blocksize

sparse

mountpoint

Configuration Example (/etc/pve/storage.cfg).

zfspool: vmdata


        pool tank/vmdata


        content rootdir,images


        sparse

File naming conventions

The backend uses the following naming scheme for VM images:

vm-<VMID>-<NAME>      // normal VM images
base-<VMID>-<NAME>    // template VM image (read-only)
subvol-<VMID>-<NAME>  // subvolumes (ZFS filesystem for containers)

<VMID>

<NAME>

Storage Features

ZFS is probably the most advanced storage type regarding snapshot and cloning. The backend uses ZFS datasets for both VM images (format raw) and container data (format subvol). ZFS properties are inherited from the parent dataset, so you can simply set defaults on the parent dataset.

Table 8. Storage features for backend zfs

Examples

It is recommended to create an extra ZFS file system to store your VM images:

# zfs create tank/vmdata

To enable compression on that newly allocated file system:

# zfs set compression=on tank/vmdata

You can get a list of available ZFS filesystems with:

# pvesm zfsscan

Configuration

The LVM backend supports the common storage properties content, nodes, disable, and the following LVM specific properties:

vgname

base

saferemove

saferemove_throughput

Configuration Example (/etc/pve/storage.cfg).

lvm: myspace


        vgname myspace


        content rootdir,images

File naming conventions

The backend use basically the same naming conventions as the ZFS pool backend.

vm-<VMID>-<NAME>      // normal VM images

Storage Features

LVM is a typical block storage, but this backend does not support snapshots and clones. Unfortunately, normal LVM snapshots are quite inefficient, because they interfere with all writes on the entire volume group during snapshot time.

One big advantage is that you can use it on top of a shared storage, for example, an iSCSI LUN. The backend itself implements proper cluster-wide locking.

Table 9. Storage features for backend lvm

Examples

List available volume groups:

# pvesm lvmscan

Configuration

The LVM thin backend supports the common storage properties content, nodes, disable, and the following LVM specific properties:

vgname

thinpool

Configuration Example (/etc/pve/storage.cfg).

lvmthin: local-lvm


        thinpool data


        vgname pve


        content rootdir,images

File naming conventions

The backend use basically the same naming conventions as the ZFS pool backend.

vm-<VMID>-<NAME>      // normal VM images

Storage Features

LVM thin is a block storage, but fully supports snapshots and clones efficiently. New volumes are automatically initialized with zero.

It must be mentioned that LVM thin pools cannot be shared across multiple nodes, so you can only use them as local storage.

Table 10. Storage features for backend lvmthin

Examples

List available LVM thin pools on volume group pve:

# pvesm lvmthinscan pve

Configuration

The backend supports the common storage properties content, nodes, disable, and the following iSCSI specific properties:

portal

target

Configuration Example (/etc/pve/storage.cfg).

iscsi: mynas


     portal 10.10.10.1


     target iqn.2006-01.openfiler.com:tsn.dcb5aaaddd


     content none

File naming conventions

The iSCSI protocol does not define an interface to allocate or delete data. Instead, that needs to be done on the target side and is vendor specific. The target simply exports them as numbered LUNs. So Proxmox VE iSCSI volume names just encodes some information about the LUN as seen by the linux kernel.

Storage Features

iSCSI is a block level type storage, and provides no management interface. So it is usually best to export one big LUN, and setup LVM on top of that LUN. You can then use the LVM plugin to manage the storage on that iSCSI LUN.

Table 11. Storage features for backend iscsi

Examples

Scan a remote iSCSI portal, and returns a list of possible targets:

pvesm scan iscsi <HOST[:PORT]>

Configuration

The user mode iSCSI backend uses the same configuration options as the Open-iSCSI backed.

Configuration Example (/etc/pve/storage.cfg).

iscsidirect: faststore


     portal 10.10.10.1


     target iqn.2006-01.openfiler.com:tsn.dcb5aaaddd

Storage Features

Table 12. Storage features for backend iscsidirect

Configuration

This backend supports the common storage properties nodes, disable, content, and the following rbd specific properties:

monhost

pool

username

krbd

Configuration Example for a external Ceph cluster (/etc/pve/storage.cfg).

rbd: ceph-external


        monhost 10.1.1.20 10.1.1.21 10.1.1.22


        pool ceph-external


        content images


        username admin

Authentication

If you use cephx authentication, which is enabled by default, you need to provide the keyring from the external Ceph cluster.

To configure the storage via the CLI, you first need to make the file containing the keyring available. One way is to copy the file from the external Ceph cluster directly to one of the Proxmox VE nodes. The following example will copy it to the /root directory of the node on which we run it:

# scp <external cephserver>:/etc/ceph/ceph.client.admin.keyring /root/rbd.keyring

Then use the pvesm CLI tool to configure the external RBD storage, use the --keyring parameter, which needs to be a path to the keyring file that you copied. For example:

# pvesm add rbd <name> --monhost "10.1.1.20 10.1.1.21 10.1.1.22" --content images --keyring /root/rbd.keyring

When configuring an external RBD storage via the GUI, you can copy and paste the keyring into the appropriate field.

The keyring will be stored at

# /etc/pve/priv/ceph/<STORAGE_ID>.keyring

Ceph client configuration (optional)

Connecting to an external Ceph storage doesn’t always allow setting client-specific options in the config DB on the external cluster. You can add a ceph.conf beside the Ceph keyring to change the Ceph client configuration for the storage.

The ceph.conf needs to have the same name as the storage.

# /etc/pve/priv/ceph/<STORAGE_ID>.conf

See the RBD configuration reference [6] for possible settings.

Storage Features

The rbd backend is a block level storage, and implements full snapshot and clone functionality.

Table 13. Storage features for backend rbd

Configuration

This backend supports the common storage properties nodes, disable, content, as well as the following cephfs specific properties:

monhost

path

username

subdir

fuse

Configuration example for an external Ceph cluster (/etc/pve/storage.cfg).

cephfs: cephfs-external


        monhost 10.1.1.20 10.1.1.21 10.1.1.22


        path /mnt/pve/cephfs-external


        content backup


        username admin

Authentication

If you use cephx authentication, which is enabled by default, you need to provide the secret from the external Ceph cluster.

To configure the storage via the CLI, you first need to make the file containing the secret available. One way is to copy the file from the external Ceph cluster directly to one of the Proxmox VE nodes. The following example will copy it to the /root directory of the node on which we run it:

# scp <external cephserver>:/etc/ceph/cephfs.secret /root/cephfs.secret

Then use the pvesm CLI tool to configure the external RBD storage, use the --keyring parameter, which needs to be a path to the secret file that you copied. For example:

# pvesm add cephfs <name> --monhost "10.1.1.20 10.1.1.21 10.1.1.22" --content backup --keyring /root/cephfs.secret

When configuring an external RBD storage via the GUI, you can copy and paste the secret into the appropriate field.

The secret is only the key itself, as opposed to the rbd backend which also contains a [client.userid] section.

The secret will be stored at

# /etc/pve/priv/ceph/<STORAGE_ID>.secret

A secret can be received from the Ceph cluster (as Ceph admin) by issuing the command below, where userid is the client ID that has been configured to access the cluster. For further information on Ceph user management, see the Ceph docs.[1]

# ceph auth get-key client.userid > cephfs.secret

Storage Features

The cephfs backend is a POSIX-compliant filesystem, on top of a Ceph cluster.

Table 14. Storage features for backend cephfs

[1] While no known bugs exist, snapshots are not yet guaranteed to be stable, as they lack sufficient testing.

Configuration

This backend is configured similarly to the directory storage. Note that when adding a directory as a BTRFS storage, which is not itself also the mount point, it is highly recommended to specify the actual mount point via the is_mountpoint option.

For example, if a BTRFS file system is mounted at /mnt/data2 and its pve-storage/ subdirectory (which may be a snapshot, which is recommended) should be added as a storage pool called data2, you can use the following entry:

btrfs: data2


        path /mnt/data2/pve-storage


        content rootdir,images


        is_mountpoint /mnt/data2

Snapshots

When taking a snapshot of a subvolume or raw file, the snapshot will be created as a read-only subvolume with the same path followed by an @ and the snapshot’s name.

Configuration

In order to use the ZFS over iSCSI plugin you need to configure the remote machine (target) to accept ssh connections from the Proxmox VE node. Proxmox VE connects to the target for creating the ZVOLs and exporting them via iSCSI. Authentication is done through a ssh-key (without password protection) stored in /etc/pve/priv/zfs/<target_ip>_id_rsa

The following steps create a ssh-key and distribute it to the storage machine with IP 192.0.2.1:

mkdir /etc/pve/priv/zfs
ssh-keygen -f /etc/pve/priv/zfs/192.0.2.1_id_rsa
ssh-copy-id -i /etc/pve/priv/zfs/192.0.2.1_id_rsa.pub root@192.0.2.1
ssh -i /etc/pve/priv/zfs/192.0.2.1_id_rsa root@192.0.2.1

The backend supports the common storage properties content, nodes, disable, and the following ZFS over ISCSI specific properties:

pool

portal

target

iscsiprovider

comstar_tg

comstar_hg

lio_tpg

nowritecache

blocksize

sparse

Configuration Examples (/etc/pve/storage.cfg).

zfs: lio


   blocksize 4k


   iscsiprovider LIO


   pool tank


   portal 192.0.2.111


   target iqn.2003-01.org.linux-iscsi.lio.x8664:sn.xxxxxxxxxxxx


   content images


   lio_tpg tpg1


   sparse 1
zfs: solaris


   blocksize 4k


   target iqn.2010-08.org.illumos:02:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx:tank1


   pool tank


   iscsiprovider comstar


   portal 192.0.2.112


   content images
zfs: freebsd


   blocksize 4k


   target iqn.2007-09.jp.ne.peach.istgt:tank1


   pool tank


   iscsiprovider istgt


   portal 192.0.2.113


   content images
zfs: iet


   blocksize 4k


   target iqn.2001-04.com.example:tank1


   pool tank


   iscsiprovider iet


   portal 192.0.2.114


   content images

Storage Features

The ZFS over iSCSI plugin provides a shared storage, which is capable of snapshots. You need to make sure that the ZFS appliance does not become a single point of failure in your deployment.

Table 15. Storage features for backend iscsi