Create via Web GUI

Under Datacenter → Cluster, click on Create Cluster. Enter the cluster name and select a network connection from the drop-down list to serve as the main cluster network (Link 0). It defaults to the IP resolved via the node’s hostname.

As of Proxmox VE 6.2, up to 8 fallback links can be added to a cluster. To add a redundant link, click the Add button and select a link number and IP address from the respective fields. Prior to Proxmox VE 6.2, to add a second link as fallback, you can select the Advanced checkbox and choose an additional network interface (Link 1, see also Corosync Redundancy (man pvecm(1))).

Create via the Command Line

Login via ssh to the first Proxmox VE node and run the following command:

 hp1# pvecm create CLUSTERNAME

To check the state of the new cluster use:

 hp1# pvecm status

Multiple Clusters in the Same Network

It is possible to create multiple clusters in the same physical or logical network. In this case, each cluster must have a unique name to avoid possible clashes in the cluster communication stack. Furthermore, this helps avoid human confusion by making clusters clearly distinguishable.

While the bandwidth requirement of a corosync cluster is relatively low, the latency of packets and the packets per second (PPS) rate is the limiting factor. Different clusters in the same network can compete with each other for these resources, so it may still make sense to use separate physical network infrastructure for bigger clusters.

Join Node to Cluster via GUI

Log in to the web interface on an existing cluster node. Under Datacenter → Cluster, click the Join Information button at the top. Then, click on the button Copy Information. Alternatively, copy the string from the Information field manually.

Next, log in to the web interface on the node you want to add. Under Datacenter → Cluster, click on Join Cluster. Fill in the Information field with the Join Information text you copied earlier. Most settings required for joining the cluster will be filled out automatically. For security reasons, the cluster password has to be entered manually.

After clicking the Join button, the cluster join process will start immediately. After the node has joined the cluster, its current node certificate will be replaced by one signed from the cluster certificate authority (CA). This means that the current session will stop working after a few seconds. You then might need to force-reload the web interface and log in again with the cluster credentials.

Now your node should be visible under Datacenter → Cluster.

Join Node to Cluster via Command Line

Log in to the node you want to join into an existing cluster via ssh.

 # pvecm add IP-ADDRESS-CLUSTER

For IP-ADDRESS-CLUSTER, use the IP or hostname of an existing cluster node. An IP address is recommended (see Link Address Types (man pvecm(1))).

By default, pvecm add authenticates against the existing cluster node’s API, prompting for that node’s root@pam password and verifying its SHA-256 certificate fingerprint. SSH password authentication is not required for this default flow, so administrators may keep PasswordAuthentication no set in sshd_config.

Pass --use_ssh to fall back to the legacy SSH-based join, which uses ssh-copy-id to merge the existing cluster’s root key onto the new node. That flow does require root SSH login (with password or key) on the existing node.

To check the state of the cluster use:

 # pvecm status

Cluster status after adding 4 nodes.

 # pvecm status
Cluster information
~~~~~~~~~~~~~~~~~~~
Name:             prod-central
Config Version:   3
Transport:        knet
Secure auth:      on
Quorum information
~~~~~~~~~~~~~~~~~~
Date:             Tue Sep 14 11:06:47 2021
Quorum provider:  corosync_votequorum
Nodes:            4
Node ID:          0x00000001
Ring ID:          1.1a8
Quorate:          Yes
Votequorum information
~~~~~~~~~~~~~~~~~~~~~~
Expected votes:   4
Highest expected: 4
Total votes:      4
Quorum:           3
Flags:            Quorate
Membership information
~~~~~~~~~~~~~~~~~~~~~~


    Nodeid      Votes Name
0x00000001          1 192.168.15.91
0x00000002          1 192.168.15.92 (local)
0x00000003          1 192.168.15.93
0x00000004          1 192.168.15.94

If you only want a list of all nodes, use:

 # pvecm nodes

List nodes in a cluster.

 # pvecm nodes
Membership information
~~~~~~~~~~~~~~~~~~~~~~


    Nodeid      Votes Name


         1          1 hp1


         2          1 hp2 (local)


         3          1 hp3


         4          1 hp4

Adding Nodes with Separated Cluster Network

When adding a node to a cluster with a separated cluster network, you need to use the link0 parameter to set the nodes address on that network:

# pvecm add IP-ADDRESS-CLUSTER --link0 LOCAL-IP-ADDRESS-LINK0

If you want to use the built-in redundancy (man pvecm(1)) of the Kronosnet transport layer, also use the link1 parameter.

Using the GUI, you can select the correct interface from the corresponding Link X fields in the Cluster Join dialog.

Prerequisites

Remove the Cluster Node

Before a node can be removed from a cluster, you must ensure that it is no longer part of a Ceph cluster and that no Ceph resources or services are residing on it.

In the following example, the cluster node node4 will be removed from both the Ceph-, and Proxmox VE corosync-cluster.

node4# pvecm nodes
Membership information
~~~~~~~~~~~~~~~~~~~~~~


    Nodeid      Votes Name


         1          1 node1


         2          1 node2


         3          1 node3


         4          1 node4 (local)

node4# pveceph mds destroy node4

node4# ceph osd crush remove node4

node1# pvecm delnode node4

node1# pvecm nodes
Membership information
~~~~~~~~~~~~~~~~~~~~~~


    Nodeid      Votes Name


         1          1 node1 (local)


         2          1 node2


         3          1 node3

Cleanup Steps After Node Removal

Rejoin the same node again:. If you want the same server to join the same cluster again, you have to:

Network Requirements

The Proxmox VE cluster stack requires a reliable network with latencies under 5 milliseconds (LAN performance) between all nodes to operate stably. While on setups with a small node count a network with higher latencies may work, this is not guaranteed and gets rather unlikely with more than three nodes and latencies above around 10 ms.

The network should not be used heavily by other members, as while corosync does not use much bandwidth, it is sensitive to latency jitters; ideally corosync runs on its own physically separated network. Especially do not use a shared network for corosync and storage (except as a potential low-priority fallback in a redundant (man pvecm(1)) configuration).

Before setting up a cluster, it is good practice to check if the network is fit for that purpose. To ensure that the nodes can connect to each other on the cluster network, you can test the connectivity between them with the ping tool.

If the Proxmox VE firewall is enabled, ACCEPT rules for corosync will automatically be generated - no manual action is required.

Corosync Over Bonds

Recommendations

Background

Separate Cluster Network

When creating a cluster without any parameters, the corosync cluster network is generally shared with the web interface and the VMs' network. Depending on your setup, even storage traffic may get sent over the same network. It’s recommended to change that, as corosync is a time-critical, real-time application.

Setting Up a New Network

Separate On Cluster Creation

pvecm create test --link0 10.10.10.1

systemctl status corosync

Separate After Cluster Creation

logging {


  debug: off


  to_syslog: yes
}
nodelist {


  node {


    name: due


    nodeid: 2


    quorum_votes: 1


    ring0_addr: due


  }


  node {


    name: tre


    nodeid: 3


    quorum_votes: 1


    ring0_addr: tre


  }


  node {


    name: uno


    nodeid: 1


    quorum_votes: 1


    ring0_addr: uno


  }
}
quorum {


  provider: corosync_votequorum
}
totem {


  cluster_name: testcluster


  config_version: 3


  ip_version: ipv4-6


  secauth: on


  version: 2


  interface {


    linknumber: 0


  }
}

logging {


  debug: off


  to_syslog: yes
}
nodelist {


  node {


    name: due


    nodeid: 2


    quorum_votes: 1


    ring0_addr: 10.10.10.2


  }


  node {


    name: tre


    nodeid: 3


    quorum_votes: 1


    ring0_addr: 10.10.10.3


  }


  node {


    name: uno


    nodeid: 1


    quorum_votes: 1


    ring0_addr: 10.10.10.1


  }
}
quorum {


  provider: corosync_votequorum
}
totem {


  cluster_name: testcluster


  config_version: 4


  ip_version: ipv4-6


  secauth: on


  version: 2


  interface {


    linknumber: 0


  }
}

systemctl restart corosync

systemctl status corosync

Corosync Addresses

A corosync link address (for backwards compatibility denoted by ringX_addr in corosync.conf) can be specified in two ways:

A separate, static hostname specifically for corosync is recommended, if hostnames are preferred. Also, make sure that every node in the cluster can resolve all hostnames correctly.

Since Proxmox VE 5.1, while supported, hostnames will be resolved at the time of entry. Only the resolved IP is saved to the configuration.

Nodes that joined the cluster on earlier versions likely still use their unresolved hostname in corosync.conf. It might be a good idea to replace them with IPs or a separate hostname, as mentioned above.

Adding Redundant Links To An Existing Cluster

To add a new link to a running configuration, first check how to edit the corosync.conf file (man pvecm(1)).

Then, add a new ringX_addr to every node in the nodelist section. Make sure that your X is the same for every node you add it to, and that it is unique for each node.

Lastly, add a new interface, as shown below, to your totem section, replacing X with the link number chosen above.

Assuming you added a link with number 1, the new configuration file could look like this:

logging {


  debug: off


  to_syslog: yes
}
nodelist {


  node {


    name: due


    nodeid: 2


    quorum_votes: 1


    ring0_addr: 10.10.10.2


    ring1_addr: 10.20.20.2


  }


  node {


    name: tre


    nodeid: 3


    quorum_votes: 1


    ring0_addr: 10.10.10.3


    ring1_addr: 10.20.20.3


  }


  node {


    name: uno


    nodeid: 1


    quorum_votes: 1


    ring0_addr: 10.10.10.1


    ring1_addr: 10.20.20.1


  }
}
quorum {


  provider: corosync_votequorum
}
totem {


  cluster_name: testcluster


  config_version: 4


  ip_version: ipv4-6


  secauth: on


  version: 2


  interface {


    linknumber: 0


  }


  interface {


    linknumber: 1


  }
}

The new link will be enabled as soon as you follow the last steps to edit the corosync.conf file (man pvecm(1)). A restart should not be necessary. You can check that corosync loaded the new link using:

journalctl -b -u corosync

It might be a good idea to test the new link by temporarily disconnecting the old link on one node and making sure that its status remains online while disconnected:

pvecm status

If you see a healthy cluster state, it means that your new link is being used.

SSH setup

On Proxmox VE systems, the following changes are made to the SSH configuration/setup:

Pitfalls due to automatic execution of .bashrc and siblings

In case you have a custom .bashrc, or similar files that get executed on login by the configured shell, ssh will automatically run it once the session is established successfully. This can cause some unexpected behavior, as those commands may be executed with root permissions on any of the operations described above. This can cause possible problematic side-effects!

In order to avoid such complications, it’s recommended to add a check in /root/.bashrc to make sure the session is interactive, and only then run .bashrc commands.

You can add this snippet at the beginning of your .bashrc file:

# Early exit if not running interactively to avoid side-effects!
case $- in


    *i*) ;;


      *) return;;
esac

QDevice Technical Overview

The Corosync Quorum Device (QDevice) is a daemon which runs on each cluster node. It provides a configured number of votes to the cluster’s quorum subsystem, based on an externally running third-party arbitrator’s decision. Its primary use is to allow a cluster to sustain more node failures than standard quorum rules allow. This can be done safely as the external device can see all nodes and thus choose only one set of nodes to give its vote. This will only be done if said set of nodes can have quorum (again) after receiving the third-party vote.

Currently, only QDevice Net is supported as a third-party arbitrator. This is a daemon which provides a vote to a cluster partition, if it can reach the partition members over the network. It will only give votes to one partition of a cluster at any time. It’s designed to support multiple clusters and is almost configuration and state free. New clusters are handled dynamically and no configuration file is needed on the host running a QDevice.

The only requirements for the external host are that it needs network access to the cluster and to have a corosync-qnetd package available. We provide a package for Debian based hosts, and other Linux distributions should also have a package available through their respective package manager.

Supported Setups

We support QDevices for clusters with an even number of nodes and recommend it for 2 node clusters, if they should provide higher availability. For clusters with an odd node count, we currently discourage the use of QDevices. The reason for this is the difference in the votes which the QDevice provides for each cluster type. Even numbered clusters get a single additional vote, which only increases availability, because if the QDevice itself fails, you are in the same position as with no QDevice at all.

On the other hand, with an odd numbered cluster size, the QDevice provides (N-1) votes — where N corresponds to the cluster node count. This alternative behavior makes sense; if it had only one additional vote, the cluster could get into a split-brain situation. This algorithm allows for all nodes but one (and naturally the QDevice itself) to fail. However, there are two drawbacks to this:

If you understand the drawbacks and implications, you can decide yourself if you want to use this technology in an odd numbered cluster setup.

QDevice-Net Setup

We recommend running any daemon which provides votes to corosync-qdevice as an unprivileged user. Proxmox VE and Debian provide a package which is already configured to do so. The traffic between the daemon and the cluster must be encrypted to ensure a safe and secure integration of the QDevice in Proxmox VE.

First, install the corosync-qnetd package on your external server

external# apt install corosync-qnetd

and the corosync-qdevice package on all cluster nodes

pve# apt install corosync-qdevice

After doing this, ensure that all the nodes in the cluster are online.

You can now set up your QDevice by running the following command on one of the Proxmox VE nodes:

pve# pvecm qdevice setup <QDEVICE-IP>

The SSH key from the cluster will be automatically copied to the QDevice.

After all the steps have successfully completed, you will see "Done". You can verify that the QDevice has been set up with:

pve# pvecm status
...
Votequorum information
~~~~~~~~~~~~~~~~~~~~~
Expected votes:   3
Highest expected: 3
Total votes:      3
Quorum:           2
Flags:            Quorate Qdevice
Membership information
~~~~~~~~~~~~~~~~~~~~~~


    Nodeid      Votes    Qdevice Name


    0x00000001      1    A,V,NMW 192.168.22.180 (local)


    0x00000002      1    A,V,NMW 192.168.22.181


    0x00000000      1            Qdevice

QDevice Status Flags

Frequently Asked Questions

Tie Breaking

Possible Negative Implications

Adding/Deleting Nodes After QDevice Setup

Removing the QDevice

pve# pvecm qdevice remove

Edit corosync.conf

Editing the corosync.conf file is not always very straightforward. There are two on each cluster node, one in /etc/pve/corosync.conf and the other in /etc/corosync/corosync.conf. Editing the one in our cluster file system will propagate the changes to the local one, but not vice versa.

The configuration will get updated automatically, as soon as the file changes. This means that changes which can be integrated in a running corosync will take effect immediately. Thus, you should always make a copy and edit that instead, to avoid triggering unintended changes when saving the file while editing.

cp /etc/pve/corosync.conf /etc/pve/corosync.conf.new

Then, open the config file with your favorite editor, such as nano or vim.tiny, which come pre-installed on every Proxmox VE node.

After making the necessary changes, create another copy of the current working configuration file. This serves as a backup if the new configuration fails to apply or causes other issues.

cp /etc/pve/corosync.conf /etc/pve/corosync.conf.bak

Then replace the old configuration file with the new one:

mv /etc/pve/corosync.conf.new /etc/pve/corosync.conf

You can check if the changes could be applied automatically, using the following commands:

systemctl status corosync
journalctl -b -u corosync

If the changes could not be applied automatically, you may have to restart the corosync service via:

systemctl restart corosync

On errors, check the troubleshooting section below.

Changing the Token Coefficient

The token coefficient can be configured in the totem section in /etc/pve/corosync.conf. If the token coefficient is not explicitly set, it defaults to 650 milliseconds. Since Proxmox VE 9.2, new clusters are created with a lower token coefficient of 125 milliseconds explicitly set in /etc/pve/corosync.conf. This reduces the time needed for reestablishing a new cluster membership after a node failure.

corosync uses the token coefficient to calculate several timeouts in relation to the cluster size [7], most importantly the token timeout and consensus timeout. corosync implements a token-passing protocol. The token timeout specifies how long a node waits for the token until it declares the token to be lost [8]. The consensus timeout [9] specifies the time nodes wait for a consensus on a new cluster membership. The sum of token and consensus timeouts defines the minimum time needed to reestablish a new cluster membership after a node goes offline.

Keeping the sum of token and consensus timeouts below 30 seconds reduces the time needed for reestablishing a new cluster membership after a node failure. When HA is enabled, it is especially important that this time stays below 45 seconds to ensure that a new cluster membership is formed before the watchdog timeout (man ha-manager(1)) of 60 seconds expires, which would trigger a node fence. A safety margin below this 45 second limit is advisable so that transient timing variations do not push the cluster past the watchdog threshold. The recommended mechanism for lowering the token and consensus timeouts is lowering the token coefficient as explained below.

You can check the current token and consensus timeouts (in milliseconds) with the following command:

corosync-cmapctl | grep -Ew 'runtime.config.totem.token|runtime.config.totem.consensus'

For example:

runtime.config.totem.consensus (u32) = 5940
runtime.config.totem.token (u32) = 4950

The sum of these two values (10.89 seconds in the example) defines the minimum time needed to reestablish a new cluster membership after a node goes offline. Lowering the token coefficient is

To lower the token coefficient, first make sure your setup adheres to the network requirements (man pvecm(1)), especially with regards to latency. Then:

token = 3000 + (number_of_nodes - 2) * token_coefficient
consensus = 1.2 * token

After adjusting the token_coefficient in corosync.conf, recent corosync versions will automatically adopt the new value for the cluster. For corosync versions below 3.1.10-pve1, corosync needs to be restarted on all nodes for the change to take effect.

Troubleshooting

Issue: quorum.expected_votes must be configured

[...]
corosync[1647]:  [QUORUM] Quorum provider: corosync_votequorum failed to initialize.
corosync[1647]:  [SERV  ] Service engine 'corosync_quorum' failed to load for reason


    'configuration error: nodelist or quorum.expected_votes must be configured!'
[...]

Write Configuration When Not Quorate

pvecm expected 1

Corosync Configuration Glossary

ringX_addr

Migration Type

The migration type defines if the migration data should be sent over an encrypted (secure) channel or an unencrypted (insecure) one. Setting the migration type to insecure means that the RAM content of a virtual guest is also transferred unencrypted, which can lead to information disclosure of critical data from inside the guest (for example, passwords or encryption keys).

Therefore, we strongly recommend using the secure channel if you do not have full control over the network and can not guarantee that no one is eavesdropping on it.

Encryption requires a lot of computing power, so this setting is often changed to insecure to achieve better performance. The impact on modern systems is lower because they implement AES encryption in hardware. The performance impact is particularly evident in fast networks, where you can transfer 10 Gbps or more.

Migration Network

By default, Proxmox VE uses the network in which cluster communication takes place to send the migration traffic. This is not optimal both because sensitive cluster traffic can be disrupted and this network may not have the best bandwidth available on the node.

Setting the migration network parameter allows the use of a dedicated network for all migration traffic. In addition to the memory, this also affects the storage traffic for offline migrations.

The migration network is set as a network using CIDR notation. This has the advantage that you don’t have to set individual IP addresses for each node. Proxmox VE can determine the real address on the destination node from the network specified in the CIDR form. To enable this, the network must be specified so that each node has exactly one IP in the respective network.

Example

iface eno1 inet manual
# public network
auto vmbr0
iface vmbr0 inet static


    address 192.X.Y.57/24


    gateway 192.X.Y.1


    bridge-ports eno1


    bridge-stp off


    bridge-fd 0
# cluster network
auto eno2
iface eno2 inet static


    address  10.1.1.1/24
# fast network
auto eno3
iface eno3 inet static


    address  10.1.2.1/24

# qm migrate 106 tre --online --migration_network 10.1.2.0/24

# use dedicated migration network
migration: secure,network=10.1.2.0/24