pve-firewall

package: pve-firewall

title: pve-firewall(8)

summary: PVE Firewall Daemon

text:
# pve-firewall(8)

## NAME
pve-firewall - PVE Firewall Daemon

## SYNOPSIS
pve-firewall <COMMAND> [ARGS] [OPTIONS]

pve-firewall compile

Compile and print firewall rules. This is useful for testing.

pve-firewall help [OPTIONS]

Get help about specified command.

--extra-args <array>

Shows help for a specific command

--verbose <boolean>

Verbose output format.

pve-firewall localnet

Print information about local network.

pve-firewall restart

Restart the Proxmox VE firewall service.

pve-firewall simulate [OPTIONS]

Simulate firewall rules. This does not simulates the kernel routing table, but simply assumes that routing from source zone to destination zone is possible.

--dest <string>

Destination IP address.

--dport <integer>

Destination port.

--from (host|outside|vm\d+|ct\d+|([a-zA-Z][a-zA-Z0-9]{0,9})/(\S+)) (default = outside)

Source zone.

--protocol (tcp|udp) (default = tcp)

Protocol.

--source <string>

Source IP address.

--sport <integer>

Source port.

--to (host|outside|vm\d+|ct\d+|([a-zA-Z][a-zA-Z0-9]{0,9})/(\S+)) (default = host)

Destination zone.

--verbose <boolean> (default = 0)

Verbose output.

pve-firewall start [OPTIONS]

Start the Proxmox VE firewall service.

--debug <boolean> (default = 0)

Debug mode - stay in foreground

pve-firewall status

Get firewall status.

pve-firewall stop

Stop the Proxmox VE firewall service. Note, stopping actively removes all Proxmox VE related iptable rules rendering the host potentially unprotected.

## DESCRIPTION
Proxmox VE Firewall provides an easy way to protect your IT infrastructure. You can setup firewall rules for all hosts inside a cluster, or define rules for virtual machines and containers. Features like firewall macros, security groups, IP sets and aliases help to make that task easier.

While all configuration is stored on the cluster file system, the iptables-based firewall service runs on each cluster node, and thus provides full isolation between virtual machines. The distributed nature of this system also provides much higher bandwidth than a central firewall solution.

The firewall has full support for IPv4 and IPv6. IPv6 support is fully transparent, and we filter traffic for both protocols by default. So there is no need to maintain a different set of rules for IPv6.

## DIRECTIONS & ZONES
The Proxmox VE firewall groups the network into multiple logical zones. You can define rules for each zone independently. Depending on the zone, you can define rules for incoming, outgoing or forwarded traffic.

### Directions
There are 3 directions that you can choose from when defining rules for a zone:

In

Traffic that is arriving in a zone.

Out

Traffic that is leaving a zone.

Forward

Traffic that is passing through a zone. In the host zone this can be routed traffic (when the host is acting as a gateway or performing NAT). At a VNet-level this affects all traffic that is passing by a VNet, including traffic from/to bridged network interfaces.

Important Creating rules for forwarded traffic is currently only possible when using the new nftables-based proxmox-firewall (man pve-firewall(8)). Any forward rules will be ignored by the stock pve-firewall and have no effect!

### Zones
There are 3 different zones that you can define firewall rules for:

Host

Traffic going from/to a host, or traffic that is forwarded by a host. You can define rules for this zone either at the datacenter level or at the host level. Rules at host level take precedence over rules at datacenter level.

VM

Traffic going from/to a VM or CT. You cannot define rules for forwarded traffic, only for incoming / outgoing traffic.

VNet

Traffic passing through a SDN VNet, either from guest to guest or from host to guest and vice-versa. Since this traffic is always forwarded traffic, it is only possible to create rules with direction forward.

Important Creating rules on a VNet-level is currently only possible when using the new nftables-based proxmox-firewall (man pve-firewall(8)). Any VNet-level rules will be ignored by the stock pve-firewall and have no effect!

## CONFIGURATION FILES
All firewall related configuration is stored on the proxmox cluster file system. So those files are automatically distributed to all cluster nodes, and the pve-firewall service updates the underlying iptables rules automatically on changes.

You can configure anything using the GUI (i.e. Datacenter → Firewall, or on a Node → Firewall), or you can edit the configuration files directly using your preferred editor.

Firewall configuration files contain sections of key-value pairs. Lines beginning with a # and blank lines are considered comments. Sections start with a header line containing the section name enclosed in [ and ].

### Cluster Wide Setup
The cluster-wide firewall configuration is stored at:

```
/etc/pve/firewall/cluster.fw
```

The configuration can contain the following sections:

[OPTIONS]

This is used to set cluster-wide firewall options.

ebtables: <boolean> (default = 1)

Enable ebtables rules cluster wide.

enable: <integer> (0 - N) (default = 0)

Enable or disable the firewall cluster wide.

log_ratelimit: [enable=]<1|0> [,burst=<integer>] [,rate=<rate>]

Log ratelimiting settings burst=<integer> (0 - N) (default = 5)

Initial burst of packages which will always get logged before the rate is applied

enable=<boolean> (default = 1)

Enable or disable log rate limiting

rate=<rate> (default = 1/second)

Frequency with which the burst bucket gets refilled

policy_forward: <ACCEPT | DROP>

Forward policy.

policy_in: <ACCEPT | DROP | REJECT>

Input policy.

policy_out: <ACCEPT | DROP | REJECT>

Output policy.

[RULES]

This sections contains cluster-wide firewall rules for all nodes.

[IPSET <name>]

Cluster wide IP set definitions.

[GROUP <name>]

Cluster wide security group definitions.

[ALIASES]

Cluster wide Alias definitions.

Enabling the Firewall

The firewall is completely disabled by default, so you need to set the enable option here:

```
[OPTIONS]
# enable firewall (cluster-wide setting, default is disabled)
enable: 1
```

Important If you enable the firewall, traffic to all hosts is blocked by default. Only exceptions is WebGUI(8006) and ssh(22) from your local network.

If you want to administrate your Proxmox VE hosts from remote, you need to create rules to allow traffic from those remote IPs to the web GUI (port 8006). You may also want to allow ssh (port 22), and maybe SPICE (port 3128).

Tip Please open a SSH connection to one of your Proxmox VE hosts before enabling the firewall. That way you still have access to the host if something goes wrong .

To simplify that task, you can instead create an IPSet called “management”, and add all remote IPs there. This creates all required firewall rules to access the GUI from remote.

### Host Specific Configuration
Host related configuration is read from:

```
/etc/pve/nodes/<nodename>/host.fw
```

This is useful if you want to overwrite rules from cluster.fw config. You can also increase log verbosity, and set netfilter related options. The configuration can contain the following sections:

[OPTIONS]

This is used to set host related firewall options.

enable: <boolean> (default = 1)

Enable host firewall rules.

log_level_forward: <alert | crit | debug | emerg | err | info | nolog | notice | warning>

Log level for forwarded traffic.

log_level_in: <alert | crit | debug | emerg | err | info | nolog | notice | warning>

Log level for incoming traffic.

log_level_out: <alert | crit | debug | emerg | err | info | nolog | notice | warning>

Log level for outgoing traffic.

log_nf_conntrack: <boolean> (default = 0)

Enable logging of conntrack information.

ndp: <boolean> (default = 1)

Enable NDP (Neighbor Discovery Protocol).

nf_conntrack_allow_invalid: <boolean> (default = 0)

Allow invalid packets on connection tracking.

nf_conntrack_helpers: <string> (default = ``)

Enable conntrack helpers for specific protocols. Supported protocols: amanda, ftp, irc, netbios-ns, pptp, sane, sip, snmp, tftp

nf_conntrack_max: <integer> (32768 - N) (default = 262144)

Maximum number of tracked connections.

nf_conntrack_tcp_timeout_established: <integer> (7875 - N) (default = 432000)

Conntrack established timeout.

nf_conntrack_tcp_timeout_syn_recv: <integer> (30 - 60) (default = 60)

Conntrack syn recv timeout.

nftables: <boolean> (default = 0)

Enable nftables based firewall (tech preview)

nosmurfs: <boolean>

Enable SMURFS filter.

protection_synflood: <boolean> (default = 0)

Enable synflood protection

protection_synflood_burst: <integer> (default = 1000)

Synflood protection rate burst by ip src.

protection_synflood_rate: <integer> (default = 200)

Synflood protection rate syn/sec by ip src.

smurf_log_level: <alert | crit | debug | emerg | err | info | nolog | notice | warning>

Log level for SMURFS filter.

tcp_flags_log_level: <alert | crit | debug | emerg | err | info | nolog | notice | warning>

Log level for illegal tcp flags filter.

tcpflags: <boolean> (default = 0)

Filter illegal combinations of TCP flags.

[RULES]

This sections contains host specific firewall rules.

### VM/Container Configuration
VM firewall configuration is read from:

```
/etc/pve/firewall/<VMID>.fw
```

and contains the following data:

[OPTIONS]

This is used to set VM/Container related firewall options.

dhcp: <boolean> (default = 0)

Enable DHCP.

enable: <boolean> (default = 0)

Enable/disable firewall rules.

ipfilter: <boolean>

Enable default IP filters. This is equivalent to adding an empty ipfilter-net<id> ipset for every interface. Such ipsets implicitly contain sane default restrictions such as restricting IPv6 link local addresses to the one derived from the interface’s MAC address. For containers the configured IP addresses will be implicitly added.

log_level_in: <alert | crit | debug | emerg | err | info | nolog | notice | warning>

Log level for incoming traffic.

log_level_out: <alert | crit | debug | emerg | err | info | nolog | notice | warning>

Log level for outgoing traffic.

macfilter: <boolean> (default = 1)

Enable/disable MAC address filter.

ndp: <boolean> (default = 1)

Enable NDP (Neighbor Discovery Protocol).

policy_in: <ACCEPT | DROP | REJECT>

Input policy.

policy_out: <ACCEPT | DROP | REJECT>

Output policy.

radv: <boolean>

Allow sending Router Advertisement.

[RULES]

This sections contains VM/Container firewall rules.

[IPSET <name>]

IP set definitions.

[ALIASES]

IP Alias definitions.

Enabling the Firewall for VMs and Containers

Each virtual network device has its own firewall enable flag. So you can selectively enable the firewall for each interface. This is required in addition to the general firewall enable option.

### VNet Configuration
VNet related configuration is read from:

```
/etc/pve/sdn/firewall/<vnet_name>.fw
```

This can be used for setting firewall configuration globally on a VNet level, without having to set firewall rules for each VM inside the VNet separately. It can only contain rules for the FORWARD direction, since there is no notion of incoming or outgoing traffic. This affects all traffic travelling from one bridge port to another, including the host interface.

Warning This feature is currently only available for the new nftables-based proxmox-firewall (man pve-firewall(8))

Since traffic passing the FORWARD chain is bi-directional, you need to create rules for both directions if you want traffic to pass both ways. For instance if HTTP traffic for a specific host should be allowed, you would need to create the following rules:

```
FORWARD ACCEPT -dest 10.0.0.1 -dport 80
FORWARD ACCEPT -source 10.0.0.1 -sport 80
```

[OPTIONS]

This is used to set VNet related firewall options.

enable: <boolean> (default = 0)

Enable/disable firewall rules.

log_level_forward: <alert | crit | debug | emerg | err | info | nolog | notice | warning>

Log level for forwarded traffic.

policy_forward: <ACCEPT | DROP>

Forward policy.

[RULES]

This section contains VNet specific firewall rules.

## FIREWALL RULES
Firewall rules consists of a direction (IN, OUT or FORWARD) and an action (ACCEPT, DENY, REJECT). You can also specify a macro name. Macros contain predefined sets of rules and options. Rules can be disabled by prefixing them with |.

Firewall rules syntax.

```
[RULES]
DIRECTION ACTION [OPTIONS]
|DIRECTION ACTION [OPTIONS] # disabled rule
DIRECTION MACRO(ACTION) [OPTIONS] # use predefined macro
```

The following options can be used to refine rule matches.

--dest <string>

Restrict packet destination address. This can refer to a single IP address, an IP set (+ipsetname) or an IP alias definition. You can also specify an address range like 20.34.101.207-201.3.9.99, or a list of IP addresses and networks (entries are separated by comma). Please do not mix IPv4 and IPv6 addresses inside such lists.

--dport <string>

Restrict TCP/UDP destination port. You can use service names or simple numbers (0-65535), as defined in /etc/services. Port ranges can be specified with \d+:\d+, for example 80:85, and you can use comma separated list to match several ports or ranges.

--icmp-type <string>

Specify icmp-type. Only valid if proto equals icmp or icmpv6/ipv6-icmp.

--iface <string>

Network interface name. You have to use network configuration key names for VMs and containers (net\d+). Host related rules can use arbitrary strings.

--log <alert | crit | debug | emerg | err | info | nolog | notice | warning>

Log level for firewall rule.

--proto <string>

IP protocol. You can use protocol names (tcp/udp) or simple numbers, as defined in /etc/protocols.

--source <string>

Restrict packet source address. This can refer to a single IP address, an IP set (+ipsetname) or an IP alias definition. You can also specify an address range like 20.34.101.207-201.3.9.99, or a list of IP addresses and networks (entries are separated by comma). Please do not mix IPv4 and IPv6 addresses inside such lists.

--sport <string>

Restrict TCP/UDP source port. You can use service names or simple numbers (0-65535), as defined in /etc/services. Port ranges can be specified with \d+:\d+, for example 80:85, and you can use comma separated list to match several ports or ranges.

Here are some examples:

```
[RULES]
IN SSH(ACCEPT) -i net0
IN SSH(ACCEPT) -i net0 # a comment
IN SSH(ACCEPT) -i net0 -source 192.168.2.192 # only allow SSH from 192.168.2.192
IN SSH(ACCEPT) -i net0 -source 10.0.0.1-10.0.0.10 # accept SSH for IP range
IN SSH(ACCEPT) -i net0 -source 10.0.0.1,10.0.0.2,10.0.0.3 #accept ssh for IP list
IN SSH(ACCEPT) -i net0 -source +mynetgroup # accept ssh for ipset mynetgroup
IN SSH(ACCEPT) -i net0 -source myserveralias #accept ssh for alias myserveralias
|IN SSH(ACCEPT) -i net0 # disabled rule
IN  DROP # drop all incoming packages
OUT ACCEPT # accept all outgoing packages
```

## SECURITY GROUPS
A security group is a collection of rules, defined at cluster level, which can be used in all VMs' rules. For example you can define a group named “webserver” with rules to open the http and https ports.

```
# /etc/pve/firewall/cluster.fw
[group webserver]
IN  ACCEPT -p tcp -dport 80
IN  ACCEPT -p tcp -dport 443
```

Then, you can add this group to a VM’s firewall

```
# /etc/pve/firewall/<VMID>.fw
[RULES]
GROUP webserver
```

## IP ALIASES
IP Aliases allow you to associate IP addresses of networks with a name. You can then refer to those names:

•inside IP set definitions

•in source and dest properties of firewall rules

### Standard IP Alias local_network
This alias is automatically defined. Please use the following command to see assigned values:

```
# pve-firewall localnet
local hostname: example
local IP address: 192.168.2.100
network auto detect: 192.168.0.0/20
using detected local_network: 192.168.0.0/20
```

The firewall automatically sets up rules to allow everything needed for cluster communication (corosync, API, SSH) using this alias.

The user can overwrite these values in the cluster.fw alias section. If you use a single host on a public network, it is better to explicitly assign the local IP address

```
#  /etc/pve/firewall/cluster.fw
[ALIASES]
local_network 1.2.3.4 # use the single IP address
```

## IP SETS
IP sets can be used to define groups of networks and hosts. You can refer to them with ‘+name` in the firewall rules’ source and dest properties.

The following example allows HTTP traffic from the management IP set.

```
IN HTTP(ACCEPT) -source +management
```

### Standard IP set management
This IP set applies only to host firewalls (not VM firewalls). Those IPs are allowed to do normal management tasks (Proxmox VE GUI, VNC, SPICE, SSH).

The local cluster network is automatically added to this IP set (alias cluster_network), to enable inter-host cluster communication. (multicast,ssh,...)

```
# /etc/pve/firewall/cluster.fw
[IPSET management]
192.168.2.10
192.168.2.10/24
```

### Standard IP set blacklist
Traffic from these IPs is dropped by every host’s and VM’s firewall.

```
# /etc/pve/firewall/cluster.fw
[IPSET blacklist]
77.240.159.182
213.87.123.0/24
```

### Standard IP set ipfilter-net*
These filters belong to a VM’s network interface and are mainly used to prevent IP spoofing. If such a set exists for an interface then any outgoing traffic with a source IP not matching its interface’s corresponding ipfilter set will be dropped.

For containers with configured IP addresses these sets, if they exist (or are activated via the general IP Filter option in the VM’s firewall’s options tab), implicitly contain the associated IP addresses.

For both virtual machines and containers they also implicitly contain the standard MAC-derived IPv6 link-local address in order to allow the neighbor discovery protocol to work.

```
/etc/pve/firewall/<VMID>.fw
[IPSET ipfilter-net0] # only allow specified IPs on net0
192.168.2.10
```

## SERVICES AND COMMANDS
The firewall runs two service daemons on each node:

•pvefw-logger: NFLOG daemon (ulogd replacement).

•pve-firewall: updates iptables rules

There is also a CLI command named pve-firewall, which can be used to start and stop the firewall service:

```
# pve-firewall start
# pve-firewall stop
```

To get the status use:

```
# pve-firewall status
```

The above command reads and compiles all firewall rules, so you will see warnings if your firewall configuration contains any errors.

If you want to see the generated iptables rules you can use:

```
# iptables-save
```

## DEFAULT FIREWALL RULES
The following traffic is filtered by the default firewall configuration:

### Datacenter incoming/outgoing DROP/REJECT
If the input or output policy for the firewall is set to DROP or REJECT, the following traffic is still allowed for all Proxmox VE hosts in the cluster:

•traffic over the loopback interface

•already established connections

•traffic using the IGMP protocol

•TCP traffic from management hosts to port 8006 in order to allow access to the web interface

•TCP traffic from management hosts to the port range 5900 to 5999 allowing traffic for the VNC web console

•TCP traffic from management hosts to port 3128 for connections to the SPICE proxy

•TCP traffic from management hosts to port 22 to allow ssh access

•UDP traffic in the cluster network to ports 5405-5412 for corosync

•UDP multicast traffic in the cluster network

•ICMP traffic type 3 (Destination Unreachable), 4 (congestion control) or 11 (Time Exceeded)

The following traffic is dropped, but not logged even with logging enabled:

•TCP connections with invalid connection state

•Broadcast, multicast and anycast traffic not related to corosync, i.e., not coming through ports 5405-5412

•TCP traffic to port 43

•UDP traffic to ports 135 and 445

•UDP traffic to the port range 137 to 139

•UDP traffic form source port 137 to port range 1024 to 65535

•UDP traffic to port 1900

•TCP traffic to port 135, 139 and 445

•UDP traffic originating from source port 53

The rest of the traffic is dropped or rejected, respectively, and also logged. This may vary depending on the additional options enabled in Firewall → Options, such as NDP, SMURFS and TCP flag filtering.

Please inspect the output of the

```
 # iptables-save
```

system command to see the firewall chains and rules active on your system. This output is also included in a System Report, accessible over a node’s subscription tab in the web GUI, or through the pvereport command-line tool.

### VM/CT incoming/outgoing DROP/REJECT
This drops or rejects all the traffic to the VMs, with some exceptions for DHCP, NDP, Router Advertisement, MAC and IP filtering depending on the set configuration. The same rules for dropping/rejecting packets are inherited from the datacenter, while the exceptions for accepted incoming/outgoing traffic of the host do not apply.

Again, you can use iptables-save (see above) (man pve-firewall(8)) to inspect all rules and chains applied.

## LOGGING OF FIREWALL RULES
By default, all logging of traffic filtered by the firewall rules is disabled. To enable logging, the loglevel for incoming and/or outgoing traffic has to be set in Firewall → Options. This can be done for the host as well as for the VM/CT firewall individually. By this, logging of Proxmox VE’s standard firewall rules is enabled and the output can be observed in Firewall → Log. Further, only some dropped or rejected packets are logged for the standard rules (see default firewall rules (man pve-firewall(8))).

loglevel does not affect how much of the filtered traffic is logged. It changes a LOGID appended as prefix to the log output for easier filtering and post-processing.

loglevel is one of the following flags:

loglevel

LOGID

nolog

—

emerg

0

alert

1

crit

2

err

3

warning

4

notice

5

info

6

debug

7

A typical firewall log output looks like this:

```
VMID LOGID CHAIN TIMESTAMP POLICY: PACKET_DETAILS
```

In case of the host firewall, VMID is equal to 0.

### Logging of user defined firewall rules
In order to log packets filtered by user-defined firewall rules, it is possible to set a log-level parameter for each rule individually. This allows to log in a fine grained manner and independent of the log-level defined for the standard rules in Firewall → Options.

While the loglevel for each individual rule can be defined or changed easily in the web UI during creation or modification of the rule, it is possible to set this also via the corresponding pvesh API calls.

Further, the log-level can also be set via the firewall configuration file by appending a -log <loglevel> to the selected rule (see possible log-levels (man pve-firewall(8))).

For example, the following two are identical:

```
IN REJECT -p icmp -log nolog
IN REJECT -p icmp
```

whereas

```
IN REJECT -p icmp -log debug
```

produces a log output flagged with the debug level.

## TIPS AND TRICKS

### How to allow FTP
FTP is an old style protocol which uses port 21 and several other dynamic ports. So you need a rule to accept port 21. In addition, you need to load the ip_conntrack_ftp module. So please run:

```
modprobe ip_conntrack_ftp
```

and add ip_conntrack_ftp to /etc/modules (so that it works after a reboot).

### Suricata IPS integration
If you want to use the Suricata IPS[1] (Intrusion Prevention System), it’s possible.

Packets will be forwarded to the IPS only after the firewall ACCEPTed them.

Rejected/Dropped firewall packets don’t go to the IPS.

Install suricata on proxmox host:

```
# apt-get install suricata
# modprobe nfnetlink_queue
```

Don’t forget to add nfnetlink_queue to /etc/modules for next reboot.

Then, enable IPS for a specific VM with:

```
# /etc/pve/firewall/<VMID>.fw
[OPTIONS]
ips: 1
ips_queues: 0
```

ips_queues will bind a specific cpu queue for this VM.

Available queues are defined in

```
# /etc/default/suricata
NFQUEUE=0
```

## NOTES ON IPV6
The firewall contains a few IPv6 specific options. One thing to note is that IPv6 does not use the ARP protocol anymore, and instead uses NDP (Neighbor Discovery Protocol) which works on IP level and thus needs IP addresses to succeed. For this purpose link-local addresses derived from the interface’s MAC address are used. By default the NDP option is enabled on both host and VM level to allow neighbor discovery (NDP) packets to be sent and received.

Beside neighbor discovery NDP is also used for a couple of other things, like auto-configuration and advertising routers.

By default VMs are allowed to send out router solicitation messages (to query for a router), and to receive router advertisement packets. This allows them to use stateless auto configuration. On the other hand VMs cannot advertise themselves as routers unless the “Allow Router Advertisement” (radv: 1) option is set.

As for the link local addresses required for NDP, there’s also an “IP Filter” (ipfilter: 1) option which can be enabled which has the same effect as adding an ipfilter-net* ipset for each of the VM’s network interfaces containing the corresponding link local addresses. (See the Standard IP set ipfilter-net* (man pve-firewall(8)) section for details.)

## PORTS USED BY PROXMOX VE
•Web interface: 8006 (TCP, HTTP/1.1 over TLS)

•VNC Web console: 5900-5999 (TCP, WebSocket)

•SPICE proxy: 3128 (TCP)

•sshd (used for cluster actions): 22 (TCP)

•rpcbind: 111 (UDP)

•sendmail: 25 (TCP, outgoing)

•corosync cluster traffic: 5405-5412 UDP

•live migration (VM memory and local-disk data): 60000-60050 (TCP)

## NFTABLES
As an alternative to pve-firewall we offer proxmox-firewall, which is an implementation of the Proxmox VE firewall based on the newer nftables[2] rather than iptables.

Warning proxmox-firewall is currently in tech preview. There might be bugs or incompatibilities with the original firewall. It is currently not suited for production use.

This implementation uses the same configuration files and configuration format, so you can use your old configuration when switching. It provides the exact same functionality with a few exceptions:

•When using Linux bridges, no additional firewall bridges (fwbrX) will be created. Guest interfaces using OVS bridges will still have firewall bridges.

•REJECT is currently not possible for guest traffic (traffic will instead be dropped).

•Using the NDP, Router Advertisement or DHCP options will always create firewall rules, irregardless of your default policy.

•firewall rules for guests are evaluated even for connections that have conntrack table entries.

### Installation and Usage
Install the proxmox-firewall package:

```
apt install proxmox-firewall
```

Enable the nftables backend via the Web UI on your hosts (Host > Firewall > Options > nftables), or by enabling it in the configuration file for your hosts (/etc/pve/nodes/<node_name>/host.fw):

```
[OPTIONS]
nftables: 1
```

Note After enabling/disabling proxmox-firewall, all running VMs and containers need to be restarted for the old/new firewall to work properly.

After setting the nftables configuration key, the new proxmox-firewall service will take over. You can check if the new service is working by checking the systemctl status of proxmox-firewall:

```
systemctl status proxmox-firewall
```

You can also examine the generated ruleset. You can find more information about this in the section Helpful Commands (man pve-firewall(8)). You should also check whether pve-firewall is no longer generating iptables rules, you can find the respective commands in the Services and Commands (man pve-firewall(8)) section.

Switching back to the old firewall can be done by simply setting the configuration value back to 0 / No.

### Usage
proxmox-firewall will create two tables that are managed by the proxmox-firewall service: proxmox-firewall and proxmox-firewall-guests. If you want to create custom rules that live outside the Proxmox VE firewall configuration you can create your own tables to manage your custom firewall rules. proxmox-firewall will only touch the tables it generates, so you can easily extend and modify the behavior of the proxmox-firewall by adding your own tables.

Instead of using the pve-firewall command, the nftables-based firewall uses proxmox-firewall. It is a systemd service, so you can start and stop it via systemctl:

```
systemctl start proxmox-firewall
systemctl stop proxmox-firewall
```

Stopping the firewall service will remove all generated rules.

To query the status of the firewall, you can query the status of the systemctl service:

```
systemctl status proxmox-firewall
```

### Helpful Commands
You can check the generated ruleset via the following command:

```
nft list ruleset
```

If you want to debug proxmox-firewall you can dump the commands generated by the firewall via the compile subcommand. Additionally, setting the PVE_LOG environment variable will print log output to STDERR, which can be useful for debugging issues during the generation of the nftables ruleset:

```
PVE_LOG=trace /usr/libexec/proxmox/proxmox-firewall compile > firewall.json
```

The nftables ruleset consists of the skeleton ruleset, that is included in the proxmox-firewall binary, as well as the rules generated from the firewall configuration. You can obtain the base ruleset via the skeleton subcommand:

```
/usr/libexec/proxmox/proxmox-firewall skeleton
```

The output of both commands can be piped directly to the nft executable. The following commands will re-create the whole nftables ruleset from scratch:

```
/usr/libexec/proxmox/proxmox-firewall skeleton | nft -f -
/usr/libexec/proxmox/proxmox-firewall compile | nft -j -f -
```

You can also edit the systemctl service if you want to have detailed output for your firewall daemon while it is running:

```
systemctl edit proxmox-firewall
```

Then you need to add the override for the PVE_LOG environment variable:

```
[Service]
Environment="PVE_LOG=trace"
```

This will generate a large amount of logs very quickly, so only use this for debugging purposes. Other, less verbose, log levels are info and debug.

It can be helpful to trace packet flow through the different chains in order to debug firewall rules. This can be achieved by setting nftrace to 1 for packets that you want to track. It is advisable that you do not set this flag for all packets, in the example below we only examine ICMP packets.

```
#!/usr/sbin/nft -f
table bridge tracebridge
delete table bridge tracebridge
table bridge tracebridge {

    chain trace {

        meta l4proto icmp meta nftrace set 1

    }

    chain prerouting {

        type filter hook prerouting priority -350; policy accept;

        jump trace

    }

    chain postrouting {

        type filter hook postrouting priority -350; policy accept;

        jump trace

    }
}
```

Saving this file, making it executable, and then running it once will create the respective tracing chains. You can then inspect the tracing output via the Proxmox VE Web UI (Firewall > Log) or via nft monitor trace.

The above example traces traffic on all bridges, which is usually where guest traffic flows through. If you want to examine host traffic, create those chains in the inet table instead of the bridge table.

Note Be aware that this can generate a lot of log spam and slow down the performance of your networking stack significantly.

You can remove the tracing rules via running the following command:

```
nft delete table bridge tracebridge
```

## MACRO DEFINITIONS
Amanda

Amanda Backup

Action

proto

dport

sport

PARAM

udp

10080

PARAM

tcp

10080

Auth

Auth (identd) traffic

Action

proto

dport

sport

PARAM

tcp

113

BGP

Border Gateway Protocol traffic

Action

proto

dport

sport

PARAM

tcp

179

BitTorrent

BitTorrent traffic for BitTorrent 3.1 and earlier

Action

proto

dport

sport

PARAM

tcp

6881:6889

PARAM

udp

6881

BitTorrent32

BitTorrent traffic for BitTorrent 3.2 and later

Action

proto

dport

sport

PARAM

tcp

6881:6999

PARAM

udp

6881

CVS

Concurrent Versions System pserver traffic

Action

proto

dport

sport

PARAM

tcp

2401

Ceph

Ceph Storage Cluster traffic (Ceph Monitors, OSD & MDS Daemons)

Action

proto

dport

sport

PARAM

tcp

6789

PARAM

tcp

3300

PARAM

tcp

6800:7300

Citrix

Citrix/ICA traffic (ICA, ICA Browser, CGP)

Action

proto

dport

sport

PARAM

tcp

1494

PARAM

udp

1604

PARAM

tcp

2598

DAAP

Digital Audio Access Protocol traffic (iTunes, Rythmbox daemons)

Action

proto

dport

sport

PARAM

tcp

3689

PARAM

udp

3689

DCC

Distributed Checksum Clearinghouse spam filtering mechanism

Action

proto

dport

sport

PARAM

tcp

6277

DHCPfwd

Forwarded DHCP traffic

Action

proto

dport

sport

PARAM

udp

67:68

67:68

DHCPv6

DHCPv6 traffic

Action

proto

dport

sport

PARAM

udp

546:547

546:547

DNS

Domain Name System traffic (upd and tcp)

Action

proto

dport

sport

PARAM

udp

53

PARAM

tcp

53

Distcc

Distributed Compiler service

Action

proto

dport

sport

PARAM

tcp

3632

FTP

File Transfer Protocol

Action

proto

dport

sport

PARAM

tcp

21

Finger

Finger protocol (RFC 742)

Action

proto

dport

sport

PARAM

tcp

79

GNUnet

GNUnet secure peer-to-peer networking traffic

Action

proto

dport

sport

PARAM

tcp

2086

PARAM

udp

2086

PARAM

tcp

1080

PARAM

udp

1080

GRE

Generic Routing Encapsulation tunneling protocol

Action

proto

dport

sport

PARAM

47

Git

Git distributed revision control traffic

Action

proto

dport

sport

PARAM

tcp

9418

HKP

OpenPGP HTTP key server protocol traffic

Action

proto

dport

sport

PARAM

tcp

11371

HTTP

Hypertext Transfer Protocol (WWW)

Action

proto

dport

sport

PARAM

tcp

80

HTTPS

Hypertext Transfer Protocol (WWW) over SSL

Action

proto

dport

sport

PARAM

tcp

443

ICPV2

Internet Cache Protocol V2 (Squid) traffic

Action

proto

dport

sport

PARAM

udp

3130

ICQ

AOL Instant Messenger traffic

Action

proto

dport

sport

PARAM

tcp

5190

IMAP

Internet Message Access Protocol

Action

proto

dport

sport

PARAM

tcp

143

IMAPS

Internet Message Access Protocol over SSL

Action

proto

dport

sport

PARAM

tcp

993

IPIP

IPIP capsulation traffic

Action

proto

dport

sport

PARAM

94

IPsec

IPsec traffic

Action

proto

dport

sport

PARAM

udp

500

500

PARAM

50

IPsecah

IPsec authentication (AH) traffic

Action

proto

dport

sport

PARAM

udp

500

500

PARAM

51

IPsecnat

IPsec traffic and Nat-Traversal

Action

proto

dport

sport

PARAM

udp

500

PARAM

udp

4500

PARAM

50

IRC

Internet Relay Chat traffic

Action

proto

dport

sport

PARAM

tcp

6667

Jetdirect

HP Jetdirect printing

Action

proto

dport

sport

PARAM

tcp

9100

L2TP

Layer 2 Tunneling Protocol traffic

Action

proto

dport

sport

PARAM

udp

1701

LDAP

Lightweight Directory Access Protocol traffic

Action

proto

dport

sport

PARAM

tcp

389

LDAPS

Secure Lightweight Directory Access Protocol traffic

Action

proto

dport

sport

PARAM

tcp

636

MDNS

Multicast DNS

Action

proto

dport

sport

PARAM

udp

5353

MSNP

Microsoft Notification Protocol

Action

proto

dport

sport

PARAM

tcp

1863

MSSQL

Microsoft SQL Server

Action

proto

dport

sport

PARAM

tcp

1433

Mail

Mail traffic (SMTP, SMTPS, Submission)

Action

proto

dport

sport

PARAM

tcp

25

PARAM

tcp

465

PARAM

tcp

587

Munin

Munin networked resource monitoring traffic

Action

proto

dport

sport

PARAM

tcp

4949

MySQL

MySQL server

Action

proto

dport

sport

PARAM

tcp

3306

NNTP

NNTP traffic (Usenet).

Action

proto

dport

sport

PARAM

tcp

119

NNTPS

Encrypted NNTP traffic (Usenet)

Action

proto

dport

sport

PARAM

tcp

563

NTP

Network Time Protocol (ntpd)

Action

proto

dport

sport

PARAM

udp

123

NeighborDiscovery

IPv6 neighbor solicitation, neighbor and router advertisement

Action

proto

dport

sport

PARAM

icmpv6

router-solicitation

PARAM

icmpv6

router-advertisement

PARAM

icmpv6

neighbor-solicitation

PARAM

icmpv6

neighbor-advertisement

OSPF

OSPF multicast traffic

Action

proto

dport

sport

PARAM

89

OpenVPN

OpenVPN traffic

Action

proto

dport

sport

PARAM

udp

1194

PCA

Symantec PCAnywere (tm)

Action

proto

dport

sport

PARAM

udp

5632

PARAM

tcp

5631

PMG

Proxmox Mail Gateway web interface

Action

proto

dport

sport

PARAM

tcp

8006

POP3

POP3 traffic

Action

proto

dport

sport

PARAM

tcp

110

POP3S

Encrypted POP3 traffic

Action

proto

dport

sport

PARAM

tcp

995

PPtP

Point-to-Point Tunneling Protocol

Action

proto

dport

sport

PARAM

47

PARAM

tcp

1723

Ping

ICMP echo request

Action

proto

dport

sport

PARAM

icmp

echo-request

PostgreSQL

PostgreSQL server

Action

proto

dport

sport

PARAM

tcp

5432

Printer

Line Printer protocol printing

Action

proto

dport

sport

PARAM

tcp

515

RDP

Microsoft Remote Desktop Protocol traffic

Action

proto

dport

sport

PARAM

tcp

3389

RIP

Routing Information Protocol (bidirectional)

Action

proto

dport

sport

PARAM

udp

520

RNDC

BIND remote management protocol

Action

proto

dport

sport

PARAM

tcp

953

Razor

Razor Antispam System

Action

proto

dport

sport

PARAM

tcp

2703

Rdate

Remote time retrieval (rdate)

Action

proto

dport

sport

PARAM

tcp

37

Rsync

Rsync server

Action

proto

dport

sport

PARAM

tcp

873

SANE

SANE network scanning

Action

proto

dport

sport

PARAM

tcp

6566

SMB

Microsoft SMB traffic

Action

proto

dport

sport

PARAM

udp

135,445

PARAM

udp

137:139

PARAM

udp

1024:65535

137

PARAM

tcp

135,139,445

SMBswat

Samba Web Administration Tool

Action

proto

dport

sport

PARAM

tcp

901

SMTP

Simple Mail Transfer Protocol

Action

proto

dport

sport

PARAM

tcp

25

SMTPS

Encrypted Simple Mail Transfer Protocol

Action

proto

dport

sport

PARAM

tcp

465

SNMP

Simple Network Management Protocol

Action

proto

dport

sport

PARAM

udp

161:162

PARAM

tcp

161

SPAMD

Spam Assassin SPAMD traffic

Action

proto

dport

sport

PARAM

tcp

783

SPICEproxy

Proxmox VE SPICE display proxy traffic

Action

proto

dport

sport

PARAM

tcp

3128

SSH

Secure shell traffic

Action

proto

dport

sport

PARAM

tcp

22

SVN

Subversion server (svnserve)

Action

proto

dport

sport

PARAM

tcp

3690

SixXS

SixXS IPv6 Deployment and Tunnel Broker

Action

proto

dport

sport

PARAM

tcp

3874

PARAM

udp

3740

PARAM

41

PARAM

udp

5072,8374

Squid

Squid web proxy traffic

Action

proto

dport

sport

PARAM

tcp

3128

Submission

Mail message submission traffic

Action

proto

dport

sport

PARAM

tcp

587

Syslog

Syslog protocol (RFC 5424) traffic

Action

proto

dport

sport

PARAM

udp

514

PARAM

tcp

514

TFTP

Trivial File Transfer Protocol traffic

Action

proto

dport

sport

PARAM

udp

69

Telnet

Telnet traffic

Action

proto

dport

sport

PARAM

tcp

23

Telnets

Telnet over SSL

Action

proto

dport

sport

PARAM

tcp

992

Time

RFC 868 Time protocol

Action

proto

dport

sport

PARAM

tcp

37

Trcrt

Traceroute (for up to 30 hops) traffic

Action

proto

dport

sport

PARAM

udp

33434:33524

PARAM

icmp

echo-request

VNC

VNC traffic for VNC display’s 0 - 99

Action

proto

dport

sport

PARAM

tcp

5900:5999

VNCL

VNC traffic from Vncservers to Vncviewers in listen mode

Action

proto

dport

sport

PARAM

tcp

5500

Web

WWW traffic (HTTP and HTTPS)

Action

proto

dport

sport

PARAM

tcp

80

PARAM

tcp

443

Webcache

Web Cache/Proxy traffic (port 8080)

Action

proto

dport

sport

PARAM

tcp

8080

Webmin

Webmin traffic

Action

proto

dport

sport

PARAM

tcp

10000

Whois

Whois (nicname, RFC 3912) traffic

Action

proto

dport

sport

PARAM

tcp

43

## COPYRIGHT AND DISCLAIMER
Copyright © 2007-2022 Proxmox Server Solutions GmbH

This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details.

You should have received a copy of the GNU Affero General Public License along with this program. If not, see https://www.gnu.org/licenses/

## AUTHOR
Proxmox Server Solutions Gmbh

www.proxmox.com

## NOTES
1.

Suricata IPS

https://suricata.io/

2.

nftables

https://wiki.nftables.org/wiki-nftables/index.php/What_is_nftables%3F


pve-ha-crm

package: pve-ha-manager

title: pve-ha-crm(8)

summary: PVE Cluster Resource Manager Daemon

text:
# pve-ha-crm(8)

## NAME
pve-ha-crm - PVE Cluster Resource Manager Daemon

## SYNOPSIS
pve-ha-crm <COMMAND> [ARGS] [OPTIONS]

pve-ha-crm help [OPTIONS]

Get help about specified command.

--extra-args <array>

Shows help for a specific command

--verbose <boolean>

Verbose output format.

pve-ha-crm start [OPTIONS]

Start the daemon.

--debug <boolean> (default = 0)

Debug mode - stay in foreground

pve-ha-crm status

Get daemon status.

pve-ha-crm stop

Stop the daemon.

## DESCRIPTION
This is the Cluster Resource Manager Daemon.

## COPYRIGHT AND DISCLAIMER
Copyright © 2007-2022 Proxmox Server Solutions GmbH

This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details.

You should have received a copy of the GNU Affero General Public License along with this program. If not, see https://www.gnu.org/licenses/

## AUTHOR
Proxmox Server Solutions Gmbh

www.proxmox.com

branch: sisyphus


pve-ha-lrm

title: pve-ha-lrm(8)

summary: PVE Local Resource Manager Daemon

text:
# pve-ha-lrm(8)

## NAME
pve-ha-lrm - PVE Local Resource Manager Daemon

## SYNOPSIS
pve-ha-lrm <COMMAND> [ARGS] [OPTIONS]

pve-ha-lrm help [OPTIONS]

Get help about specified command.

--extra-args <array>

Shows help for a specific command

--verbose <boolean>

Verbose output format.

pve-ha-lrm start [OPTIONS]

Start the daemon.

--debug <boolean> (default = 0)

Debug mode - stay in foreground

pve-ha-lrm status

Get daemon status.

pve-ha-lrm stop

Stop the daemon.

## DESCRIPTION
This is the Local Resource Manager Daemon.

## COPYRIGHT AND DISCLAIMER
Copyright © 2007-2022 Proxmox Server Solutions GmbH

This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details.

You should have received a copy of the GNU Affero General Public License along with this program. If not, see https://www.gnu.org/licenses/

## AUTHOR
Proxmox Server Solutions Gmbh

www.proxmox.com


pvedaemon

package: pve-manager

title: pvedaemon(8)

summary: PVE API Daemon

text:
# pvedaemon(8)

## NAME
pvedaemon - PVE API Daemon

## SYNOPSIS
pvedaemon <COMMAND> [ARGS] [OPTIONS]

pvedaemon help [OPTIONS]

Get help about specified command.

--extra-args <array>

Shows help for a specific command

--verbose <boolean>

Verbose output format.

pvedaemon restart

Restart the daemon (or start if not running).

pvedaemon start [OPTIONS]

Start the daemon.

--debug <boolean> (default = 0)

Debug mode - stay in foreground

pvedaemon status

Get daemon status.

pvedaemon stop

Stop the daemon.

## DESCRIPTION
This daemon exposes the whole Proxmox VE API on 127.0.0.1:85. It runs as root and has permission to do all privileged operations.

Note The daemon listens to a local address only, so you cannot access it from outside. The pveproxy daemon exposes the API to the outside world.

## NUMBER OF WORKERS
pvedaemon delegates handling of incoming requests to worker processes. By default, pvedaemon spawns 3 worker processes, which is sufficient for most workloads. For automation-heavy workloads that issue a huge volume of API requests and that experience slow request handling or timeouts, the number of worker processes can be increased by setting MAX_WORKERS in /etc/default/pvedaemon, for example:

```
MAX_WORKERS=5
```

Note that a higher number of worker processes may result in higher CPU usage. The number of worker processes must be greater than 0 and smaller than 128.

The same setting exists for pveproxy (man pveproxy(8)).

## COPYRIGHT AND DISCLAIMER
Copyright © 2007-2022 Proxmox Server Solutions GmbH

This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details.

You should have received a copy of the GNU Affero General Public License along with this program. If not, see https://www.gnu.org/licenses/

## AUTHOR
Proxmox Server Solutions Gmbh

www.proxmox.com


pveproxy

title: pveproxy(8)

summary: PVE API Proxy Daemon

text:
# pveproxy(8)

## NAME
pveproxy - PVE API Proxy Daemon

## SYNOPSIS
pveproxy <COMMAND> [ARGS] [OPTIONS]

pveproxy help [OPTIONS]

Get help about specified command.

--extra-args <array>

Shows help for a specific command

--verbose <boolean>

Verbose output format.

pveproxy restart

Restart the daemon (or start if not running).

pveproxy start [OPTIONS]

Start the daemon.

--debug <boolean> (default = 0)

Debug mode - stay in foreground

pveproxy status

Get daemon status.

pveproxy stop

Stop the daemon.

## DESCRIPTION
This daemon exposes the whole Proxmox VE API on TCP port 8006 using HTTPS. It runs as user www-data and has very limited permissions. Operation requiring more permissions are forwarded to the local pvedaemon.

Requests targeted for other nodes are automatically forwarded to those nodes. This means that you can manage your whole cluster by connecting to a single Proxmox VE node.

## HOST BASED ACCESS CONTROL
It is possible to configure “apache2”-like access control lists. Values are read from file /etc/default/pveproxy. For example:

```
ALLOW_FROM="10.0.0.1-10.0.0.5,192.168.0.0/22"
DENY_FROM="all"
POLICY="allow"
```

IP addresses can be specified using any syntax understood by Net::IP. The name all is an alias for 0/0 and ::/0 (meaning all IPv4 and IPv6 addresses).

The default policy is allow.

Match

POLICY=deny

POLICY=allow

Match Allow only

allow

allow

Match Deny only

deny

deny

No match

deny

allow

Match Both Allow & Deny

deny

allow

## LISTENING IP ADDRESS
By default the pveproxy and spiceproxy daemons listen on the wildcard address and accept connections from both IPv4 and IPv6 clients.

By setting LISTEN_IP in /etc/default/pveproxy you can control to which IP address the pveproxy and spiceproxy daemons bind. The IP-address needs to be configured on the system.

Setting the sysctl net.ipv6.bindv6only to the non-default 1 will cause the daemons to only accept connection from IPv6 clients, while usually also causing lots of other issues. If you set this configuration we recommend to either remove the sysctl setting, or set the LISTEN_IP to 0.0.0.0 (which will only allow IPv4 clients).

LISTEN_IP can be used to only to restricting the socket to an internal interface and thus have less exposure to the public internet, for example:

```
LISTEN_IP="192.0.2.1"
```

Similarly, you can also set an IPv6 address:

```
LISTEN_IP="2001:db8:85a3::1"
```

Note that if you want to specify a link-local IPv6 address, you need to provide the interface name itself. For example:

```
LISTEN_IP="fe80::c463:8cff:feb9:6a4e%vmbr0"
```

Warning The nodes in a cluster need access to pveproxy for communication, possibly on different sub-nets. It is not recommended to set LISTEN_IP on clustered systems.

To apply the change you need to either reboot your node or fully restart the pveproxy and spiceproxy service:

```
systemctl restart pveproxy.service spiceproxy.service
```

Note Unlike reload, a restart of the pveproxy service can interrupt some long-running worker processes, for example a running console or shell from a virtual guest. So, please use a maintenance window to bring this change in effect.

## SSL CIPHER SUITE
You can define the cipher list in /etc/default/pveproxy via the CIPHERS (TLS ⟨ 1.2) and CIPHERSUITES (TLS >= 1.3) keys. For example

```
CIPHERS="ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256"
CIPHERSUITES="TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256"
```

Above is the default. See the ciphers(1) man page from the openssl package for a list of all available options.

Additionally, you can set the client to choose the cipher used in /etc/default/pveproxy (default is the first cipher in the list available to both client and pveproxy):

```
HONOR_CIPHER_ORDER=0
```

## SUPPORTED TLS VERSIONS
The insecure SSL versions 2 and 3 are unconditionally disabled for pveproxy. TLS versions below 1.1 are disabled by default on recent OpenSSL versions, which is honored by pveproxy (see /etc/ssl/openssl.cnf).

To disable TLS version 1.2 or 1.3, set the following in /etc/default/pveproxy:

```
DISABLE_TLS_1_2=1
```

or, respectively:

```
DISABLE_TLS_1_3=1
```

Note Unless there is a specific reason to do so, it is not recommended to manually adjust the supported TLS versions.

## DIFFIE-HELLMAN PARAMETERS
You can define the used Diffie-Hellman parameters in /etc/default/pveproxy by setting DHPARAMS to the path of a file containing DH parameters in PEM format, for example

```
DHPARAMS="/path/to/dhparams.pem"
```

If this option is not set, the built-in skip2048 parameters will be used.

Note DH parameters are only used if a cipher suite utilizing the DH key exchange algorithm is negotiated.

## ALTERNATIVE HTTPS CERTIFICATE
You can change the certificate used to an external one or to one obtained via ACME.

pveproxy uses /etc/pve/local/pveproxy-ssl.pem and /etc/pve/local/pveproxy-ssl.key, if present, and falls back to /etc/pve/local/pve-ssl.pem and /etc/pve/local/pve-ssl.key. The private key may not use a passphrase.

It is possible to override the location of the certificate private key /etc/pve/local/pveproxy-ssl.key by setting TLS_KEY_FILE in /etc/default/pveproxy, for example:

```
TLS_KEY_FILE="/secrets/pveproxy.key"
```

Note The included ACME integration does not honor this setting.

See the Host System Administration chapter of the documentation for details.

## RESPONSE COMPRESSION
By default pveproxy uses gzip HTTP-level compression for compressible content, if the client supports it. This can be disabled in /etc/default/pveproxy:

```
COMPRESSION=0
```

Note Disabling compression mitigates BREACH-class side-channel attacks against authenticated session cookies, at the cost of slightly more bandwidth used by the web interface. Most deployments do not need to change the default; consider it on hosts where the web interface is exposed across networks you do not fully control.

## REAL CLIENT IP LOGGING
By default, pveproxy logs the IP address of the client that sent the request. In cases where a proxy server is in front of pveproxy, it may be desirable to log the IP of the client making the request instead of the proxy IP.

To enable processing of a HTTP header set by the proxy for logging purposes, set PROXY_REAL_IP_HEADER to the name of the header to retrieve the client IP from. For example:

```
PROXY_REAL_IP_HEADER="X-Forwarded-For"
```

Any invalid values passed in this header will be ignored.

The default behavior is log the value in this header on all incoming requests. To define a list of proxy servers that should be trusted to set the above HTTP header, set PROXY_REAL_IP_ALLOW_FROM, for example:

```
PROXY_REAL_IP_ALLOW_FROM="192.168.0.2"
```

The PROXY_REAL_IP_ALLOW_FROM setting also supports values similar to the ALLOW_FROM and DENY_FROM settings.

IP addresses can be specified using any syntax understood by Net::IP. The name all is an alias for 0/0 and ::/0 (meaning all IPv4 and IPv6 addresses).

## NUMBER OF WORKERS
pveproxy delegates handling of incoming requests to worker processes. By default, pveproxy spawns 3 worker processes, which is sufficient for most workloads. For automation-heavy workloads that issue a huge volume of API requests and that experience slow request handling or timeouts, the number of worker processes can be increased by setting MAX_WORKERS in /etc/default/pveproxy, for example:

```
MAX_WORKERS=5
```

Note that a higher number of worker processes may result in higher CPU usage. The number of worker processes must be greater than 0 and smaller than 128.

The same setting exists for pvedaemon (man pvedaemon(8)).

## COPYRIGHT AND DISCLAIMER
Copyright © 2007-2022 Proxmox Server Solutions GmbH

This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details.

You should have received a copy of the GNU Affero General Public License along with this program. If not, see https://www.gnu.org/licenses/

## AUTHOR
Proxmox Server Solutions Gmbh

www.proxmox.com


pvescheduler

title: pvescheduler(8)

summary: PVE Scheduler Daemon

text:
# pvescheduler(8)

## NAME
pvescheduler - PVE Scheduler Daemon

## SYNOPSIS
pvescheduler <COMMAND> [ARGS] [OPTIONS]

pvescheduler help [OPTIONS]

Get help about specified command.

--extra-args <array>

Shows help for a specific command

--verbose <boolean>

Verbose output format.

pvescheduler restart

Restart the daemon (or start if not running).

pvescheduler start [OPTIONS]

Start the daemon.

--debug <boolean> (default = 0)

Debug mode - stay in foreground

pvescheduler status

Get daemon status.

pvescheduler stop

Stop the daemon.

## DESCRIPTION
This daemon is responsible for starting jobs according to the schedule, such as replication and vzdump jobs.

For vzdump jobs, it gets its configuration from the file /etc/pve/jobs.cfg

## COPYRIGHT AND DISCLAIMER
Copyright © 2007-2022 Proxmox Server Solutions GmbH

This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details.

You should have received a copy of the GNU Affero General Public License along with this program. If not, see https://www.gnu.org/licenses/

## AUTHOR
Proxmox Server Solutions Gmbh

www.proxmox.com


pvestatd

title: pvestatd(8)

summary: PVE Status Daemon

text:
# pvestatd(8)

## NAME
pvestatd - PVE Status Daemon

## SYNOPSIS
pvestatd <COMMAND> [ARGS] [OPTIONS]

pvestatd help [OPTIONS]

Get help about specified command.

--extra-args <array>

Shows help for a specific command

--verbose <boolean>

Verbose output format.

pvestatd restart

Restart the daemon (or start if not running).

pvestatd start [OPTIONS]

Start the daemon.

--debug <boolean> (default = 0)

Debug mode - stay in foreground

pvestatd status

Get daemon status.

pvestatd stop

Stop the daemon.

## DESCRIPTION
This daemon queries the status of VMs, storages and containers at regular intervals. The result is sent to all nodes in the cluster.

## COPYRIGHT AND DISCLAIMER
Copyright © 2007-2022 Proxmox Server Solutions GmbH

This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details.

You should have received a copy of the GNU Affero General Public License along with this program. If not, see https://www.gnu.org/licenses/

## AUTHOR
Proxmox Server Solutions Gmbh

www.proxmox.com


spiceproxy

title: spiceproxy(8)

summary: SPICE Proxy Service

text:
# spiceproxy(8)

## NAME
spiceproxy - SPICE Proxy Service

## SYNOPSIS
spiceproxy <COMMAND> [ARGS] [OPTIONS]

spiceproxy help [OPTIONS]

Get help about specified command.

--extra-args <array>

Shows help for a specific command

--verbose <boolean>

Verbose output format.

spiceproxy restart

Restart the daemon (or start if not running).

spiceproxy start [OPTIONS]

Start the daemon.

--debug <boolean> (default = 0)

Debug mode - stay in foreground

spiceproxy status

Get daemon status.

spiceproxy stop

Stop the daemon.

## DESCRIPTION
SPICE[1] (the Simple Protocol for Independent Computing Environments) is an open remote computing solution, providing client access to remote displays and devices (e.g. keyboard, mouse, audio). The main use case is to get remote access to virtual machines and container.

This daemon listens on TCP port 3128, and implements an HTTP proxy to forward CONNECT request from the SPICE client to the correct Proxmox VE VM. It runs as user www-data and has very limited permissions.

## HOST BASED ACCESS CONTROL
It is possible to configure "apache2" like access control lists. Values are read from file /etc/default/pveproxy. See pveproxy documentation for details.

## COPYRIGHT AND DISCLAIMER
Copyright © 2007-2022 Proxmox Server Solutions GmbH

This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details.

You should have received a copy of the GNU Affero General Public License along with this program. If not, see https://www.gnu.org/licenses/

## AUTHOR
Proxmox Server Solutions Gmbh

www.proxmox.com

## NOTES
1.

SPICE

http://www.spice-space.org


qemu-nbd

package: pve-qemu-img

title: qemu-nbd(8)

summary: QEMU Disk Network Block Device Server

text:
# qemu-nbd(8)

## NAME
qemu-nbd - QEMU Disk Network Block Device Server

## SYNOPSIS
qemu-nbd [OPTION]... filename

qemu-nbd -L [OPTION]...

qemu-nbd -d dev

## DESCRIPTION
Export a QEMU disk image using the NBD protocol.

Other uses:

- Bind a /dev/nbdX block device to a QEMU server (on Linux).

- As a client to query exports of a remote NBD server.

## OPTIONS
filename is a disk image filename, or a set of block driver options if --image-opts is specified.

dev is an NBD device.

--object type,id=ID,...

Define a new instance of the type object class identified by ID. See the qemu(1) manual page for full details of the properties supported. The common object types that it makes sense to define are the secret object, which is used to supply passwords and/or encryption keys, and the tls-creds object, which is used to supply TLS credentials for the qemu-nbd server or client.

-p, --port=PORT

TCP port to listen on as a server, or connect to as a client (default 10809).

-o, --offset=OFFSET

The offset into the image.

-b, --bind=IFACE

The interface to bind to as a server, or connect to as a client (default 0.0.0.0).

-k, --socket=PATH

Use a unix socket with path PATH.

--image-opts

Treat filename as a set of image options, instead of a plain filename. If this flag is specified, the -f flag should not be used, instead the format= option should be set.

-f, --format=FMT

Force the use of the block driver for format FMT instead of auto-detecting.

-r, --read-only

Export the disk as read-only.

-A, --allocation-depth

Expose allocation depth information via the qemu:allocation-depth metadata context accessible through NBD_OPT_SET_META_CONTEXT.

-B, --bitmap=NAME

If filename has a qcow2 persistent bitmap NAME, expose that bitmap via the qemu:dirty-bitmap:NAME metadata context accessible through NBD_OPT_SET_META_CONTEXT.

-s, --snapshot

Use filename as an external snapshot, create a temporary file with backing_file=filename, redirect the write to the temporary one.

-l, --load-snapshot=SNAPSHOT_PARAM

Load an internal snapshot inside filename and export it as an read-only device, SNAPSHOT_PARAM format is snapshot.id=[ID],snapshot.name=[NAME] or [ID_OR_NAME]

--cache=CACHE

The cache mode to be used with the file. Valid values are: none, writeback (the default), writethrough, directsync and unsafe. See the documentation of the emulator's -drive cache=... option for more info.

-n, --nocache

Equivalent to --cache=none.

--aio=AIO

Set the asynchronous I/O mode between threads (the default), native (Linux only), and io_uring (Linux 5.1+).

--discard=DISCARD

Control whether discard (also known as trim or unmap) requests are ignored or passed to the filesystem. DISCARD is one of ignore (or off), unmap (or on). The default is ignore.

--detect-zeroes=DETECT_ZEROES

Control the automatic conversion of plain zero writes by the OS to driver-specific optimized zero write commands. DETECT_ZEROES is one of off, on, or unmap. unmap converts a zero write to an unmap operation and can only be used if DISCARD is set to unmap. The default is off.

-c, --connect=DEV

Connect filename to NBD device DEV (Linux only).

-d, --disconnect

Disconnect the device DEV (Linux only).

-e, --shared=NUM

Allow up to NUM clients to share the device (default 1), 0 for unlimited.

-t, --persistent

Don't exit on the last connection.

-x, --export-name=NAME

Set the NBD volume export name (default of a zero-length string).

-D, --description=DESCRIPTION

Set the NBD volume export description, as a human-readable string.

--handshake-limit=N

Set the timeout for a client to successfully complete its handshake to N seconds (default 10), or 0 for no limit.

-L, --list

Connect as a client and list all details about the exports exposed by a remote NBD server. This enables list mode, and is incompatible with options that change behavior related to a specific export (such as --export-name, --offset, ...).

--tls-creds=ID

Enable mandatory TLS encryption for the server by setting the ID of the TLS credentials object previously created with the --object option; or provide the credentials needed for connecting as a client in list mode.

--tls-hostname=hostname

When validating an x509 certificate received over a TLS connection, the hostname that the NBD client used to connect will be checked against information in the server provided certificate. Sometimes it might be required to override the hostname used to perform this check. For example, if the NBD client is using a tunnel from localhost to connect to the remote server, the --tls-hostname option should be used to set the officially expected hostname of the remote NBD server. This can also be used if accessing NBD over a UNIX socket where there is no inherent hostname available. This is only permitted when acting as a NBD client with the --list option.

--fork

Fork off the server process and exit the parent once the server is running.

--pid-file=PATH

Store the server's process ID in the given file.

--tls-authz=ID

Specify the ID of a qauthz object previously created with the --object option. This will be used to authorize connecting users against their x509 distinguished name.

-v, --verbose

Display extra debugging information. This option also keeps the original STDERR stream open if the qemu-nbd process is daemonized due to other options like --fork or -c.

-h, --help

Display this help and exit.

-V, --version

Display version information and exit.

-T, --trace [[enable=]PATTERN][,events=FILE][,file=FILE]

Specify tracing options. [enable=]PATTERN

Immediately enable events matching PATTERN (either event name or a globbing pattern). This option is only available if QEMU has been compiled with the simple, log or ftrace tracing backend. To specify multiple events or patterns, specify the -trace option multiple times. Use -trace help to print a list of names of trace points.

events=FILE

Immediately enable events listed in FILE. The file must contain one event name (as listed in the trace-events-all file) per line; globbing patterns are accepted too. This option is only available if QEMU has been compiled with the simple, log or ftrace tracing backend.

file=FILE

Log output traces to FILE. This option is only available if QEMU has been compiled with the simple tracing backend.

## EXAMPLES
Start a server listening on port 10809 that exposes only the guest-visible contents of a qcow2 file, with no TLS encryption, and with the default export name (an empty string). The command is one-shot, and will block until the first successful client disconnects:

```
qemu-nbd -f qcow2 file.qcow2
```

Start a long-running server listening with encryption on port 10810, and allow clients with a specific X.509 certificate to connect to a 1 megabyte subset of a raw file, using the export name 'subset':

```
qemu-nbd \

  --object tls-creds-x509,id=tls0,endpoint=server,dir=/path/to/qemutls \

  --object 'authz-simple,id=auth0,identity=CN=laptop.example.com,,\

            O=Example Org,,L=London,,ST=London,,C=GB' \

  --tls-creds tls0 --tls-authz auth0 \

  -t -x subset -p 10810 \

  --image-opts driver=raw,offset=1M,size=1M,file.driver=file,file.filename=file.raw
```

Serve a read-only copy of a guest image over a Unix socket with as many as 5 simultaneous readers, with a persistent process forked as a daemon:

```
qemu-nbd --fork --persistent --shared=5 --socket=/path/to/sock \

  --read-only --format=qcow2 file.qcow2
```

Expose the guest-visible contents of a qcow2 file via a block device /dev/nbd0 (and possibly creating /dev/nbd0p1 and friends for partitions found within), then disconnect the device when done. Access to bind qemu-nbd to a /dev/nbd device generally requires root privileges, and may also require the execution of modprobe nbd to enable the kernel NBD client module. CAUTION: Do not use this method to mount filesystems from an untrusted guest image - a malicious guest may have prepared the image to attempt to trigger kernel bugs in partition probing or file system mounting.

```
qemu-nbd -c /dev/nbd0 -f qcow2 file.qcow2
qemu-nbd -d /dev/nbd0
```

Query a remote server to see details about what export(s) it is serving on port 10809, and authenticating via PSK:

```
qemu-nbd \

  --object tls-creds-psk,id=tls0,dir=/tmp/keys,username=eblake,endpoint=client \

  --tls-creds tls0 -L -b remote.example.com
```

## SEE ALSO
qemu(1), qemu-img(1)

## Author
Anthony Liguori <anthony@codemonkey.ws>

## Copyright
2025, The QEMU Project Developers


qmeventd

package: pve-qemu-server

title: qmeventd(8)

summary: PVE QEMU Eventd Daemon

text:
# qmeventd(8)

## NAME
qmeventd - PVE QEMU Eventd Daemon

## SYNOPSIS
qmeventd [-f] [-v] PATH

-v

Turn on verbose messages

-f

Do not daemonize server

PATH

The path to listen on

This service is usually started and managed using systemd toolset. The service is called qmeventd.

```
systemctl start qmeventd
```

```
systemctl stop qmeventd
```

```
systemctl status qmeventd
```

## DESCRIPTION
qmeventd is a daemon that listens on PATH for incoming connections from a qemu qmp socket, and waits for SHUTDOWN events. When a client then disconnects, it executes /usr/sbin/qm cleanup. This makes it easy to clean up leftover tap devices, vgpus, etc.

## COPYRIGHT AND DISCLAIMER
Copyright © 2007-2022 Proxmox Server Solutions GmbH

This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details.

You should have received a copy of the GNU Affero General Public License along with this program. If not, see https://www.gnu.org/licenses/

## AUTHOR
Proxmox Server Solutions Gmbh

www.proxmox.com


qemu-pr-helper

package: pve-qemu-system

title: qemu-pr-helper(8)

summary: QEMU persistent reservation helper

text:
# qemu-pr-helper(8)

## NAME
qemu-pr-helper - QEMU persistent reservation helper

## SYNOPSIS
qemu-pr-helper [OPTION]

## DESCRIPTION
Implements the persistent reservation helper for QEMU.

SCSI persistent reservations allow restricting access to block devices to specific initiators in a shared storage setup. When implementing clustering of virtual machines, it is a common requirement for virtual machines to send persistent reservation SCSI commands. However, the operating system restricts sending these commands to unprivileged programs because incorrect usage can disrupt regular operation of the storage fabric. QEMU's SCSI passthrough devices scsi-block and scsi-generic support passing guest persistent reservation requests to a privileged external helper program. qemu-pr-helper is that external helper; it creates a listener socket which will accept incoming connections for communication with QEMU.

If you want to run VMs in a setup like this, this helper should be started as a system service, and you should read the QEMU manual section on "persistent reservation managers" to find out how to configure QEMU to connect to the socket created by qemu-pr-helper.

After connecting to the socket, qemu-pr-helper can optionally drop root privileges, except for those capabilities that are needed for its operation.

qemu-pr-helper can also use the systemd socket activation protocol. In this case, the systemd socket unit should specify a Unix stream socket, like this:

```
[Socket]
ListenStream=/var/run/qemu-pr-helper.sock
```

## OPTIONS
-d, --daemon

run in the background (and create a PID file)

-q, --quiet

decrease verbosity

-v, --verbose

increase verbosity

-f, --pidfile=PATH

PID file when running as a daemon. By default the PID file is created in the system runtime state directory, for example /var/run/qemu-pr-helper.pid.

-k, --socket=PATH

path to the socket. By default the socket is created in the system runtime state directory, for example /var/run/qemu-pr-helper.sock.

-T, --trace [[enable=]PATTERN][,events=FILE][,file=FILE]

Specify tracing options. [enable=]PATTERN

Immediately enable events matching PATTERN (either event name or a globbing pattern). This option is only available if QEMU has been compiled with the simple, log or ftrace tracing backend. To specify multiple events or patterns, specify the -trace option multiple times. Use -trace help to print a list of names of trace points.

events=FILE

Immediately enable events listed in FILE. The file must contain one event name (as listed in the trace-events-all file) per line; globbing patterns are accepted too. This option is only available if QEMU has been compiled with the simple, log or ftrace tracing backend.

file=FILE

Log output traces to FILE. This option is only available if QEMU has been compiled with the simple tracing backend.

-u, --user=USER

user to drop privileges to

-g, --group=GROUP

group to drop privileges to

-h, --help

Display a help message and exit.

-V, --version

Display version information and exit.

## Copyright
2025, The QEMU Project Developers


pethtool

title: pethtool(8)

text:
# pethtool(8)

## NAME
pethtool - Display or change ethernet card settings

## SYNOPSIS
pethtool [OPTIONS] [<INTERFACE>]

## DESCRIPTION
This script mimic ethtool behavior, but is written purely in python and python module ethtool.

## OPTIONS
INTERFACE

Is the name of the ethernet device on which pethtool should operate.

-h, --help

Show help message and exit.

-c|--show-coalesce

Show coalesce options

-C|--coalesce

Set coalesce options

```
adaptive-rx on|off
adaptive-tx on|off
rx-usecs N
rx-frames N
rx-usecs-irq N
rx-frames-irq N
tx-usecs N
tx-frames N
tx-usecs-irq N
tx-frames-irq N
stats-block-usecs N
pkt-rate-low N
rx-usecs-low N
rx-frames-low N
tx-usecs-low N
tx-frames-low N
pkt-rate-high N
rx-usecs-high N
rx-frames-high N
tx-usecs-high N
tx-frames-high N
sample-interval N
```

-i|--driver

Show driver information

-k|--show-offload

Get protocol offload information

-K|--offload

Set protocol offload

[ tso on|off ]

## SEE ALSO
ethtool(8)

## AUTHORS
Arnaldo Carvalho de Melo <acme@redhat.com>

Man page written by Miroslav Suchý <msuchy@redhat.com>


pifconfig

title: pifconfig(8)

text:
# pifconfig(8)

## NAME
pifconfig - display information about a network interface

## SYNOPSIS
pifconfig [INTERFACE [INTERFACE [INTERFACE] ...]]

## DESCRIPTION
This script mimic ifconfig behavior, but is written purely in python and python module ethtool.

## OPTIONS
INTERFACE

Display information about only the listed interfaces. If no interface is given all interfaces are displayed.

-h, --help

Show help message and exit.

## SEE ALSO
ifconfig(8)

## AUTHORS
Arnaldo Carvalho de Melo <acme@redhat.com>

Man page written by Miroslav Suchý <msuchy@redhat.com>, David Sommerseth <davids@redhat.com>