NAME

drbd-reactor.promoter — drbd-reactor promoter plugin

DESCRIPTION

drbd-reactor.promoter is the promoter plugin for drbd-reactor(1).

The promoter plugin monitors events on resources an executes systemd units. This plugin can be used for simple high-availability.

By default the plugin generates a series of systemd service overrides (i.e., what systemd calls "Drop-In") and a systemd target unit that contains dependencies on these generated services. The services, and their order, is defined via the list specified in start.

The plugin generates two implicit extra units:

• a drbd-promote@ override that promotes the DRBD resource (i.e., switches it to Primary). This is a dependency for all the other units from start (according overrides are generated).
• a drbd-services@ target that subsumes all the generated dependencies from the start list.

Let's look at a simple example to see which overrides get generated from a dummy start list like this:

[[promoter]]
[promoter.resources.foo]
start = [ "a.service", "b.service", "c.service" ]

• /var/run/systemd/system/drbd-promote@foo.d/reactor.conf containing the necessary pieces to wait for the backing devices of DRBD resource “foo” and to promote it to Primary.
• /var/run/systemd/system/a.service.d/reactor.conf containing a dependency on drbd-promote@foo
• /var/run/systemd/system/b.service.d/reactor.conf containing dependencies on drbd-promote@foo and on a.service.
• /var/run/systemd/system/c.service.d/reactor.conf containing dependencies on drbd-promote@foo and on b.service.
• /var/run/systemd/system/drbd-services@foo.target.d/reactor.conf containing dependencies on a.service, b.service, and c.service.

If a DRBD resource changes its state to “may promote”, the plugin (i.e., all plugins on all nodes in the cluster) start the generated systemd target (e.g., drbd-services@foo.target ). All will try to start the drbd-promote@ unit first, but only one will succeed and continue to start the rest of the services. All the others will fail intentionally.

In order to promote the resource with the best disk state, the plugin evaluates its local disk states and waits a short period of time, which might give a promoter on another host the chance (i.e., because its sleep time is shorter) to promote the resource first. Sleep times are calculated as follows:

The actual sleep time is calculated as the worst case of all the volumes in a resource and can be scaled by setting sleep-before-promote-factor.

If a resource loses “quorum”, it stops the systemd drbd-services@ target and all the dependencies. Stopping services on the node that lost quorum is the standard behavior one would expect from a cluster manger. There might be scenarios where it is preferable to freeze the started service until quorum is gained again. As this requires multiple prerequisites to hold true, freezing a resource on quorum loss is described in FREEZING RESOURCES.

The plugin's configuration can contain an action that is executed if a stop action fails (e.g., triggering a reboot). Start actions in start are interpreted as systemd.unit(5). and have to have an according postfix like .service or .mount. ocf resource agents are supported via the ocf.rs@ service. Please see OCF RESOURCE AGENTS for a detailed overview.

The configuration can contain a setting that specifies that resources are stopped whenever the plugin exits (e.g., on service restart).

It also contains a runner that can be set to shell. Then the items in start are interpreted as shell scripts and started in order (no explicit targets or anything) and stopped in reverse order or as defined via stop. This can be used on systems without systemd and might be useful for Windows systems in the future. If you can, use the default systemd method, it is the preferred one.

In order for DRBD to use quorum as needed by this plugin, make sure the resource file of the DRBD resource contains the following option (this is the default for LINSTOR, but your resources needs to qualify for quorum).

options {
   auto-promote no;
   quorum majority;
   on-suspended-primary-outdated force-secondary;
   on-no-quorum io-error; # for the default drbd-reactor on-quorum-loss policy (i.e., Shutdown)
   # on-no-quorum suspend-io; # for freezing resources
   on-no-data-accessible io-error # always set this to the value of on-no-quorum!
   # on-no-data-accessible suspend-io # for freezing, always set this to the value of on-no-quorum!
}
# net { rr-conflict retry-connect; } # for freezing resources

SERVICE DEPENDENCIES

Let's get back to our simple example:

[[promoter]]
[promoter.resources.foo]
start = [ "a.service", "b.service", "c.service" ]

As we noted in previously we generate a dependency chain for these services (i.e., all depend on drbd-promote@ as well as on the previous services). The strictness of these dependencies can be set via dependencies-as , where the default is Requires systemd.unit(5)

We also generate the mentioned drbd-services@.target , which lists all the services from start. The dependencies for that are generated via the value set in target-as.

Especially when one debugs services it might make sense to lower these defaults to for example Wants. Otherwise a failed service might prohibit a successful start of the drbd-services@.target , which then triggers a stop of the target and its dependencies, which might again trigger a start because the resource is DRBD promotable again and so on.

It is really up to you and how strict/hard you want your dependencies and what their outcome should be. Requires should be a good default, you might lower or increase the strictness depending on the scenario.

OCF RESOURCE AGENTS

It is possible to use resource agents in the start list of services via ocf:$vendor:$agent instance-id name=value ... The instance-id is user defined and gets postfixed with _$resourcename. For example the generated systemd unit for an instance-id of “p_iscsi_demo1” for a DRBD resource “foo” would be ocf.rs@p_iscsi_demo1_foo. name / value pairs are passed to the unit as environment variables prefixed with OCF_RESKEY_. In a concrete example using the “heartbeat:IPaddr2” agent this could look like this:

start = [
  "foo.service",
  "ocf:heartbeat:IPaddr2 p_iscsi_demo1_ip ip=10.43.7.223 cidr_netmask=16",
  "bar.service"
]

FREEZING RESOURCES

The default behavior when a DRBD Primary loses quorum is to immediately stop the generated target unit and hope that other nodes still having quorum will successfully start the service. This works well if services can be failed over/started on another node in reasonable time. Unfortunately there are services that take a very long time to start, for example huge data bases.

When a DRBD Primary loses its quorum we basically have two possibilities:

• the rest of the nodes, or at least parts of it still have quorum: Then these have to start the service, they are the only ones with quorum, but still we could keep the old Primary in a frozen state. And then, when the nodes with quorum come into contact with the old Primary, then it should stop the service and its storage should become in sync with the other nodes.
• the rest of the nodes are not able to form a partition with quorum. In such a scenario there are no alternatives anyways, we would need to keep the Primary frozen. But if the nodes eventually join the old Primary again, and quorum would be restored, we could just unfreeze/thaw the old Primary (which is also the new Primary).

There are several requirements for this to work properly:

• A system with unified cgroups. If the file /sys/fs/cgroup/cgroup.controllers exists you should be fine. That requires a relatively “new” kernel. Note that “even” RHEL8 for example needs the addition of systemd.unified_cgroup_hierarchy on the kernel command line.
• a service that can tolerate to be frozen
• DRBD option on-suspended-primary-outdated set to force-secondary
• DRBD option on-no-quorum set to suspend-io
• DRBD option on-no-data-accessible set to suspend-io
• DRBD net option rr-conflict set to retry-connect

PREFERRED NODES

While in a HA cluster that deserves the name every node needs to be able to run all services, some users like to add preferences for nodes. This can be done by setting a list of preferred-nodes. On resource startup a delay based on the node's position in the list is added. Node names need to match the output of uname -n. Nodes with a lower preference will sleep longer. By default, if a node joins on DRBD level, and that peer's disk becomes UpToDate, and the peer has a higher preference, then the active node stops the services locally. As it will then have a higher sleep penalty as the preferred node, the preferred one will take over the service (if it can). If preferred-nodes-policy is set to always the system behaves as described above. If it is set to start-only, then preferred nodes are only taken into account when a service can be startet, but it will not fall back to a more preferred node when the service is already running on a lower priority node and a higher priority nodes joins later.

TWO NODE CLUSTERS WITH FENCING

If possible the promoter plugin should be used in setups that allow for DRBD quorum. Two node clusters with properly configured fencing are the exception. We only support fencing as split-brain avoidance mechanism in two node clusters, everything else should use DRBD quorum. The policy gets auto detected as follows:

• if DRBD quorum is something other than “off”, quorum is used
• if DRBD quorum is “off” and DRBD fencing is something other than “dont-care”, fencing is used

ADJUSTING RESOURCES ON START

When the promoter plugin starts, it runs drbdadm adjust on all configured resources by default. This ensures that the DRBD resource configuration on disk matches the running kernel state. The plugin waits for the backing devices to become available before running adjust.

This behavior can be disabled per resource by setting adjust-resource-on-start to false. Disabling it may be useful if drbdadm adjust is managed externally, or if its execution during startup causes undesirable side effects in a specific setup.

LOCAL DISK FAILURE

If the local DRBD disk on a Primary node gets detached (i.e., becomes Diskless), the behavior is controlled by the on-disk-detach policy. By default the policy is set to “ignore”, which means no action is taken. If set to “fail-over”, the promoter will stop services if an UpToDate peer is found, triggering a failover to that peer.

FILES

/usr/lib/ocf/resource.d/

Path used for for OCF agents.

EXAMPLES

This configures a promoter for resource “foo”. If the resource might be promoted the “foo.mount” and “foo.service” units get started. On DRBD demote failures a systemd "reboot" is triggered.

[promoter.resources.foo]
start = ["foo.mount", "foo.service"]
on-drbd-demote-failure =  "reboot"
stop-services-on-exit = false
preferred-nodes = ["nodeA", "nodeB" ]

SEE ALSO

For further design notes and handled failure scenarios see doc/promoter.md in the source code. An online version is available at promoter.md

AUTHORS

The drbd-reactor(1) program was written by the following authors:
Roland Kammerer <rck@linbit.com>
Moritz Wanzenböck <moritz.wanzenboeck@linbit.com>