sanlock

package: sanlock

title: sanlock(8)

summary: shared storage lock manager

text:
# sanlock(8)

## NAME
sanlock - shared storage lock manager

## SYNOPSIS
sanlock [COMMAND] [ACTION] ...

## DESCRIPTION
sanlock is a lock manager built on shared storage. Hosts with access to the storage can perform locking. An application running on the hosts is given a small amount of space on the shared block device or file, and uses sanlock for its own application-specific synchronization. Internally, the sanlock daemon manages locks using two disk-based lease algorithms: delta leases and paxos leases.

•

delta leases are slow to acquire and demand regular i/o to shared storage. sanlock only uses them internally to hold a lease on its "host_id" (an integer host identifier from 1-2000). They prevent two hosts from using the same host identifier. The delta lease renewals also indicate if a host is alive. ("Light-Weight Leases for Storage-Centric Coordination", Chockler and Malkhi.)

•

paxos leases are fast to acquire and sanlock makes them available to applications as general purpose resource leases. The disk paxos algorithm uses host_id's internally to represent different hosts, and the owner of a paxos lease. delta leases provide unique host_id's for implementing paxos leases, and delta lease renewals serve as a proxy for paxos lease renewal. ("Disk Paxos", Eli Gafni and Leslie Lamport.)

Externally, the sanlock daemon exposes a locking interface through libsanlock in terms of "lockspaces" and "resources". A lockspace is a locking context that an application creates for itself on shared storage. When the application on each host is started, it "joins" the lockspace. It can then create "resources" on the shared storage. Each resource represents an application-specific entity. The application can acquire and release leases on resources.

To use sanlock from an application:

•

Allocate shared storage for an application, e.g. a shared LUN or LV from a SAN, or files from NFS.

•

Provide the storage to the application.

•

The application uses this storage with libsanlock to create a lockspace and resources for itself.

•

The application joins the lockspace when it starts.

•

The application acquires and releases leases on resources.

How lockspaces and resources translate to delta leases and paxos leases within sanlock:

Lockspaces

•

A lockspace is based on delta leases held by each host using the lockspace.

•

A lockspace is a series of 2000 delta leases on disk, and requires 1MB of storage. (See Storage below for size variations.)

•

A lockspace can support up to 2000 concurrent hosts using it, each using a different delta lease.

•

Applications can i) create, ii) join and iii) leave a lockspace, which corresponds to i) initializing the set of delta leases on disk, ii) acquiring one of the delta leases and iii) releasing the delta lease.

•

When a lockspace is created, a unique lockspace name and disk location is provided by the application.

•

When a lockspace is created/initialized, sanlock formats the sequence of 2000 on-disk delta lease structures on the file or disk, e.g. /mnt/leasefile (NFS) or /dev/vg/lv (SAN).

•

The 2000 individual delta leases in a lockspace are identified by number: 1,2,3,...,2000.

•

Each delta lease is a 512 byte sector in the 1MB lockspace, offset by its number, e.g. delta lease 1 is offset 0, delta lease 2 is offset 512, delta lease 2000 is offset 1023488. (See Storage below for size variations.)

•

When an application joins a lockspace, it must specify the lockspace name, the lockspace location on shared disk/file, and the local host's host_id. sanlock then acquires the delta lease corresponding to the host_id, e.g. joining the lockspace with host_id 1 acquires delta lease 1.

•

The terms delta lease, lockspace lease, and host_id lease are used interchangeably.

•

sanlock acquires a delta lease by writing the host's unique name to the delta lease disk sector, reading it back after a delay, and verifying it is the same.

•

If a unique host name is not specified, sanlock uses the product_uuid if one is available, otherwise generates a uuid to use as the host's name. The delta lease algorithm depends on hosts using unique names.

•

The application on each host should be configured with a unique host_id, where the host_id is an integer 1-2000.

•

If hosts are misconfigured and have the same host_id, the delta lease algorithm is designed to detect this conflict, and only one host will be able to acquire the delta lease for that host_id.

•

A delta lease ensures that a lockspace host_id is being used by a single host with the unique name specified in the delta lease.

•

Resolving delta lease conflicts is slow, because the algorithm is based on waiting and watching for some time for other hosts to write to the same delta lease sector. If multiple hosts try to use the same delta lease, the delay is increased substantially. So, it is best to configure applications to use unique host_id's that will not conflict.

•

After sanlock acquires a delta lease, the lease must be renewed until the application leaves the lockspace (which corresponds to releasing the delta lease on the host_id.)

•

sanlock renews delta leases every 20 seconds (by default) by writing a new timestamp into the delta lease sector.

•

When a host acquires a delta lease in a lockspace, it can be referred to as "joining" the lockspace. Once it has joined the lockspace, it can use resources associated with the lockspace.

Resources

•

A lockspace is a context for resources that can be locked and unlocked by an application.

•

sanlock uses paxos leases to implement leases on resources. The terms paxos lease and resource lease are used interchangeably.

•

A paxos lease exists on shared storage and requires 1MB of space. It contains a unique resource name and the name of the lockspace.

•

An application assigns its own meaning to a sanlock resource and the leases on it. A sanlock resource could represent some shared object like a file, or some unique role among the hosts.

•

Resource leases are associated with a specific lockspace and can only be used by hosts that have joined that lockspace (they are holding a delta lease on a host_id in that lockspace.)

•

An application must keep track of the disk locations of its lockspaces and resources. sanlock does not maintain any persistent index or directory of lockspaces or resources that have been created by applications, so applications need to remember where they have placed their own leases (which files or disks and offsets).

•

sanlock does not renew paxos leases directly (although it could). Instead, the renewal of a host's delta lease represents the renewal of all that host's paxos leases in the associated lockspace. In effect, many paxos lease renewals are factored out into one delta lease renewal. This reduces i/o when many paxos leases are used.

•

The disk paxos algorithm allows multiple hosts to all attempt to acquire the same paxos lease at once, and will produce a single winner/owner of the resource lease. (Shared resource leases are also possible in addition to the default exclusive leases.)

•

The disk paxos algorithm involves a specific sequence of reading and writing the sectors of the paxos lease disk area. Each host has a dedicated 512 byte sector in the paxos lease disk area where it writes its own "ballot", and each host reads the entire disk area to see the ballots of other hosts. The first sector of the disk area is the "leader record" that holds the result of the last paxos ballot. The winner of the paxos ballot writes the result of the ballot to the leader record (the winner of the ballot may have selected another contending host as the owner of the paxos lease.)

•

After a paxos lease is acquired, no further i/o is done in the paxos lease disk area.

•

Releasing the paxos lease involves writing a single sector to clear the current owner in the leader record.

•

If a host holding a paxos lease fails, the disk area of the paxos lease still indicates that the paxos lease is owned by the failed host. If another host attempts to acquire the paxos lease, and finds the lease is held by another host_id, it will check the delta lease of that host_id. If the delta lease of the host_id is being renewed, then the paxos lease is owned and cannot be acquired. If the delta lease of the owner's host_id has expired, then the paxos lease is expired and can be taken (by going through the paxos lease algorithm.)

•

The "interaction" or "awareness" between hosts of each other is limited to the case where they attempt to acquire the same paxos lease, and need to check if the referenced delta lease has expired or not.

•

When hosts do not attempt to lock the same resources concurrently, there is no host interaction or awareness. The state or actions of one host have no effect on others.

•

To speed up checking delta lease expiration (in the case of a paxos lease conflict), sanlock keeps track of past renewals of other delta leases in the lockspace.

Resource Index

The resource index (rindex) is an optional sanlock feature that applications can use to keep track of resource lease offsets. Without the rindex, an application must keep track of where its resource leases exist on disk and find available locations when creating new leases.

The sanlock rindex uses two align-size areas on disk following the lockspace. The first area holds rindex entries; each entry records a resource lease name and location. The second area holds a private paxos lease, used by sanlock internally to protect rindex updates.

The application creates the rindex on disk with the "format" function. Format is a disk-only operation and does not interact with the live lockspace, so it can be called without first calling add_lockspace. The application needs to follow the convention of writing the lockspace at the start of the device (offset 0) and formatting the rindex immediately following the lockspace area. When formatting, the application must set flags for sector size and align size to match those for the lockspace.

To use the rindex, the application:

•

Uses the "create" function to create a new resource lease on disk. This takes the place of the write_resource function. The create function requires the location of the rindex and the name of the new resource lease. sanlock finds a free lease area, writes the new resource lease at that location, updates the rindex with the name:offset, and returns the offset to the caller. The caller uses this offset when acquiring the resource lease.

•

Uses the "delete" function to remove a resource disk on disk (also corresponding to the write_resource function.) sanlock clears the resource lease and the rindex entry for it. A subsequent call to create may use this same disk location for a different resource lease.

•

Uses the "lookup" function to discover the offset of a resource lease given the resource lease name. The caller would typically call this prior to acquiring the resource lease.

•

Uses the "rebuild" function to recreate the rindex if it is damaged or becomes inconsistent. This function scans the disk for resource leases and creates new rindex entries to match the leases it finds.

•

The "update" function manipulates rindex entries directly and should not normally be used by the application. In normal usage, the create and delete functions manipulate rindex entries. Update is mainly useful for testing or repairs.

Expiration

•

If a host fails to renew its delta lease, e.g. it loses access to the storage, its delta lease will eventually expire and another host will be able to take over any resource leases held by the host. sanlock must ensure that the application on two different hosts is not holding and using the same lease concurrently.

•

When sanlock has failed to renew a delta lease for a period of time, it will begin taking measures to stop local processes (applications) from using any resource leases associated with the expiring lockspace delta lease. sanlock enters this "recovery mode" well ahead of the time when another host could take over the locally owned leases. sanlock must have sufficient time to stop all local processes that are using the expiring leases.

•

sanlock uses three methods to stop local processes that are using expiring leases: 1. Graceful shutdown. sanlock will execute a "graceful shutdown" program that the application previously specified for this case. The shutdown program tells the application to shut down because its leases are expiring. The application must respond by stopping its activities and releasing its leases (or exit). If an application does not specify a graceful shutdown program, sanlock sends SIGTERM to the process instead. The process must release its leases or exit in a prescribed amount of time (see -g), or sanlock proceeds to the next method of stopping.

2. Forced shutdown. sanlock will send SIGKILL to processes using the expiring leases. The processes have a fixed amount of time to exit after receiving SIGKILL. If any do not exit in this time, sanlock will proceed to the next method.

3. Host reset. sanlock will trigger the host's watchdog device to forcibly reset it. sanlock carefully manages the timing of the watchdog device so that it fires shortly before any other host could take over the resource leases held by local processes.

NO_TIMEOUT and DEAD_EXT

NO_TIMEOUT is a flag that can be set in a host_id delta lease, which disables timeout-based lease expiration. If the delta lease stops being renewed, the host will be reported as failed after the standard renewal_fail_seconds, but will not be considered dead after the standard host_dead_seconds. If access to the delta lease is restored, the host can continue renewals, and will return to the live state. The host with the unrenewed delta lease will not react to reaching renewal_fail_seconds, but will continue holding its paxos leases in the lockspace indefinitely. Other hosts will be unable to acquire the paxos leases while the delta lease remains in the failed state. The watchdog is not used for NO_TIMEOUT delta leases.

DEAD_EXT is a flag that one host can set in another host's delta lease, to indicate that the other host is effectively dead. paxos leases held by the dead host become available to others. Before DEAD_EXT is set in another host's delta lease, the dead host must be prevented from future updates to resources on which it holds paxos leases (in the given lockspace.) DEAD_EXT is generally used with NO_TIMEOUT leases.

Setting DEAD_EXT is accompanied by the generation number of the host being declared dead, e.g. host A observes that host B gen X (B.X) has failed, host A fences host B from the lockspace resources, host A tells sanlock to set DEAD_EXT for host B gen X (B.X becomes B.X.DEAD_EXT). paxos leases owned by B.X become available to host A. host B could at the same time be restarted with gen Y (B.X.DEAD_EXT becomes B.Y), and acquire paxos leases owned by B.Y. host A cannot acquire paxos leases owned by host B gen Y. The delta lease for host B gen X will have DEAD_EXT set, but the delta lease for host B gen Y will not have DEAD_EXT set. (In the event of a timing race, after host B.Y rejoins the lockspace, its delta lease may revert from B.Y back to B.X.DEAD_EXT temporarily before returning to B.Y.)

From the command line, use -N 1 to include the NO_TIMEOUT flag when initializing a delta lease (init or init_host.) See set_host for setting DEAD_EXT from the command line.

Failures

If a process holding resource leases fails or exits without releasing its leases, sanlock will release the leases for it automatically (unless persistent resource leases were used.)

If the sanlock daemon cannot renew a lockspace delta lease for a specific period of time (see Expiration), sanlock will enter "recovery mode" where it attempts to stop and/or kill any processes holding resource leases in the expiring lockspace. If the processes do not exit in time, sanlock will force the host to be reset using the local watchdog device.

If the sanlock daemon crashes or hangs, it will not renew the expiry time of the per-lockspace connections it had to the wdmd daemon. This will lead to the expiration of the local watchdog device, and the host will be reset.

Watchdog

sanlock uses the wdmd(8) daemon to access /dev/watchdog. wdmd multiplexes multiple timeouts onto the single watchdog timer. This is required because delta leases for each lockspace are renewed and expire independently.

sanlock maintains a wdmd connection for each lockspace delta lease being renewed. Each connection has an expiry time for some seconds in the future. After each successful delta lease renewal, the expiry time is renewed for the associated wdmd connection. If wdmd finds any connection expired, it will not renew the /dev/watchdog timer. Given enough successive failed renewals, the watchdog device will fire and reset the host. (Given the multiplexing nature of wdmd, shorter overlapping renewal failures from multiple lockspaces could cause spurious watchdog firing.)

The direct link between delta lease renewals and watchdog renewals provides a predictable watchdog firing time based on delta lease renewal timestamps that are visible from other hosts. sanlock knows the time the watchdog on another host has fired based on the delta lease time. Furthermore, if the watchdog device on another host fails to fire when it should, the continuation of delta lease renewals from the other host will make this evident and prevent leases from being taken from the failed host.

If sanlock is able to stop/kill all processing using an expiring lockspace, the associated wdmd connection for that lockspace is removed. The expired wdmd connection will no longer block /dev/watchdog renewals, and the host should avoid being reset.

Storage

The sector size and the align size should be specified when creating lockspaces and resources (and rindex). The "align size" is the size on disk of a lockspace or a resource, i.e. the amount of disk space it uses. Lockspaces and resources should use matching sector and align sizes, and must use offsets in multiples of the align size. The max number of hosts that can use a lockspace or resource depends on the combination of sector size and align size, shown below. The host_id of hosts using the lockspace can be no larger than the max_hosts value for the lockspace.

Accepted combinations of sector size and align size, and the corresponding max_hosts (and max host_id) are:

sector_size 512, align_size 1M, max_hosts 2000

sector_size 4096, align_size 1M, max_hosts 250

sector_size 4096, align_size 2M, max_hosts 500

sector_size 4096, align_size 4M, max_hosts 1000

sector_size 4096, align_size 8M, max_hosts 2000

When sector_size and align_size are not specified, the behavior matches the behavior before these sizes could be configured: on devices which report sector size 512, 512/1M/2000 is used, on devices which report sector size 4096, 4096/8M/2000 is used, and on files, 512/1M/2000 is always used. (Other combinations are not compatible with sanlock version 3.6 or earlier.)

Using sanlock on shared block devices that do host based mirroring or replication is not likely to work correctly. When using sanlock on shared files, all sanlock io should go to one file server.

Example

This is an example of creating and using lockspaces and resources from the command line. (Most applications would use sanlock through libsanlock rather than through the command line.)

1.

Allocate shared storage for sanlock leases. This example assumes 512 byte sectors on the device, in which case the lockspace needs 1MB and each resource needs 1MB.

The example shared block device accessible to all hosts is /dev/leases.

2.

Start sanlock on all hosts. The -w 0 disables use of the watchdog for testing.

```
# sanlock daemon -w 0
```

3.

Start a dummy application on all hosts. This sanlock command registers with sanlock, then execs the sleep command which inherits the registered fd. The sleep process acts as the dummy application. Because the sleep process is registered with sanlock, leases can be acquired for it.

```
# sanlock client command -c /bin/sleep 600 &
```

4.

Create a lockspace for the application (from one host). The lockspace is named "test".

```
# sanlock client init -s test:0:/dev/leases:0
```

5.

Join the lockspace for the application. Use a unique host_id on each host.

```
host1:
# sanlock client add_lockspace -s test:1:/dev/leases:0
host2:
# sanlock client add_lockspace -s test:2:/dev/leases:0
```

6.

Create two resources for the application (from one host). The resources are named "RA" and "RB". Offsets are used on the same device as the lockspace. Different LVs or files could also be used.

```
# sanlock client init -r test:RA:/dev/leases:1048576
# sanlock client init -r test:RB:/dev/leases:2097152
```

7.

Acquire resource leases for the application on host1. Acquire an exclusive lease (the default) on the first resource, and a shared lease (SH) on the second resource.

```
# export P=`pidof sleep`
# sanlock client acquire -r test:RA:/dev/leases:1048576 -p $P
# sanlock client acquire -r test:RB:/dev/leases:2097152:SH -p $P
```

8.

Acquire resource leases for the application on host2. Acquiring the exclusive lease on the first resource will fail because it is held by host1. Acquiring the shared lease on the second resource will succeed.

```
# export P=`pidof sleep`
# sanlock client acquire -r test:RA:/dev/leases:1048576 -p $P
# sanlock client acquire -r test:RB:/dev/leases:2097152:SH -p $P
```

9.

Release resource leases for the application on both hosts. The sleep pid could also be killed, which will result in the sanlock daemon releasing its leases when it exits.

```
# sanlock client release -r test:RA:/dev/leases:1048576 -p $P
# sanlock client release -r test:RB:/dev/leases:2097152 -p $P
```

10.

Leave the lockspace for the application.

```
host1:
# sanlock client rem_lockspace -s test:1:/dev/leases:0
host2:
# sanlock client rem_lockspace -s test:2:/dev/leases:0
```

11.

Stop sanlock on all hosts.

```
# sanlock shutdown
```

## OPTIONS
COMMAND can be one of three primary top level choices

sanlock daemon start daemon

sanlock client send request to daemon (default command if none given)

sanlock direct access storage directly (no coordination with daemon)

### Daemon Command
sanlock daemon [options]

-D no fork and print all logging to stderr

-Q 0|1 quiet error messages for common lock contention

-R 0|1 renewal debugging, log debug info for each renewal

-L pri write logging at priority level and up to logfile (-1 none)

-S pri write logging at priority level and up to syslog (-1 none)

-U uid user id

-G gid group id

-H num renewal history size

-t num max worker threads

-g sec seconds for graceful recovery

-w 0|1 use watchdog through wdmd

-o sec io timeout

-h 0|1 use high priority (RR) scheduling

-l num use mlockall (0 none, 1 current, 2 current and future)

-b sec seconds a host id bit will remain set in delta lease bitmap

-e str unique local host name used in delta leases as host_id owner

### Client Command
sanlock client action [options]

sanlock client status

Print processes, lockspaces, and resources being managed by the sanlock daemon. Add -D to show extra internal daemon status for debugging. Add -o p to show resources by pid, or -o s to show resources by lockspace.

sanlock client host_status

Print state of host_id delta leases read during the last renewal. State of all lockspaces is shown (use -s to select one). Add -D to show extra internal daemon status for debugging.

sanlock client gets

Print lockspaces being managed by the sanlock daemon. The LOCKSPACE string will be followed by ADD or REM if the lockspace is currently being added or removed. Add -h 1 to also show hosts in each lockspace.

sanlock client renewal -s LOCKSPACE

Print a history of renewals with timing details. See the Renewal history section below.

sanlock client log_dump

Print the sanlock daemon internal debug log.

sanlock client shutdown

Ask the sanlock daemon to exit. Without the force option (-f 0), the command will be ignored if any lockspaces exist. With the force option (-f 1), any registered processes will be killed, their resource leases released, and lockspaces removed. With the wait option (-w 1), the command will wait for a result from the daemon indicating that it has shut down and is exiting, or cannot shut down because lockspaces exist (command fails).

sanlock client init -s LOCKSPACE

Tell the sanlock daemon to initialize a lockspace on disk. The -o option can be used to specify the io timeout to be written in the host_id leases. The -Z and -A options can be used to specify the sector size and align size, and both should be set together. Use -N 1 to include the NO_TIMEOUT flag in the newly formatted leases. (Also see sanlock direct init.)

sanlock client init -r RESOURCE

Tell the sanlock daemon to initialize a resource lease on disk. The -Z and -A options can be used to specify the sector size and align size, and both should be set together. (Also see sanlock direct init.)

sanlock client read -s LOCKSPACE

Tell the sanlock daemon to read a lockspace from disk. Only the LOCKSPACE path and offset are required. If host_id is zero, the first record at offset (host_id 1) is used. The complete LOCKSPACE is printed. Add -D to print other details. (Also see sanlock direct read_leader.)

sanlock client read -r RESOURCE

Tell the sanlock daemon to read a resource lease from disk. Only the RESOURCE path and offset are required. The complete RESOURCE is printed. Add -D to print other details. (Also see sanlock direct read_leader.)

sanlock client init_host -s LOCKSPACE

Tell the sanlock daemon to initialize a single delta lease on disk. The host_id specified in the -s arg will be used, and written as the lease owner. Optionally specify host name with -e, generation with -g, and timestamp with -t. Use -Z to specify sector size. Use -N 1 to include the NO_TIMEOUT flag in the reformatted lease. (Also see sanlock direct init_host for more information.)

sanlock client add_lockspace -s LOCKSPACE

Tell the sanlock daemon to acquire the delta lease for the host_id specified in LOCKSPACE. With a host_id/delta lease held for the lockspace, the host is then able to acquire resource/paxos leases in the lockspace. Use -o <sec> to specify the io timeout of the acquiring host, which will be written in the host_id delta lease.

sanlock client inq_lockspace -s LOCKSPACE

Inquire about the state of the lockspace in the sanlock daemon, whether it is being added or removed, or is joined.

sanlock client rem_lockspace -s LOCKSPACE

Tell the sanlock daemon to release the specified host_id in the lockspace. Any processes holding resource leases in this lockspace will be killed, and the resource leases not released.

sanlock client command -r RESOURCE -c path args

Register with the sanlock daemon, acquire the specified resource lease, and exec the command at path with args. When the command exits, the sanlock daemon will release the lease. -c must be the final option.

sanlock client acquire -r RESOURCE -p pid

sanlock client release -r RESOURCE -p pid

Tell the sanlock daemon to acquire or release the specified resource lease for the given pid. The pid must be registered with the sanlock daemon. acquire can optionally take a versioned RESOURCE string RESOURCE:lver, where lver is the version of the lease that must be acquired, or fail. Use -C in place of -p to specify client_id.

sanlock client convert -r RESOURCE -p pid

Tell the sanlock daemon to convert the mode of the specified resource lease for the given pid. If the existing mode is exclusive (default), the mode of the lease can be converted to shared with RESOURCE:SH. If the existing mode is shared, the mode of the lease can be converted to exclusive with RESOURCE (no :SH suffix). Use -C in place of -p to specify client_id.

sanlock client inquire -p pid

Print the resource leases held the given pid. The format is a versioned RESOURCE string "RESOURCE:lver" where lver is the version of the lease held. Use -C in place of -p to specify client_id.

sanlock client request -r RESOURCE -f force_mode

Request the owner of a resource do something specified by force_mode. A versioned RESOURCE:lver string must be used with a greater version than is presently held. Zero lver and force_mode clears the request.

sanlock client examine -r RESOURCE

Examine the request record for the currently held resource lease and carry out the action specified by the requested force_mode.

sanlock client examine -s LOCKSPACE

Examine requests for all resource leases currently held in the named lockspace. Only lockspace_name is used from the LOCKSPACE argument.

sanlock client set_event -s LOCKSPACE -i host_id -g gen -e num -d num

Set an event for another host. When the sanlock daemon next renews its delta lease for the lockspace it will: set the bit for the host_id in its bitmap, and set the generation, event and data values in its own delta lease. An application that has registered for events from this lockspace on the destination host will get the event that has been set when the destination sees the event during its next delta lease renewal.

sanlock client set_config -s LOCKSPACE

Set a configuration value for a lockspace. Only lockspace_name is used from the LOCKSPACE argument. The USED flag has the same effect on a lockspace as a process holding a resource lease that will not exit. The USED_BY_ORPHANS flag means that an orphan resource lease will have the same effect as the USED. The -o option can be used to update the lockspace's io timeout.

-u 0|1 Set (1) or clear (0) the USED flag.

-O 0|1 Set (1) or clear (0) the USED_BY_ORPHANS flag.

-o sec io timeout in seconds.

sanlock client set_host -s LOCKSPACE -i host_id -g gen -F flag_name

When flag_name is DEAD_EXT, the DEAD_EXT flag is set in the delta lease for the specified host_id. If the current delta lease generation does not match the specified generation, then the command will fail. With DEAD_EXT set, the host_id+generation will be considered dead, and paxos leases held by the specified host_id+generation will be free for other hosts to acquire. DEAD_EXT should only be set for a host if that host can no longer modify the shared resources that were protected by the paxos leases in the lockspace.

sanlock client format -x RINDEX

Create a resource index on disk. Use -Z and -A to set the sector size and align size to match the lockspace.

sanlock client create -x RINDEX -e resource_name

Create a new resource lease on disk, using the rindex to find a free offset.

sanlock client delete -x RINDEX -e resource_name[:offset]

Delete an existing resource lease on disk.

sanlock client lookup -x RINDEX -e resource_name

Look up the offset of an existing resource lease by name on disk, using the rindex. With no -e option, lookup returns the next free lease offset. If -e specifies both name and offset, the lookup verifies both are correct.

sanlock client update -x RINDEX -e resource_name[:offset] [-z 0|1]

Add (-z 0) or remove (-z 1) an rindex entry on disk.

sanlock client rebuild -x RINDEX

Rebuild the rindex entries by scanning the disk for resource leases.

### Direct Command
sanlock direct action [options]

-o sec io timeout in seconds

sanlock direct init -s LOCKSPACE

sanlock direct init -r RESOURCE

Initialize storage for a lockspace or resource. Use the -Z and -A flags to specify the sector size and align size. The max hosts that can use the lockspace/resource (and the max possible host_id) is determined by the sector/align size combination. Possible combinations are: 512/1M, 4096/1M, 4096/2M, 4096/4M, 4096/8M. Lockspaces and resources both use the same amount of space (align_size) for each combination. When initializing a lockspace, sanlock initializes delta leases for max_hosts in the given space. When initializing a resource, sanlock initializes a single paxos lease in the space. With -s, the -o option specifies the io timeout to be written in the host_id leases. With -r, the -z 1 option invalidates the resource lease on disk so it cannot be used until reinitialized normally. Use -N 1 to include NO_TIMEOUT in newly formatted lockspace host_id leases.

sanlock direct init_host -s LOCKSPACE

Initialize a single delta lease. The host_id specified in the -s arg will be used, and written as the lease owner (leader.owner_id). Optionally specify host name (leader.resource_name) with -e, generation number (leader.owner_generation) with -g, and timestamp (leader.timestamp) with -t (timestamp value 1 is special, and causes the current time to be written in the timestamp field. A timestamp value of 0 is means the delta lease is free, as usual.) The -Z and -o options apply as with direct init. Use -N 1 to include NO_TIMEOUT in the reformatted host_id lease.

sanlock direct read_leader -s LOCKSPACE

sanlock direct read_leader -r RESOURCE

Read a leader record from disk and print the fields. The leader record is the single sector of a delta lease, or the first sector of a paxos lease.

sanlock direct read -s LOCKSPACE

sanlock direct read -r RESOURCE

Read a complete lockspace or resource from disk and print it.

sanlock direct dump path[:offset[:size]]

Read disk sectors and print leader records for delta or paxos leases. Add -f 1 to print the request record values for paxos leases, host_ids set in delta lease bitmaps, and rindex entries.

sanlock direct format -x RINDEX

sanlock direct lookup -x RINDEX -e resource_name

sanlock direct update -x RINDEX -e resource_name[:offset] [-z 0|1]

sanlock direct rebuild -x RINDEX

Access the resource index on disk without going through the sanlock daemon. This precludes using the internal paxos lease to protect rindex modifications. See client equivalents for descriptions.

### LOCKSPACE option string
-s lockspace_name:host_id:path:offset

lockspace_name name of lockspace

host_id local host identifier in lockspace

path path to storage to use for leases

offset offset on path (bytes)

### RESOURCE option string
-r lockspace_name:resource_name:path:offset

lockspace_name name of lockspace

resource_name name of resource

path path to storage to use leases

offset offset on path (bytes)

### RESOURCE option string with suffix
-r lockspace_name:resource_name:path:offset:lver

lver leader version

-r lockspace_name:resource_name:path:offset:SH

SH indicates shared mode

### RINDEX option string
-x lockspace_name:path:offset

lockspace_name name of lockspace

path path to storage to use for leases

offset offset on path (bytes) of rindex

### Defaults
sanlock help shows the default values for the options above.

sanlock version shows the build version.

## OTHER

### Request/Examine
The first part of making a request for a resource is writing the request record of the resource (the sector following the leader record). To make a successful request:

- RESOURCE:lver must be greater than the lver presently held by the other host. This implies the leader record must be read to discover the lver, prior to making a request.

- RESOURCE:lver must be greater than or equal to the lver presently written to the request record. Two hosts may write a new request at the same time for the same lver, in which case both would succeed, but the force_mode from the last would win.

- The force_mode must be greater than zero.

- To unconditionally clear the request record (set both lver and force_mode to 0), make request with RESOURCE:0 and force_mode 0.

The owner of the requested resource will not know of the request unless it is explicitly told to examine its resources via the "examine" api/command, or otherwise notfied.

The second part of making a request is notifying the resource lease owner that it should examine the request records of its resource leases. The notification will cause the lease owner to automatically run the equivalent of "sanlock client examine -s LOCKSPACE" for the lockspace of the requested resource.

The notification is made using a bitmap in each host_id delta lease. Each bit represents each of the possible host_ids (1-2000). If host A wants to notify host B to examine its resources, A sets the bit in its own bitmap that corresponds to the host_id of B. When B next renews its delta lease, it reads the delta leases for all hosts and checks each bitmap to see if its own host_id has been set. It finds the bit for its own host_id set in A's bitmap, and examines its resource request records. (The bit remains set in A's bitmap for set_bitmap_seconds.)

force_mode determines the action the resource lease owner should take:

•

FORCE (1): kill the process holding the resource lease. When the process has exited, the resource lease will be released, and can then be acquired by anyone. The kill signal is SIGKILL (or SIGTERM if SIGKILL is restricted.)

•

GRACEFUL (2): run the program configured by sanlock_killpath against the process holding the resource lease. If no killpath is defined, then FORCE is used.

### Persistent and orphan resource leases
A resource lease can be acquired with the PERSISTENT flag (-P 1). If the process holding the lease exits, the lease will not be released, but kept on an orphan list. Another local process can acquire an orphan lease using the ORPHAN flag (-O 1), or release the orphan lease using the ORPHAN flag (-O 1). All orphan leases can be released by setting the lockspace name (-s lockspace_name) with no resource name.

### Renewal history
sanlock saves a limited history of lease renewal information in each lockspace. See sanlock.conf renewal_history_size to set the amount of history or to disable (set to 0).

IO times are measured in delta lease renewal (each delta lease renewal includes one read and one write).

For each successful renewal, a record is saved that includes:

•

the timestamp written in the delta lease by the renewal

•

the time in milliseconds taken by the delta lease read

•

the time in milliseconds taken by the delta lease write

Also counted and recorded are the number io timeouts and other io errors that occur between successful renewals.

Two consecutive successful renewals would be recorded as:

```
timestamp=5332 read_ms=482 write_ms=5525 next_timeouts=0 next_errors=0
timestamp=5353 read_ms=99 write_ms=3161 next_timeouts=0 next_errors=0
```

Those fields are:

•

timestamp is the value written into the delta lease during that renewal.

•

read_ms/write_ms are the milliseconds taken for the renewal read/write ios.

•

next_timeouts are the number of io timeouts that occurred after the renewal recorded on that line, and before the next successful renewal on the following line.

•

next_errors are the number of io errors (not timeouts) that occurred after renewal recorded on that line, and before the next successful renewal on the following line.

The command 'sanlock client renewal -s lockspace_name' reports the full history of renewals saved by sanlock, which by default is 180 records, about 1 hour of history when using a 20 second renewal interval for a 10 second io timeout.

### Configurable watchdog timeout
Watchdog devices usually have a 60 second timeout, but some devices have a configurable timeout. To use a different watchdog timeout, set sanlock.conf watchdog_fire_timeout (in seconds) to a value supported by the device. The same watchdog_fire_timeout must be configured on all hosts (so all hosts must have watchdog devices that support the same timeout). Unmatching values will invalidate the lease protection provided by the watchdog.

watchdog_fire_timeout and io_timeout should usually be configured together. By default, sanlock uses watchdog_fire_timeout=60 with io_timeout=10. Other combinations to consider are:

watchdog_fire_timeout=30 with io_timeout=5

watchdog_fire_timeout=10 with io_timeout=2

Smaller values make it more likely that a host will be reset by the watchdog while waiting for slow io to complete or for temporary io failures to be resolved. Spurious watchdog resets will also become more likely due to independent, overlapping lockspace outages, each of which would be inconsequential by itself.

## INTERNALS

### Disk Format
•

This example uses 512 byte sectors.

•

Each lockspace is 1MB. It holds 2000 delta_leases, one per sector, supporting up to 2000 hosts.

•

Each paxos_lease is 1MB. It is used as a lease for one resource.

•

The leader_record structure is used differently by each lease type.

•

To display all leader_record fields, see sanlock direct read_leader.

•

A lockspace is often followed on disk by the paxos_leases used within that lockspace, but this layout is not required.

•

The request_record and host_id bitmap are used for requests/events.

•

The mode_block contains the SHARED flag indicating a lease is held in the shared mode.

•

In a lockspace, the host using host_id N writes to a single delta_lease in sector N-1. No other hosts write to this sector. All hosts read all lockspace sectors when renewing their own delta_lease, and are able to monitor renewals of all delta_leases.

•

In a paxos_lease, each host has a dedicated sector it writes to, containing its own paxos_dblock and mode_block structures. Its sector is based on its host_id; host_id 1 writes to the dblock/mode_block in sector 2 of the paxos_lease.

•

The paxos_dblock structures are used by the paxos_lease algorithm, and the result is written to the leader_record.

0x000000 lockspace foo:0:/path:0

(There is no representation on disk of the lockspace in general, only the sequence of specific delta_leases which collectively represent the lockspace.)

delta_lease foo:1:/path:0

```
0x000 0         leader_record         (sector 0, for host_id 1)

                magic: 0x12212010

                space_name: foo

                resource_name: host uuid/name

                ...

                host_id bitmap        (leader_record + 256)
```

delta_lease foo:2:/path:0

```
0x200 512       leader_record         (sector 1, for host_id 2)

                magic: 0x12212010

                space_name: foo

                resource_name: host uuid/name

                ...

                host_id bitmap        (leader_record + 256)
```

delta_lease foo:3:/path:0

```
0x400 1024      leader_record         (sector 2, for host_id 3)

                magic: 0x12212010

                space_name: foo

                resource_name: host uuid/name

                ...

                host_id bitmap        (leader_record + 256)
```

delta_lease foo:2000:/path:0

```
0xF9E00         leader_record         (sector 1999, for host_id 2000)

                magic: 0x12212010

                space_name: foo

                resource_name: host uuid/name

                ...

                host_id bitmap        (leader_record + 256)
```

0x100000 paxos_lease foo:example1:/path:1048576

```
0x000 0         leader_record         (sector 0)

                magic: 0x06152010

                space_name: foo

                resource_name: example1
0x200 512       request_record        (sector 1)

                magic: 0x08292011
0x400 1024      paxos_dblock          (sector 2, for host_id 1)
0x480 1152      mode_block            (paxos_dblock + 128)
0x600 1536      paxos_dblock          (sector 3, for host_id 2)
0x680 1664      mode_block            (paxos_dblock + 128)
0x800 2048      paxos_dblock          (sector 4, for host_id 3)
0x880 2176      mode_block            (paxos_dblock + 128)
0xFA200         paxos_dblock          (sector 2001, for host_id 2000)
0xFA280         mode_block            (paxos_dblock + 128)
```

0x200000 paxos_lease foo:example2:/path:2097152

```
0x000 0         leader_record         (sector 0)

                magic: 0x06152010

                space_name: foo

                resource_name: example2
0x200 512       request_record        (sector 1)

                magic: 0x08292011
0x400 1024      paxos_dblock          (sector 2, for host_id 1)
0x480 1152      mode_block            (paxos_dblock + 128)
0x600 1536      paxos_dblock          (sector 3, for host_id 2)
0x680 1664      mode_block            (paxos_dblock + 128)
0x800 2048      paxos_dblock          (sector 4, for host_id 3)
0x880 2176      mode_block            (paxos_dblock + 128)
0xFA200         paxos_dblock          (sector 2001, for host_id 2000)
0xFA280         mode_block            (paxos_dblock + 128)
```

### Lease ownership
Not shown in the leader_record structures above are the owner_id, owner_generation and timestamp fields. These are the fields that define the lease owner.

The delta_lease at sector N for host_id N+1 has leader_record.owner_id N+1. The leader_record.owner_generation is incremented each time the delta_lease is acquired. When a delta_lease is acquired, the leader_record.timestamp field is set to the time of the host and the leader_record.resource_name is set to the unique name of the host. When the host renews the delta_lease, it writes a new leader_record.timestamp. When a host releases a delta_lease, it writes zero to leader_record.timestamp.

When a host acquires a paxos_lease, it uses the host_id/generation value from the delta_lease it holds in the lockspace. It uses this host_id/generation to identify itself in the paxos_dblock when running the paxos algorithm. The result of the algorithm is the winning host_id/generation - the new owner of the paxos_lease. The winning host_id/generation are written to the paxos_lease leader_record.owner_id and leader_record.owner_generation fields and leader_record.timestamp is set. When a host releases a paxos_lease, it sets leader_record.timestamp to 0.

When a paxos_lease is free (leader_record.timestamp is 0), multiple hosts may attempt to acquire it. The paxos algorithm, using the paxos_dblock structures, will select only one of the hosts as the new owner, and that owner is written in the leader_record. The paxos_lease will no longer be free (non-zero timestamp). Other hosts will see this and will not attempt to acquire the paxos_lease until it is free again.

If a paxos_lease is owned (non-zero timestamp), but the owner has not renewed its delta_lease for a specific length of time, then the owner value in the paxos_lease becomes expired, and other hosts will use the paxos algorithm to acquire the paxos_lease, and set a new owner.

## FILES
/etc/sanlock/sanlock.conf

The current settings in use by the sanlock daemon can be seen in the output of 'sanlock status -D'.

•

quiet_fail = 1

See -Q

•

debug_renew = 0

See -R

•

logfile_priority = 4

See -L

•

logfile_use_utc = 0

Use UTC instead of local time in log messages.

•

syslog_priority = 3

See -S

•

names_log_priority = 6

Log resource names at this priority level (uses syslog priority numbers). If this number less than or equal to logfile_priority, each requested resource name and location is recorded in sanlock.log.

•

use_watchdog = 1

See -w

•

high_priority = 1

See -h

•

mlock_level = 1

See -l

•

sh_retries = 8

The number of times to try acquiring a paxos lease when acquiring a shared lease when the paxos lease is held by another host acquiring a shared lease.

•

uname = sanlock

See -U

•

gname = sanlock

See -G

•

our_host_name = <str>

A unique name that a host uses to ensure exclusive ownership of a lockspace host_id (delta lease owner.) The maximum length is 48 characters. If no value is provided in sanlock.conf or on the command line (-e), sanlock attempts to set our_host_name from /sys/devices/virtual/dmi/id/product_uuid. If that is not available, sanlock generates a random uuid to use as our_host_name. Using a fixed our_host_name value will reduce delays when using a lockspace. Using product_uuid will reduce delays further.

•

renewal_read_extend_sec = <seconds>

If a renewal read i/o times out, wait this many additional seconds for that read to complete at the start of the subsequent renewal attempt. When not configured, sanlock waits for an additional io_timeout seconds for a previous timed out read to complete.

•

renewal_history_size = 180

See -H

•

paxos_debug_all = 0

Include all details in the paxos debug logging.

•

debug_io = <str>

Add debug logging for each i/o. "submit" (no quotes) produces debug output at submission time, "complete" produces debug output at completion time, and "submit,complete" (no space) produces both.

•

max_sectors_kb = <str>|<num>

Set to "ignore" (no quotes) to prevent sanlock from checking or changing max_sectors_kb for the lockspace disk when starting a lockspace. Set to "align" (no quotes) to set max_sectors_kb for the lockspace disk to the align size of the lockspace. Set to a number to set a specific number of KB for all lockspace disks. A larger existing max_sectors_kb value will not be reduced by this setting.

•

debug_clients = 0

Enable or disable debug logging for all client connections to the sanlock daemon.

•

debug_cmd = +|-<name>

Enable (+name) or disable (-name) debug logging at the command processing level for specifically named commands, e.g. "debug_cmd = +acquire", or "debug_cmd = -inq_lockspace". Repeat this line for each command name. Use a plus prefix before the name to enable and a minus prefix to disable. By default sanlock disables some command level debugging for commands that are often repetitive and fill the in memory debug buffer. This only affects debug logging, not errors or warnings, and disabling command level debugging for a command does not disable lower level debugging for that command. Special values +all and -all can be used to enable or disable all commands, and can be used before or after other debug_cmd lines.

•

debug_hosts = 1

Log information about other host_id lease renewals. When set to 1 (the default), messages are logged when a host_id lease is observed reaching the failed and dead states. When set to 2, messages are logged when any update (e.g. renewal) is observed for another host_id lease. When set to 0, neither are logged.

•

write_init_io_timeout = <seconds>

The io timeout to use when initializing ondisk lease structures for a lockspace or resource. This timeout is not used as a part of either lease algorithm (as the standard io_timeout is.)

•

max_worker_threads = <num>

See -t

•

io_timeout = <seconds>

The io timeout for disk operations, most notably delta lease renewals. This value is basis for calculating most other timeout values. (Some special cases may use a different io timeout.) Tune this value with caution, it can substantially alter the overall sanlock behavior.

•

watchdog_fire_timeout = <seconds>

The watchdog device timeout. The watchdog device must support the specified value. It is critical that all hosts use the same value. Not doing so will invalidate the lease protection provided by sanlock. The io_timeout should usually be tuned along with this value, e.g. watchdog_fire_timeout = 30 with io_timeout = 5.

•

use_hugepages = <str>

Set to "all" to use transparent hugepages (2MB via MADV_HUGEPAGE.) This should minimize, or prevent, splitting read io's on lease areas. 2MB is allocated for 1MB lease areas, causing some extra memory usage. Set to "none" to disable.

•

create_old_delta_disk_version = <X.Y>

Format delta leases using the specified old version number, e.g. 3.4. This allows new versions to remain compatible with old versions. Some new features will not be usable with the old version.

## SEE ALSO
wdmd(8)


wdmd

title: wdmd(8)

summary: watchdog multiplexing daemon

text:
# wdmd(8)

## NAME
wdmd - watchdog multiplexing daemon

## SYNOPSIS
wdmd [OPTIONS]

## DESCRIPTION
This daemon opens /dev/watchdog and allows multiple independent sources to detmermine whether each KEEPALIVE is done. Every test interval (default 10 seconds), the daemon tests each source. If any test fails, the KEEPALIVE is not done. In the default configuration, the watchdog timer will reset the system if no KEEPALIVE is done for 60 seconds ("fire timeout"). This means that if a single test fails 5-6 times in row, the watchdog will fire and reset the system. With multiple test sources, fewer separate failures back to back can also cause a reset, e.g.

T seconds, P pass, F fail

T00: test1 P, test2 P, test3 P: KEEPALIVE done

T10: test1 F, test2 F, test3 P: KEEPALIVE skipped

T20: test1 F, test2 P, test3 P: KEEPALIVE skipped

T30: test1 P, test2 F, test3 P: KEEPALIVE skipped

T40: test1 P, test2 P, test3 F: KEEPALIVE skipped

T50: test1 F, test2 F, test3 P: KEEPALIVE skipped

T60: test1 P, test2 F, test3 P: KEEPALIVE skipped

T60: watchdog fires, system resets

(Depending on timings, the system may be reset sometime shortly before T60, and the tests at T60 would not be run.)

A crucial aspect to the design and function of wdmd is that if any single source does not pass the test for the length of the fire timeout, the watchdog is guaranteed to fire, regardless of whether other sources on the system have passed or failed. A spurious reset due to the combined effects of multiple failing tests as shown above, is an accepted side effect.

The wdmd init script will load the softdog module if no other watchdog module has been loaded.

wdmd cannot be used on the system with any other program that needs to open /dev/watchdog, e.g. watchdog(8).

### Test Source: clients
Using libwdmd, programs connect to wdmd via a unix socket, and send regular messages to wdmd to update an expiry time for their connection. Every test interval, wdmd will check if the expiry time for a connection has been reached. If so, the test for that client fails.

### Test Source: scripts
wdmd will run scripts from a designated directory every test interval. If a script exits with 0, the test is considered a success, otherwise a failure. If a script does not exit by the end of the test interval, it is considered a failure.

## OPTIONS
--version, -V

Print version.

--help, -h

Print usage.

--dump, -d

Print debug information from the daemon.

--probe, -p

Print path of functional watchdog device. Exit code 0 indicates a

functional device was found. Exit code 1 indicates a functional device

was not found.

-D

Enable debugging to stderr and don't fork.

-H 0|1

Enable (1) or disable (0) high priority features such as realtime

scheduling priority and mlockall.

-G name

Group ownership for the socket.

-S 0|1

Enable (1) or disable (0) script tests.

-s path

Path to scripts dir.

-k num

Kill unfinished scripts after num seconds.

-w path

The path to the watchdog device to try first.

--trytimeout, -t seconds

Set the timeout for the watchdog device. Use this to check for supported

timeout values.

--forcefire, -F

Force the watchdog to fire and reset the machine. Use with -t.

branch: sisyphus


sbctl

package: sbctl

title: sbctl(8)

summary: Secure Boot Manager

text:
# sbctl(8)

## NAME
sbctl - Secure Boot Manager

## SYNOPSIS
sbctl <command>

## DESCRIPTION
sbctl is a tool that allows one to create keys for secure boot, securely enroll them and keep track of files to sign.

## EFI SIGNING COMMANDS
status

Shows the current secure boot status of the system. It checks if you are currently booted in UEFI with Secure Boot, and whether Setup Mode has been enabled.

create-keys

Creates a set of signing keys used to sign EFI binaries. Currently, it will create the following keys:

•Platform Key

•Key Exchange Key

•Signature Database Key -e, --export

The directory to persist the exported keys.

-d, --database-path

Path to save the GUID file when generating keys.

enroll-keys

Enrolls the created key into the EFI variables.

```
Note that some devices have hardware firmware that is signed and
validated when Secure Boot is enabled. Failing to validate this firmware
could brick devices. It's recommended to enroll your own keys with
Microsoft certificates.
```

-m, --microsoft

Enroll UEFI vendor certificates from Microsoft into the signature database. See Option ROM*.

-t, --tpm-eventlog

Enroll checksums from the TPM Eventlog into the signature database. See Option ROM*.

This feature is experimental

-c, --custom

Enroll custom KEK and db certificates from "/var/lib/sbctl/keys/custom/KEK/", "/var/lib/sbctl/keys/custom/db/", respectively.

-f, --firmware-builtin

Enroll signatures from dbDefault, KEKDefault or PKDefault. This is usefull if sbctl does not vendor your OEM certificates, or doesn’t include all of them. Valid values are "db", "KEK" or "PK" passed as a comma delimitered string.

Default: "db,KEK"

--yes-this-might-brick-my-machine, --yolo

Ignore the Option ROM error and continue enrolling keys into the UEFI firmware. See Option ROM*.

-i, --ignore-immutable

Ignore checking /sys/firmware/efi/efivars/ for immutable files and unset the immutable attribute before enrolling certificates.

--export

Export the keys we intend to enroll as EFI Signature Lists (esl), or EFI Authenticated Variables (auth) into the current working directory. Valid values are: esl, auth.

-p, --partial

Enroll keys only for the hierarchy specified. Valid values are: db, KEK, PK.

--custom-bytes

Enroll a custom bytefile provided by its path to the efivar specified by partial.

-a, --append

Instead of replacing the currently enrolled keys, append the provided one.

--keytype

Set the keytype for all signing keys used by sbctl. This includes PK, KEK and db keys. Default: file

Valid values are: file, tpm

--pk-keytype

Set the PK key type. Default: file

Valid values are: file, tpm

--kek-keytype

Set the KEK key type. Default: file

Valid values are: file, tpm

--db-keytype

Set the db key type. Default: file

Valid values are: file, tpm

sign <FILE>...

Signs an EFI binary with the created key. The file will be checked for valid signatures to avoid duplicates. -o PATH, --output PATH

Output filename. Default replaces the file.

-s, --save

Save file to the database.

sign-all

Signs all enrolled EFI binaries. -g, --generate

Generate all bundles before signing.

import-keys

Imports existing keys into sbctl. --db-cert PATH

Path to a valid Database (db) certificate.

--db-key PATH

Path to a valid Database (db) private key.

--kek-cert PATH

Path to a valid Key Exchange Key (KEK) certificate.

--kek-key PATH

Path to a valid Key Exchange Key (KEK) private key.

--pk-cert PATH

Path to a valid Platform Key(PK) certificate.

--pk-key PATH

Path to a valid Platform Key (PK) private key.

--directory PATH

Path to a key directory. The expected file locations inside this directory are:

•PK/PK.key

•PK/PK.pem

•KEK/KEK.key

•KEK/KEK.pem

•db/db.key

•db/db.pem

--force

Overwrite the existing key directory used by sbctl.

list-files, ls-files, ls

Lists all enrolled EFI binaries.

remove-file <FILE>, rm-file <FILE>, rm <FILE>

Removes the file from the signing database.

list-enrolled-keys, ls-enrolled-keys

Lists all enrolled keys on the system.

verify [FILE...]

Looks for EFI binaries with the mime type application/x-dosexec in the ESP partition, and looks at the file database. Checks if they have been signed with the Signature Database Key. Takes an optional file argument to check specific files.

reset

Resets the Platform Key. This sets the machine out of Secure Boot mode and allows key rotation. -p, --partial

Reset keys only for the hierarchy specified. Valid values are: db, KEK, PK.

rotate-keys

Rotate the secure boot keys and replace them with newly generated keys. Saves the old keys to a directory in /var/tmp and resigns any files from the file database. --backup-dir PATH

Choose backup directory for old keys.

-p, --partial

Rotate keys only for the hierarchy specified. Valid values are: db, KEK, PK.

-k, --key-file

Key file to be appended for the specified hierarchy.

-c, --cert-file

Certificate file to be appended for the specified hierarchy.

--keytype

Set the keytype for all signing keys used by sbctl. This includes PK, KEK and db keys. Default: file

Valid values are: file, tpm

--pk-keytype

Set the PK key type. Default: file

Valid values are: file, tpm

--kek-keytype

Set the KEK key type. Default: file

Valid values are: file, tpm

--db-keytype

Set the db key type. Default: file

Valid values are: file, tpm

export-enrolled-keys

Export already enrolled keys from the system. --dir PATH

Output directory for the exported keys.

--format TYPE

Format of the exported keys. Default: der Valid values: esl, auth.

setup

Setup an sbctl installation.

```
This commands enables declarative configuration of sbctl through the
configuration file. It can also print the current configuration and the
current state of the installation.
```

--setup

Setup an sbctl install based off on the provided configuration file, or the default read configuration from /etc/sbctl/sbctl.conf. This will automatically do the create-keys and enroll-keys commands and also setup the files database for signing.

See sbctl.conf(5) for details.

--migrate

Migrate the configuration and setup of sbctl to a new iteration. Currently the only migration for sbctl is moving from /usr/share/secureboot to /var/lib/sbctl.

--print-config

Prints a serialized version of the current configuration of sbctl. See sbctl.conf(5) for details.

--print-state

Prints the state of the sbctl installation. Currently only prints whether or not sbctl has been installed on the system. Note: This option requires passing --json.

help

Displays a help message.

## EFI BINARY COMMANDS
bundle [FLAGS] <NAME>

Creates a bundle that should produce EFI binaries. See BUNDLES below for more details. -a PATH, --amducode PATH

AMD microcode location.

-c PATH, --cmdline PATH

Cmdline location. (default "/etc/kernel/cmdline")

-e PATH, --efi-stub PATH

EFI Stub location. (default "/usr/lib/systemd/boot/efi/linuxx64.efi.stub")

-p PATH, --esp PATH

ESP location. (default "/efi")

-h, --help

Help for bundle.

-f PATH, --initramfs PATH

Initramfs location. (default "/boot/initramfs-linux.img")

-i PATH, --intelucode PATH

Intel microcode location.

-k PATH, --kernel-img PATH

Kernel image location. (default "/boot/vmlinuz-linux")

-o PATH, --os-release PATH

OS Release file location. (default "/usr/lib/os-release")

-s, --save

Save bundle to the database.

-l PATH, --splash-img PATH

Boot splash image location.

generate-bundles

This command generates all bundles. -s, --sign

Sign all the generated bundles.

remove-bundle <NAME>, rm-bundle <NAME>

Removes a bundle from the list. This does not delete the bundle itself.

list-bundles, ls-bundle

List all registered bundles to generate.

## OPTIONS
-j, --json

This enables supported commands to output their values in json instead of human-readable text. This is practical for parsing data with tools like jq.

-c, --config

An optionally provided path to the configuration file that should be used by sbctl. Default: /etc/sbctl/sbctl.conf

--disable-landlock

Disables landlock sandboxing in sbctl. See landlock(7).

--debug

Enable verbose debug logging. This will break the pretty printed text.

## BUNDLES
Note: This is being deprecated. Please move to dracut/mkinitcpio/ukify.

Normally, only the kernel is signed with your secure boot keys. This means the kernel command line and initramfs can be changed without possibility of verification.

Bundles are EFI executables which pack all three (initramfs, kernel and cmdline) into a single file which is easy to sign. Avoiding any unsigned files during boot makes the whole process more tamper-proof.

When a bundle is generated, its configuration is stored into the bundle database (see FILES). Subsequent executions of sbctl generate-bundles will rebuild these bundles, so you don’t need to re-specify all parameters after each system update.

Tip: systemd-boot will automatically show entries for any bundles found in esp/EFI/Linux/*.efi.

## SUPPORTED KEY TYPES
sbctl currently supports storing keys as plain unencrypted files, or as TPM shielded keys.

Plain unencrypted files (file) should only be used when the root partition is encrypted. This is the default key type for historic reasons. File keys are hardcoded to RSA 4096.

TPM shielded keys (tpm) are shielded inside the TPM and available if there is an accessible TPM on the system. TPM policies are not supported which means we can’t seal keys towards the system state. Note that TPM keys are hardcoded to RSA 2048, which is usually the highest bit strength supported by TPMs.

When creating a key hierarchy for Secure Boot, the user can decide which key type each of the keys in the hierarchy gets. Because TPMs can be slow to sign, it’s generally adviced to keep PK and KEK in the TPM, while the db key can be a standard file key.

Note that password protection is currently not supported.

## LANDLOCK
sbctl supports landlock and will attempt to restrict access to filepaths to where it’s needed during execution. Any attempts at establishing network access is also restricted.

This feature can be disabled by setting landlock: false in the configuration file, or by passing --disable-landlock to sbctl.

## OPTION ROM
See https://github.com/Foxboron/sbctl/wiki/FAQ#option-rom

## USAGE
Note: To use custom Secure Boot keys it’s important to reboot into firmware setup (systemctl reboot --firmware-setup) and navigate into the Secure Boot menu to enter Setup Mode. This is normally achieved by deleting/clearing the secure boot keys (or at a minimum the Platform Key) while leaving secure boot mode enabled. Some firmwares have a Custom Mode which only disables signature verification and should therefore not be enabled unless no other way to enter key management is provided. If this step is not completed, enrolling custom keys will be rejected by the firmware.

Next is creating the keys for secure boot. create-keys creates the key hierarchy needed for secure boot into "/var/lib/sbctl/keys".

```
# sbctl create-keys
Created Owner UUID a9fbbdb7-a05f-48d5-b63a-08c5df45ee70
Creating secure boot keys...✔
Secure boot keys created!
```

Next up is enrolling the keys into the efi firmware. sbctl supports doing this on a live system instead of having to boot or run a key management tool from the UEFI shell.

Note: This can fail because of firmware issues and unique options in the machine BIOS menu. Also, some devices have hardware firmware that is signed and validated when Secure Boot is enabled. Failing to validate this firmware could brick devices. It’s recommended to enroll your own keys with Microsoft certificates using the -m option. See Option ROM* above.

```
# sbctl enroll-keys
Enrolling keys to EFI variables...✔
Enrolled keys to the EFI variables!
```

After we have successfully enrolled the keys, we need to sign our current boot chain. Traditionally on UEFI systems one can have an EFI System Partition (ESP) on /efi, /boot or /boot/efi. One can usually find the correct one by looking at mount points or finding the EFI directory on the ESP.

The most important file to sign is the kernel. This location differs between distributions but can usually be found on the ESP or /boot. We use --save to store the file path, so we don’t need to manually sign it later.

Note that sbctl can only keep track of file paths. On versioned kernels this might prove tricky.

```
# sbctl sign --save /efi/vmlinuz-linux
✔ Signed /efi/vmlinuz-linux
```

Next is to sign the bootloader. This can usually be found on the standard path below, but might differ between installations.

```
# sbctl sign --save /efi/EFI/BOOT/BOOTX64.EFI
✔ Signed /efi/EFI/BOOT/BOOTX64.EFI
```

sbctl is able to find and verify the ESP, along with any saved files to verify we have signed the files we need.

```
# sbctl verify
Verifying file database and EFI images in /efi...
✔ /efi/EFI/BOOT/BOOTX64.EFI is signed
✔ /efi/vmlinuz-linux is signed
```

Once we have confirmed everything works, we can reboot. Once we have logged back in, we can verify the state of the system. There should be no need to re-enable Secure Boot or enter User Mode in the firmware.

```
$ sbctl status
Installed:  ✓ sbctl is installed
Owner GUID: a7b893cc-949d-408c-b5cc-6e7d0370fdb6
Setup Mode: ✓ Disabled
Secure Boot:    ✓ Enabled
```

When we do a system update, we can run sign-all to resign all the saved files from earlier.

```
# sbctl sign-all
File has already been signed /boot/vmlinuz-linux
✓ Signed /efi/EFI/BOOT/BOOTX64.EFI
```

sbctl supports creating unified kernel images. These UEFI executables bundles the initramfs, kernel and cmdline into one executable which can be signed for secure boot. This allows you to authenticate larger parts of the bootchain instead of only signing the kernel.

```
# sbctl bundle -i /boot/intel-ucode.img

               -l /usr/share/systemd/bootctl/splash-arch.bmp

               -k /boot/vmlinuz-linux

               -f /boot/initramfs-linux-lts.img

               -c /etc/kernel/cmdline

               /efi/EFI/Linux/linux-linux.efi
```

Note that dracut(8) and mkinitcpio(8) supports unified kernel features, and they should be preferred over the sbctl implementation. It is mostly provided in the cases where this feature is not supported by the initramfs generator of the distribution.

## NOTES
All commands that take path arguments convert them into absolute paths when saving them to the database.

## EXIT STATUS
On success, 0 is returned, a non-zero failure code otherwise.

## ENVIRONMENT VARIABLES
SYSTEMD_ESP_PATH, ESP_PATH

Defines the EFI system partition (ESP) location. This overrides the behaviour from sbctl where we query for the correct partition with lsblk. No checks are performed on this path and can be usefull for testing purposes.

SBCTL_UNICODE

If this value is "0" sbctl will replace the unicode symbols to equivalent ascii ones. The default value is assumed to be 1.

## FILES
/etc/sbctl/sbctl.conf:; Default configuration file. See sbctl.conf(5)

/var/lib/sbctl

Default storage directory.

/var/lib/sbctl/GUID

Owner identification. This is a randomly generated UUID.

/var/lib/sbctl/files.db

Contains a list of EFI binaries to be signed by the generated key.

/var/lib/sbctl/bundles.db

Contains a list of EFI bundles to be generated.

/var/lib/sbctl/keys/db/db.{pem,key}

Contains the Signature Database key used for signing EFI binaries.

/var/lib/sbctl/keys/KEK/KEK.{pem,key}

Contains the Key Exchange Key.

/var/lib/sbctl/keys/PK/PK.{pem,key}

Contains the Platform Key.

/var/lib/sbctl/keys/custom/KEK/*

Contains custom certificates which will be added to the firmware as additional Key Exchange Keys.

/var/lib/sbctl/keys/custom/db/*

Contains custom certificates which will be added to the firmware Signature Database.

## SEE ALSO
bootctl(1) jq(1) sbctl.conf(5) landlock(7)

## AUTHORS
•Morten Linderud <morten@linderud.pw>


sbd

package: sbd

title: sbd(8)

summary: STONITH Block Device daemon

text:
# sbd(8)

## NAME
sbd - STONITH Block Device daemon

## SYNOPSIS
sbd <-d /dev/...> [options] "command"

## SUMMARY
SBD provides a node fencing mechanism (Shoot the other node in the head, STONITH) for Pacemaker-based clusters through the exchange of messages via shared block storage such as for example a SAN, iSCSI, FCoE. This isolates the fencing mechanism from changes in firmware version or dependencies on specific firmware controllers, and it can be used as a STONITH mechanism in all configurations that have reliable shared storage.

SBD can also be used without any shared storage. In this mode, the watchdog device will be used to reset the node if it loses quorum, if any monitored daemon is lost and not recovered or if Pacemaker decides that the node requires fencing.

The sbd binary implements both the daemon that watches the message slots as well as the management tool for interacting with the block storage device(s). This mode of operation is specified via the "command" parameter; some of these modes take additional parameters.

To use SBD with shared storage, you must first "create" the messaging layout on one to three block devices. Second, configure /etc/sysconfig/sbd to list those devices (and possibly adjust other options), and restart the cluster stack on each node to ensure that "sbd" is started. Third, configure the "external/sbd" fencing resource in the Pacemaker CIB.

Each of these steps is documented in more detail below the description of the command options.

"sbd" can only be used as root.

### GENERAL OPTIONS
-d /dev/...

Specify the block device(s) to be used. If you have more than one, specify this option up to three times. This parameter is mandatory for all modes, since SBD always needs a block device to interact with. This man page uses /dev/sda1, /dev/sdb1, and /dev/sdc1 as example device names for brevity. However, in your production environment, you should instead always refer to them by using the long, stable device name (e.g., /dev/disk/by-id/dm-uuid-part1-mpath-3600508b400105b5a0001500000250000).

-v|-vv|-vvv

Enable verbose|debug|debug-library logging (optional)

-h

Display a concise summary of "sbd" options.

-n node

Set local node name; defaults to "uname -n". This should not need to be set.

-R

Do not enable realtime priority. By default, "sbd" runs at realtime priority, locks itself into memory, and also acquires highest IO priority to protect itself against interference from other processes on the system. This is a debugging-only option.

-I N

Async IO timeout (defaults to 3 seconds, optional). You should not need to adjust this unless your IO setup is really very slow. (In daemon mode, the watchdog is refreshed when the majority of devices could be read within this time.)

### create
Example usage:

```
        sbd -d /dev/sdc2 -d /dev/sdd3 create
```

If you specify the create command, sbd will write a metadata header to the device(s) specified and also initialize the messaging slots for up to 255 nodes.

Warning: This command will not prompt for confirmation. Roughly the first megabyte of the specified block device(s) will be overwritten immediately and without backup.

This command accepts a few options to adjust the default timings that are written to the metadata (to ensure they are identical across all nodes accessing the device).

-1 N

Set watchdog timeout to N seconds. This depends mostly on your storage latency; the majority of devices must be successfully read within this time, or else the node will self-fence. If your sbd device(s) reside on a multipath setup or iSCSI, this should be the time required to detect a path failure. You may be able to reduce this if your device outages are independent, or if you are using the Pacemaker integration.

-2 N

Set slot allocation timeout to N seconds. You should not need to tune this.

-3 N

Set daemon loop timeout to N seconds. You should not need to tune this.

-4 N

Set msgwait timeout to N seconds. This should be twice the watchdog timeout. This is the time after which a message written to a node's slot will be considered delivered. (Or long enough for the node to detect that it needed to self-fence.) This also affects the stonith-timeout in Pacemaker's CIB; see below.

### list
Example usage:

```
        # sbd -d /dev/sda1 list
        0       hex-0   clear
        1       hex-7   clear
        2       hex-9   clear
```

List all allocated slots on device, and messages. You should see all cluster nodes that have ever been started against this device. Nodes that are currently running should have a clear state; nodes that have been fenced, but not yet restarted, will show the appropriate fencing message.

### dump
Example usage:

```
        # sbd -d /dev/sda1 dump
        ==Dumping header on disk /dev/sda1
        Header version     : 2
        Number of slots    : 255
        Sector size        : 512
        Timeout (watchdog) : 15
        Timeout (allocate) : 2
        Timeout (loop)     : 1
        Timeout (msgwait)  : 30
        ==Header on disk /dev/sda1 is dumped
```

Dump meta-data header from device.

### watch
Example usage:

```
        sbd -d /dev/sdc2 -d /dev/sdd3 -P watch
```

This command will make "sbd" start in daemon mode. It will constantly monitor the message slot of the local node for incoming messages, reachability, and optionally take Pacemaker's state into account.

"sbd" must be started on boot before the cluster stack! See below for enabling this according to your boot environment.

The options for this mode are rarely specified directly on the commandline directly, but most frequently set via /etc/sysconfig/sbd.

It also constantly monitors connectivity to the storage device, and self-fences in case the partition becomes unreachable, guaranteeing that it does not disconnect from fencing messages.

A node slot is automatically allocated on the device(s) the first time the daemon starts watching the device; hence, manual allocation is not usually required.

If a watchdog is used together with the "sbd" as is strongly recommended, the watchdog is activated at initial start of the sbd daemon. The watchdog is refreshed every time the majority of SBD devices has been successfully read. Using a watchdog provides additional protection against "sbd" crashing.

If the Pacemaker integration is activated, "sbd" will not self-fence if device majority is lost, if:

1.

The partition the node is in is still quorate according to the CIB;

2.

it is still quorate according to Corosync's node count;

3.

the node itself is considered online and healthy by Pacemaker.

This allows "sbd" to survive temporary outages of the majority of devices. However, while the cluster is in such a degraded state, it can neither successfully fence nor be shutdown cleanly (as taking the cluster below the quorum threshold will immediately cause all remaining nodes to self-fence). In short, it will not tolerate any further faults. Please repair the system before continuing.

There is one "sbd" process that acts as a master to which all watchers report; one per device to monitor the node's slot; and, optionally, one that handles the Pacemaker integration.

-W

Enable or disable use of the system watchdog to protect against the sbd processes failing and the node being left in an undefined state. Specify this once to enable, twice to disable. Defaults to enabled.

-w /dev/watchdog

This can be used to override the default watchdog device used and should not usually be necessary.

-p /run/sbd.pid

This option can be used to specify a pidfile for the main sbd process.

-F N

Number of failures before a failing servant process will not be restarted immediately until the dampening delay has expired. If set to zero, servants will be restarted immediately and indefinitely. If set to one, a failed servant will be restarted once every -t seconds. If set to a different value, the servant will be restarted that many times within the dampening period and then delay. Defaults to 1.

-t N

Dampening delay before faulty servants are restarted. Combined with "-F 1", the most logical way to tune the restart frequency of servant processes. Default is 5 seconds. If set to zero, processes will be restarted indefinitely and immediately.

-P

Enable Pacemaker integration which checks Pacemaker quorum and node health. Specify this once to enable, twice to disable. Defaults to enabled.

-S N

Set the start mode. (Defaults to 0.) If this is set to zero, sbd will always start up unconditionally, regardless of whether the node was previously fenced or not.

If set to one, sbd will only start if the node was previously shutdown cleanly (as indicated by an exit request message in the slot), or if the slot is empty. A reset, crashdump, or power-off request in any slot will halt the start up.

This is useful to prevent nodes from rejoining if they were faulty. The node must be manually "unfenced" by sending an empty message to it:

```
        sbd -d /dev/sda1 message node1 clear

```

-s N

Set the start-up wait time for devices. (Defaults to 120.) Dynamic block devices such as iSCSI might not be fully initialized and present yet. This allows one to set a timeout for waiting for devices to appear on start-up. If set to 0, start-up will be aborted immediately if no devices are available.

-Z

Enable trace mode. Warning: this is unsafe for production, use at your own risk! Specifying this once will turn all reboots or power-offs, be they caused by self-fence decisions or messages, into a crashdump. Specifying this twice will just log them but not continue running.

-T

By default, the daemon will set the watchdog timeout as specified in the device metadata. However, this does not work for every watchdog device. In this case, you must manually ensure that the watchdog timeout used by the system correctly matches the SBD settings, and then specify this option to allow "sbd" to continue with start-up.

-5 N

Warn if the time interval for tickling the watchdog exceeds this many seconds. Since the node is unable to log the watchdog expiry (it reboots immediately without a chance to write its logs to disk), this is very useful for getting an indication that the watchdog timeout is too short for the IO load of the system. Default is about 3/5 of watchdog timeout, set to zero to disable.

-C N

Watchdog timeout to set before crashdumping. If SBD is set to crashdump instead of reboot - either via the trace mode settings or the external/sbd fencing agent's parameter -, SBD will adjust the watchdog timeout to this setting before triggering the dump. Otherwise, the watchdog might trigger and prevent a successful crashdump from ever being written. Set to zero (= default) to disable.

-r N

Actions to be executed when the watchers don't timely report to the sbd master process or one of the watchers detects that the master process has died. Set timeout-action to comma-separated combination of noflush|flush plus reboot|crashdump|off. If just one of both is given the other stays at the default.

This doesn't affect actions like off, crashdump, reboot explicitly triggered via message slots. And it does as well not configure the action a watchdog would trigger should it run off (there is no generic interface).

Defaults to flush,reboot.

### allocate
Example usage:

```
        sbd -d /dev/sda1 allocate node1
```

Explicitly allocates a slot for the specified node name. This should rarely be necessary, as every node will automatically allocate itself a slot the first time it starts up on watch mode.

### message
Example usage:

```
        sbd -d /dev/sda1 message node1 test
```

Writes the specified message to node's slot. This is rarely done directly, but rather abstracted via the "external/sbd" fencing agent configured as a cluster resource.

Supported message types are:

test

This only generates a log message on the receiving node and can be used to check if SBD is seeing the device. Note that this could overwrite a fencing request send by the cluster, so should not be used during production.

reset

Reset the target upon receipt of this message.

off

Power-off the target.

crashdump

Cause the target node to crashdump.

exit

This will make the "sbd" daemon exit cleanly on the target. You should not send this message manually; this is handled properly during shutdown of the cluster stack. Manually stopping the daemon means the node is unprotected!

clear

This message indicates that no real message has been sent to the node. You should not set this manually; "sbd" will clear the message slot automatically during start-up, and setting this manually could overwrite a fencing message by the cluster.

### query-watchdog
Example usage:

```
        sbd query-watchdog
```

Check for available watchdog devices and print some info.

Warning: This command will arm the watchdog during query, and if your watchdog refuses disarming (for example, if its kernel module has the 'nowayout' parameter set) this will reset your system.

### test-watchdog
Example usage:

```
        sbd test-watchdog [-w /dev/watchdog3]
```

Test specified watchdog device (/dev/watchdog by default).

Warning: This command will arm the watchdog and have your system reset in case your watchdog is working properly! If issued from an interactive session, it will prompt for confirmation.

## Base system configuration

### Configure a watchdog
It is highly recommended that you configure your Linux system to load a watchdog driver with hardware assistance (as is available on most modern systems), such as hpwdt, iTCO_wdt, or others. As a fall-back, you can use the softdog module.

No other software must access the watchdog timer; it can only be accessed by one process at any given time. Some hardware vendors ship systems management software that use the watchdog for system resets (f.e. HP ASR daemon). Such software has to be disabled if the watchdog is to be used by SBD.

### Choosing and initializing the block device(s)
First, you have to decide if you want to use one, two, or three devices.

If you are using multiple ones, they should reside on independent storage setups. Putting all three of them on the same logical unit for example would not provide any additional redundancy.

The SBD device can be connected via Fibre Channel, Fibre Channel over Ethernet, or even iSCSI. Thus, an iSCSI target can become a sort-of network-based quorum server; the advantage is that it does not require a smart host at your third location, just block storage.

The SBD partitions themselves must not be mirrored (via MD, DRBD, or the storage layer itself), since this could result in a split-mirror scenario. Nor can they reside on cLVM2 volume groups, since they must be accessed by the cluster stack before it has started the cLVM2 daemons; hence, these should be either raw partitions or logical units on (multipath) storage.

The block device(s) must be accessible from all nodes. (While it is not necessary that they share the same path name on all nodes, this is considered a very good idea.)

SBD will only use about one megabyte per device, so you can easily create a small partition, or very small logical units. (The size of the SBD device depends on the block size of the underlying device. Thus, 1MB is fine on plain SCSI devices and SAN storage with 512 byte blocks. On the IBM s390x architecture in particular, disks default to 4k blocks, and thus require roughly 4MB.)

The number of devices will affect the operation of SBD as follows:

One device

In its most simple implementation, you use one device only. This is appropriate for clusters where all your data is on the same shared storage (with internal redundancy) anyway; the SBD device does not introduce an additional single point of failure then. If the SBD device is not accessible, the daemon will fail to start and inhibit startup of cluster services.

Two devices

This configuration is a trade-off, primarily aimed at environments where host-based mirroring is used, but no third storage device is available. SBD will not commit suicide if it loses access to one mirror leg; this allows the cluster to continue to function even in the face of one outage.

However, SBD will not fence the other side while only one mirror leg is available, since it does not have enough knowledge to detect an asymmetric split of the storage. So it will not be able to automatically tolerate a second failure while one of the storage arrays is down. (Though you can use the appropriate crm command to acknowledge the fence manually.)

It will not start unless both devices are accessible on boot.

Three devices

In this most reliable and recommended configuration, SBD will only self-fence if more than one device is lost; hence, this configuration is resilient against temporary single device outages (be it due to failures or maintenance). Fencing messages can still be successfully relayed if at least two devices remain accessible. This configuration is appropriate for more complex scenarios where storage is not confined to a single array. For example, host-based mirroring solutions could have one SBD per mirror leg (not mirrored itself), and an additional tie-breaker on iSCSI.

It will only start if at least two devices are accessible on boot.

After you have chosen the devices and created the appropriate partitions and perhaps multipath alias names to ease management, use the "sbd create" command described above to initialize the SBD metadata on them.

Sharing the block device(s) between multiple clusters

It is possible to share the block devices between multiple clusters, provided the total number of nodes accessing them does not exceed 255 nodes, and they all must share the same SBD timeouts (since these are part of the metadata).

If you are using multiple devices this can reduce the setup overhead required. However, you should not share devices between clusters in different security domains.

### Configure SBD to start on boot
On systems using "sysvinit", the "openais" or "corosync" system start-up scripts must handle starting or stopping "sbd" as required before starting the rest of the cluster stack.

For "systemd", sbd simply has to be enabled using

```
        systemctl enable sbd.service
```

The daemon is brought online on each node before corosync and Pacemaker are started, and terminated only after all other cluster components have been shut down - ensuring that cluster resources are never activated without SBD supervision.

### Configuration via sysconfig
The system instance of "sbd" is configured via /etc/sysconfig/sbd. In this file, you must specify the device(s) used, as well as any options to pass to the daemon:

```
        SBD_DEVICE="/dev/sda1;/dev/sdb1;/dev/sdc1"
        SBD_PACEMAKER="true"
```

"sbd" will fail to start if no "SBD_DEVICE" is specified. See the installed template or section for configuration via environment for more options that can be configured here. In general configuration done via parameters takes precedence over the configuration from the configuration file.

### Configuration via environment
SBD_DEVICE

Allows "string" defaulting to "" SBD_DEVICE specifies the devices to use for exchanging sbd messages and to monitor. If specifying more than one path, use ";" as separator.

SBD_PACEMAKER

Allows "yesno" defaulting to "yes" Whether to enable the pacemaker integration.

SBD_STARTMODE

Allows "always / clean" defaulting to "always" Specify the start mode for sbd. Setting this to "clean" will only allow sbd to start if it was not previously fenced. See the -S option in the man page.

SBD_DELAY_START

Allows "yesno / integer" defaulting to "no" Whether to delay after starting sbd on boot for "msgwait" seconds. This may be necessary if your cluster nodes reboot so fast that the other nodes are still waiting in the fence acknowledgement phase. This is an occasional issue with virtual machines.

This can also be enabled by being set to a specific delay value, in seconds. Sometimes a longer delay than the default, "msgwait", is needed, for example in the cases where it's considered to be safer to wait longer than: corosync token timeout + consensus timeout + pcmk_delay_max + msgwait

Be aware that the special value "1" means "yes" rather than "1s".

Consider that you might have to adapt the startup-timeout accordingly if the default isn't sufficient. (TimeoutStartSec for systemd)

This option may be ignored at a later point, once pacemaker handles this case better.

SBD_WATCHDOG_DEV

Allows "string" defaulting to "/dev/watchdog" Watchdog device to use. If set to /dev/null, no watchdog device will be used.

SBD_WATCHDOG_TIMEOUT

Allows "integer" defaulting to 5 How long, in seconds, the watchdog will wait before panicking the node if no-one tickles it.

This depends mostly on your storage latency; the majority of devices must be successfully read within this time, or else the node will self-fence.

If your sbd device(s) reside on a multipath setup or iSCSI, this should be the time required to detect a path failure.

Be aware that watchdog timeout set in the on-disk metadata takes precedence.

SBD_TIMEOUT_ACTION

Allows "string" defaulting to "flush,reboot" Actions to be executed when the watchers don't timely report to the sbd master process or one of the watchers detects that the master process has died.

Set timeout-action to comma-separated combination of noflush|flush plus reboot|crashdump|off. If just one of both is given the other stays at the default.

This doesn't affect actions like off, crashdump, reboot explicitly triggered via message slots. And it does as well not configure the action a watchdog would trigger should it run off (there is no generic interface).

SBD_MOVE_TO_ROOT_CGROUP

Allows "yesno / auto" defaulting to "auto" If CPUAccounting is enabled default is not to assign any RT-budget to the system.slice which prevents sbd from running RR-scheduled.

One way to escape that issue is to move sbd-processes from the slice they were originally started to root-slice. Of course starting sbd in a certain slice might be intentional. Thus in auto-mode sbd will check if the slice has RT-budget assigned. If that is the case sbd will stay in that slice while it will be moved to root-slice otherwise.

SBD_SYNC_RESOURCE_STARTUP

Allows "yesno" defaulting to "yes" If resource startup syncing is enabled then pacemakerd is gonna wait to be pinged via IPC before it starts resources. On shutdown pacemakerd is going to wait in a state where it has cleanly shutdown resources till sbd fetches that state.

The default is set when building SBD and Pacemaker from source. Going for 'no' is safer if it can't be assured that SBD and Pacemaker installed do both support the synchronization feature. When going with 'yes' - also using package dependencies to assure SBD & Pacemaker both support the synchronization feature and are assuming the same default - an SBD configuration inherited via an upgrade doesn't have to be altered to still benefit from the new feature.

SBD_OPTS

Allows "string" defaulting to "" Additional options for starting sbd

### Testing the sbd installation
After a restart of the cluster stack on this node, you can now try sending a test message to it as root, from this or any other node:

```
        sbd -d /dev/sda1 message node1 test
```

The node will acknowledge the receipt of the message in the system logs:

```
        Aug 29 14:10:00 node1 sbd: [13412]: info: Received command test from node2
```

This confirms that SBD is indeed up and running on the node, and that it is ready to receive messages.

Make sure that /etc/sysconfig/sbd is identical on all cluster nodes, and that all cluster nodes are running the daemon.

## Pacemaker CIB integration

### Fencing resource
Pacemaker can only interact with SBD to issue a node fence if there is a configure fencing resource. This should be a primitive, not a clone, as follows:

```
        primitive fencing-sbd stonith:external/sbd \
                params pcmk_delay_max=30
```

This will automatically use the same devices as configured in /etc/sysconfig/sbd.

While you should not configure this as a clone (as Pacemaker will register the fencing device on each node automatically), the pcmk_delay_max setting enables random fencing delay which ensures, in a scenario where a split-brain scenario did occur in a two node cluster, that one of the nodes has a better chance to survive to avoid double fencing.

SBD also supports turning the reset request into a crash request, which may be helpful for debugging if you have kernel crashdumping configured; then, every fence request will cause the node to dump core. You can enable this via the "crashdump="true"" parameter on the fencing resource. This is not recommended for production use, but only for debugging phases.

### General cluster properties
You must also enable STONITH in general, and set the STONITH timeout to be at least twice the msgwait timeout you have configured, to allow enough time for the fencing message to be delivered. If your msgwait timeout is 60 seconds, this is a possible configuration:

```
        property stonith-enabled="true"
        property stonith-timeout="120s"
```

Caution: if stonith-timeout is too low for msgwait and the system overhead, sbd will never be able to successfully complete a fence request. This will create a fencing loop.

Note that the sbd fencing agent will try to detect this and automatically extend the stonith-timeout setting to a reasonable value, on the assumption that sbd modifying your configuration is preferable to not fencing.

## Management tasks

### Recovering from temporary SBD device outage
If you have multiple devices, failure of a single device is not immediately fatal. "sbd" will retry to restart the monitor for the device every 5 seconds by default. However, you can tune this via the options to the watch command.

In case you wish the immediately force a restart of all currently disabled monitor processes, you can send a SIGUSR1 to the SBD inquisitor process.

## LICENSE
Copyright (C) 2008-2013 Lars Marowsky-Bree

This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

This software is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

For details see the GNU General Public License at http://www.gnu.org/licenses/gpl-2.0.html (version 2) and/or http://www.gnu.org/licenses/gpl.html (the newest as per "any later").