drbdmeta-9.0(8)

DRBDMETA(8) System Administration DRBDMETA(8)

NAME

drbdmeta - Manipulate the DRBD on-disk metadata

SYNOPSIS

drbdmeta [--force] [--quiet] [--ignore-sanity-checks] {device} {v06 minor | v07 meta_dev index | v08 meta_dev index | v09 meta_dev index} {command} [cmd args...]

DESCRIPTION

The drbdmeta utility is used for creating, displaying, and modifying DRBD's on-disk metadata. Users usually interact with the drbdadm utility, which provides a more high-level interface to DRBD than drbdmeta. (See drbdadm's --dry-run option to see how drbdadm uses drbdmeta.)

This utility can only be used on devices which are not currently in use by the kernel.

The first argument (device) specifies the drbd device associated with a volume, or “-” if no device is associated with that volume. If the drbd device is specified, the drbdmeta utility makes sure that the drbd device does not currently have a volume attached to prevent meta-data of an active volume from being destroyed.

The second argument specifies the metadata version to use (v06, v07, v08, v09). In most metadata versions, the third argument (meta_dev) specifies the device which contains the metadata; this argument can be the same as device. The fourth argument (index) can be one of the keywords internal (for internal metadata), flex-internal (in v07 for variable-sized metadata; v07 otherwise defaults to fixed-size internal metadata), flex-external (for variable-sized external metadata), or a numeric matadata index (for fixed-size external metadata). See the meta-disk parameter in drbd.conf(5).

OPTIONS

--force

Assume yes as the answer to all questions drbdmeta would ask. Each auto-confirmed prompt is reported to standard error. Combine with --quiet to suppress these messages.

--quiet

When combined with --force, suppresses the informational messages and confirmation prompts that --force would otherwise auto-confirm and report. At the end of the run, a single summary line is printed to standard error stating how many confirmations were suppressed, but only if at least one was. Has no effect without --force.

--ignore-sanity-checks

Normally, drbdmeta performs some sanity checks before writing to the metadata device: for example, if the device appears to contain a file system, it refuses to destroy the file system by writing into it. Use this option to ignore these checks.

COMMANDS

create-md [--peer-max-bio-size=val] (metadata versions v06, v07, and v08),
create-md {number-of-bitmap-slots} [--peer-max-bio-size=val] [--initialize-bitmap={automatic|zeroout|pwrite|set-all|skip}] [--al-stripes=val] [--al-stripe-size-kB=val] (metadata version v09) [--bitmap-block-size=val] [--peers=node-id-list] [--bitmap-slots=slot-list] [--initial-current-uuid={generate|<hex>}] [--consistent] [--uptodate] [--peers-outdated] [--rotate-uuids]

Initialize the metadata. This is necessary before a DRBD resource can be attached. If drbdmeta finds an older version of DRBD metadata on the device, it asks if the format should be converted.

When drbdadm calls drbdmeta's create-md command for a device, it sets the number-of-bitmap-slots argument to the number of peers in the resource. To reserve additional bitmap slots (which allows to add more peers in the future), call drbdmeta directly instead.

When a device is used before being connected to its peers the first time, DRBD assumes that peers can only handle 4 KiB requests by default. The --peer-max-bio-size option allows to set more optimistic values; use this if the versions of DRBD that this device will connect to are known. DRBD supports a maximum bio size of 32 KiB since version 8.3.8, of 128 KiB since version 8.3.9, and of 1 MiB since version 8.4.0.

By default, we explicitly initialize the bitmap area to all zero. With --initialize-bitmap you can chose to only try the fast method (zeroout, try ioctl BLKZEROOUT only), only use explicit pwrite calls, or skip this bitmap initialization phase completely. If you intend to do an initial full sync anyways, you can use skip to leave the bitmap initialization to the kernel. This can make a noticable difference when initializing huge volumes. Default is automatic, which is zeroout with an implicit fallback to pwrite. set-all marks the whole device out of sync instead.

When create-md converts existing metadata, --initialize-bitmap also overrides what the conversion would do to the bitmap on its own: set-all asks for a full resync, the zeroing variants declare the device in sync with the peer that keeps its bitmap slot, and skip leaves the bitmap area as it is on disk. Use skip with care: if the conversion changes the bitmap geometry, the bitmap area then holds bits that describe other blocks than DRBD will read them as.

If you want to use more than 6433 activity log extents, or live on top of a spriped RAID, you may specify the number of stripes (--al-stripes, default 1), and the stripe size (--al-stripe-size-kB, default 32). To just use a larger linear on-disk ring-buffer, leave the number of stripes at 1, and increase the size only:

drbdmeta 0 v08 /dev/vg23/lv42 internal create-md --al-stripe-size 1M

To avoid a single "spindle" from becoming a bottleneck, increase the number of stripes, to achieve an interleaved layout of the on-disk activity-log transactions. What you give as "stripe-size" should be what is a.k.a. "chunk size" or "granularity" or "strip unit": the minimum skip to the next "spindle".

drbdmeta 0 v08 /dev/vg23/lv42 internal create-md --al-stripes 7 --al-stripe-size 64

When create-md converts existing metadata, the two options change the activity log geometry of that metadata as well. The activity log sits next to the bitmap, so another geometry moves the bitmap: the activity log is re-created, and the bitmap with it. Where the bitmap being replaced tracks out-of-sync blocks, that costs a resync of the whole device, and create-md asks first. Without the two options the conversion keeps the geometry the metadata carries, and the out-of-sync blocks with it. Unapplied activity log transactions are lost either way.

A numeric index names a slot of fixed size and fixed offsets, which holds one 32 KiB activity log stripe and 4096 bytes per bitmap bit, and nothing else; DRBD refuses to attach anything else. Both --al-stripes / --al-stripe-size-kB and --bitmap-block-size therefore need a metadata device of their own, or one of the flexible layouts, which size themselves after the device.

With the --bitmap-block-size option you can change the change-tracking bitmap's granularity from 4KiB to another value. Allowed values are: 4k, 8k, 16k, 32k, 64k, 128k, 256k, 512k, 1M. The benefit of a larger block size is a reduced bitmap size and a smaller memory footprint for the DRBD volume. The disadvantage is a higher resync volume if the workload writes to the DRBD devices at a finer granularity, since drbd rounds each modified block up to a complete bitmap block for change tracking, and later resync.

Note that DRBD 8.4 and older only know the 4KiB granularity. Converting such metadata to v08 therefore re-creates the bitmap with 4KiB per bit, which needs more bits, and with internal metadata thus more space, for the same device. If the last agreed device size leaves no room for that, the conversion is refused: shrink the file system and the DRBD device first, while still running DRBD 9.

DRBD 8.4 and older also have a single bitmap slot, so converting metadata that has more than one drops all but one of them, and re-creates the bitmap. Which slot survives has to be clear: --peers names it by the node id of the peer that owns it, --bitmap-slots by slot number. Use the slot number for a slot that no peer owns any more, which is what forget-peer leaves behind. Both options take a comma separated list, or a mask in hexadecimal, so that they can also express a conversion that keeps more than one slot; naming more than the conversion keeps is refused, and a conversion to v08 keeps one. If nothing is named, and more than one slot is in use, the conversion is refused and lists the candidates; a single slot in use is unambiguous and is kept. The peer that owns the kept slot provides the bitmap UUID and the peer flags of the v08 metadata. If that slot tracks anything out of sync, the whole device has to be marked out of sync, and drbdmeta asks before it does: a few out-of-sync blocks would become a resync of the whole device, so bringing the device in sync while still running DRBD 9 is usually the better order. Only the slot that is kept decides the state of the new bitmap: if it tracks nothing out of sync, the new bitmap is all clear, even when a dropped slot did track something. If it does, the whole device is marked out of sync, because a bit of the old bitmap does not describe the same blocks as a bit of the new one.

Both of these re-create the bitmap, because the metadata is converted where it lies and the bitmap area moves. The convert-dump command converts a dump instead. A dump has one bitmap section per slot, rather than the interleaved on-disk layout, so the out-of-sync state survives. Use it where a full resync after the conversion is worth avoiding.

In the other direction, v08 metadata knows one peer, and no node ids: neither the one of that peer, nor its own. What it says about that peer -- the peer flags and the bitmap UUID -- goes to every peer entry, since none of them has seen our data. The one bitmap slot stays unassigned; DRBD 9 hands it to the peer that connects first.

The following options are available for metadata version v09 only. They are intended for use when pre-provisioning metadata without an initial full sync.

--initial-current-uuid={generate|<hex>} sets the initial current_uuid instead of leaving it at the sentinel value UUID_JUST_CREATED. Use generate to let drbdmeta pick a random value, or supply an explicit 64-bit hex value. Bit 0 is silently cleared (it is reserved by the DRBD module); the resulting value must not be zero.

--consistent sets the MDF_CONSISTENT flag on the local device, indicating that its data is consistent. If no --initial-current-uuid is given, a random UUID is generated automatically.

--uptodate sets both MDF_CONSISTENT and MDF_WAS_UP_TO_DATE on the local device, marking it as the authoritative copy. If no --initial-current-uuid is given, a random UUID is generated automatically.

--peers-outdated sets MDF_PEER_OUTDATED on every peer slot.

--rotate-uuids performs a UUID rotation: the current UUID (which will be generated if not provided using --initial-current-uuid) is written into every peer's bitmap_uuid slot and into history_uuids[0], and a fresh random value becomes the new current_uuid.

get-gi [--node-id=id]

Show the data generation identifiers for a device on a particular connection. DRBD version 9.0.0 and beyond support multiple peers; use the node-id option to define which peer's data generation identifiers to show.

show-gi [--node-id=id]

Similar to get-gi, but with explanatory information.

dump-md

Dump the metadata of a device in text form, including the bitmap and activity log.

outdate

Mark the data on a lower-level device as outdated. See drbdsetup(8) for details.

dstate

Show the current disk state of a lower-level device.

check-resize

Examine the device size of a lower-level device and its last known device size (saved in /var/lib/drbd/drbd-minor-minor.lkbd by drbdsetup check-resize). For internal metadata, if the size of the lower-level device has changed and the metadata can be found at the previous position, move the metadata to the new position at the end of the block device.

apply-al

Apply the activity log of the specified device. This is necessary before the device can be attached by the kernel again.

repair-md [--tentative]

Repair known problems in metadata. Only problems that can be automatically fixed in a safe manner will be repaired. When --tentative is set, the metadata will only be checked, it will not be changed.

EXPERT COMMANDS

The drbdmeta utility can be used to fine tune metdata. Please note that this can lead to destroyed metadata or even silent data corruption; use with great care only.

set-gi gi [--node-id=id]

Set the generation identifiers. The gi argument is a generation counter for the v06 and v07 formats, and a set of UUIDs for v08 and beyond. Accepts the same syntax as in the get-gi output. DRBD version 9.0.0 and beyond support multiple peers; use the --node-id option to define which peer's data generation identifiers to set.

restore-md dump_file [--max-peers=val] [--peers=node-id-list]

Replace the metadata on the device with the contents of dump_file. The dump file format is defined by the output of the dump-md command. The metadata is laid out for what the dump says, including a bm-byte-per-bit other than 4096 and an activity log geometry other than the default.

--max-peers writes a different number of bitmap slots than the dump has, keeping the data generation identifiers that create-md would discard. The dump holds one bitmap[N] section per slot, so the slots are re-numbered rather than re-interleaved.

Fewer slots than the dump has drop the ones that no peer owns first. If the peers still own more slots than remain, the choice is refused and the candidates are listed; --peers then names the peers whose slots to keep. A kept slot takes the lowest free number, and its peer entry follows it.

More slots than the dump has take their content from a slot that no peer owns. Those all track from day0, so any of them will do; if the dump has none, the added slots come out all set, which is the state forget-peer leaves behind. A peer entry that the dump points at one of the added slots gets the day0 bitmap UUID, the one shared by the peers that have not seen our data. With internal metadata the added slots take their space from the data area; if the last agreed device size no longer fits, the restore is refused.

The same holds for a dump whose bm-byte-per-bit differs from the metadata in place: the offsets follow the bitmap the dump asks for, and a finer bitmap that no longer leaves room for the last agreed device size is refused.

convert-dump dump_file [--peers=node-id-list] [--bitmap-slots=slot-list] [--bitmap-block-size=val]

Read dump_file and write a dump of the metadata version named on the command line to standard output. A dump names its own version, so the dump is the source and the command line names the target. The device is not touched; feed the result to restore-md to write it.

drbdmeta - v08 /dev/vg23/lv42 internal convert-dump v09.txt --peers 2 > v08.txt

drbdmeta - v08 /dev/vg23/lv42 internal restore-md v08.txt

v09 to v08 is the rollback to DRBD 8.4. create-md converts the metadata where it lies and has to re-create the bitmap; convert-dump re-writes the bitmap section of the slot that survives, so the out-of-sync state of that peer reaches the v08 metadata instead of a full resync. --peers and --bitmap-slots name the slot to keep, as they do for the conversion in place, and drbdmeta reports the out-of-sync bits it drops with the other slots. Where no slot survives at all, there is nobody to attribute those bits to, and the conversion is refused.

Either version written as itself changes nothing, except that --bitmap-block-size gives a v09 dump a different number of bytes per bitmap bit. A target bit is set if any source bit covering the same byte range is set. A finer bitmap repeats every source bit and loses nothing; a coarser one or-s them together, which can only add out-of-sync, never drop it.

Other directions are refused. Converting v08 metadata to v09 is what create-md does, and it keeps the bitmap as it is.

VERSION

This document was revised for version 9.0.0 of the DRBD distribution.

AUTHOR

Written by Philipp Reisner <philipp.reisner@linbit.com> and Lars Ellenberg <lars.ellenberg@linbit.com>.

REPORTING BUGS

Report bugs at https://github.com/LINBIT/drbd-utils/issues.

COPYRIGHT

Copyright 2001-2008,2012 LINBIT Information Technologies, Philipp Reisner, Lars Ellenberg. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

SEE ALSO

drbdadm(8) drbd.conf(5)

6 December 2012 DRBD 9.0.0