mcelog

package: mcelog

title: mcelog(8)

text:
# mcelog(8)

## NAME
mcelog - Decode kernel machine check log on x86 machines

## SYNOPSIS
mcelog [options] [device]

mcelog [options] --daemon

mcelog [options] --client

mcelog [options] --ascii

mcelog [options] --is-cpu-supported

mcelog --ping

mcelog --version

## DESCRIPTION
X86 CPUs report errors detected by the CPU as machine check events (MCEs). These can be data corruption detected in the CPU caches, in main memory by an integrated memory controller, data transfer errors on the front side bus or CPU interconnect or other internal errors. Possible causes can be cosmic radiation, instable power supplies, cooling problems, broken hardware, running systems out of specification, or bad luck.

Most errors can be corrected by the CPU by internal error correction mechanisms. Uncorrected errors cause machine check exceptions which may kill processes or panic the machine. A small number of corrected errors is usually not a cause for worry, but a large number can indicate future failure.

When a corrected or recovered error happens, the x86 kernel writes a record describing the MCE into a internal ring buffer available through the /dev/mcelog device. mcelog retrieves errors from /dev/mcelog, decodes them into a human readable format and prints them on the standard output or optionally into the system log.

Optionally it can also take more options like keeping statistics or triggering shell scripts on specific events. By default mcelog supports offlining memory pages with persistent corrected errors, offlining CPU cores if they developed cache problems, and otherwise logging specific events to the system log after they crossed a threshold.

The normal operating modes for mcelog are: running as a regular cron job (traditional way, deprecated), running as a trigger directly executed by the kernel, or running as a daemon with the --daemon option.

When an uncorrected machine check error happens that the kernel cannot recover from then it will usually panic the system. In this case when there was a warm reset after the panic mcelog should pick up the machine check errors after reboot. This is not possible after a cold reset.

In addition mcelog can be used on the command line to decode the kernel output for a fatal machine check panic in text format using the --ascii option. This is typically used to decode the panic console output of a fatal machine check, if the system was power cycled or mcelog didn't run immediately after reboot.

When the panic triggers a kdump kexec crash kernel the crash kernel boot up script should log the machine checks to disk, otherwise they might be lost.

Note that after mcelog retrieves an error the kernel doesn't store it anymore (different from dmesg(1)), so the output should be always saved somewhere and mcelog not run in uncontrolled ways.

When invoked with the --is-cpu-supported option mcelog exits with code 0 if the current CPU is supported, 1 otherwise.

## OPTIONS
When the --syslog option is specified redirect output to system log. The --syslog-error option causes the normal machine checks to be logged as LOG_ERR (implies --syslog ). Normally only fatal errors or high level remarks are logged with error level. High level one line summaries of specific errors are also logged to the syslog by default unless mcelog operates in --ascii mode.

When the --logfile=file option is specified append log output to the specified file. With the --no-syslog option mcelog will never log anything to the syslog.

When the --cpu=cputype option is specified set the to be decoded CPU to cputype. See mcelog --help for a list of valid CPUs. Note that specifying an incorrect CPU can lead to incorrect decoding output. Default is either the CPU of the machine that reported the machine check (needs a newer kernel version) or the CPU of the machine mcelog is running on, so normally this option doesn't have to be used. Older versions of mcelog had separate options for different CPU types. These are still implemented, but deprecated and undocumented now.

With the --dmi option mcelog will look up the DIMMs reported in machine checks in the SMBIOS/DMI tables of the BIOS and map the DIMMs to board identifiers. This only works when the BIOS reports the identifiers correctly. Unfortunately often the information reported by the BIOS is either subtly or obviously wrong or useless. This option requires that mcelog has read access to /dev/mem (normally requires root) and runs on the same machine in the same hardware configuration as when the machine check event happened.

When --ignorenodev is specified then mcelog will exit silently when the device cannot be opened. This is useful in virtualized environment with limited devices.

When --filter is specified mcelog will filter out known broken machine check events (default on). When the --no-filter option is specified mcelog does not filter events.

When --raw is specified mcelog will not decode, but just dump the mcelog in a raw hex format. This can be useful for automatic post processing.

When a device is specified the machine check logs are read from device instead of the default /dev/mcelog.

With the --ascii option mcelog decodes a fatal machine check panic generated by the kernel ("CPU n: Machine Check Exception ...") in ASCII from standard input and exits afterwards. Note that when the panic comes from a different machine than where mcelog is running on you might need to specify the correct cputype on older kernels. On newer kernels which output the PROCESSOR field this is not needed anymore.

When the --file filename option is specified mcelog --ascii will read the ASCII machine check record from input file filename instead of standard input.

With the --config-file file option mcelog reads the specified config file. Default is /etc/mcelog/mcelog.conf See also CONFIG FILE below.

With the --daemon option mcelog will run in the background. This gives the fastest reaction time and is the recommended operating mode. If an output option isn't selected ( --logfile or --syslog or --syslog-error ), this option implies --logfile=/var/log/mcelog. Important messages will be logged as one-liner summaries to syslog unless --no-syslog is given. The option --foreground will prevent mcelog from giving up the terminal in daemon mode. This is intended for debugging.

With the --client option mcelog will query a running daemon for accumulated errors.

With the --cpumhz=mhz option assume the CPU has mhz frequency for decoding the time of the event using the CPU time stamp counter. This also forces decoding. Note this can be unreliable. on some systems with CPU frequency scaling or deep C states, where the CPU time stamp counter does not increase linearly. By default the frequency of the current CPU is used when mcelog determines it is safe to use. Newer kernels report the time directly in the event and don't need this anymore.

The --pidfile file option writes the process id of the daemon into file file. Only valid in daemon mode.

Mcelog will enable extended error reporting from the memory controller on processors that support it unless you tell it not to with the --no-imc-log option. You might need this option when decoding old logs from a system where this mode was not enabled.

With the --binary option mcelog reads from a regular file given and first argument instead of from /dev/mcelog. Useful for decoding errors saved in binary format to the pstore file system.

Users can utilize the --ping option to check the availability of the mcelog server. If the mcelog server is running, the command returns the string pong followed by a newline character.

--version displays the version of mcelog and exits.

## CONFIG FILE
mcelog supports a config file to set defaults. Command line options override the config file. By default the config file is read from /etc/mcelog/mcelog.conf unless overridden with the --config-file option.

The general format is optionname = value White space is not allowed in value currently, except at the end where it is dropped Comments start with #.

All command line options that are not commands can be specified in the config file. For example t to enable the --no-syslog option use no-syslog = yes (or no to disable). When the option has a argument use logfile = /tmp/logfile

For more information on the config file please see mcelog.conf(5).

## NOTES
The kernel prefers old messages over new. If the log buffer overflows only old ones will be kept.

The exact output in the log file depends on the CPU, unless the --raw option is used.

mcelog will report serious errors to the syslog during decoding.

## SIGNALS
When mcelog runs in daemon mode and receives a SIGUSR1 it will close and reopen the log files. This can be used to rotate logs without restarting the daemon.

## FILES
/dev/mcelog (char 10, minor 227)

/etc/mcelog/mcelog.conf

/var/log/mcelog

/var/run/mcelog.pid

## SEE ALSO
mcelog.conf(5), mcelog.triggers(5)

http://www.mcelog.org

AMD x86-64 architecture programmer's manual, Volume 2, System programming

Intel 64 and IA32 Architectures Software Developer's manual, Volume 3, System programming guide Chapter 15 and 16. http://www.intel.com/sdm

Datasheet of your CPU.


mdmon

package: mdadm

title: mdmon(8)

summary: monitor MD external metadata arrays

text:
# mdmon(8)

## NAME
mdmon - monitor MD external metadata arrays

## SYNOPSIS
mdmon [--all] [--takeover] [--foreground] CONTAINER

## OVERVIEW
The 2.6.27 kernel brings the ability to support external metadata arrays. External metadata implies that user space handles all updates to the metadata. The kernel's responsibility is to notify user space when a "metadata event" occurs, like disk failures and clean-to-dirty transitions. The kernel, in important cases, waits for user space to take action on these notifications.

## DESCRIPTION

### Metadata updates:
To service metadata update requests a daemon, mdmon, is introduced. Mdmon is tasked with polling the sysfs namespace looking for changes in array_state, sync_action, and per disk state attributes. When a change is detected it calls a per metadata type handler to make modifications to the metadata. The following actions are taken:

array_state - inactive

Clear the dirty bit for the volume and let the array be stopped

array_state - write pending

Set the dirty bit for the array and then set array_state to active. Writes are blocked until userspace writes active.

array_state - active-idle

The safe mode timer has expired so set array state to clean to block writes to the array

array_state - clean

Clear the dirty bit for the volume

array_state - read-only

This is the initial state that all arrays start at. mdmon takes one of the three actions:

1/

Transition the array to read-auto keeping the dirty bit clear if the metadata handler determines that the array does not need resyncing or other modification

2/

Transition the array to active if the metadata handler determines a resync or some other manipulation is necessary

3/

Leave the array read-only if the volume is marked to not be monitored; for example, the metadata version has been set to "external:-dev/md127" instead of "external:/dev/md127"

sync_action - resync-to-idle

Notify the metadata handler that a resync may have completed. If a resync process is idled before it completes this event allows the metadata handler to checkpoint resync.

sync_action - recover-to-idle

A spare may have completed rebuilding so tell the metadata handler about the state of each disk. This is the metadata handler's opportunity to clear any "out-of-sync" bits and clear the volume's degraded status. If a recovery process is idled before it completes this event allows the metadata handler to checkpoint recovery.

<disk>/state - faulty

A disk failure kicks off a series of events. First, notify the metadata handler that a disk has failed, and then notify the kernel that it can unblock writes that were dependent on this disk. After unblocking the kernel this disk is set to be removed+ from the member array. Finally the disk is marked failed in all other member arrays in the container.

+ Note This behavior differs slightly from native MD arrays where removal is reserved for a mdadm --remove event. In the external metadata case the container holds the final reference on a block device and a mdadm --remove <container> <victim> call is still required.

### Containers:
External metadata formats, like DDF, differ from the native MD metadata formats in that they define a set of disks and a series of sub-arrays within those disks. MD metadata in comparison defines a 1:1 relationship between a set of block devices and a RAID array. For example to create 2 arrays at different RAID levels on a single set of disks, MD metadata requires the disks be partitioned and then each array can be created with a subset of those partitions. The supported external formats perform this disk carving internally.

Container devices simply hold references to all member disks and allow tools like mdmon to determine which active arrays belong to which container. Some array management commands like disk removal and disk add are now only valid at the container level. Attempts to perform these actions on member arrays are blocked with error messages like:

"mdadm: Cannot remove disks from a ´member´ array, perform this operation on the parent container"

Containers are identified in /proc/mdstat with a metadata version string "external:<metadata name>". Member devices are identified by "external:/<container device>/<member index>", or "external:-<container device>/<member index>" if the array is to remain readonly.

## OPTIONS
CONTAINER

The container device to monitor. It can be a full path like /dev/md/container, or a simple md device name like md127.

--foreground

Normally, mdmon will fork and continue in the background. Adding this option will skip that step and run mdmon in the foreground.

--takeover

This instructs mdmon to replace any active mdmon which is currently monitoring the array. This is primarily used late in the boot process to replace any mdmon which was started from an initramfs before the root filesystem was mounted. This avoids holding a reference on that initramfs indefinitely and ensures that the pid and sock files used to communicate with mdmon are in a standard place.

--all

This tells mdmon to find any active containers and start monitoring each of them if appropriate. This is normally used with --takeover late in the boot sequence. A separate mdmon process is started for each container as the --all argument is over-written with the name of the container. To allow for containers with names longer than 5 characters, this argument can be arbitrarily extended, e.g. to --all-active-arrays.

Note that mdmon is automatically started by mdadm when needed and so does not need to be considered when working with RAID arrays. The only times it is run other than by mdadm is when the boot scripts need to restart it after mounting the new root filesystem.

## START UP AND SHUTDOWN
As mdmon needs to be running whenever any filesystem on the monitored device is mounted there are special considerations when the root filesystem is mounted from an mdmon monitored device. Note that in general mdmon is needed even if the filesystem is mounted read-only as some filesystems can still write to the device in those circumstances, for example to replay a journal after an unclean shutdown.

When the array is assembled by the initramfs code, mdadm will automatically start mdmon as required. This means that mdmon must be installed on the initramfs and there must be a writable filesystem (typically tmpfs) in which mdmon can create a .pid and .sock file. The particular filesystem to use is given to mdmon at compile time and defaults to /run/mdadm.

This filesystem must persist through to shutdown time.

After the final root filesystem has be instantiated (usually with pivot_root) mdmon should be run with --all --takeover so that the mdmon running from the initramfs can be replaced with one running in the main root, and so the memory used by the initramfs can be released.

At shutdown time, mdmon should not be killed along with other processes. Also as it holds a file (socket actually) open in /dev (by default) it will not be possible to unmount /dev if it is a separate filesystem.

## EXAMPLES
mdmon --all-active-arrays --takeover

Any mdmon which is currently running is killed and a new instance is started. This should be run during in the boot sequence if an initramfs was used, so that any mdmon running from the initramfs will not hold the initramfs active.

## SEE ALSO
mdadm(8), md(4).

branch: sisyphus


title: raid6check(8)

summary: check MD RAID6 device for errors

text:
# raid6check(8)

## NAME
raid6check - check MD RAID6 device for errors aka Linux Software RAID

## SYNOPSIS
raid6check <raid6_device> <start_stripe> <number_of_stripes> [autorepair]

raid6check <raid6_device> repair <stripe> <failed_slot_1> <failed_slot_2>

## DESCRIPTION
RAID6 devices in which one single component drive has errors can use the double parity in order to find out which component drive. The "raid6check" tool has two modes: check mode and repair mode.

"raid6check" requires a non-degraded RAID6 MD device as first parameter. If the given MD device is not a RAID6, "raid6check" will, of course, not continue.

If the RAID6 MD device is degraded, "raid6check" will report an error and it will not proceed further.

### Check mode
In the check mode, the "raid6check" tool checks, for each stripe, the double parity consistency, reports mismatches and, if possible, which component drive has the mismatch. Since it works at stripe level, it can report different drives with mismatches at different stripes.

In addition to the MD device, "raid6check" requries a starting stripe (usually 0) and the number of stripes to be checked. If this third parameter is also 0, it will check the array up to the end. If the fourth parameter is "autorepair", it will overwrite single-slot errors.

"raid6check" will start printing information about the RAID6, then for each stripe, it will report the parity rotation status. In case of parity mismatches, "raid6check" reports, if possible, which component drive could be responsible. Otherwise it reports that it is not possible to find the component drive.

No write operations are performed on the array or the components, unless "autorepair" is specified. Furthermore, the checked array can be online and in use during the operation of "raid6check".

### Repair mode
In the repair mode, the "raid6check" tool checks the given stripe. If inconsistencies are found, it attempts to repair the strip assuming that the two given slots are incorrect.

-1 may be used to specify parity P and -2 parity Q.

## EXAMPLES
raid6check /dev/md0 0 0

This will check /dev/md0 from start to end.

raid6check /dev/md3 0 1 autorepair

This will check the first stripe of /dev/md3. Any error will be reported, and if single-slot, repaired.

raid6check /dev/md3 0 repair 1 -1 -2

This will check the first stripe of /dev/md3. If any inconsistencies are found, repair is performed assuming the two parities are incorrect.

raid6check /dev/md1 1000 0

This will check /dev/md1 from stripe 1000 up to the end.

raid6check /dev/m127 128 256

This will check 256 stripes of /dev/md127 starting from stripe 128.

raid6check /dev/md0 0 0 | grep -i error > md0_err.log

This will check /dev/md0 completely and create a log file only with errors, if any.

## FILES
"raid6check" uses directly the component drives as found in /dev. Furthermore, the sysfs interface is needed in order to find out the RAID6 parameters.

## BUGS
Negative stripe parameters can lead to unexpected results due to strtoull. (Negative slot numbers should work for Q [-1] and P [-2] but currently involve compiler-defined behavior.)

It is not clear what will happen if the RAID6 MD device gets degraded during the check.

The latest version of raid6check should always be available from

https://www.kernel.org/pub/linux/utils/raid/mdadm/

Related man pages:

mdadm(8) mdmon(8), mdadm.conf(5), md(4).