cpuset

title: cpuset(7)

text:
# cpuset(7)

## NAME
cpuset - confine processes to processor and memory node subsets

## DESCRIPTION
The cpuset filesystem is a pseudo-filesystem interface to the kernel cpuset mechanism, which is used to control the processor placement and memory placement of processes. It is commonly mounted at /dev/cpuset.

On systems with kernels compiled with built in support for cpusets, all processes are attached to a cpuset, and cpusets are always present. If a system supports cpusets, then it will have the entry nodev cpuset in the file /proc/filesystems. By mounting the cpuset filesystem (see the EXAMPLES section below), the administrator can configure the cpusets on a system to control the processor and memory placement of processes on that system. By default, if the cpuset configuration on a system is not modified or if the cpuset filesystem is not even mounted, then the cpuset mechanism, though present, has no effect on the system's behavior.

A cpuset defines a list of CPUs and memory nodes.

The CPUs of a system include all the logical processing units on which a process can execute, including, if present, multiple processor cores within a package and Hyper-Threads within a processor core. Memory nodes include all distinct banks of main memory; small and SMP systems typically have just one memory node that contains all the system's main memory, while NUMA (non-uniform memory access) systems have multiple memory nodes.

Cpusets are represented as directories in a hierarchical pseudo-filesystem, where the top directory in the hierarchy (/dev/cpuset) represents the entire system (all online CPUs and memory nodes) and any cpuset that is the child (descendant) of another parent cpuset contains a subset of that parent's CPUs and memory nodes. The directories and files representing cpusets have normal filesystem permissions.

Every process in the system belongs to exactly one cpuset. A process is confined to run only on the CPUs in the cpuset it belongs to, and to allocate memory only on the memory nodes in that cpuset. When a process fork(2)s, the child process is placed in the same cpuset as its parent. With sufficient privilege, a process may be moved from one cpuset to another and the allowed CPUs and memory nodes of an existing cpuset may be changed.

When the system begins booting, a single cpuset is defined that includes all CPUs and memory nodes on the system, and all processes are in that cpuset. During the boot process, or later during normal system operation, other cpusets may be created, as subdirectories of this top cpuset, under the control of the system administrator, and processes may be placed in these other cpusets.

Cpusets are integrated with the sched_setaffinity(2) scheduling affinity mechanism and the mbind(2) and set_mempolicy(2) memory-placement mechanisms in the kernel. Neither of these mechanisms let a process make use of a CPU or memory node that is not allowed by that process's cpuset. If changes to a process's cpuset placement conflict with these other mechanisms, then cpuset placement is enforced even if it means overriding these other mechanisms. The kernel accomplishes this overriding by silently restricting the CPUs and memory nodes requested by these other mechanisms to those allowed by the invoking process's cpuset. This can result in these other calls returning an error, if for example, such a call ends up requesting an empty set of CPUs or memory nodes, after that request is restricted to the invoking process's cpuset.

Typically, a cpuset is used to manage the CPU and memory-node confinement for a set of cooperating processes such as a batch scheduler job, and these other mechanisms are used to manage the placement of individual processes or memory regions within that set or job.

## FILES
Each directory below /dev/cpuset represents a cpuset and contains a fixed set of pseudo-files describing the state of that cpuset.

New cpusets are created using the mkdir(2) system call or the mkdir(1) command. The properties of a cpuset, such as its flags, allowed CPUs and memory nodes, and attached processes, are queried and modified by reading or writing to the appropriate file in that cpuset's directory, as listed below.

The pseudo-files in each cpuset directory are automatically created when the cpuset is created, as a result of the mkdir(2) invocation. It is not possible to directly add or remove these pseudo-files.

A cpuset directory that contains no child cpuset directories, and has no attached processes, can be removed using rmdir(2) or rmdir(1). It is not necessary, or possible, to remove the pseudo-files inside the directory before removing it.

The pseudo-files in each cpuset directory are small text files that may be read and written using traditional shell utilities such as cat(1), and echo(1), or from a program by using file I/O library functions or system calls, such as open(2), read(2), write(2), and close(2).

The pseudo-files in a cpuset directory represent internal kernel state and do not have any persistent image on disk. Each of these per-cpuset files is listed and described below.

tasks

List of the process IDs (PIDs) of the processes in that cpuset. The list is formatted as a series of ASCII decimal numbers, each followed by a newline. A process may be added to a cpuset (automatically removing it from the cpuset that previously contained it) by writing its PID to that cpuset's tasks file (with or without a trailing newline).

Warning: only one PID may be written to the tasks file at a time. If a string is written that contains more than one PID, only the first one will be used.

notify_on_release

Flag (0 or 1). If set (1), that cpuset will receive special handling after it is released, that is, after all processes cease using it (i.e., terminate or are moved to a different cpuset) and all child cpuset directories have been removed. See the Notify On Release section, below.

cpuset.cpus

List of the physical numbers of the CPUs on which processes in that cpuset are allowed to execute. See List Format below for a description of the format of cpus.

The CPUs allowed to a cpuset may be changed by writing a new list to its cpus file.

cpuset.cpu_exclusive

Flag (0 or 1). If set (1), the cpuset has exclusive use of its CPUs (no sibling or cousin cpuset may overlap CPUs). By default, this is off (0). Newly created cpusets also initially default this to off (0).

Two cpusets are sibling cpusets if they share the same parent cpuset in the /dev/cpuset hierarchy. Two cpusets are cousin cpusets if neither is the ancestor of the other. Regardless of the cpu_exclusive setting, if one cpuset is the ancestor of another, and if both of these cpusets have nonempty cpus, then their cpus must overlap, because the cpus of any cpuset are always a subset of the cpus of its parent cpuset.

cpuset.mems

List of memory nodes on which processes in this cpuset are allowed to allocate memory. See List Format below for a description of the format of mems.

cpuset.mem_exclusive

Flag (0 or 1). If set (1), the cpuset has exclusive use of its memory nodes (no sibling or cousin may overlap). Also if set (1), the cpuset is a Hardwall cpuset (see below). By default, this is off (0). Newly created cpusets also initially default this to off (0).

Regardless of the mem_exclusive setting, if one cpuset is the ancestor of another, then their memory nodes must overlap, because the memory nodes of any cpuset are always a subset of the memory nodes of that cpuset's parent cpuset.

cpuset.mem_hardwall (since Linux 2.6.26)

Flag (0 or 1). If set (1), the cpuset is a Hardwall cpuset (see below). Unlike mem_exclusive, there is no constraint on whether cpusets marked mem_hardwall may have overlapping memory nodes with sibling or cousin cpusets. By default, this is off (0). Newly created cpusets also initially default this to off (0).

cpuset.memory_migrate (since Linux 2.6.16)

Flag (0 or 1). If set (1), then memory migration is enabled. By default, this is off (0). See the Memory Migration section, below.

cpuset.memory_pressure (since Linux 2.6.16)

A measure of how much memory pressure the processes in this cpuset are causing. See the Memory Pressure section, below. Unless memory_pressure_enabled is enabled, always has value zero (0). This file is read-only. See the WARNINGS section, below.

cpuset.memory_pressure_enabled (since Linux 2.6.16)

Flag (0 or 1). This file is present only in the root cpuset, normally /dev/cpuset. If set (1), the memory_pressure calculations are enabled for all cpusets in the system. By default, this is off (0). See the Memory Pressure section, below.

cpuset.memory_spread_page (since Linux 2.6.17)

Flag (0 or 1). If set (1), pages in the kernel page cache (filesystem buffers) are uniformly spread across the cpuset. By default, this is off (0) in the top cpuset, and inherited from the parent cpuset in newly created cpusets. See the Memory Spread section, below.

cpuset.memory_spread_slab (since Linux 2.6.17)

Flag (0 or 1). If set (1), the kernel slab caches for file I/O (directory and inode structures) are uniformly spread across the cpuset. By default, is off (0) in the top cpuset, and inherited from the parent cpuset in newly created cpusets. See the Memory Spread section, below.

cpuset.sched_load_balance (since Linux 2.6.24)

Flag (0 or 1). If set (1, the default) the kernel will automatically load balance processes in that cpuset over the allowed CPUs in that cpuset. If cleared (0) the kernel will avoid load balancing processes in this cpuset, unless some other cpuset with overlapping CPUs has its sched_load_balance flag set. See Scheduler Load Balancing, below, for further details.

cpuset.sched_relax_domain_level (since Linux 2.6.26)

Integer, between -1 and a small positive value. The sched_relax_domain_level controls the width of the range of CPUs over which the kernel scheduler performs immediate rebalancing of runnable tasks across CPUs. If sched_load_balance is disabled, then the setting of sched_relax_domain_level does not matter, as no such load balancing is done. If sched_load_balance is enabled, then the higher the value of the sched_relax_domain_level, the wider the range of CPUs over which immediate load balancing is attempted. See Scheduler Relax Domain Level, below, for further details.

In addition to the above pseudo-files in each directory below /dev/cpuset, each process has a pseudo-file, /proc/pid/cpuset, that displays the path of the process's cpuset directory relative to the root of the cpuset filesystem.

Also the /proc/pid/status file for each process has four added lines, displaying the process's Cpus_allowed (on which CPUs it may be scheduled) and Mems_allowed (on which memory nodes it may obtain memory), in the two formats Mask Format and List Format (see below) as shown in the following example:

```
Cpus_allowed:   ffffffff,ffffffff,ffffffff,ffffffff
Cpus_allowed_list:     0-127
Mems_allowed:   ffffffff,ffffffff
Mems_allowed_list:     0-63
```

The "allowed" fields were added in Linux 2.6.24; the "allowed_list" fields were added in Linux 2.6.26.

## EXTENDED CAPABILITIES
In addition to controlling which cpus and mems a process is allowed to use, cpusets provide the following extended capabilities.

### Exclusive cpusets
If a cpuset is marked cpu_exclusive or mem_exclusive, no other cpuset, other than a direct ancestor or descendant, may share any of the same CPUs or memory nodes.

A cpuset that is mem_exclusive restricts kernel allocations for buffer cache pages and other internal kernel data pages commonly shared by the kernel across multiple users. All cpusets, whether mem_exclusive or not, restrict allocations of memory for user space. This enables configuring a system so that several independent jobs can share common kernel data, while isolating each job's user allocation in its own cpuset. To do this, construct a large mem_exclusive cpuset to hold all the jobs, and construct child, non-mem_exclusive cpusets for each individual job. Only a small amount of kernel memory, such as requests from interrupt handlers, is allowed to be placed on memory nodes outside even a mem_exclusive cpuset.

### Hardwall
A cpuset that has mem_exclusive or mem_hardwall set is a hardwall cpuset. A hardwall cpuset restricts kernel allocations for page, buffer, and other data commonly shared by the kernel across multiple users. All cpusets, whether hardwall or not, restrict allocations of memory for user space.

This enables configuring a system so that several independent jobs can share common kernel data, such as filesystem pages, while isolating each job's user allocation in its own cpuset. To do this, construct a large hardwall cpuset to hold all the jobs, and construct child cpusets for each individual job which are not hardwall cpusets.

Only a small amount of kernel memory, such as requests from interrupt handlers, is allowed to be taken outside even a hardwall cpuset.

### Notify on release
If the notify_on_release flag is enabled (1) in a cpuset, then whenever the last process in the cpuset leaves (exits or attaches to some other cpuset) and the last child cpuset of that cpuset is removed, the kernel will run the command /sbin/cpuset_release_agent, supplying the pathname (relative to the mount point of the cpuset filesystem) of the abandoned cpuset. This enables automatic removal of abandoned cpusets.

The default value of notify_on_release in the root cpuset at system boot is disabled (0). The default value of other cpusets at creation is the current value of their parent's notify_on_release setting.

The command /sbin/cpuset_release_agent is invoked, with the name (/dev/cpuset relative path) of the to-be-released cpuset in argv[1].

The usual contents of the command /sbin/cpuset_release_agent is simply the shell script:

```
#!/bin/sh
rmdir /dev/cpuset/$1
```

As with other flag values below, this flag can be changed by writing an ASCII number 0 or 1 (with optional trailing newline) into the file, to clear or set the flag, respectively.

### Memory pressure
The memory_pressure of a cpuset provides a simple per-cpuset running average of the rate that the processes in a cpuset are attempting to free up in-use memory on the nodes of the cpuset to satisfy additional memory requests.

This enables batch managers that are monitoring jobs running in dedicated cpusets to efficiently detect what level of memory pressure that job is causing.

This is useful both on tightly managed systems running a wide mix of submitted jobs, which may choose to terminate or reprioritize jobs that are trying to use more memory than allowed on the nodes assigned them, and with tightly coupled, long-running, massively parallel scientific computing jobs that will dramatically fail to meet required performance goals if they start to use more memory than allowed to them.

This mechanism provides a very economical way for the batch manager to monitor a cpuset for signs of memory pressure. It's up to the batch manager or other user code to decide what action to take if it detects signs of memory pressure.

Unless memory pressure calculation is enabled by setting the pseudo-file /dev/cpuset/cpuset.memory_pressure_enabled, it is not computed for any cpuset, and reads from any memory_pressure always return zero, as represented by the ASCII string "0\n". See the WARNINGS section, below.

A per-cpuset, running average is employed for the following reasons:

•

Because this meter is per-cpuset rather than per-process or per virtual memory region, the system load imposed by a batch scheduler monitoring this metric is sharply reduced on large systems, because a scan of the tasklist can be avoided on each set of queries.

•

Because this meter is a running average rather than an accumulating counter, a batch scheduler can detect memory pressure with a single read, instead of having to read and accumulate results for a period of time.

•

Because this meter is per-cpuset rather than per-process, the batch scheduler can obtain the key information—memory pressure in a cpuset—with a single read, rather than having to query and accumulate results over all the (dynamically changing) set of processes in the cpuset.

The memory_pressure of a cpuset is calculated using a per-cpuset simple digital filter that is kept within the kernel. For each cpuset, this filter tracks the recent rate at which processes attached to that cpuset enter the kernel direct reclaim code.

The kernel direct reclaim code is entered whenever a process has to satisfy a memory page request by first finding some other page to repurpose, due to lack of any readily available already free pages. Dirty filesystem pages are repurposed by first writing them to disk. Unmodified filesystem buffer pages are repurposed by simply dropping them, though if that page is needed again, it will have to be reread from disk.

The cpuset.memory_pressure file provides an integer number representing the recent (half-life of 10 seconds) rate of entries to the direct reclaim code caused by any process in the cpuset, in units of reclaims attempted per second, times 1000.

### Memory spread
There are two Boolean flag files per cpuset that control where the kernel allocates pages for the filesystem buffers and related in-kernel data structures. They are called cpuset.memory_spread_page and cpuset.memory_spread_slab.

If the per-cpuset Boolean flag file cpuset.memory_spread_page is set, then the kernel will spread the filesystem buffers (page cache) evenly over all the nodes that the faulting process is allowed to use, instead of preferring to put those pages on the node where the process is running.

If the per-cpuset Boolean flag file cpuset.memory_spread_slab is set, then the kernel will spread some filesystem-related slab caches, such as those for inodes and directory entries, evenly over all the nodes that the faulting process is allowed to use, instead of preferring to put those pages on the node where the process is running.

The setting of these flags does not affect the data segment (see brk(2)) or stack segment pages of a process.

By default, both kinds of memory spreading are off and the kernel prefers to allocate memory pages on the node local to where the requesting process is running. If that node is not allowed by the process's NUMA memory policy or cpuset configuration or if there are insufficient free memory pages on that node, then the kernel looks for the nearest node that is allowed and has sufficient free memory.

When new cpusets are created, they inherit the memory spread settings of their parent.

Setting memory spreading causes allocations for the affected page or slab caches to ignore the process's NUMA memory policy and be spread instead. However, the effect of these changes in memory placement caused by cpuset-specified memory spreading is hidden from the mbind(2) or set_mempolicy(2) calls. These two NUMA memory policy calls always appear to behave as if no cpuset-specified memory spreading is in effect, even if it is. If cpuset memory spreading is subsequently turned off, the NUMA memory policy most recently specified by these calls is automatically reapplied.

Both cpuset.memory_spread_page and cpuset.memory_spread_slab are Boolean flag files. By default, they contain "0", meaning that the feature is off for that cpuset. If a "1" is written to that file, that turns the named feature on.

Cpuset-specified memory spreading behaves similarly to what is known (in other contexts) as round-robin or interleave memory placement.

Cpuset-specified memory spreading can provide substantial performance improvements for jobs that:

•

need to place thread-local data on memory nodes close to the CPUs which are running the threads that most frequently access that data; but also

•

need to access large filesystem data sets that must to be spread across the several nodes in the job's cpuset in order to fit.

Without this policy, the memory allocation across the nodes in the job's cpuset can become very uneven, especially for jobs that might have just a single thread initializing or reading in the data set.

### Memory migration
Normally, under the default setting (disabled) of cpuset.memory_migrate, once a page is allocated (given a physical page of main memory), then that page stays on whatever node it was allocated, so long as it remains allocated, even if the cpuset's memory-placement policy mems subsequently changes.

When memory migration is enabled in a cpuset, if the mems setting of the cpuset is changed, then any memory page in use by any process in the cpuset that is on a memory node that is no longer allowed will be migrated to a memory node that is allowed.

Furthermore, if a process is moved into a cpuset with memory_migrate enabled, any memory pages it uses that were on memory nodes allowed in its previous cpuset, but which are not allowed in its new cpuset, will be migrated to a memory node allowed in the new cpuset.

The relative placement of a migrated page within the cpuset is preserved during these migration operations if possible. For example, if the page was on the second valid node of the prior cpuset, then the page will be placed on the second valid node of the new cpuset, if possible.

### Scheduler load balancing
The kernel scheduler automatically load balances processes. If one CPU is underutilized, the kernel will look for processes on other more overloaded CPUs and move those processes to the underutilized CPU, within the constraints of such placement mechanisms as cpusets and sched_setaffinity(2).

The algorithmic cost of load balancing and its impact on key shared kernel data structures such as the process list increases more than linearly with the number of CPUs being balanced. For example, it costs more to load balance across one large set of CPUs than it does to balance across two smaller sets of CPUs, each of half the size of the larger set. (The precise relationship between the number of CPUs being balanced and the cost of load balancing depends on implementation details of the kernel process scheduler, which is subject to change over time, as improved kernel scheduler algorithms are implemented.)

The per-cpuset flag sched_load_balance provides a mechanism to suppress this automatic scheduler load balancing in cases where it is not needed and suppressing it would have worthwhile performance benefits.

By default, load balancing is done across all CPUs, except those marked isolated using the kernel boot time "isolcpus=" argument. (See Scheduler Relax Domain Level, below, to change this default.)

This default load balancing across all CPUs is not well suited to the following two situations:

•

On large systems, load balancing across many CPUs is expensive. If the system is managed using cpusets to place independent jobs on separate sets of CPUs, full load balancing is unnecessary.

•

Systems supporting real-time on some CPUs need to minimize system overhead on those CPUs, including avoiding process load balancing if that is not needed.

When the per-cpuset flag sched_load_balance is enabled (the default setting), it requests load balancing across all the CPUs in that cpuset's allowed CPUs, ensuring that load balancing can move a process (not otherwise pinned, as by sched_setaffinity(2)) from any CPU in that cpuset to any other.

When the per-cpuset flag sched_load_balance is disabled, then the scheduler will avoid load balancing across the CPUs in that cpuset, except in so far as is necessary because some overlapping cpuset has sched_load_balance enabled.

So, for example, if the top cpuset has the flag sched_load_balance enabled, then the scheduler will load balance across all CPUs, and the setting of the sched_load_balance flag in other cpusets has no effect, as we're already fully load balancing.

Therefore in the above two situations, the flag sched_load_balance should be disabled in the top cpuset, and only some of the smaller, child cpusets would have this flag enabled.

When doing this, you don't usually want to leave any unpinned processes in the top cpuset that might use nontrivial amounts of CPU, as such processes may be artificially constrained to some subset of CPUs, depending on the particulars of this flag setting in descendant cpusets. Even if such a process could use spare CPU cycles in some other CPUs, the kernel scheduler might not consider the possibility of load balancing that process to the underused CPU.

Of course, processes pinned to a particular CPU can be left in a cpuset that disables sched_load_balance as those processes aren't going anywhere else anyway.

### Scheduler relax domain level
The kernel scheduler performs immediate load balancing whenever a CPU becomes free or another task becomes runnable. This load balancing works to ensure that as many CPUs as possible are usefully employed running tasks. The kernel also performs periodic load balancing off the software clock described in time(7). The setting of sched_relax_domain_level applies only to immediate load balancing. Regardless of the sched_relax_domain_level setting, periodic load balancing is attempted over all CPUs (unless disabled by turning off sched_load_balance.) In any case, of course, tasks will be scheduled to run only on CPUs allowed by their cpuset, as modified by sched_setaffinity(2) system calls.

On small systems, such as those with just a few CPUs, immediate load balancing is useful to improve system interactivity and to minimize wasteful idle CPU cycles. But on large systems, attempting immediate load balancing across a large number of CPUs can be more costly than it is worth, depending on the particular performance characteristics of the job mix and the hardware.

The exact meaning of the small integer values of sched_relax_domain_level will depend on internal implementation details of the kernel scheduler code and on the non-uniform architecture of the hardware. Both of these will evolve over time and vary by system architecture and kernel version.

As of this writing, when this capability was introduced in Linux 2.6.26, on certain popular architectures, the positive values of sched_relax_domain_level have the following meanings.

1

Perform immediate load balancing across Hyper-Thread siblings on the same core.

2

Perform immediate load balancing across other cores in the same package.

3

Perform immediate load balancing across other CPUs on the same node or blade.

4

Perform immediate load balancing across over several (implementation detail) nodes [On NUMA systems].

5

Perform immediate load balancing across over all CPUs in system [On NUMA systems].

The sched_relax_domain_level value of zero (0) always means don't perform immediate load balancing, hence that load balancing is done only periodically, not immediately when a CPU becomes available or another task becomes runnable.

The sched_relax_domain_level value of minus one (-1) always means use the system default value. The system default value can vary by architecture and kernel version. This system default value can be changed by kernel boot-time "relax_domain_level=" argument.

In the case of multiple overlapping cpusets which have conflicting sched_relax_domain_level values, then the highest such value applies to all CPUs in any of the overlapping cpusets. In such cases, -1 is the lowest value, overridden by any other value, and 0 is the next lowest value.

## FORMATS
The following formats are used to represent sets of CPUs and memory nodes.

### Mask format
The Mask Format is used to represent CPU and memory-node bit masks in the /proc/pid/status file.

This format displays each 32-bit word in hexadecimal (using ASCII characters "0" - "9" and "a" - "f"); words are filled with leading zeros, if required. For masks longer than one word, a comma separator is used between words. Words are displayed in big-endian order, which has the most significant bit first. The hex digits within a word are also in big-endian order.

The number of 32-bit words displayed is the minimum number needed to display all bits of the bit mask, based on the size of the bit mask.

Examples of the Mask Format:

```
00000001                        # just bit 0 set
40000000,00000000,00000000      # just bit 94 set
00000001,00000000,00000000      # just bit 64 set
000000ff,00000000               # bits 32-39 set
00000000,000e3862               # 1,5,6,11-13,17-19 set
```

A mask with bits 0, 1, 2, 4, 8, 16, 32, and 64 set displays as:

```
00000001,00000001,00010117
```

The first "1" is for bit 64, the second for bit 32, the third for bit 16, the fourth for bit 8, the fifth for bit 4, and the "7" is for bits 2, 1, and 0.

### List format
The List Format for cpus and mems is a comma-separated list of CPU or memory-node numbers and ranges of numbers, in ASCII decimal.

Examples of the List Format:

```
0-4,9           # bits 0, 1, 2, 3, 4, and 9 set
0-2,7,12-14     # bits 0, 1, 2, 7, 12, 13, and 14 set
```

## RULES
The following rules apply to each cpuset:

•

Its CPUs and memory nodes must be a (possibly equal) subset of its parent's.

•

It can be marked cpu_exclusive only if its parent is.

•

It can be marked mem_exclusive only if its parent is.

•

If it is cpu_exclusive, its CPUs may not overlap any sibling.

•

If it is mem_exclusive, its memory nodes may not overlap any sibling.

## PERMISSIONS
The permissions of a cpuset are determined by the permissions of the directories and pseudo-files in the cpuset filesystem, normally mounted at /dev/cpuset.

For instance, a process can put itself in some other cpuset (than its current one) if it can write the tasks file for that cpuset. This requires execute permission on the encompassing directories and write permission on the tasks file.

An additional constraint is applied to requests to place some other process in a cpuset. One process may not attach another to a cpuset unless it would have permission to send that process a signal (see kill(2)).

A process may create a child cpuset if it can access and write the parent cpuset directory. It can modify the CPUs or memory nodes in a cpuset if it can access that cpuset's directory (execute permissions on the each of the parent directories) and write the corresponding cpus or mems file.

There is one minor difference between the manner in which these permissions are evaluated and the manner in which normal filesystem operation permissions are evaluated. The kernel interprets relative pathnames starting at a process's current working directory. Even if one is operating on a cpuset file, relative pathnames are interpreted relative to the process's current working directory, not relative to the process's current cpuset. The only ways that cpuset paths relative to a process's current cpuset can be used are if either the process's current working directory is its cpuset (it first did a cd or chdir(2) to its cpuset directory beneath /dev/cpuset, which is a bit unusual) or if some user code converts the relative cpuset path to a full filesystem path.

In theory, this means that user code should specify cpusets using absolute pathnames, which requires knowing the mount point of the cpuset filesystem (usually, but not necessarily, /dev/cpuset). In practice, all user level code that this author is aware of simply assumes that if the cpuset filesystem is mounted, then it is mounted at /dev/cpuset. Furthermore, it is common practice for carefully written user code to verify the presence of the pseudo-file /dev/cpuset/tasks in order to verify that the cpuset pseudo-filesystem is currently mounted.

## WARNINGS

### Enabling memory_pressure
By default, the per-cpuset file cpuset.memory_pressure always contains zero (0). Unless this feature is enabled by writing "1" to the pseudo-file /dev/cpuset/cpuset.memory_pressure_enabled, the kernel does not compute per-cpuset memory_pressure.

### Using the echo command
When using the echo command at the shell prompt to change the values of cpuset files, beware that the built-in echo command in some shells does not display an error message if the write(2) system call fails. For example, if the command:

```
echo 19 > cpuset.mems
```

failed because memory node 19 was not allowed (perhaps the current system does not have a memory node 19), then the echo command might not display any error. It is better to use the /bin/echo external command to change cpuset file settings, as this command will display write(2) errors, as in the example:

```
/bin/echo 19 > cpuset.mems
/bin/echo: write error: Invalid argument
```

## EXCEPTIONS

### Memory placement
Not all allocations of system memory are constrained by cpusets, for the following reasons.

If hot-plug functionality is used to remove all the CPUs that are currently assigned to a cpuset, then the kernel will automatically update the cpus_allowed of all processes attached to CPUs in that cpuset to allow all CPUs. When memory hot-plug functionality for removing memory nodes is available, a similar exception is expected to apply there as well. In general, the kernel prefers to violate cpuset placement, rather than starving a process that has had all its allowed CPUs or memory nodes taken offline. User code should reconfigure cpusets to refer only to online CPUs and memory nodes when using hot-plug to add or remove such resources.

A few kernel-critical, internal memory-allocation requests, marked GFP_ATOMIC, must be satisfied immediately. The kernel may drop some request or malfunction if one of these allocations fail. If such a request cannot be satisfied within the current process's cpuset, then we relax the cpuset, and look for memory anywhere we can find it. It's better to violate the cpuset than stress the kernel.

Allocations of memory requested by kernel drivers while processing an interrupt lack any relevant process context, and are not confined by cpusets.

### Renaming cpusets
You can use the rename(2) system call to rename cpusets. Only simple renaming is supported; that is, changing the name of a cpuset directory is permitted, but moving a directory into a different directory is not permitted.

## ERRORS
The Linux kernel implementation of cpusets sets errno to specify the reason for a failed system call affecting cpusets.

The possible errno settings and their meaning when set on a failed cpuset call are as listed below.

E2BIG

Attempted a write(2) on a special cpuset file with a length larger than some kernel-determined upper limit on the length of such writes.

EACCES

Attempted to write(2) the process ID (PID) of a process to a cpuset tasks file when one lacks permission to move that process.

EACCES

Attempted to add, using write(2), a CPU or memory node to a cpuset, when that CPU or memory node was not already in its parent.

EACCES

Attempted to set, using write(2), cpuset.cpu_exclusive or cpuset.mem_exclusive on a cpuset whose parent lacks the same setting.

EACCES

Attempted to write(2) a cpuset.memory_pressure file.

EACCES

Attempted to create a file in a cpuset directory.

EBUSY

Attempted to remove, using rmdir(2), a cpuset with attached processes.

EBUSY

Attempted to remove, using rmdir(2), a cpuset with child cpusets.

EBUSY

Attempted to remove a CPU or memory node from a cpuset that is also in a child of that cpuset.

EEXIST

Attempted to create, using mkdir(2), a cpuset that already exists.

EEXIST

Attempted to rename(2) a cpuset to a name that already exists.

EFAULT

Attempted to read(2) or write(2) a cpuset file using a buffer that is outside the writing processes accessible address space.

EINVAL

Attempted to change a cpuset, using write(2), in a way that would violate a cpu_exclusive or mem_exclusive attribute of that cpuset or any of its siblings.

EINVAL

Attempted to write(2) an empty cpuset.cpus or cpuset.mems list to a cpuset which has attached processes or child cpusets.

EINVAL

Attempted to write(2) a cpuset.cpus or cpuset.mems list which included a range with the second number smaller than the first number.

EINVAL

Attempted to write(2) a cpuset.cpus or cpuset.mems list which included an invalid character in the string.

EINVAL

Attempted to write(2) a list to a cpuset.cpus file that did not include any online CPUs.

EINVAL

Attempted to write(2) a list to a cpuset.mems file that did not include any online memory nodes.

EINVAL

Attempted to write(2) a list to a cpuset.mems file that included a node that held no memory.

EIO

Attempted to write(2) a string to a cpuset tasks file that does not begin with an ASCII decimal integer.

EIO

Attempted to rename(2) a cpuset into a different directory.

ENAMETOOLONG

Attempted to read(2) a /proc/pid/cpuset file for a cpuset path that is longer than the kernel page size.

ENAMETOOLONG

Attempted to create, using mkdir(2), a cpuset whose base directory name is longer than 255 characters.

ENAMETOOLONG

Attempted to create, using mkdir(2), a cpuset whose full pathname, including the mount point (typically "/dev/cpuset/") prefix, is longer than 4095 characters.

ENODEV

The cpuset was removed by another process at the same time as a write(2) was attempted on one of the pseudo-files in the cpuset directory.

ENOENT

Attempted to create, using mkdir(2), a cpuset in a parent cpuset that doesn't exist.

ENOENT

Attempted to access(2) or open(2) a nonexistent file in a cpuset directory.

ENOMEM

Insufficient memory is available within the kernel; can occur on a variety of system calls affecting cpusets, but only if the system is extremely short of memory.

ENOSPC

Attempted to write(2) the process ID (PID) of a process to a cpuset tasks file when the cpuset had an empty cpuset.cpus or empty cpuset.mems setting.

ENOSPC

Attempted to write(2) an empty cpuset.cpus or cpuset.mems setting to a cpuset that has tasks attached.

ENOTDIR

Attempted to rename(2) a nonexistent cpuset.

EPERM

Attempted to remove a file from a cpuset directory.

ERANGE

Specified a cpuset.cpus or cpuset.mems list to the kernel which included a number too large for the kernel to set in its bit masks.

ESRCH

Attempted to write(2) the process ID (PID) of a nonexistent process to a cpuset tasks file.

## VERSIONS
Cpusets appeared in Linux 2.6.12.

## NOTES
Despite its name, the pid parameter is actually a thread ID, and each thread in a threaded group can be attached to a different cpuset. The value returned from a call to gettid(2) can be passed in the argument pid.

## BUGS
cpuset.memory_pressure cpuset files can be opened for writing, creation, or truncation, but then the write(2) fails with errno set to EACCES, and the creation and truncation options on open(2) have no effect.

## EXAMPLES
The following examples demonstrate querying and setting cpuset options using shell commands.

### Creating and attaching to a cpuset.
To create a new cpuset and attach the current command shell to it, the steps are:

(1)

mkdir /dev/cpuset (if not already done)

(2)

mount -t cpuset none /dev/cpuset (if not already done)

(3)

Create the new cpuset using mkdir(1).

(4)

Assign CPUs and memory nodes to the new cpuset.

(5)

Attach the shell to the new cpuset.

For example, the following sequence of commands will set up a cpuset named "Charlie", containing just CPUs 2 and 3, and memory node 1, and then attach the current shell to that cpuset.

```
$ mkdir /dev/cpuset
$ mount -t cpuset cpuset /dev/cpuset
$ cd /dev/cpuset
$ mkdir Charlie
$ cd Charlie
$ /bin/echo 2-3 > cpuset.cpus
$ /bin/echo 1 > cpuset.mems
$ /bin/echo $$ > tasks
# The current shell is now running in cpuset Charlie
# The next line should display '/Charlie'
$ cat /proc/self/cpuset
```

### Migrating a job to different memory nodes.
To migrate a job (the set of processes attached to a cpuset) to different CPUs and memory nodes in the system, including moving the memory pages currently allocated to that job, perform the following steps.

(1)

Let's say we want to move the job in cpuset alpha (CPUs 4–7 and memory nodes 2–3) to a new cpuset beta (CPUs 16–19 and memory nodes 8–9).

(2)

First create the new cpuset beta.

(3)

Then allow CPUs 16–19 and memory nodes 8–9 in beta.

(4)

Then enable memory_migration in beta.

(5)

Then move each process from alpha to beta.

The following sequence of commands accomplishes this.

```
$ cd /dev/cpuset
$ mkdir beta
$ cd beta
$ /bin/echo 16-19 > cpuset.cpus
$ /bin/echo 8-9 > cpuset.mems
$ /bin/echo 1 > cpuset.memory_migrate
$ while read i; do /bin/echo $i; done < ../alpha/tasks > tasks
```

The above should move any processes in alpha to beta, and any memory held by these processes on memory nodes 2–3 to memory nodes 8–9, respectively.

Notice that the last step of the above sequence did not do:

```
$ cp ../alpha/tasks tasks
```

The while loop, rather than the seemingly easier use of the cp(1) command, was necessary because only one process PID at a time may be written to the tasks file.

The same effect (writing one PID at a time) as the while loop can be accomplished more efficiently, in fewer keystrokes and in syntax that works on any shell, but alas more obscurely, by using the -u (unbuffered) option of sed(1):

```
$ sed -un p < ../alpha/tasks > tasks
```

## SEE ALSO
taskset(1), get_mempolicy(2), getcpu(2), mbind(2), sched_getaffinity(2), sched_setaffinity(2), sched_setscheduler(2), set_mempolicy(2), CPU_SET(3), proc(5), cgroups(7), numa(7), sched(7), migratepages(8), numactl(8)

Documentation/admin-guide/cgroup-v1/cpusets.rst in the Linux kernel source tree (or Documentation/cgroup-v1/cpusets.txt before Linux 4.18, and Documentation/cpusets.txt before Linux 2.6.29)


credentials

package: man-pages

title: credentials(7)

summary: process identifiers

text:
# credentials(7)

## NAME
credentials - process identifiers

## DESCRIPTION

### Process ID (PID)
Each process has a unique nonnegative integer identifier that is assigned when the process is created using fork(2). A process can obtain its PID using getpid(2). A PID is represented using the type pid_t (defined in <sys/types.h>).

PIDs are used in a range of system calls to identify the process affected by the call, for example: kill(2), ptrace(2), setpriority(2), setpgid(2), setsid(2), sigqueue(3), and waitpid(2).

A process's PID is preserved across an execve(2).

### Parent process ID (PPID)
A process's parent process ID identifies the process that created this process using fork(2). A process can obtain its PPID using getppid(2). A PPID is represented using the type pid_t.

A process's PPID is preserved across an execve(2).

### Process group ID and session ID
Each process has a session ID and a process group ID, both represented using the type pid_t. A process can obtain its session ID using getsid(2), and its process group ID using getpgrp(2).

A child created by fork(2) inherits its parent's session ID and process group ID. A process's session ID and process group ID are preserved across an execve(2).

Sessions and process groups are abstractions devised to support shell job control. A process group (sometimes called a "job") is a collection of processes that share the same process group ID; the shell creates a new process group for the process(es) used to execute single command or pipeline (e.g., the two processes created to execute the command "ls | wc" are placed in the same process group). A process's group membership can be set using setpgid(2). The process whose process ID is the same as its process group ID is the process group leader for that group.

A session is a collection of processes that share the same session ID. All of the members of a process group also have the same session ID (i.e., all of the members of a process group always belong to the same session, so that sessions and process groups form a strict two-level hierarchy of processes.) A new session is created when a process calls setsid(2), which creates a new session whose session ID is the same as the PID of the process that called setsid(2). The creator of the session is called the session leader.

All of the processes in a session share a controlling terminal. The controlling terminal is established when the session leader first opens a terminal (unless the O_NOCTTY flag is specified when calling open(2)). A terminal may be the controlling terminal of at most one session.

At most one of the jobs in a session may be the foreground job; other jobs in the session are background jobs. Only the foreground job may read from the terminal; when a process in the background attempts to read from the terminal, its process group is sent a SIGTTIN signal, which suspends the job. If the TOSTOP flag has been set for the terminal (see termios(3)), then only the foreground job may write to the terminal; writes from background jobs cause a SIGTTOU signal to be generated, which suspends the job. When terminal keys that generate a signal (such as the interrupt key, normally control-C) are pressed, the signal is sent to the processes in the foreground job.

Various system calls and library functions may operate on all members of a process group, including kill(2), killpg(3), getpriority(2), setpriority(2), ioprio_get(2), ioprio_set(2), waitid(2), and waitpid(2). See also the discussion of the F_GETOWN, F_GETOWN_EX, F_SETOWN, and F_SETOWN_EX operations in fcntl(2).

### User and group identifiers
Each process has various associated user and group IDs. These IDs are integers, respectively represented using the types uid_t and gid_t (defined in <sys/types.h>).

On Linux, each process has the following user and group identifiers:

•

Real user ID and real group ID. These IDs determine who owns the process. A process can obtain its real user (group) ID using getuid(2) (getgid(2)).

•

Effective user ID and effective group ID. These IDs are used by the kernel to determine the permissions that the process will have when accessing shared resources such as message queues, shared memory, and semaphores. On most UNIX systems, these IDs also determine the permissions when accessing files. However, Linux uses the filesystem IDs described below for this task. A process can obtain its effective user (group) ID using geteuid(2) (getegid(2)).

•

Saved set-user-ID and saved set-group-ID. These IDs are used in set-user-ID and set-group-ID programs to save a copy of the corresponding effective IDs that were set when the program was executed (see execve(2)). A set-user-ID program can assume and drop privileges by switching its effective user ID back and forth between the values in its real user ID and saved set-user-ID. This switching is done via calls to seteuid(2), setreuid(2), or setresuid(2). A set-group-ID program performs the analogous tasks using setegid(2), setregid(2), or setresgid(2). A process can obtain its saved set-user-ID (set-group-ID) using getresuid(2) (getresgid(2)).

•

Filesystem user ID and filesystem group ID (Linux-specific). These IDs, in conjunction with the supplementary group IDs described below, are used to determine permissions for accessing files; see path_resolution(7) for details. Whenever a process's effective user (group) ID is changed, the kernel also automatically changes the filesystem user (group) ID to the same value. Consequently, the filesystem IDs normally have the same values as the corresponding effective ID, and the semantics for file-permission checks are thus the same on Linux as on other UNIX systems. The filesystem IDs can be made to differ from the effective IDs by calling setfsuid(2) and setfsgid(2).

•

Supplementary group IDs. This is a set of additional group IDs that are used for permission checks when accessing files and other shared resources. Before Linux 2.6.4, a process can be a member of up to 32 supplementary groups; since Linux 2.6.4, a process can be a member of up to 65536 supplementary groups. The call sysconf(_SC_NGROUPS_MAX) can be used to determine the number of supplementary groups of which a process may be a member. A process can obtain its set of supplementary group IDs using getgroups(2).

A child process created by fork(2) inherits copies of its parent's user and groups IDs. During an execve(2), a process's real user and group ID and supplementary group IDs are preserved; the effective and saved set IDs may be changed, as described in execve(2).

Aside from the purposes noted above, a process's user IDs are also employed in a number of other contexts:

•

when determining the permissions for sending signals (see kill(2));

•

when determining the permissions for setting process-scheduling parameters (nice value, real time scheduling policy and priority, CPU affinity, I/O priority) using setpriority(2), sched_setaffinity(2), sched_setscheduler(2), sched_setparam(2), sched_setattr(2), and ioprio_set(2);

•

when checking resource limits (see getrlimit(2));

•

when checking the limit on the number of inotify instances that the process may create (see inotify(7)).

### Modifying process user and group IDs
Subject to rules described in the relevant manual pages, a process can use the following APIs to modify its user and group IDs:

setuid(2) (setgid(2))

Modify the process's real (and possibly effective and saved-set) user (group) IDs.

seteuid(2) (setegid(2))

Modify the process's effective user (group) ID.

setfsuid(2) (setfsgid(2))

Modify the process's filesystem user (group) ID.

setreuid(2) (setregid(2))

Modify the process's real and effective (and possibly saved-set) user (group) IDs.

setresuid(2) (setresgid(2))

Modify the process's real, effective, and saved-set user (group) IDs.

setgroups(2)

Modify the process's supplementary group list.

Any changes to a process's effective user (group) ID are automatically carried over to the process's filesystem user (group) ID. Changes to a process's effective user or group ID can also affect the process "dumpable" attribute, as described in prctl(2).

Changes to process user and group IDs can affect the capabilities of the process, as described in capabilities(7).

## STANDARDS
Process IDs, parent process IDs, process group IDs, and session IDs are specified in POSIX.1. The real, effective, and saved set user and groups IDs, and the supplementary group IDs, are specified in POSIX.1.

The filesystem user and group IDs are a Linux extension.

## NOTES
Various fields in the /proc/pid/status file show the process credentials described above. See proc(5) for further information.

The POSIX threads specification requires that credentials are shared by all of the threads in a process. However, at the kernel level, Linux maintains separate user and group credentials for each thread. The NPTL threading implementation does some work to ensure that any change to user or group credentials (e.g., calls to setuid(2), setresuid(2)) is carried through to all of the POSIX threads in a process. See nptl(7) for further details.

## SEE ALSO
bash(1), csh(1), groups(1), id(1), newgrp(1), ps(1), runuser(1), setpriv(1), sg(1), su(1), access(2), execve(2), faccessat(2), fork(2), getgroups(2), getpgrp(2), getpid(2), getppid(2), getsid(2), kill(2), setegid(2), seteuid(2), setfsgid(2), setfsuid(2), setgid(2), setgroups(2), setpgid(2), setresgid(2), setresuid(2), setsid(2), setuid(2), waitpid(2), euidaccess(3), initgroups(3), killpg(3), tcgetpgrp(3), tcgetsid(3), tcsetpgrp(3), group(5), passwd(5), shadow(5), capabilities(7), namespaces(7), path_resolution(7), pid_namespaces(7), pthreads(7), signal(7), system_data_types(7), unix(7), user_namespaces(7), sudo(8)

branch: sisyphus


ddp

title: ddp(7)

summary: Linux AppleTalk protocol implementation

text:
# ddp(7)

## NAME
ddp - Linux AppleTalk protocol implementation

## SYNOPSIS

```
#include <sys/socket.h>
#include <netatalk/at.h>
```

```
ddp_socket = socket(AF_APPLETALK, SOCK_DGRAM, 0);
raw_socket = socket(AF_APPLETALK, SOCK_RAW, protocol);
```

## DESCRIPTION
Linux implements the AppleTalk protocols described in Inside AppleTalk. Only the DDP layer and AARP are present in the kernel. They are designed to be used via the netatalk protocol libraries. This page documents the interface for those who wish or need to use the DDP layer directly.

The communication between AppleTalk and the user program works using a BSD-compatible socket interface. For more information on sockets, see socket(7).

An AppleTalk socket is created by calling the socket(2) function with a AF_APPLETALK socket family argument. Valid socket types are SOCK_DGRAM to open a ddp socket or SOCK_RAW to open a raw socket. protocol is the AppleTalk protocol to be received or sent. For SOCK_RAW you must specify ATPROTO_DDP.

Raw sockets may be opened only by a process with effective user ID 0 or when the process has the CAP_NET_RAW capability.

### Address format
An AppleTalk socket address is defined as a combination of a network number, a node number, and a port number.

```
struct at_addr {

    unsigned short s_net;

    unsigned char  s_node;
};
struct sockaddr_atalk {

    sa_family_t    sat_family;    /* address family */

    unsigned char  sat_port;      /* port */

    struct at_addr sat_addr;      /* net/node */
};
```

sat_family is always set to AF_APPLETALK. sat_port contains the port. The port numbers below 129 are known as reserved ports. Only processes with the effective user ID 0 or the CAP_NET_BIND_SERVICE capability may bind(2) to these sockets. sat_addr is the host address. The net member of struct at_addr contains the host network in network byte order. The value of AT_ANYNET is a wildcard and also implies “this network.” The node member of struct at_addr contains the host node number. The value of AT_ANYNODE is a wildcard and also implies “this node.” The value of ATADDR_BCAST is a link local broadcast address.

### Socket options
No protocol-specific socket options are supported.

### /proc interfaces
IP supports a set of /proc interfaces to configure some global AppleTalk parameters. The parameters can be accessed by reading or writing files in the directory /proc/sys/net/atalk/.

aarp-expiry-time

The time interval (in seconds) before an AARP cache entry expires.

aarp-resolve-time

The time interval (in seconds) before an AARP cache entry is resolved.

aarp-retransmit-limit

The number of retransmissions of an AARP query before the node is declared dead.

aarp-tick-time

The timer rate (in seconds) for the timer driving AARP.

The default values match the specification and should never need to be changed.

### Ioctls
All ioctls described in socket(7) apply to DDP.

## ERRORS
EACCES

The user tried to execute an operation without the necessary permissions. These include sending to a broadcast address without having the broadcast flag set, and trying to bind to a reserved port without effective user ID 0 or CAP_NET_BIND_SERVICE.

EADDRINUSE

Tried to bind to an address already in use.

EADDRNOTAVAIL

A nonexistent interface was requested or the requested source address was not local.

EAGAIN

Operation on a nonblocking socket would block.

EALREADY

A connection operation on a nonblocking socket is already in progress.

ECONNABORTED

A connection was closed during an accept(2).

EHOSTUNREACH

No routing table entry matches the destination address.

EINVAL

Invalid argument passed.

EISCONN

connect(2) was called on an already connected socket.

EMSGSIZE

Datagram is bigger than the DDP MTU.

ENODEV

Network device not available or not capable of sending IP.

ENOENT

SIOCGSTAMP was called on a socket where no packet arrived.

ENOMEM and ENOBUFS

Not enough memory available.

ENOPKG

A kernel subsystem was not configured.

ENOPROTOOPT and EOPNOTSUPP

Invalid socket option passed.

ENOTCONN

The operation is defined only on a connected socket, but the socket wasn't connected.

EPERM

User doesn't have permission to set high priority, make a configuration change, or send signals to the requested process or group.

EPIPE

The connection was unexpectedly closed or shut down by the other end.

ESOCKTNOSUPPORT

The socket was unconfigured, or an unknown socket type was requested.

## VERSIONS
AppleTalk is supported by Linux 2.0 or higher. The /proc interfaces exist since Linux 2.2.

## NOTES
Be very careful with the SO_BROADCAST option; it is not privileged in Linux. It is easy to overload the network with careless sending to broadcast addresses.

### Compatibility
The basic AppleTalk socket interface is compatible with netatalk on BSD-derived systems. Many BSD systems fail to check SO_BROADCAST when sending broadcast frames; this can lead to compatibility problems.

The raw socket mode is unique to Linux and exists to support the alternative CAP package and AppleTalk monitoring tools more easily.

## BUGS
There are too many inconsistent error values.

The ioctls used to configure routing tables, devices, AARP tables, and other devices are not yet described.

## SEE ALSO
recvmsg(2), sendmsg(2), capabilities(7), socket(7)


environ

title: environ(7)

summary: user environment

text:
# environ(7)

## NAME
environ - user environment

## SYNOPSIS

```
extern char **environ;
```

## DESCRIPTION
The variable environ points to an array of pointers to strings called the "environment". The last pointer in this array has the value NULL. This array of strings is made available to the process by the execve(2) call when a new program is started. When a child process is created via fork(2), it inherits a copy of its parent's environment.

By convention, the strings in environ have the form "name=value". The name is case-sensitive and may not contain the character "=". The value can be anything that can be represented as a string. The name and the value may not contain an embedded null byte ('\0'), since this is assumed to terminate the string.

Environment variables may be placed in the shell's environment by the export command in sh(1), or by the setenv command if you use csh(1).

The initial environment of the shell is populated in various ways, such as definitions from /etc/environment that are processed by pam_env(8) for all users at login time (on systems that employ pam(8)). In addition, various shell initialization scripts, such as the system-wide /etc/profile script and per-user initializations script may include commands that add variables to the shell's environment; see the manual page of your preferred shell for details.

Bourne-style shells support the syntax

```
NAME=value command
```

to create an environment variable definition only in the scope of the process that executes command. Multiple variable definitions, separated by white space, may precede command.

Arguments may also be placed in the environment at the point of an exec(3). A C program can manipulate its environment using the functions getenv(3), putenv(3), setenv(3), and unsetenv(3).

What follows is a list of environment variables typically seen on a system. This list is incomplete and includes only common variables seen by average users in their day-to-day routine. Environment variables specific to a particular program or library function are documented in the ENVIRONMENT section of the appropriate manual page.

USER

The name of the logged-in user (used by some BSD-derived programs). Set at login time, see section NOTES below.

LOGNAME

The name of the logged-in user (used by some System-V derived programs). Set at login time, see section NOTES below.

HOME

A user's login directory. Set at login time, see section NOTES below.

LANG

The name of a locale to use for locale categories when not overridden by LC_ALL or more specific environment variables such as LC_COLLATE, LC_CTYPE, LC_MESSAGES, LC_MONETARY, LC_NUMERIC, and LC_TIME (see locale(7) for further details of the LC_* environment variables).

PATH

The sequence of directory prefixes that sh(1) and many other programs employ when searching for an executable file that is specified as a simple filename (i.a., a pathname that contains no slashes). The prefixes are separated by colons (:). The list of prefixes is searched from beginning to end, by checking the pathname formed by concatenating a prefix, a slash, and the filename, until a file with execute permission is found.

As a legacy feature, a zero-length prefix (specified as two adjacent colons, or an initial or terminating colon) is interpreted to mean the current working directory. However, use of this feature is deprecated, and POSIX notes that a conforming application shall use an explicit pathname (e.g., .) to specify the current working directory.

Analogously to PATH, one has CDPATH used by some shells to find the target of a change directory command, MANPATH used by man(1) to find manual pages, and so on.

PWD

Absolute path to the current working directory; required to be partially canonical (no . or .. components).

SHELL

The absolute pathname of the user's login shell. Set at login time, see section NOTES below.

TERM

The terminal type for which output is to be prepared.

PAGER

The user's preferred utility to display text files. Any string acceptable as a command-string operand to the sh -c command shall be valid. If PAGER is null or is not set, then applications that launch a pager will default to a program such as less(1) or more(1).

EDITOR/VISUAL

The user's preferred utility to edit text files. Any string acceptable as a command_string operand to the sh -c command shall be valid.

Note that the behavior of many programs and library routines is influenced by the presence or value of certain environment variables. Examples include the following:

•

The variables LANG, LANGUAGE, NLSPATH, LOCPATH, LC_ALL, LC_MESSAGES, and so on influence locale handling; see catopen(3), gettext(3), and locale(7).

•

TMPDIR influences the path prefix of names created by tempnam(3) and other routines, and the temporary directory used by sort(1) and other programs.

•

LD_LIBRARY_PATH, LD_PRELOAD, and other LD_* variables influence the behavior of the dynamic loader/linker. See also ld.so(8).

•

POSIXLY_CORRECT makes certain programs and library routines follow the prescriptions of POSIX.

•

The behavior of malloc(3) is influenced by MALLOC_* variables.

•

The variable HOSTALIASES gives the name of a file containing aliases to be used with gethostbyname(3).

•

TZ and TZDIR give timezone information used by tzset(3) and through that by functions like ctime(3), localtime(3), mktime(3), strftime(3). See also tzselect(8).

•

TERMCAP gives information on how to address a given terminal (or gives the name of a file containing such information).

•

COLUMNS and LINES tell applications about the window size, possibly overriding the actual size.

•

PRINTER or LPDEST may specify the desired printer to use. See lpr(1).

## NOTES
Historically and by standard, environ must be declared in the user program. However, as a (nonstandard) programmer convenience, environ is declared in the header file <unistd.h> if the _GNU_SOURCE feature test macro is defined (see feature_test_macros(7)).

The prctl(2) PR_SET_MM_ENV_START and PR_SET_MM_ENV_END operations can be used to control the location of the process's environment.

The HOME, LOGNAME, SHELL, and USER variables are set when the user is changed via a session management interface, typically by a program such as login(1) from a user database (such as passwd(5)). (Switching to the root user using su(1) may result in a mixed environment where LOGNAME and USER are retained from old user; see the su(1) manual page.)

## BUGS
Clearly there is a security risk here. Many a system command has been tricked into mischief by a user who specified unusual values for IFS or LD_LIBRARY_PATH.

There is also the risk of name space pollution. Programs like make and autoconf allow overriding of default utility names from the environment with similarly named variables in all caps. Thus one uses CC to select the desired C compiler (and similarly MAKE, AR, AS, FC, LD, LEX, RM, YACC, etc.). However, in some traditional uses such an environment variable gives options for the program instead of a pathname. Thus, one has MORE and LESS. Such usage is considered mistaken, and to be avoided in new programs.

## SEE ALSO
bash(1), csh(1), env(1), login(1), printenv(1), sh(1), su(1), tcsh(1), execve(2), clearenv(3), exec(3), getenv(3), putenv(3), setenv(3), unsetenv(3), locale(7), ld.so(8), pam_env(8)


epoll

title: epoll(7)

summary: I/O event notification facility

text:
# epoll(7)

## NAME
epoll - I/O event notification facility

## SYNOPSIS

```
#include <sys/epoll.h>
```

## DESCRIPTION
The epoll API performs a similar task to poll(2): monitoring multiple file descriptors to see if I/O is possible on any of them. The epoll API can be used either as an edge-triggered or a level-triggered interface and scales well to large numbers of watched file descriptors.

The central concept of the epoll API is the epoll instance, an in-kernel data structure which, from a user-space perspective, can be considered as a container for two lists:

•

The interest list (sometimes also called the epoll set): the set of file descriptors that the process has registered an interest in monitoring.

•

The ready list: the set of file descriptors that are "ready" for I/O. The ready list is a subset of (or, more precisely, a set of references to) the file descriptors in the interest list. The ready list is dynamically populated by the kernel as a result of I/O activity on those file descriptors.

The following system calls are provided to create and manage an epoll instance:

•

epoll_create(2) creates a new epoll instance and returns a file descriptor referring to that instance. (The more recent epoll_create1(2) extends the functionality of epoll_create(2).)

•

Interest in particular file descriptors is then registered via epoll_ctl(2), which adds items to the interest list of the epoll instance.

•

epoll_wait(2) waits for I/O events, blocking the calling thread if no events are currently available. (This system call can be thought of as fetching items from the ready list of the epoll instance.)

### Level-triggered and edge-triggered
The epoll event distribution interface is able to behave both as edge-triggered (ET) and as level-triggered (LT). The difference between the two mechanisms can be described as follows. Suppose that this scenario happens:

(1)

The file descriptor that represents the read side of a pipe (rfd) is registered on the epoll instance.

(2)

A pipe writer writes 2 kB of data on the write side of the pipe.

(3)

A call to epoll_wait(2) is done that will return rfd as a ready file descriptor.

(4)

The pipe reader reads 1 kB of data from rfd.

(5)

A call to epoll_wait(2) is done.

If the rfd file descriptor has been added to the epoll interface using the EPOLLET (edge-triggered) flag, the call to epoll_wait(2) done in step 5 will probably hang despite the available data still present in the file input buffer; meanwhile the remote peer might be expecting a response based on the data it already sent. The reason for this is that edge-triggered mode delivers events only when changes occur on the monitored file descriptor. So, in step 5 the caller might end up waiting for some data that is already present inside the input buffer. In the above example, an event on rfd will be generated because of the write done in 2 and the event is consumed in 3. Since the read operation done in 4 does not consume the whole buffer data, the call to epoll_wait(2) done in step 5 might block indefinitely.

An application that employs the EPOLLET flag should use nonblocking file descriptors to avoid having a blocking read or write starve a task that is handling multiple file descriptors. The suggested way to use epoll as an edge-triggered (EPOLLET) interface is as follows:

(1)

with nonblocking file descriptors; and

(2)

by waiting for an event only after read(2) or write(2) return EAGAIN.

By contrast, when used as a level-triggered interface (the default, when EPOLLET is not specified), epoll is simply a faster poll(2), and can be used wherever the latter is used since it shares the same semantics.

Since even with edge-triggered epoll, multiple events can be generated upon receipt of multiple chunks of data, the caller has the option to specify the EPOLLONESHOT flag, to tell epoll to disable the associated file descriptor after the receipt of an event with epoll_wait(2). When the EPOLLONESHOT flag is specified, it is the caller's responsibility to rearm the file descriptor using epoll_ctl(2) with EPOLL_CTL_MOD.

If multiple threads (or processes, if child processes have inherited the epoll file descriptor across fork(2)) are blocked in epoll_wait(2) waiting on the same epoll file descriptor and a file descriptor in the interest list that is marked for edge-triggered (EPOLLET) notification becomes ready, just one of the threads (or processes) is awoken from epoll_wait(2). This provides a useful optimization for avoiding "thundering herd" wake-ups in some scenarios.

### Interaction with autosleep
If the system is in autosleep mode via /sys/power/autosleep and an event happens which wakes the device from sleep, the device driver will keep the device awake only until that event is queued. To keep the device awake until the event has been processed, it is necessary to use the epoll_ctl(2) EPOLLWAKEUP flag.

When the EPOLLWAKEUP flag is set in the events field for a struct epoll_event, the system will be kept awake from the moment the event is queued, through the epoll_wait(2) call which returns the event until the subsequent epoll_wait(2) call. If the event should keep the system awake beyond that time, then a separate wake_lock should be taken before the second epoll_wait(2) call.

### /proc interfaces
The following interfaces can be used to limit the amount of kernel memory consumed by epoll:

/proc/sys/fs/epoll/max_user_watches (since Linux 2.6.28)

This specifies a limit on the total number of file descriptors that a user can register across all epoll instances on the system. The limit is per real user ID. Each registered file descriptor costs roughly 90 bytes on a 32-bit kernel, and roughly 160 bytes on a 64-bit kernel. Currently, the default value for max_user_watches is 1/25 (4%) of the available low memory, divided by the registration cost in bytes.

### Example for suggested usage
While the usage of epoll when employed as a level-triggered interface does have the same semantics as poll(2), the edge-triggered usage requires more clarification to avoid stalls in the application event loop. In this example, listener is a nonblocking socket on which listen(2) has been called. The function do_use_fd() uses the new ready file descriptor until EAGAIN is returned by either read(2) or write(2). An event-driven state machine application should, after having received EAGAIN, record its current state so that at the next call to do_use_fd() it will continue to read(2) or write(2) from where it stopped before.

```
#define MAX_EVENTS 10
struct epoll_event ev, events[MAX_EVENTS];
int listen_sock, conn_sock, nfds, epollfd;
/* Code to set up listening socket, 'listen_sock',

   (socket(), bind(), listen()) omitted. */
epollfd = epoll_create1(0);
if (epollfd == -1) {

    perror("epoll_create1");

    exit(EXIT_FAILURE);
}
ev.events = EPOLLIN;
ev.data.fd = listen_sock;
if (epoll_ctl(epollfd, EPOLL_CTL_ADD, listen_sock, &ev) == -1) {

    perror("epoll_ctl: listen_sock");

    exit(EXIT_FAILURE);
}
for (;;) {

    nfds = epoll_wait(epollfd, events, MAX_EVENTS, -1);

    if (nfds == -1) {

        perror("epoll_wait");

        exit(EXIT_FAILURE);

    }

    for (n = 0; n < nfds; ++n) {

        if (events[n].data.fd == listen_sock) {

            conn_sock = accept(listen_sock,

                               (struct sockaddr *) &addr, &addrlen);

            if (conn_sock == -1) {

                perror("accept");

                exit(EXIT_FAILURE);

            }

            setnonblocking(conn_sock);

            ev.events = EPOLLIN | EPOLLET;

            ev.data.fd = conn_sock;

            if (epoll_ctl(epollfd, EPOLL_CTL_ADD, conn_sock,

                        &ev) == -1) {

                perror("epoll_ctl: conn_sock");

                exit(EXIT_FAILURE);

            }

        } else {

            do_use_fd(events[n].data.fd);

        }

    }
}
```

When used as an edge-triggered interface, for performance reasons, it is possible to add the file descriptor inside the epoll interface (EPOLL_CTL_ADD) once by specifying (EPOLLIN|EPOLLOUT). This allows you to avoid continuously switching between EPOLLIN and EPOLLOUT calling epoll_ctl(2) with EPOLL_CTL_MOD.

### Questions and answers
•

What is the key used to distinguish the file descriptors registered in an interest list?

The key is the combination of the file descriptor number and the open file description (also known as an "open file handle", the kernel's internal representation of an open file).

•

What happens if you register the same file descriptor on an epoll instance twice?

You will probably get EEXIST. However, it is possible to add a duplicate (dup(2), dup2(2), fcntl(2) F_DUPFD) file descriptor to the same epoll instance. This can be a useful technique for filtering events, if the duplicate file descriptors are registered with different events masks.

•

Can two epoll instances wait for the same file descriptor? If so, are events reported to both epoll file descriptors?

Yes, and events would be reported to both. However, careful programming may be needed to do this correctly.

•

Is the epoll file descriptor itself poll/epoll/selectable?

Yes. If an epoll file descriptor has events waiting, then it will indicate as being readable.

•

What happens if one attempts to put an epoll file descriptor into its own file descriptor set?

The epoll_ctl(2) call fails (EINVAL). However, you can add an epoll file descriptor inside another epoll file descriptor set.

•

Can I send an epoll file descriptor over a UNIX domain socket to another process?

Yes, but it does not make sense to do this, since the receiving process would not have copies of the file descriptors in the interest list.

•

Will closing a file descriptor cause it to be removed from all epoll interest lists?

Yes, but be aware of the following point. A file descriptor is a reference to an open file description (see open(2)). Whenever a file descriptor is duplicated via dup(2), dup2(2), fcntl(2) F_DUPFD, or fork(2), a new file descriptor referring to the same open file description is created. An open file description continues to exist until all file descriptors referring to it have been closed.

A file descriptor is removed from an interest list only after all the file descriptors referring to the underlying open file description have been closed. This means that even after a file descriptor that is part of an interest list has been closed, events may be reported for that file descriptor if other file descriptors referring to the same underlying file description remain open. To prevent this happening, the file descriptor must be explicitly removed from the interest list (using epoll_ctl(2) EPOLL_CTL_DEL) before it is duplicated. Alternatively, the application must ensure that all file descriptors are closed (which may be difficult if file descriptors were duplicated behind the scenes by library functions that used dup(2) or fork(2)).

•

If more than one event occurs between epoll_wait(2) calls, are they combined or reported separately?

They will be combined.

•

Does an operation on a file descriptor affect the already collected but not yet reported events?

You can do two operations on an existing file descriptor. Remove would be meaningless for this case. Modify will reread available I/O.

•

Do I need to continuously read/write a file descriptor until EAGAIN when using the EPOLLET flag (edge-triggered behavior)?

Receiving an event from epoll_wait(2) should suggest to you that such file descriptor is ready for the requested I/O operation. You must consider it ready until the next (nonblocking) read/write yields EAGAIN. When and how you will use the file descriptor is entirely up to you.

For packet/token-oriented files (e.g., datagram socket, terminal in canonical mode), the only way to detect the end of the read/write I/O space is to continue to read/write until EAGAIN.

For stream-oriented files (e.g., pipe, FIFO, stream socket), the condition that the read/write I/O space is exhausted can also be detected by checking the amount of data read from / written to the target file descriptor. For example, if you call read(2) by asking to read a certain amount of data and read(2) returns a lower number of bytes, you can be sure of having exhausted the read I/O space for the file descriptor. The same is true when writing using write(2). (Avoid this latter technique if you cannot guarantee that the monitored file descriptor always refers to a stream-oriented file.)

### Possible pitfalls and ways to avoid them
•

Starvation (edge-triggered)

If there is a large amount of I/O space, it is possible that by trying to drain it the other files will not get processed causing starvation. (This problem is not specific to epoll.)

The solution is to maintain a ready list and mark the file descriptor as ready in its associated data structure, thereby allowing the application to remember which files need to be processed but still round robin amongst all the ready files. This also supports ignoring subsequent events you receive for file descriptors that are already ready.

•

If using an event cache...

If you use an event cache or store all the file descriptors returned from epoll_wait(2), then make sure to provide a way to mark its closure dynamically (i.e., caused by a previous event's processing). Suppose you receive 100 events from epoll_wait(2), and in event #47 a condition causes event #13 to be closed. If you remove the structure and close(2) the file descriptor for event #13, then your event cache might still say there are events waiting for that file descriptor causing confusion.

One solution for this is to call, during the processing of event 47, epoll_ctl(EPOLL_CTL_DEL) to delete file descriptor 13 and close(2), then mark its associated data structure as removed and link it to a cleanup list. If you find another event for file descriptor 13 in your batch processing, you will discover the file descriptor had been previously removed and there will be no confusion.

## VERSIONS
Some other systems provide similar mechanisms; for example, FreeBSD has kqueue, and Solaris has /dev/poll.

## STANDARDS
Linux.

## HISTORY
Linux 2.5.44. glibc 2.3.2.

## NOTES
The set of file descriptors that is being monitored via an epoll file descriptor can be viewed via the entry for the epoll file descriptor in the process's /proc/pid/fdinfo directory. See proc(5) for further details.

The kcmp(2) KCMP_EPOLL_TFD operation can be used to test whether a file descriptor is present in an epoll instance.

## SEE ALSO
epoll_create(2), epoll_create1(2), epoll_ctl(2), epoll_wait(2), poll(2), select(2)