The cpuset facility is primarily a workload manager tool permitting a system administrator to restrict the number of processor and memory resources that a process or set of processes may use. A cpuset defines a list of CPUs and memory nodes. A process contained in a cpuset may only execute on the CPUs in that cpuset and may only allocate memory on the memory nodes in that cpuset. Essentially, cpusets provide you with a CPU and memory containers or “soft partitions” within which you can run sets of related tasks. Using cpusets on an SGI Altix system improves cache locality and memory access times and can substantially improve an application's performance and runtime repeatability. Restraining all other jobs from using any of the CPUs or memory resources assigned to a critical job minimizes interference from other jobs on the system. For example, Message Passing Interface (MPI) jobs frequently consist of a number of threads that communicate using message passing interfaces. All threads need to be executing at the same time. If a single thread loses a CPU, all threads stop making forward progress and spin at a barrier.

Cpusets can eliminate the need for a gang scheduler, provide isolation of one such job from other tasks on a system, and facilitate providing equal resources to each thread in a job. This results in both optimum and repeatable performance.

In addition to their traditional use to control the placement of jobs on the CPUs and memory nodes of a system, cpusets also provide a convenient mechanism to control the use of Hyper-Threading Technology.

Cpusets are represented in a hierarchical virtual file system. Cpusets can be nested and they have file-like permissions.

The sched_setaffinity, mbind, and set_mempolicy system calls allow you to specify the CPU and memory placement for individual tasks. On smaller or limited-use systems, these calls may be sufficient.

The kernel cpuset facility provides additional support for system-wide management of CPU and memory resources by related sets of tasks. It provides a hierarchical structure to the resources, with filesystem-like namespace and permissions, and support for guaranteed exclusive use of resources.

You can have a boot cpuset running the traditional daemon and server tasks and a second cpuset to hold interactive telnet, rlogin and/or secure shell (SSH) user sessions called the user cpuset.

Creating a user cpuset provides additional isolation between interactive user login sessions and essential system tasks. For example, a user process in the user cpuset consuming excessive CPU or memory resources will not seriously impact essential system services in the boot cpuset. For more information, see “Configuring a User Cpuset for Interactive Sessions”.

This section covers the following topics:

Cpuset Interaction with Other Placement Mechanism

The Linux 2.6 kernel supports additional processor and memory placement mechanisms, as follows:

---

Note: Use the uname(1) command to print out system information to make sure you are running the Linux 2.6.x sn2 kernel, as follows: % uname -r -s Linux 2.6.16.14-6-default

---

• The sched_setaffinity(2) and sched_getaffinity(2) system calls set and get the CPU affinity mask of a process. This determines the set of CPUs on which the process is eligible to run. The taskset(1) command provides a command line utility for manipulating the CPU affinity mask of a process using these system calls. For more information, see the appropriate man page.

• The set_mempolicy system call sets the NUMA memory policy of the current process to policy. A NUMA machine has different memory controllers with different distances to specific CPUs. The memory policy defines in which node memory is allocated for the process.

  The get_mempolicy(2) system retrieves the NUMA policy of the calling process or of a memory address, depending on the setting of flags. The numactl(8) command provides a command line utility for manipulating the NUMA memory policy of a process using these system calls.

• The mbind(2) system call sets the NUMA memory policy for the pages in a specific range of a task's virtual address space.

Cpusets are designed to interact cleanly with other placement mechanisms. For example, a batch manager can use cpusets to control the CPU and memory placement of various jobs; while within each job, these other kernel mechanisms are used to manage placement in more detail. It is possible for a batch manager to change a job's cpuset placement while preserving the internel CPU affinity and NUMA memory placement policy, without requiring any special coding or awareness by the affected job.

Most jobs initialize their placement early in their timeslot, and jobs are rarely migrated until they have been running for a while. As long a a batch manager does not try to migrate a job at the same time as it is adjusting its own CPU or memory placement, there is little risk of interaction between cpusets and other kernel placement mechanisms.

The CPU and memory node placement constraints imposed by cpusets always override those of these other mechanisms.

Calls to the sched_setaffinity(2) system call automatically mask off CPUs that are not allowed by the affected task's cpuset. If a request results in all the CPUs being masked off, the call fails with errno set to EINVAL. If some of the requested CPUs are allowed by the task's cpuset, the call proceeds as if only the allowed CPUs were requested. The unallowed CPUs are silently ignored. If a task is moved to a different cpuset, or if the CPUs of a cpuset are changed, the CPU affinity of the affected task or tasks is lost. If a batch manager needs to preserve the CPU affinity of the tasks in a job that is being moved, it should use the sched_setaffinity(2) and sched_getaffinity(2) calls to save and restore each affected task's CPU affinity across the move, relative to the cpuset. The cpu_set_t mask data type supported by the C library for use with the CPU affinity calls is different from the libbitmask bitmasks used by libcpuset, so some coding will be required to convert between the two, in order to calculate and preserve cpuset relative CPU affinity.

Similar to CPU affinity, calls to modify a task's NUMA memory policy silently mask off requested memory nodes outside the task's allowed cpuset, and will fail if that results in requested an empty set of memory nodes. Unlike CPU affinity, the NUMA memory policy system calls to not support one task querying or modifying another task's policy. So the kernel automatically handles preserving cpuset relative NUMA memory policy when either a task is attached to a different cpuset, or a cpusets mems value setting is changed. If the old and new mems value sets have the same size, the cpuset relative offset of affected NUMA memory policies is preserved. If the new mems value is smaller, the old mems value relative offsets are folded onto the new mems value, modulo the size of the new mems. If the new mems value is larger, then just the first N nodes are used, where N is the size of the old mems value.

Cpusets are named, nested sets of CPUs and memory nodes. Each cpuset is represented by a directory in the cpuset virtual file system, normally mounted at /dev/cpuset, as described earlier.

The state of each cpuset is represented by small text files in the directory for the cpuset. These files may be read and written using traditional shell utilities such as cat(1) and echo(1) or using ordinary file access routines from programming languages, such as open(2), read(2), write (2) and close(2) from the C programming library.

To view the files in a cpuset that can be either read or written, perform the following commands:

% cd /dev/cpuset
% ls
cpu_exclusive  memory_migrate           memory_spread_page  notify_on_release
cpus           memory_pressure          memory_spread_slab  tasks
mem_exclusive  memory_pressure_enabled  mems

Descriptions of the files in the cpuset directory are, as follows:

A new file has been added to /proc file system, as follows:

There are two control fields used by the kernel scheduler and memory allocation mechanism to constrain scheduling and memory allocation to the allowed CPUs. These are two fields in the status file of each task, as follows:

There are several reasons why a tasks Cpus_allowed and Mems_allowed values may differ from the the values in the cpus and mems file for that are allowed in its current cpuset, as follows:

There is one other condition under which the confines of a cpuset may be violated. A few kernel critical internal memory allocation requests, marked GFP_ATOMIC, must be satisfied immediately. The kernel may drop some request or malfunction if one of these allocations fail. If such a request cannot be satisfied within the current task's cpuset, the kernel relaxes the cpuset, and looks for memory anywhere it can find it. It is better to violate the cpuset than stress the kernel operation.

New cpusets are created using the mkdir command at the shell (see“Using Cpusets at the Shell Prompt”) or via the C programming language (see Appendix A, “Cpuset Library Functions”). Old cpusets are removed using the rmdir(1) command. The above files are accessed using read(2) and write(2) system calls, or shell commands such as cat(1) and echo(1).

The CPUs and memory nodes in a given cpuset are always a subset of its parent. The root cpuset has all possible CPUs and memory nodes in the system. A cpuset may be exclusive (CPU or memory) only if its parent is similarly exclusive.

Each task has a pointer to a cpuset. Multiple tasks may reference the same cpuset. Requests by a task, using the sched_setaffinity (2) system call to include CPUs in its CPU affinity mask, and using the mbind(2) and set_mempolicy(2) system calls to include memory nodes in its memory policy, are both filtered through that task's cpuset, filtering out any CPUs or memory nodes not in that cpuset. The scheduler will not schedule a task on a CPU that is not allowed in its cpus_allowed vector and the kernel page allocator will not allocate a page on a node that is not allowed in the requesting task's mems_allowed vector.

If a cpuset is marked cpu_exclusive or mem_exclusive, no other cpuset, other than a direct ancestor or descendant, may share any of the same CPUs or memory nodes.

A cpuset that is cpu_exclusive has a scheduler (sched) domain associated with it. The sched domain consists of all CPUs in the current cpuset that are not part of any exclusive child cpusets. This ensures that the scheduler load balancing code only balances against the CPUs that are in the sched domain as described in“Cpuset File System Directories” and not all of the CPUs in the system. This removes any overhead due to load balancing code trying to pull tasks outside of the cpu_exclusive cpuset only to be prevented by the Cpus_allowed mask of the task.

A cpuset that is mem_exclusive restricts kernel allocations for page, buffer, and other data commonly shared by the kernel across multiple users. All cpusets, whether mem_exclusive or not, restrict allocations of memory for user space. This enables configuring a system so that several independent jobs can share common kernel data, such as file system pages, while isolating the user allocation of each job to its own cpuset. To do this, construct a large mem_exclusive cpuset to hold all the jobs, and construct child, non- mem_exclusive cpusets for each individual job. Only a small amount of typical kernel memory, such as requests from interrupt handlers, is allowed to be taken outside even a mem_exclusive cpuset.

If the notify_on_release flag is enabled (1) in a cpuset, whenever the last task in the cpuset leaves (exits or attaches to some other cpuset) and the last child cpuset of that cpuset is removed, the kernel runs the /sbin/cpuset_release_agent command, supplying the pathname (relative to the mount point of the cpuset file system) of the abandoned cpuset. This enables automatic removal of abandoned cpusets.

The default value of notify_on_release in the root cpuset at system boot is disabled (0). The default value of other cpusets at creation is the current value of their parents notify_on_release setting.

The /sbin/cpuset_release_agent command is invoked, with the name (/dev/cpuset relative path) of that cpuset in argv[1] argument. This supports automatic cleanup of abandoned cpusets.

The usual contents of the /sbin/cpuset_release_agent command is a simple shell script, as follows:

#!/bin/sh
rmdir /dev/cpuset/$1

By default, the notify_on_release flag is off (0). Newly created cpusets inherit their notify_on_release flag setting from their parent cpuset. As with other flag values, this flag can be changed by writing an ASCII number 0 or 1 (with optional trailing newline) into the file, to clear or set the flag, respectively.

The memory_pressure of a cpuset provides a simple per-cpuset metric of the rate that the tasks in a cpuset are attempting to free up in use memory on the nodes of the cpuset to satisfy additional memory requests. This enables batch managers, monitoring jobs running in dedicated cpusets, to efficiently detect what level of memory pressure that job is causing.

This is useful in the following situations:

This mechanism provides a very economical way for the batch manager to monitor a cpuset for signs of memory pressure. It is up to the batch manager or other user code to decide when to take action to alleviate memory pressures.

If the memory_pressure_enabled flag in the top cpuset is (0), that is, it is not set, the kernel does not compute this filter and the per-cpuset files memory_pressure contain the value zero (0).

If the memory_pressure_enabled flag in the top cpuset is set (1), the kernel computes this filter for each cpuset in the system, and the memory_pressure file for each cpuset reflects the recent rate of such low memory page allocation attempts by tasks in said cpuset.

Reading the memory_pressure file of a cpuset is very efficient. This mechanism allows batch schedulers to poll these files and detect jobs that are causing memory stress. They can then take action to avoid impacting the rest of the system with a job that is trying to aggressively exceed its allowed memory.

---

Note: Unless enabled by setting memory_pressure_enabled in the top cpuset, memory_pressure is not computed for any cpuset and always reads a value of zero.

---

A running average per cpuset has the following advantages:

A simple, per-cpuset digital filter is kept within the kernel and updated by any task attached to that cpuset if it enters the synchronous (direct) page reclaim code.

The per-cpuset memory_pressure file provides an integer number representing the recent (half-life of 10 seconds) rate of direct page reclaims caused by the tasks in the cpuset in units of reclaims attempted per second, times 1000.

The kernel computes this value using a single-pole, low-pass recursive digital filter coded with 32-bit integer arithmetic. The value decays at an exponential rate.

Given the simple 32-bit integer arithmetic used in the kernel to compute this value, this meter works best for reporting page reclaim rates between one per millisecond (msec) and one per 32 (approximate) seconds. At constant rates faster than one per msec, it reaches maximum at values just under 1,000,000. At constant rates between one per msec and one per second, it stabilizes to a value N*1000, where N is the rate of events per second. At constant rates between one per second and one per 32 seconds, it is choppy, moving up on the seconds that have an event, and then decaying until the next event. At rates slower than about one in 32 seconds, it decays all the way back to zero between each event.

There are two Boolean flag files per cpuset that control where the kernel allocates pages for the file system buffers and related in kernel data structures. They are called memory_spread_page and memory_spread_slab.

If the per-cpuset, memory_spread_page flag is set, the kernel spreads the file system buffers (page cache) evenly over all the nodes that the faulting task is allowed to use, instead of preferring to put those pages on the node where the task is running.

If the per-cpuset, memory_spread_slab flag is set, the kernel spreads some file system related slab caches, such as for inodes and directory entries, evenly over all the nodes that the faulting task is allowed to use, instead of preferring to put those pages on the node where the task is running.

The setting of these flags does not affect the anonymous data segment or stack segment pages of a task.

By default, both kinds of memory spreading are off, and memory pages are allocated on the node local to where the task is running, except perhaps as modified by the tasks NUMA memory policy or cpuset configuration. This is true as long as sufficient free memory pages are available.

When new cpusets are created, they inherit the memory spread settings of their parent.

Setting memory spreading causes allocations for the affected page or slab caches to ignore the task's NUMA memory policy and be spread instead. Tasks using mbind() or set_mempolicy() calls to set NUMA memory policies will not notice any change in these calls, as a result of their containing tasks memory spread settings. If memory spreading is turned off, the currently specified NUMA memory policy once again applies to memory page allocations.

Both memory_spread_page and memory_spread_slab are Boolean flag files. By default, they contain 0. This means the feature is off for the cpuset. If a 1 is written to this file, the named feature is turned on for the cpuset.

This memory placement policy is also known (in other contexts) as round-robin or interleave.

This policy can provide substantial improvements for jobs that need to place thread local data on the corresponding node, but that need to access large file system data sets that need to be spread across the several nodes in the job's cpuset in order to fit. Without this policy, especially for jobs that might have one thread reading in the data set, the memory allocation across the nodes in the jobs cpuset can become very uneven.

Normally, under the default setting of memory_migrate, once a page is allocated (given a physical page of main memory), that page stays on whatever node it was allocated, as long as it remains allocated, even if the cpuset's memory placement mems policy subsequently changes. The default setting has the memory_migrate flag disabled.

When memory migration is enabled in a cpuset, if the mems setting of the cpuset is changed, any memory page in use by any task in the cpuset that is on a memory node no longer allowed is migrated to a memory node that is allowed.

Also, if a task is moved into a cpuset with memory_migrate enabled, any memory pages it uses that were on memory nodes allowed in its previous cpuset, but which are not allowed in its new cpuset, are migrated to a memory node allowed in the new cpuset.

The relative placement of a migrated page within the cpuset is preserved during these migration operations if possible. For example, if the page was on the second valid node of the prior cpuset then the page will be placed on the second valid node of the new cpuset, if possible.

In order to maintain the cpuset relative position of pages, even pages on memory nodes allowed in both the old and new cpusets may be migrated. For example, if memory_migrate is enabled in a cpuset, and that cpuset's mems file is written, changing it from say memory nodes "4-7", to memory nodes "5-8", the following page migrations is done, in order, for all pages in the address space of tasks in that cpuset:

In this example, pages on any memory node other than "4 through 7" will not be migrated. The order in which nodes are handled in a migration is intentionally chosen so as to avoid migrating memory to a node until any migrations from that node have first been accomplished.

The mask format is used to represent CPU and memory node bitmasks in the /proc/pid/status file. It is hexadecimal, using ASCII characters "0" - "9" and "a" - "f". This format displays each 32-bit word in hex (zero filled), and for masks longer than one word, uses a comma separator between words. Words are displayed in big-endian order (most significant first). And hexadecimal digits within a word are also in big-endian order. The number of 32-bit words displayed is the minimum number needed to display all bits of the bitmask, based on the size of the bitmask. An example of the mask format is, as follows:

00000001                        # just bit 0 set
80000000,00000000,00000000      # just bit 95 set
00000001,00000000,00000000      # just bit 64 set
000000ff,00000000               # bits 32-39 set
00000000,000E3862               # bits 1,5,6,11-13,17-19 set

A mask with bits 0, 1, 2, 4, 8, 16, 32 and 64 set displays as 00000001,00000001,00010117. The first "1" is for bit 64, the second for bit 32, the third for bit 16, the fourth for bit 8, the fifth for bit 4, and the "7" is for bits 2, 1 and 0.

The list format is used to represent CPU and memory node bitmasks (sets of CPU and memory node numbers) in the /dev/cpuset file system. It is a comma separated list of CPU or memory node numbers and ranges of numbers, in ASCII decimal. An example of list format is, as follows:

0-4,9           # bits 0, 1, 2, 3, 4, and 9 set
0-3,7,12-15     # bits 0, 1, 2, 3, 7, 12, 13, 14, and 15 set

The permissions of a cpuset are determined by the permissions of the special files and directories in the cpuset file system, normally mounted at /dev/cpuset.

For example, a task can put itself in some other cpuset (than its current one) if it can write the tasks file (see “Cpuset File System Directories”) for that cpuset (requires execute permission on the encompassing directories and write permission on that tasks file).

An additional constraint is applied to requests to place some other task in a cpuset. One task may not attach another task to a cpuset unless it has permission to send that task a signal.

A task may create a child cpuset if it can access and write the parent cpuset directory. It can modify the CPUs or memory nodes in a cpuset if it can access that cpuset's directory (execute permissions on the encompassing directories) and write the corresponding cpus or mems file (see “Cpuset File System Directories”).

It should be noted, however, that changes to the CPUs of a cpuset do not apply to any task in that cpuset until the task is reattached to that cpuset. If a task can write the cpus file, it should also be able to write the tasks file and might be expected to have permission to reattach the tasks therein (equivalent to permission to send them a signal).

There is one minor difference between the manner in which cpuset path permissions are evaluated by libcpuset and the manner in which file system operation permissions are evaluated by direct system calls. System calls that operate on file pathnames, such as the open(2) system call, rely on direct kernel support for a task's current directory. Therefore, such calls can successfully operate on files in or below a task's current directory, even if the task lacks search permission on some ancestor directory. Calls in libcpuset that operate on cpuset pathnames, such as the cpuset_query() call, rely on libcpuset internal conversion of all cpuset pathnames to full, root-based paths. They cannot successfully operate on a cpuset unless the task has search permission on all ancestor directories, starting with the usual cpuset mount point (/dev/cpuset).

This section describes CPU scheduling and memory allocation for cpusets and covers these topics:

This section describes the use of cpusets using shell commands. For information on the cpuset(1) command line utility, see “Cpuset Command Line Utility”. For information on using the cpuset library functions, see Appendix A, “Cpuset Library Functions”.

When modifying the CPUs in a cpuset from the from the shell prompt, you must write the process ID (PID) of each task attached to that cpuset back into the cpuset's tasks file. Wwhen using the libcpuset API, use the cpuset_reattach() routine to perform this step. The reasons for performing this step are described in “Modifying the CPUs in a Cpuset and Kernel Processing”.

In this procedure, you will create a new cpuset called green, assign CPUs 2 and 3 and memory node 1 to the new cpuset, and start a subshell running in the cpuset.

To start a new job and contain it within a cpuset, perform the following steps:

To remove the cpuset green from the/dev/cpuset directory, perform the following:

The cpuset (1) command is used to create and destroy cpusets, to retrieve information about existing cpusets, and to attach processes to cpusets. The cpuset(1) command line utility is not essential to the use of cpusets. This utility provides an alternative that may be convenient for some uses. Users of earlier versions of cpusets may find this utility familiar, though the details of the options have changed in order reflect the current implementation of cpusets.

A cpuset is defined by a cpuset configuration file and a name. For a definition of the cpuset configuration file format, see “Cpuset Text Format”. The cpuset configuration file is used to list the CPUs and memory nodes that are members of the cpuset. It also contains any additional parameters required to define the cpuset. For more information on the cpuset configuration file, see “bootcpuset.conf File”.

This command automatically handles reattaching tasks to their cpuset whenever necessary, as described in the cpuset_reattach routine in Appendix A, “Cpuset Library Functions”.

The cpuset command accepts the following options:

Action Options (choose exactly one):

cpuset -F foo 2 bar 6 baz 4

Modifier Options (may be used in any combination):

-r, --recursive		When used with -p or -s option, applies to all descendants recursively of the named cpuset csname.
-I cmd, --invokecmd= cmd		When used with the -i option, the command cmd is invoked, with any optional unused arguments. The following example invokes an interactive subshell in cpuset foo: cpuset -i foo -I sh -- -i The next example invokes a second cpuset command in cpuset foo, which then displays the full cpuset path of foo: cpuset -i foo -I cpuset -- -w 0 Note: The double minus -- is needed to end option parsing by the initial cpuset command.
-f fname, --file= fname		Uses file named fname for command input or output stream, instead of stdin or stdout.
--move_tasks_from=csname1 --move_tasks_to= csname2		Move all tasks from cpuset csname1 to cpuset csname2. Retries up to ten times to move all tasks, in case it is racing against parallel attempts to fork or add tasks into cpuset csname1. Fails with non-zero exit status and an error message to stderr if unable to move all tasks out of csname1.

Help Option (overrides all other options):

Notes

The csname of "/" (slash) refers to the top cpuset, which encompasses all CPUs and memory nodes in the system. The csname of "." (dot) refers to the cpuset of the current task. If a csname begins with the "/" (slash) character, it is resolved relative to the top cpuset, otherwise it is resolved relative to the cpuset of the current task.

The 'command input stream' and 'command output stream' refer to the stdin (file descriptor 0) and stdout (file descriptor 1) of the command, unless the -f option is specified, in which case they refer to the file specified to -f option. Specifying the filename - to the -f option, as in -f -, is equivalent to not specifying the -f option at all.

Exactly one of the action options must be specified. They are, as follows:

-c, -m, -x, -d, -p, -a, -i, -w, -s, -R

The additional modifier options may be specified in any order. All modifier options are evaluated first, before the action option. If the help option is present, no action option is evaluated. The modifier options are, as follows:

-r, -I, -f

You can use the bootcpuset (8) command to create a “boot”cpuset during the system boot that you can use to restrict the default placement of almost all UNIX processes on your system. You can use the bootcpuset to reduce the interference of system processes with applications running on dedicated cpusets.

The default cpuset for the init process, classic UNIX daemons, and user login shells is the root cpuset that contains the entire system. For systems dedicated to running particular applications, it is better to restrict init, the kernel daemons, and login shells to a particular set of CPUs and memory nodes called the bootcpuset.

This section covers the following topics:

You can have a boot cpuset running the traditional daemon and server tasks and a second cpuset to hold interactive telnet, rlogin and/or secure shell (SSH) user sessions called the user cpuset.

Creating a user cpuset provides additional isolation between interactive user login sessions and essential system tasks. For example, a user process in the user cpuset consuming excessive CPU or memory resources will not seriously impact essential system services in the boot cpuset.

The following init script provides an example of how you can set up user cpuset. It runs as one of the last init scripts when the system is being booted.

If the system has a boot cpuset configured, the following script creates a second cpuset called user and place the sshd and xinetd daemons that create interactive login sessions in this new user cpuset. The user cpuset configuration is defined in the/etc/usercpuset.conf file.

To isolate the boot cpuset from the user cpuset, the set of cpus and mems values in the /etc/bootcpuset.conf file should not overlap the cpus and mems values in the /etc/usercpuset.conf file.

Instructions for using this script are included in comments within the script, as follows:

#! /bin/sh
# /etc/init.d/usercpuset
#
### BEGIN INIT INFO
# Provides: usercpuset
# Required-Start: sshd xinetd
# Required-Stop:
# Default-Start: 3 5
# Default-Stop: 0 1 2 6
# Description: Put login sessions in user cpuset
### END INIT INFO

# This init script creates a 'user' cpuset and places the login servers 
# (sshd and xinetd) in that cpuset, 
# 
# This script presumes you also have a 'boot' cpuset configured,
# and does nothing if you don't.
#
# By using this init script, one can isolate essential daemon and
# server tasks from interactive user login sessions in separate
# cpusets.
#
# To use this usercpuset init script:
#
#  1) Using your editor, create a file /etc/init.d/usercpuset
#     containing this script.
#  2) Run the command "insserv usercpuset" to insert this script
#     into the sequence of init scripts executed during system boot.
#  3) Create a /etc/usercpuset.conf, in the "Cpuset Text Format"
#     described in the libcpuset(3) man page, describing what CPUs
#     ("cpus") and memory nodes ("mems") are to be used by the
#     user cpuset.
#  4) Also configure and enable a boot cpuset, as documented in
#     the bootcpuset(8) man page.
#  5) Beginning with the next system reboot, login sessions under
#     either the SSH daemon (sshd) or xinetd (telnet, rlogin) will
#     be started in the 'user' cpuset, whileother daemons and
#     system services, including the consolelogin, will be in the
#     'boot' cpuset.
#  6) If you did not do both steps (3) and (4) above, then this
#     usercpuset script will do nothing, quietly, with no harm.

CPUSET_CMD=/usr/bin/cpuset

# Define the 'mems' and 'cpus' for user cpuset in this configuration file:
CONF=/etc/usercpuset.conf

USERCPUSET=/user

SSHD_PIDFILE=/var/run/sshd.init.pid

test -x $CPUSET_CMD || exit 5
test -r $CONF || exit 6

# Skip this if we didn't have a boot cpuset

test -d /dev/cpuset/boot || exit 7
if [ -f $CONF ]; then
        $CPUSET_CMD -c $USERCPUSET -f $CONF
fi
. /etc/rc.status

# sshd
$CPUSET_CMD -a $USERCPUSET < $SSHD_PIDFILE

# xinetd
echo $(pidof xinetd) | $CPUSET_CMD -a $USERCPUSET

Cpuset settings may be exported to and imported from text files using a text format representation of cpusets.

Permissions of files holding these text representations have no special significance to the implementation of cpusets. Rather, the permissions of the special cpuset files in the cpuset file system, normally mounted at /dev/cpuset, control reading and writing of and attaching to cpusets.

The text representation of cpusets is not essential to the use of cpusets. One can directly manipulate the special files in the cpuset file system. This text representation provides an alternative that may be convenient for some uses and a form for representing cpusets that users of earlier versions of cpusets will find familiar.

The exported cpuset text format has fewer directives than earlier Linux versions. Additional directives may be added in the future.

The cpuset text format supports one directive per line. Comments begin with the '#' character and extend to the end of line.

After stripping comments, the first white space separated token on each remaining line selects from the following possible directives:

Additional unnecessary tokens on a line are quietly ignored. Lines containing only comments and white space are ignored.

The token cpu is allowed for cpus and mem for mems. Matching is case insensitive.

See the libcpuset routines cpuset_import and cpuset_export to handle converting the internal struct cpuset representation of cpusets to (export) and from (import) this text representation.

For information on manipulating cpuset text files at the shell prompt or in shell scripts using the cpuset(1) command, see “Cpuset Command Line Utility”.

In order to minimize the impact of cpusets on critical kernel code, such as the scheduler, and due to the fact that the Linux kernel does not support one task updating the memory placement of another task directly, the impact on a task of changing its cpuset CPU or memory node placement or of changing to which cpuset a task is attached, is subtle and is described in the following paragraphs.

When a cpuset has its memory nodes modified, for each task attached to that cpuset, the next time that the kernel attempts to allocate a page of memory for a particular task, the kernel notices the change in the task's cpuset, and updates its per-task memory placement to remain within the new cpusets memory placement. If the task was using memory policy MPOL_BIND and the nodes to which it was bound overlaps with its new cpuset, the task continues to use whatever subset of MPOL_BIND nodes that are still allowed in the new cpuset. If the task was using MPOL_BIND and now none of its MPOL_BIND nodes are allowed in the new cpuset, the task is essentially treated as if it was MPOL_BIND bound to the new cpuset (even though its NUMA placement, as queried by the get_mempolicy () routine, does not change). If a task is moved from one cpuset to another, the kernel adjusts the task's memory placement, as above, the next time that the kernel attempts to allocate a page of memory for that task.

When a cpuset has its CPUs modified, each task using that cpuset does not change its behavior automatically. In order to minimize the impact on the critical kernel scheduling code, tasks continue to use their prior CPU placement until they are rebound to their cpuset by rewriting their PID to the tasks file of their cpuset. If a task is moved from one cpuset to another, its CPU placement is updated in the same way as if the task's PID is rewritten to the tasks file of its current cpuset.

In summary, the memory placement of a task whose cpuset is changed is automatically updated by the kernel, on the next allocation of a page for that task but the processor placement is not updated until that task's PID is rewritten to the tasks file of its cpuset. The delay in rebinding a task's memory placement is necessary because the kernel does not support one task changing memory placement of another task. The added user level step in rebinding a task's CPU placement is necessary to avoid impacting the scheduler code in the kernel with a check for changes in a task's processor placement.

Threading in a software application splits instructions into multiple streams so that multiple processors can act on them.

Hyper-Threading (HT) Technology, developed by Intel Corporation, provides thread-level parallelism on each processor, resulting in more efficient use of processor resources, higher processing throughput, and improved performance. One physical CPU can appear as two logical CPUs by having additional registers to overlap two instruction streams or a single processor can have dual-cores executing instructions in parallel.

In addition to their traditional use to control the placement of jobs on the CPUs and memory nodes of a system, cpusets also provide a convenient mechanism to control the use of Hyper-Threading Technology.

Some jobs achieve better performance by using both of the Hyper-Thread sides, A and B, of a processor core, and some run better by using just one of the sides, allowing the other side to idle.

Since each logical (Hyper-Threaded) processor in a core has a distinct CPU number, you can specify a cpuset that contains both sides of a processor core or a cpuset that contains just one side from a processor core.

Cpusets can be configured to include any combination of the logical CPUs in a system.

For example, the following cpuset configuration file called cpuset.cfg includes the A sides of an HT enabled system, along with all the memory, on the first 32 nodes (assuming 2 cores per node). The colon ':' prefixes the stride. The stride of '2' in this example means use every other logical CPU.

cpus 0-127:2    # the even numbered CPUs 0, 2, 4, ... 126
mems 0-63       # all memory nodes 0, 1, 2, ... 63

To create a cpuset called foo and run a job called bar in that cpuset, defined by the cpuset configuration file cpuset.cfg shown above, use the following commands:

cpuset -c /foo < cpuset.cfg
cpuset -i /foo -I bar

To specify both sides of the first 64 cores, use the following entry in your cpuset configuration file:

cpus 0-127

To specify just the B sides of the processor cores of an HT enabled system, use the following entry in your cpuset configuration file:

cpus 1-127:2

The examples above assume that the CPUs are uniformly numbered with the even numbers for the A side and odd numbers for the B side of the processors cores. This is usually the case, but not guaranteed. You can still place a job on a system that is not uniformly numbered. Currently, it involves a longer argument list to the cpus option, that is, you must explicitly list the desired CPUs.

If you are using a bootcpuset to keep other tasks confined, you do not need to create a separate cpuset with just the B side CPUs to avoid having some tasks running on the B sides of the processor cores. If there is no cpuset for the B sides of the processor cores, except the all encompassing root cpuset, and if only root can put tasks in the root cpuset, then no one other tasks can run on the B sides.

You can use the dplace(1) command to manage more detailed placement of job tasks within a cpuset. Since the dplace command numbering of CPUs is relative to the cpuset, it does not affect the dplace configuration. This is true in the case where the cpuset includes both sides of Hyper-Threaded cores, just one side of the Hyper-Threaded cores, or even is on a system that does not support hyperthreading.

Typically, the logical numbering of CPUs puts the even numbered CPUs on the A sides of processor cores and the odd numbered CPUs on the B sides. You can easily specify that only every other side is used using the stride suffix ":2", described above. If the CPU number range starts with an even number, the A sides of the processor cores are used. If the CPU range starts with an odd number, the be B sides of the processor cores are used.

To setup a job to run only on the A sides of the system's Hyper-Threaded cores and to ensure that no other tasks run on the B sides (they remain idle), perform the following steps:

If you use a bootcpuset to confine the traditional UNIX load processes, nothing will run on the other CPUs in the system, except when those CPUs are included in a cpuset to which a job has been assigned. These CPUs are of course in the root cpuset, however, this cpuset is normally only usable by a system administrator or batch scheduler with root permissions. This prevents any user without root permission from running a task on those CPUs, unless an administrator or service with root permission allows it. For more information, see “Boot Cpuset”.

A ps(1) or top(1) invocation will show a handful of threads on unused CPUs. These are kernel threads assigned to every CPU in support of user applications running on those CPUs to handle tasks such as asynchronous file system writes and task migration between CPUs. If no application is actually using a CPU, the kernel threads on that CPU will be almost always idle.

The programming model for this version of cpusets is an extension of the cpuset model provided on IRIX and and earlier versions of SGI Linux.

The flat name space of earlier cpuset versions on SGI systems is extended to a hierarchical name space. This will become more important as systems become larger. The name space remains visible to all tasks on a system. Once created, a cpuset remains in existence until it is deleted or until the system is rebooted, even if no tasks are currently running in that cpuset.

The key properties of a cpuset are its pathname, the list of which CPUs and memory nodes it contains, and whether the cpuset has exclusive rights to these resources.

Every task (process) in the system is attached to (running inside) a cpuset. Tasks inherit their parents cpuset attachment when forked. This binding of task to a cpuset can subsequently be changed, either by the task itself, or externally from another task, given sufficient authority.

Tasks have their CPU and memory placement constrained to whatever their containing cpuset allows. A cpuset may have exclusive rights to its CPUs and memory, which provides certain guarantees that other cpusets will not overlap.

At system boot, a top level root cpuset is created, which includes all CPUs and memory nodes on the system. The usual mount point of the cpuset file system and therefore the usual file system path to this root cpuset, is /dev/cpuset.

Optionally, a "boot" cpuset may be created, at /dev/cpuset/boot , to include typically just a one or a few CPUs and memory nodes. A typical use for a "boot" cpuset is to contain the general purpose UNIX daemons and login sessions, while reserving the rest of the system for running specific major applications on dedicated cpusets. For more information, see “Boot Cpuset”.

Moved tasks do not have the memory they might have allocated on their old nodes moved to the new nodes. On kernels that support such memory migration, use the [optional] cpuset_migrate to move allocated memory as well.

Cpusets have a permission structure which determines which users have rights to query, modify, and attach to any given cpuset. Rights are based on the hierarchical model provided by the underlying Linux 2.6 kernel cpuset file system.

To create a cpuset from within a C language application, your program obtains a handle to a new struct cpuset, sets the desired attributes via that handle, and issues a cpuset_create() call to create the desired cpuset and bind it to the specified name. Your program can also issue calls to list by name what cpusets exist, query their properties, move tasks between cpusets, list what tasks are currently attached to a cpuset, and delete cpusets.

The names of cpusets in this C library are always relative to the root cpuset mount point, typically /dev/cpuset. For more information on the libcpuset C language application programming interface (API) functions, see Appendix A, “Cpuset Library Functions”.

The Linux kernel implementation of cpusets sets errno to specify the reason for a failed system call that affects cpusets. These errno values are available when a cpuset library call fails. They can be displayed by shell commands used to directly manipulate files below the /dev/cpuset directory and can be displayed by the cpuset(1) command.

The possible errno settings and their meaning when set on a failed cpuset call are, as follows: