Provided by: manpages_6.03-2_all bug

NAME

       cpuset - confine processes to processor and memory node subsets

DESCRIPTION

       The  cpuset  filesystem  is  a pseudo-filesystem interface to the kernel cpuset mechanism,
       which is used to control the processor placement and memory placement of processes.  It is
       commonly mounted at /dev/cpuset.

       On  systems  with  kernels  compiled  with built in support for cpusets, all processes are
       attached to a cpuset, and cpusets are always present.  If a system supports cpusets,  then
       it will have the entry nodev cpuset in the file /proc/filesystems.  By mounting the cpuset
       filesystem (see the EXAMPLES section below), the administrator can configure  the  cpusets
       on a system to control the processor and memory placement of processes on that system.  By
       default, if the cpuset configuration on  a  system  is  not  modified  or  if  the  cpuset
       filesystem  is  not even mounted, then the cpuset mechanism, though present, has no effect
       on the system's behavior.

       A cpuset defines a list of CPUs and memory nodes.

       The CPUs of a system include all the logical processing  units  on  which  a  process  can
       execute,  including,  if  present,  multiple  processor  cores within a package and Hyper-
       Threads within a processor core.  Memory nodes include all distinct banks of main  memory;
       small  and  SMP systems typically have just one memory node that contains all the system's
       main memory, while NUMA (non-uniform memory access) systems have multiple memory nodes.

       Cpusets are represented as directories in a hierarchical pseudo-filesystem, where the  top
       directory in the hierarchy (/dev/cpuset) represents the entire system (all online CPUs and
       memory nodes) and any cpuset that is the  child  (descendant)  of  another  parent  cpuset
       contains  a  subset  of  that  parent's  CPUs and memory nodes.  The directories and files
       representing cpusets have normal filesystem permissions.

       Every process in the system belongs to exactly one cpuset.  A process is confined  to  run
       only  on  the  CPUs in the cpuset it belongs to, and to allocate memory only on the memory
       nodes in that cpuset.  When a process fork(2)s, the child process is placed  in  the  same
       cpuset  as  its parent.  With sufficient privilege, a process may be moved from one cpuset
       to another and the allowed CPUs and memory nodes of an existing cpuset may be changed.

       When the system begins booting, a single cpuset is defined  that  includes  all  CPUs  and
       memory  nodes  on  the  system,  and  all  processes  are in that cpuset.  During the boot
       process, or later during normal  system  operation,  other  cpusets  may  be  created,  as
       subdirectories  of  this  top  cpuset,  under the control of the system administrator, and
       processes may be placed in these other cpusets.

       Cpusets are integrated with the sched_setaffinity(2) scheduling affinity mechanism and the
       mbind(2) and set_mempolicy(2) memory-placement mechanisms in the kernel.  Neither of these
       mechanisms let a process make use of a CPU or memory node that  is  not  allowed  by  that
       process's  cpuset.   If  changes to a process's cpuset placement conflict with these other
       mechanisms, then cpuset placement is enforced even if  it  means  overriding  these  other
       mechanisms.   The kernel accomplishes this overriding by silently restricting the CPUs and
       memory nodes requested by  these  other  mechanisms  to  those  allowed  by  the  invoking
       process's  cpuset.   This  can  result  in  these  other  calls returning an error, if for
       example, such a call ends up requesting an empty set of CPUs or memory nodes,  after  that
       request is restricted to the invoking process's cpuset.

       Typically,  a  cpuset  is  used to manage the CPU and memory-node confinement for a set of
       cooperating processes such as a batch scheduler job, and these other mechanisms  are  used
       to manage the placement of individual processes or memory regions within that set or job.

FILES

       Each  directory  below /dev/cpuset represents a cpuset and contains a fixed set of pseudo-
       files describing the state of that cpuset.

       New cpusets are created using the mkdir(2) system  call  or  the  mkdir(1)  command.   The
       properties  of  a  cpuset,  such as its flags, allowed CPUs and memory nodes, and attached
       processes, are queried and modified by reading or writing to the appropriate file in  that
       cpuset's directory, as listed below.

       The  pseudo-files  in  each  cpuset directory are automatically created when the cpuset is
       created, as a result of the mkdir(2) invocation.  It is not possible to  directly  add  or
       remove these pseudo-files.

       A  cpuset  directory  that  contains  no  child  cpuset  directories,  and has no attached
       processes, can be removed using rmdir(2) or rmdir(1).  It is not necessary,  or  possible,
       to remove the pseudo-files inside the directory before removing it.

       The  pseudo-files  in  each  cpuset  directory  are  small text files that may be read and
       written using traditional shell utilities such as cat(1), and echo(1), or from  a  program
       by  using  file I/O library functions or system calls, such as open(2), read(2), write(2),
       and close(2).

       The pseudo-files in a cpuset directory represent internal kernel state and do not have any
       persistent image on disk.  Each of these per-cpuset files is listed and described below.

       tasks  List  of  the  process  IDs  (PIDs)  of  the processes in that cpuset.  The list is
              formatted as a series of ASCII decimal numbers, each  followed  by  a  newline.   A
              process  may  be  added to a cpuset (automatically removing it from the cpuset that
              previously contained it) by writing its PID to that cpuset's tasks  file  (with  or
              without a trailing newline).

              Warning:  only  one PID may be written to the tasks file at a time.  If a string is
              written that contains more than one PID, only the first one will be used.

       notify_on_release
              Flag (0 or 1).  If set (1), that cpuset will receive special handling after  it  is
              released, that is, after all processes cease using it (i.e., terminate or are moved
              to a different cpuset) and all child cpuset directories have been removed.  See the
              Notify On Release section, below.

       cpuset.cpus
              List  of  the  physical  numbers  of the CPUs on which processes in that cpuset are
              allowed to execute.  See List Format below for a description of the format of cpus.

              The CPUs allowed to a cpuset may be changed by writing a new list to its cpus file.

       cpuset.cpu_exclusive
              Flag (0 or 1).  If set (1), the cpuset has exclusive use of its CPUs (no sibling or
              cousin  cpuset  may  overlap  CPUs).   By  default, this is off (0).  Newly created
              cpusets also initially default this to off (0).

              Two cpusets are sibling cpusets if  they  share  the  same  parent  cpuset  in  the
              /dev/cpuset  hierarchy.   Two cpusets are cousin cpusets if neither is the ancestor
              of the other.  Regardless of the  cpu_exclusive  setting,  if  one  cpuset  is  the
              ancestor  of  another,  and if both of these cpusets have nonempty cpus, then their
              cpus must overlap, because the cpus of any cpuset are always a subset of  the  cpus
              of its parent cpuset.

       cpuset.mems
              List  of  memory  nodes  on  which processes in this cpuset are allowed to allocate
              memory.  See List Format below for a description of the format of mems.

       cpuset.mem_exclusive
              Flag (0 or 1).  If set (1), the cpuset has exclusive use of its  memory  nodes  (no
              sibling  or  cousin may overlap).  Also if set (1), the cpuset is a Hardwall cpuset
              (see below).  By default, this is off (0).  Newly created  cpusets  also  initially
              default this to off (0).

              Regardless  of the mem_exclusive setting, if one cpuset is the ancestor of another,
              then their memory nodes must overlap, because the memory nodes of  any  cpuset  are
              always a subset of the memory nodes of that cpuset's parent cpuset.

       cpuset.mem_hardwall (since Linux 2.6.26)
              Flag  (0  or  1).  If set (1), the cpuset is a Hardwall cpuset (see below).  Unlike
              mem_exclusive, there is no constraint on whether cpusets  marked  mem_hardwall  may
              have  overlapping memory nodes with sibling or cousin cpusets.  By default, this is
              off (0).  Newly created cpusets also initially default this to off (0).

       cpuset.memory_migrate (since Linux 2.6.16)
              Flag (0 or 1).  If set (1), then memory migration is enabled.  By default, this  is
              off (0).  See the Memory Migration section, below.

       cpuset.memory_pressure (since Linux 2.6.16)
              A  measure  of  how  much memory pressure the processes in this cpuset are causing.
              See the Memory Pressure section, below.  Unless memory_pressure_enabled is enabled,
              always  has  value  zero  (0).   This file is read-only.  See the WARNINGS section,
              below.

       cpuset.memory_pressure_enabled (since Linux 2.6.16)
              Flag (0 or 1).  This file is present only in the root cpuset, normally /dev/cpuset.
              If  set  (1),  the  memory_pressure calculations are enabled for all cpusets in the
              system.  By default, this is off (0).  See the Memory Pressure section, below.

       cpuset.memory_spread_page (since Linux 2.6.17)
              Flag (0 or 1).  If set (1), pages in the kernel page cache (filesystem buffers) are
              uniformly spread across the cpuset.  By default, this is off (0) in the top cpuset,
              and inherited from the parent cpuset in newly  created  cpusets.   See  the  Memory
              Spread section, below.

       cpuset.memory_spread_slab (since Linux 2.6.17)
              Flag  (0  or  1).   If  set (1), the kernel slab caches for file I/O (directory and
              inode structures) are uniformly spread across the cpuset.  By default, is  off  (0)
              in  the  top cpuset, and inherited from the parent cpuset in newly created cpusets.
              See the Memory Spread section, below.

       cpuset.sched_load_balance (since Linux 2.6.24)
              Flag (0 or 1).  If set (1, the default) the kernel will automatically load  balance
              processes  in that cpuset over the allowed CPUs in that cpuset.  If cleared (0) the
              kernel will avoid load balancing processes in this cpuset, unless some other cpuset
              with  overlapping  CPUs  has  its  sched_load_balance flag set.  See Scheduler Load
              Balancing, below, for further details.

       cpuset.sched_relax_domain_level (since Linux 2.6.26)
              Integer, between -1 and  a  small  positive  value.   The  sched_relax_domain_level
              controls  the  width  of the range of CPUs over which the kernel scheduler performs
              immediate rebalancing of runnable tasks  across  CPUs.   If  sched_load_balance  is
              disabled,  then the setting of sched_relax_domain_level does not matter, as no such
              load balancing is done.  If sched_load_balance is  enabled,  then  the  higher  the
              value  of  the  sched_relax_domain_level,  the  wider  the range of CPUs over which
              immediate load balancing is attempted.  See Scheduler Relax  Domain  Level,  below,
              for further details.

       In  addition  to  the above pseudo-files in each directory below /dev/cpuset, each process
       has a pseudo-file, /proc/<pid>/cpuset, that displays the  path  of  the  process's  cpuset
       directory relative to the root of the cpuset filesystem.

       Also  the  /proc/<pid>/status  file  for each process has four added lines, displaying the
       process's Cpus_allowed (on which CPUs it may be  scheduled)  and  Mems_allowed  (on  which
       memory  nodes  it  may obtain memory), in the two formats Mask Format and List Format (see
       below) as shown in the following example:

           Cpus_allowed:   ffffffff,ffffffff,ffffffff,ffffffff
           Cpus_allowed_list:     0-127
           Mems_allowed:   ffffffff,ffffffff
           Mems_allowed_list:     0-63

       The "allowed" fields were added in Linux 2.6.24; the "allowed_list" fields were  added  in
       Linux 2.6.26.

EXTENDED CAPABILITIES

       In  addition  to  controlling  which  cpus  and  mems a process is allowed to use, cpusets
       provide the following extended capabilities.

   Exclusive cpusets
       If a cpuset is marked cpu_exclusive or mem_exclusive, no other cpuset, other than a direct
       ancestor or descendant, may share any of the same CPUs or memory nodes.

       A  cpuset  that  is  mem_exclusive restricts kernel allocations for buffer cache pages and
       other internal kernel data pages commonly shared by the kernel across multiple users.  All
       cpusets,  whether  mem_exclusive  or  not,  restrict allocations of memory for user space.
       This enables configuring a system so that several independent jobs can share common kernel
       data, while isolating each job's user allocation in its own cpuset.  To do this, construct
       a large mem_exclusive cpuset to hold all the jobs, and construct child,  non-mem_exclusive
       cpusets  for  each individual job.  Only a small amount of kernel memory, such as requests
       from interrupt handlers,  is  allowed  to  be  placed  on  memory  nodes  outside  even  a
       mem_exclusive cpuset.

   Hardwall
       A  cpuset  that  has  mem_exclusive  or mem_hardwall set is a hardwall cpuset.  A hardwall
       cpuset restricts kernel allocations for page, buffer, and other data  commonly  shared  by
       the  kernel  across  multiple  users.   All  cpusets,  whether  hardwall  or not, restrict
       allocations of memory for user space.

       This enables configuring a system so that several independent jobs can share common kernel
       data,  such  as  filesystem  pages,  while isolating each job's user allocation in its own
       cpuset.  To do this, construct a large hardwall cpuset to hold all the jobs, and construct
       child cpusets for each individual job which are not hardwall cpusets.

       Only a small amount of kernel memory, such as requests from interrupt handlers, is allowed
       to be taken outside even a hardwall cpuset.

   Notify on release
       If the notify_on_release flag is enabled (1) in a cpuset, then whenever the  last  process
       in the cpuset leaves (exits or attaches to some other cpuset) and the last child cpuset of
       that cpuset is removed,  the  kernel  will  run  the  command  /sbin/cpuset_release_agent,
       supplying  the  pathname  (relative  to  the  mount point of the cpuset filesystem) of the
       abandoned cpuset.  This enables automatic removal of abandoned cpusets.

       The default value of notify_on_release in the root cpuset at system boot is disabled  (0).
       The  default  value  of  other  cpusets at creation is the current value of their parent's
       notify_on_release setting.

       The command /sbin/cpuset_release_agent is invoked, with  the  name  (/dev/cpuset  relative
       path) of the to-be-released cpuset in argv[1].

       The usual contents of the command /sbin/cpuset_release_agent is simply the shell script:

           #!/bin/sh
           rmdir /dev/cpuset/$1

       As  with other flag values below, this flag can be changed by writing an ASCII number 0 or
       1 (with optional trailing newline) into the file, to clear or set the flag, respectively.

   Memory pressure
       The memory_pressure of a cpuset provides a simple per-cpuset running average of  the  rate
       that the processes in a cpuset are attempting to free up in-use memory on the nodes of the
       cpuset to satisfy additional memory requests.

       This enables batch managers that are monitoring  jobs  running  in  dedicated  cpusets  to
       efficiently detect what level of memory pressure that job is causing.

       This is useful both on tightly managed systems running a wide mix of submitted jobs, which
       may choose to terminate or reprioritize jobs that are  trying  to  use  more  memory  than
       allowed  on  the  nodes  assigned  them, and with tightly coupled, long-running, massively
       parallel  scientific  computing  jobs  that  will  dramatically  fail  to  meet   required
       performance goals if they start to use more memory than allowed to them.

       This  mechanism  provides  a very economical way for the batch manager to monitor a cpuset
       for signs of memory pressure.  It's up to the batch manager or other user code  to  decide
       what action to take if it detects signs of memory pressure.

       Unless   memory   pressure   calculation   is   enabled   by   setting   the   pseudo-file
       /dev/cpuset/cpuset.memory_pressure_enabled, it is not computed for any cpuset,  and  reads
       from  any  memory_pressure  always  return zero, as represented by the ASCII string "0\n".
       See the WARNINGS section, below.

       A per-cpuset, running average is employed for the following reasons:

       •  Because this meter is per-cpuset rather than per-process or per virtual memory  region,
          the  system load imposed by a batch scheduler monitoring this metric is sharply reduced
          on large systems, because a scan of the tasklist can be avoided on each set of queries.

       •  Because this meter is a running average rather than an accumulating  counter,  a  batch
          scheduler  can detect memory pressure with a single read, instead of having to read and
          accumulate results for a period of time.

       •  Because this meter is per-cpuset rather  than  per-process,  the  batch  scheduler  can
          obtain  the key information—memory pressure in a cpuset—with a single read, rather than
          having to query and accumulate results over  all  the  (dynamically  changing)  set  of
          processes in the cpuset.

       The  memory_pressure  of  a  cpuset is calculated using a per-cpuset simple digital filter
       that is kept within the kernel.  For each cpuset, this filter tracks the  recent  rate  at
       which processes attached to that cpuset enter the kernel direct reclaim code.

       The  kernel direct reclaim code is entered whenever a process has to satisfy a memory page
       request by first finding some other  page  to  repurpose,  due  to  lack  of  any  readily
       available already free pages.  Dirty filesystem pages are repurposed by first writing them
       to disk.  Unmodified filesystem buffer pages  are  repurposed  by  simply  dropping  them,
       though if that page is needed again, it will have to be reread from disk.

       The  cpuset.memory_pressure file provides an integer number representing the recent (half-
       life of 10 seconds) rate of entries to the direct reclaim code caused by  any  process  in
       the cpuset, in units of reclaims attempted per second, times 1000.

   Memory spread
       There  are two Boolean flag files per cpuset that control where the kernel allocates pages
       for the filesystem buffers  and  related  in-kernel  data  structures.   They  are  called
       cpuset.memory_spread_page and cpuset.memory_spread_slab.

       If the per-cpuset Boolean flag file cpuset.memory_spread_page is set, then the kernel will
       spread the filesystem buffers (page cache) evenly over all the  nodes  that  the  faulting
       process  is allowed to use, instead of preferring to put those pages on the node where the
       process is running.

       If the per-cpuset Boolean flag file cpuset.memory_spread_slab is set, then the kernel will
       spread  some  filesystem-related  slab  caches,  such  as  those  for inodes and directory
       entries, evenly over all the nodes that the faulting process is allowed to use, instead of
       preferring to put those pages on the node where the process is running.

       The  setting of these flags does not affect the data segment (see brk(2)) or stack segment
       pages of a process.

       By default, both kinds of memory spreading are off and  the  kernel  prefers  to  allocate
       memory  pages  on the node local to where the requesting process is running.  If that node
       is not allowed by the process's NUMA memory policy or cpuset configuration or if there are
       insufficient  free  memory  pages on that node, then the kernel looks for the nearest node
       that is allowed and has sufficient free memory.

       When new cpusets are created, they inherit the memory spread settings of their parent.

       Setting memory spreading causes allocations for the affected page or slab caches to ignore
       the  process's  NUMA  memory  policy  and be spread instead.  However, the effect of these
       changes in memory placement caused by cpuset-specified memory spreading is hidden from the
       mbind(2)  or  set_mempolicy(2) calls.  These two NUMA memory policy calls always appear to
       behave as if no cpuset-specified memory spreading is in effect, even if it is.  If  cpuset
       memory  spreading  is  subsequently  turned  off,  the  NUMA  memory  policy most recently
       specified by these calls is automatically reapplied.

       Both cpuset.memory_spread_page and cpuset.memory_spread_slab are Boolean flag  files.   By
       default,  they  contain "0", meaning that the feature is off for that cpuset.  If a "1" is
       written to that file, that turns the named feature on.

       Cpuset-specified memory spreading behaves similarly to what is known (in  other  contexts)
       as round-robin or interleave memory placement.

       Cpuset-specified  memory  spreading  can  provide substantial performance improvements for
       jobs that:

       •  need to place thread-local data on memory nodes close to the CPUs which are running the
          threads that most frequently access that data; but also

       •  need  to  access  large  filesystem data sets that must to be spread across the several
          nodes in the job's cpuset in order to fit.

       Without this policy, the memory allocation across the nodes in the job's cpuset can become
       very  uneven,  especially  for  jobs  that might have just a single thread initializing or
       reading in the data set.

   Memory migration
       Normally, under the default setting (disabled) of cpuset.memory_migrate, once  a  page  is
       allocated (given a physical page of main memory), then that page stays on whatever node it
       was allocated, so long as it remains allocated,  even  if  the  cpuset's  memory-placement
       policy mems subsequently changes.

       When  memory  migration  is  enabled  in  a  cpuset,  if the mems setting of the cpuset is
       changed, then any memory page in use by any process in the cpuset that is on a memory node
       that is no longer allowed will be migrated to a memory node that is allowed.

       Furthermore,  if  a process is moved into a cpuset with memory_migrate enabled, any memory
       pages it uses that were on memory nodes allowed in its previous cpuset, but which are  not
       allowed in its new cpuset, will be migrated to a memory node allowed in the new cpuset.

       The  relative  placement  of  a  migrated page within the cpuset is preserved during these
       migration operations if possible.  For example, if the page was on the second  valid  node
       of  the  prior  cpuset,  then  the page will be placed on the second valid node of the new
       cpuset, if possible.

   Scheduler load balancing
       The kernel scheduler automatically load balances processes.  If one CPU is  underutilized,
       the  kernel will look for processes on other more overloaded CPUs and move those processes
       to the underutilized CPU, within the constraints of such placement mechanisms  as  cpusets
       and sched_setaffinity(2).

       The algorithmic cost of load balancing and its impact on key shared kernel data structures
       such as the process list increases more than  linearly  with  the  number  of  CPUs  being
       balanced.  For example, it costs more to load balance across one large set of CPUs than it
       does to balance across two smaller sets of CPUs, each of half the size of the larger  set.
       (The  precise  relationship between the number of CPUs being balanced and the cost of load
       balancing depends on implementation details of the  kernel  process  scheduler,  which  is
       subject to change over time, as improved kernel scheduler algorithms are implemented.)

       The  per-cpuset  flag  sched_load_balance  provides a mechanism to suppress this automatic
       scheduler load balancing in cases where it is not needed and  suppressing  it  would  have
       worthwhile performance benefits.

       By default, load balancing is done across all CPUs, except those marked isolated using the
       kernel boot time "isolcpus=" argument.  (See  Scheduler  Relax  Domain  Level,  below,  to
       change this default.)

       This  default  load  balancing  across  all  CPUs  is not well suited to the following two
       situations:

       •  On large systems, load balancing across many CPUs  is  expensive.   If  the  system  is
          managed  using  cpusets  to  place independent jobs on separate sets of CPUs, full load
          balancing is unnecessary.

       •  Systems supporting real-time on some CPUs need to minimize  system  overhead  on  those
          CPUs, including avoiding process load balancing if that is not needed.

       When  the per-cpuset flag sched_load_balance is enabled (the default setting), it requests
       load balancing across all the CPUs in that  cpuset's  allowed  CPUs,  ensuring  that  load
       balancing  can  move a process (not otherwise pinned, as by sched_setaffinity(2)) from any
       CPU in that cpuset to any other.

       When the per-cpuset flag sched_load_balance is disabled, then  the  scheduler  will  avoid
       load  balancing  across  the CPUs in that cpuset, except in so far as is necessary because
       some overlapping cpuset has sched_load_balance enabled.

       So, for example, if the top cpuset has  the  flag  sched_load_balance  enabled,  then  the
       scheduler  will  load  balance  across all CPUs, and the setting of the sched_load_balance
       flag in other cpusets has no effect, as we're already fully load balancing.

       Therefore in the above two situations, the flag sched_load_balance should be  disabled  in
       the top cpuset, and only some of the smaller, child cpusets would have this flag enabled.

       When  doing this, you don't usually want to leave any unpinned processes in the top cpuset
       that might  use  nontrivial  amounts  of  CPU,  as  such  processes  may  be  artificially
       constrained  to  some subset of CPUs, depending on the particulars of this flag setting in
       descendant cpusets.  Even if such a process could use spare CPU cycles in some other CPUs,
       the  kernel scheduler might not consider the possibility of load balancing that process to
       the underused CPU.

       Of course, processes pinned to a particular CPU can be left  in  a  cpuset  that  disables
       sched_load_balance as those processes aren't going anywhere else anyway.

   Scheduler relax domain level
       The  kernel  scheduler  performs  immediate  load balancing whenever a CPU becomes free or
       another task becomes runnable.  This load balancing works to ensure that as many  CPUs  as
       possible  are  usefully  employed  running  tasks.  The kernel also performs periodic load
       balancing   off   the   software   clock   described   in   time(7).    The   setting   of
       sched_relax_domain_level  applies  only  to  immediate  load balancing.  Regardless of the
       sched_relax_domain_level setting, periodic load  balancing  is  attempted  over  all  CPUs
       (unless  disabled  by turning off sched_load_balance.)  In any case, of course, tasks will
       be  scheduled  to  run  only  on  CPUs  allowed  by   their   cpuset,   as   modified   by
       sched_setaffinity(2) system calls.

       On  small  systems, such as those with just a few CPUs, immediate load balancing is useful
       to improve system interactivity and to minimize wasteful idle CPU cycles.   But  on  large
       systems,  attempting  immediate  load  balancing across a large number of CPUs can be more
       costly than it is worth, depending on the particular performance  characteristics  of  the
       job mix and the hardware.

       The  exact  meaning of the small integer values of sched_relax_domain_level will depend on
       internal implementation details of the  kernel  scheduler  code  and  on  the  non-uniform
       architecture  of  the  hardware.   Both  of these will evolve over time and vary by system
       architecture and kernel version.

       As of this writing, when this capability  was  introduced  in  Linux  2.6.26,  on  certain
       popular  architectures, the positive values of sched_relax_domain_level have the following
       meanings.

       1      Perform immediate load balancing across Hyper-Thread siblings on the same core.
       2      Perform immediate load balancing across other cores in the same package.
       3      Perform immediate load balancing across other CPUs on the same node or blade.
       4      Perform immediate load balancing across over several (implementation detail)  nodes
              [On NUMA systems].
       5      Perform immediate load balancing across over all CPUs in system [On NUMA systems].

       The  sched_relax_domain_level  value of zero (0) always means don't perform immediate load
       balancing, hence that load balancing is done only periodically, not immediately when a CPU
       becomes available or another task becomes runnable.

       The  sched_relax_domain_level  value of minus one (-1) always means use the system default
       value.  The system default value can vary by architecture and kernel version.  This system
       default value can be changed by kernel boot-time "relax_domain_level=" argument.

       In    the    case    of    multiple    overlapping    cpusets   which   have   conflicting
       sched_relax_domain_level values, then the highest such value applies to all CPUs in any of
       the  overlapping  cpusets.   In  such cases, the value minus one (-1) is the lowest value,
       overridden by any other value, and the value zero (0) is the next lowest value.

FORMATS

       The following formats are used to represent sets of CPUs and memory nodes.

   Mask format
       The  Mask  Format  is  used  to  represent  CPU  and  memory-node   bit   masks   in   the
       /proc/<pid>/status file.

       This format displays each 32-bit word in hexadecimal (using ASCII characters "0" - "9" and
       "a" - "f"); words are filled with leading zeros, if required.  For masks longer  than  one
       word,  a  comma separator is used between words.  Words are displayed in big-endian order,
       which has the most significant bit first.  The hex digits within a word are also  in  big-
       endian order.

       The  number  of 32-bit words displayed is the minimum number needed to display all bits of
       the bit mask, based on the size of the bit mask.

       Examples of the Mask Format:

           00000001                        # just bit 0 set
           40000000,00000000,00000000      # just bit 94 set
           00000001,00000000,00000000      # just bit 64 set
           000000ff,00000000               # bits 32-39 set
           00000000,000e3862               # 1,5,6,11-13,17-19 set

       A mask with bits 0, 1, 2, 4, 8, 16, 32, and 64 set displays as:

           00000001,00000001,00010117

       The first "1" is for bit 64, the second for bit 32, the third for bit 16, the  fourth  for
       bit 8, the fifth for bit 4, and the "7" is for bits 2, 1, and 0.

   List format
       The  List Format for cpus and mems is a comma-separated list of CPU or memory-node numbers
       and ranges of numbers, in ASCII decimal.

       Examples of the List Format:

           0-4,9           # bits 0, 1, 2, 3, 4, and 9 set
           0-2,7,12-14     # bits 0, 1, 2, 7, 12, 13, and 14 set

RULES

       The following rules apply to each cpuset:

       •  Its CPUs and memory nodes must be a (possibly equal) subset of its parent's.

       •  It can be marked cpu_exclusive only if its parent is.

       •  It can be marked mem_exclusive only if its parent is.

       •  If it is cpu_exclusive, its CPUs may not overlap any sibling.

       •  If it is mem_exclusive, its memory nodes may not overlap any sibling.

PERMISSIONS

       The permissions of a cpuset are determined by  the  permissions  of  the  directories  and
       pseudo-files in the cpuset filesystem, normally mounted at /dev/cpuset.

       For  instance,  a process can put itself in some other cpuset (than its current one) if it
       can write the tasks file for  that  cpuset.   This  requires  execute  permission  on  the
       encompassing directories and write permission on the tasks file.

       An  additional  constraint is applied to requests to place some other process in a cpuset.
       One process may not attach another to a cpuset unless it would  have  permission  to  send
       that process a signal (see kill(2)).

       A  process  may  create  a  child  cpuset  if  it  can  access and write the parent cpuset
       directory.  It can modify the CPUs or memory nodes in a  cpuset  if  it  can  access  that
       cpuset's  directory  (execute permissions on the each of the parent directories) and write
       the corresponding cpus or mems file.

       There is one minor difference between the manner in which these permissions are  evaluated
       and the manner in which normal filesystem operation permissions are evaluated.  The kernel
       interprets relative pathnames starting at a process's current working directory.  Even  if
       one  is  operating  on  a  cpuset file, relative pathnames are interpreted relative to the
       process's current working directory, not relative to the process's  current  cpuset.   The
       only  ways  that  cpuset  paths  relative to a process's current cpuset can be used are if
       either the process's current working directory is  its  cpuset  (it  first  did  a  cd  or
       chdir(2)  to  its cpuset directory beneath /dev/cpuset, which is a bit unusual) or if some
       user code converts the relative cpuset path to a full filesystem path.

       In theory, this means that user code should  specify  cpusets  using  absolute  pathnames,
       which  requires  knowing  the  mount  point  of  the  cpuset  filesystem (usually, but not
       necessarily, /dev/cpuset).  In practice, all user level code that this author is aware  of
       simply  assumes  that  if  the  cpuset  filesystem  is  mounted,  then  it  is  mounted at
       /dev/cpuset.  Furthermore, it is common practice for carefully written user code to verify
       the  presence  of  the  pseudo-file  /dev/cpuset/tasks  in order to verify that the cpuset
       pseudo-filesystem is currently mounted.

WARNINGS

   Enabling memory_pressure
       By default, the per-cpuset file cpuset.memory_pressure always contains zero  (0).   Unless
       this     feature     is     enabled     by     writing     "1"    to    the    pseudo-file
       /dev/cpuset/cpuset.memory_pressure_enabled,  the  kernel  does  not   compute   per-cpuset
       memory_pressure.

   Using the echo command
       When  using  the  echo  command  at the shell prompt to change the values of cpuset files,
       beware that the built-in echo command in some shells does not display an error message  if
       the write(2) system call fails.  For example, if the command:

           echo 19 > cpuset.mems

       failed  because memory node 19 was not allowed (perhaps the current system does not have a
       memory node 19), then the echo command might not display any error.  It is better  to  use
       the  /bin/echo  external  command  to  change  cpuset  file settings, as this command will
       display write(2) errors, as in the example:

           /bin/echo 19 > cpuset.mems
           /bin/echo: write error: Invalid argument

EXCEPTIONS

   Memory placement
       Not all allocations of system  memory  are  constrained  by  cpusets,  for  the  following
       reasons.

       If  hot-plug functionality is used to remove all the CPUs that are currently assigned to a
       cpuset, then the kernel will  automatically  update  the  cpus_allowed  of  all  processes
       attached to CPUs in that cpuset to allow all CPUs.  When memory hot-plug functionality for
       removing memory nodes is available, a similar exception is  expected  to  apply  there  as
       well.   In general, the kernel prefers to violate cpuset placement, rather than starving a
       process that has had all its allowed CPUs or memory nodes taken offline.  User code should
       reconfigure  cpusets  to refer only to online CPUs and memory nodes when using hot-plug to
       add or remove such resources.

       A few kernel-critical, internal memory-allocation requests,  marked  GFP_ATOMIC,  must  be
       satisfied  immediately.   The  kernel may drop some request or malfunction if one of these
       allocations fail.  If such a request cannot be  satisfied  within  the  current  process's
       cpuset,  then  we  relax  the  cpuset,  and look for memory anywhere we can find it.  It's
       better to violate the cpuset than stress the kernel.

       Allocations of memory requested by kernel drivers while processing an interrupt  lack  any
       relevant process context, and are not confined by cpusets.

   Renaming cpusets
       You  can  use  the  rename(2)  system  call  to  rename  cpusets.  Only simple renaming is
       supported; that is, changing the name of a cpuset directory is  permitted,  but  moving  a
       directory into a different directory is not permitted.

ERRORS

       The  Linux  kernel implementation of cpusets sets errno to specify the reason for a failed
       system call affecting cpusets.

       The possible errno settings and their meaning when set on a  failed  cpuset  call  are  as
       listed below.

       E2BIG  Attempted  a  write(2)  on  a  special  cpuset  file with a length larger than some
              kernel-determined upper limit on the length of such writes.

       EACCES Attempted to write(2) the process ID (PID) of a process to a cpuset tasks file when
              one lacks permission to move that process.

       EACCES Attempted  to  add, using write(2), a CPU or memory node to a cpuset, when that CPU
              or memory node was not already in its parent.

       EACCES Attempted to set, using write(2), cpuset.cpu_exclusive or cpuset.mem_exclusive on a
              cpuset whose parent lacks the same setting.

       EACCES Attempted to write(2) a cpuset.memory_pressure file.

       EACCES Attempted to create a file in a cpuset directory.

       EBUSY  Attempted to remove, using rmdir(2), a cpuset with attached processes.

       EBUSY  Attempted to remove, using rmdir(2), a cpuset with child cpusets.

       EBUSY  Attempted  to  remove a CPU or memory node from a cpuset that is also in a child of
              that cpuset.

       EEXIST Attempted to create, using mkdir(2), a cpuset that already exists.

       EEXIST Attempted to rename(2) a cpuset to a name that already exists.

       EFAULT Attempted to read(2) or write(2) a cpuset file using a buffer that is  outside  the
              writing processes accessible address space.

       EINVAL Attempted  to  change  a  cpuset,  using  write(2),  in  a way that would violate a
              cpu_exclusive or mem_exclusive attribute of that cpuset or any of its siblings.

       EINVAL Attempted to write(2) an empty cpuset.cpus or cpuset.mems list to  a  cpuset  which
              has attached processes or child cpusets.

       EINVAL Attempted to write(2) a cpuset.cpus or cpuset.mems list which included a range with
              the second number smaller than the first number.

       EINVAL Attempted to write(2) a cpuset.cpus or cpuset.mems list which included  an  invalid
              character in the string.

       EINVAL Attempted  to write(2) a list to a cpuset.cpus file that did not include any online
              CPUs.

       EINVAL Attempted to write(2) a list to a cpuset.mems file that did not include any  online
              memory nodes.

       EINVAL Attempted  to  write(2) a list to a cpuset.mems file that included a node that held
              no memory.

       EIO    Attempted to write(2) a string to a cpuset tasks file that does not begin  with  an
              ASCII decimal integer.

       EIO    Attempted to rename(2) a cpuset into a different directory.

       ENAMETOOLONG
              Attempted  to  read(2)  a  /proc/<pid>/cpuset file for a cpuset path that is longer
              than the kernel page size.

       ENAMETOOLONG
              Attempted to create, using mkdir(2), a cpuset whose base directory name  is  longer
              than 255 characters.

       ENAMETOOLONG
              Attempted  to  create,  using mkdir(2), a cpuset whose full pathname, including the
              mount point (typically "/dev/cpuset/") prefix, is longer than 4095 characters.

       ENODEV The cpuset was removed by another process at  the  same  time  as  a  write(2)  was
              attempted on one of the pseudo-files in the cpuset directory.

       ENOENT Attempted  to  create,  using  mkdir(2),  a  cpuset in a parent cpuset that doesn't
              exist.

       ENOENT Attempted to access(2) or open(2) a nonexistent file in a cpuset directory.

       ENOMEM Insufficient memory is available within the kernel;  can  occur  on  a  variety  of
              system  calls  affecting  cpusets,  but  only  if  the system is extremely short of
              memory.

       ENOSPC Attempted to write(2) the process ID (PID) of a process to a cpuset tasks file when
              the cpuset had an empty cpuset.cpus or empty cpuset.mems setting.

       ENOSPC Attempted  to write(2) an empty cpuset.cpus or cpuset.mems setting to a cpuset that
              has tasks attached.

       ENOTDIR
              Attempted to rename(2) a nonexistent cpuset.

       EPERM  Attempted to remove a file from a cpuset directory.

       ERANGE Specified a cpuset.cpus or cpuset.mems list to the kernel which included  a  number
              too large for the kernel to set in its bit masks.

       ESRCH  Attempted  to  write(2)  the  process ID (PID) of a nonexistent process to a cpuset
              tasks file.

VERSIONS

       Cpusets appeared in Linux 2.6.12.

NOTES

       Despite its name, the pid parameter is actually a thread ID, and each thread in a threaded
       group  can be attached to a different cpuset.  The value returned from a call to gettid(2)
       can be passed in the argument pid.

BUGS

       cpuset.memory_pressure cpuset files can be opened for writing,  creation,  or  truncation,
       but  then  the  write(2)  fails  with errno set to EACCES, and the creation and truncation
       options on open(2) have no effect.

EXAMPLES

       The following examples  demonstrate  querying  and  setting  cpuset  options  using  shell
       commands.

   Creating and attaching to a cpuset.
       To create a new cpuset and attach the current command shell to it, the steps are:

       (1)  mkdir /dev/cpuset (if not already done)
       (2)  mount -t cpuset none /dev/cpuset (if not already done)
       (3)  Create the new cpuset using mkdir(1).
       (4)  Assign CPUs and memory nodes to the new cpuset.
       (5)  Attach the shell to the new cpuset.

       For  example,  the  following  sequence  of commands will set up a cpuset named "Charlie",
       containing just CPUs 2 and 3, and memory node 1, and then attach the current shell to that
       cpuset.

           $ mkdir /dev/cpuset
           $ mount -t cpuset cpuset /dev/cpuset
           $ cd /dev/cpuset
           $ mkdir Charlie
           $ cd Charlie
           $ /bin/echo 2-3 > cpuset.cpus
           $ /bin/echo 1 > cpuset.mems
           $ /bin/echo $$ > tasks
           # The current shell is now running in cpuset Charlie
           # The next line should display '/Charlie'
           $ cat /proc/self/cpuset

   Migrating a job to different memory nodes.
       To  migrate a job (the set of processes attached to a cpuset) to different CPUs and memory
       nodes in the system, including moving the memory pages currently allocated  to  that  job,
       perform the following steps.

       (1)  Let's  say we want to move the job in cpuset alpha (CPUs 4–7 and memory nodes 2–3) to
            a new cpuset beta (CPUs 16–19 and memory nodes 8–9).
       (2)  First create the new cpuset beta.
       (3)  Then allow CPUs 16–19 and memory nodes 8–9 in beta.
       (4)  Then enable memory_migration in beta.
       (5)  Then move each process from alpha to beta.

       The following sequence of commands accomplishes this.

           $ cd /dev/cpuset
           $ mkdir beta
           $ cd beta
           $ /bin/echo 16-19 > cpuset.cpus
           $ /bin/echo 8-9 > cpuset.mems
           $ /bin/echo 1 > cpuset.memory_migrate
           $ while read i; do /bin/echo $i; done < ../alpha/tasks > tasks

       The above should move any processes in alpha  to  beta,  and  any  memory  held  by  these
       processes on memory nodes 2–3 to memory nodes 8–9, respectively.

       Notice that the last step of the above sequence did not do:

           $ cp ../alpha/tasks tasks

       The  while  loop, rather than the seemingly easier use of the cp(1) command, was necessary
       because only one process PID at a time may be written to the tasks file.

       The same effect (writing one PID at a time) as the while loop  can  be  accomplished  more
       efficiently,  in  fewer  keystrokes  and  in syntax that works on any shell, but alas more
       obscurely, by using the -u (unbuffered) option of sed(1):

           $ sed -un p < ../alpha/tasks > tasks

SEE ALSO

       taskset(1),     get_mempolicy(2),     getcpu(2),      mbind(2),      sched_getaffinity(2),
       sched_setaffinity(2),   sched_setscheduler(2),   set_mempolicy(2),   CPU_SET(3),  proc(5),
       cgroups(7), numa(7), sched(7), migratepages(8), numactl(8)

       Documentation/admin-guide/cgroup-v1/cpusets.rst  in  the  Linux  kernel  source  tree  (or
       Documentation/cgroup-v1/cpusets.txt   before  Linux  4.18,  and  Documentation/cpusets.txt
       before Linux 2.6.29)