Ubuntu Manpage: cgroup.conf - Slurm configuration file for the cgroup support

Provided by: slurm-client_17.11.2-1build1_amd64

NAME

       cgroup.conf - Slurm configuration file for the cgroup support

DESCRIPTION

cgroup.conf is an ASCII file which defines parameters used by Slurm's Linux cgroup related
plugins. The file location can be modified at system build time using the
DEFAULT_SLURM_CONF parameter or at execution time by setting the SLURM_CONF environment
variable. The file will always be located in the same directory as the slurm.conf file.

Parameter names are case insensitive. Any text following a "#" in the configuration file
is treated as a comment through the end of that line. Changes to the configuration file
take effect upon restart of Slurm daemons, daemon receipt of the SIGHUP signal, or
execution of the command "scontrol reconfigure" unless otherwise noted.

For general Slurm Cgroups information, see the Cgroups Guide at
<https://slurm.schedmd.com/cgroups.html>.

The following cgroup.conf parameters are defined to control the general behavior of Slurm
cgroup plugins.

CgroupAutomount=<yes|no>
Slurm cgroup plugins require valid and functional cgroup subsystem to be mounted
under /sys/fs/cgroup/<subsystem_name>. When launched, plugins check their
subsystem availability. If not available, the plugin launch fails unless
CgroupAutomount is set to yes. In that case, the plugin will first try to mount the
required subsystems.

CgroupMountpoint=PATH
Specify the PATH under which cgroups should be mounted. This should be a writable
directory which will contain cgroups mounted one per subsystem. The default PATH is
/sys/fs/cgroup.

TASK/CGROUP PLUGIN

The following cgroup.conf parameters are defined to control the behavior of this
particular plugin:

AllowedDevicesFile=<path_to_allowed_devices_file>
If the ConstrainDevices field is set to "yes" then this file has to be used to
declare the devices that need to be allowed by default for all the jobs. The
current implementation of cgroup devices subsystem works as a whitelist of entries,
which means that in order to isolate the access of a job upon particular devices we
need to allow the access on all the devices, supported by default and then deny on
those that the job does not have the permission to use. The default value is
"/etc/slurm/cgroup_allowed_devices_file.conf". The syntax of the file accepts one
device per line and it permits lines like /dev/sda* or /dev/cpu/*/*. See also an
example of this file in etc/cgroup_allowed_devices_file.conf.example.

AllowedKmemSpace=<number>
Constrain the job cgroup kernel memory to this amount of the allocated memory,
specified in bytes. The AllowedKmemSpace must be between the upper and lower memory
limits, specified by MaxKmemPercent and MinKmemSpace, respectivley. If
AllowedKmemSpace goes beyond the upper or lower limit, it will be reset to that
upper or lower limit, whichever has been exceeded.

AllowedRAMSpace=<number>
Constrain the job cgroup RAM to this percentage of the allocated memory. The
percentage supplied may be expressed as floating point number, e.g. 98.5. If the
AllowedRAMSpace limit is exceeded, the job steps will be killed and a warning
message will be written to standard error. Also see ConstrainRAMSpace. The
default value is 100.

AllowedSwapSpace=<number>
Constrain the job cgroup swap space to this percentage of the allocated memory.
The default value is 0, which means that RAM+Swap will be limited to
AllowedRAMSpace. The supplied percentage may be expressed as a floating point
number, e.g. 50.5. If the limit is exceeded, the job steps will be killed and a
warning message will be written to standard error. Also see ConstrainSwapSpace.
NOTE: Setting AllowedSwapSpace to 0 does not restrict the Linux kernel from using
swap space. To control how the kernel uses swap space, see MemorySwappiness.

ConstrainCores=<yes|no>
If configured to "yes" then constrain allowed cores to the subset of allocated
resources. This functionality makes use of the cpuset subsystem. Due to a bug
fixed in version 1.11.5 of HWLOC, the task/affinity plugin may be required in
addition to task/cgroup for this to function properly. The default value is "no".

ConstrainDevices=<yes|no>
If configured to "yes" then constrain the job's allowed devices based on GRES
allocated resources. It uses the devices subsystem for that. The default value is
"no".

ConstrainKmemSpace=<yes|no>
If configured to "yes" then constrain the job's Kmem RAM usage. In addition to RAM
usage. Only takes effect if ConstrainRAMSpace is set to "yes". The default value is
"yes", in which case the job's Kmem limit will be set to its RAM limit. Also see
AllowedKmemSpace.

ConstrainRAMSpace=<yes|no>
If configured to "yes" then constrain the job's RAM usage by setting the memory
soft limit to the allocated memory and the hard limit to the allocated memory *
AllowedRAMSpace. The default value is "no", in which case the job's RAM limit will
be set to its swap space limit if ConstrainSwapSpace is set to "yes". Also see
AllowedSwapSpace, AllowedRAMSpace and ConstrainSwapSpace. NOTE: When enabled,
ConstrainRAMSpace can lead to a noticiable decline in per-node job throughout.
Sites with high-throughput requirements should carefully weigh the tradeoff between
per-node throughput, versus potential problems that can arise from unconstrained
memory usage on the node. See <https://slurm.schedmd.com/high_throughput.html> for
further discussion.

ConstrainSwapSpace=<yes|no>
If configured to "yes" then constrain the job's swap space usage. The default
value is "no". Note that when set to "yes" and ConstrainRAMSpace is set to "no",
AllowedRAMSpace is automatically set to 100% in order to limit the RAM+Swap amount
to 100% of job's requirement plus the percent of allowed swap space. This amount is
thus set to both RAM and RAM+Swap limits. This means that in that particular case,
ConstrainRAMSpace is automatically enabled with the same limit than the one used to
constrain swap space. Also see AllowedSwapSpace.

MaxRAMPercent=PERCENT
Set an upper bound in percent of total RAM on the RAM constraint for a job. This
will be the memory constraint applied to jobs that are not explicitly allocated
memory by Slurm (i.e. Slurm's select plugin is not configured to manage memory
allocations). The PERCENT may be an arbitrary floating point number. The default
value is 100.

MaxSwapPercent=PERCENT
Set an upper bound (in percent of total RAM) on the amount of RAM+Swap that may be
used for a job. This will be the swap limit applied to jobs on systems where memory
is not being explicitly allocated to job. The PERCENT may be an arbitrary floating
point number between 0 and 100. The default value is 100.

MaxKmemPercent=PERCENT
Set an upper bound in percent of total Kmem for a job. The PERCENT may be an
arbitrary floating point number. The default value is 100.

MemorySwappiness=<number>
Configure the kernel's priority for swapping out anonymous pages (such as program
data) verses file cache pages for the job cgroup. Valid values are between 0 and
100, inclusive. A value of 0 prevents the kernel from swapping out program data. A
value of 100 gives equal priorioty to swapping out file cache or anonymous pages.
If not set, then the kernel's default swappiness value will be used. Either
ConstrainRAMSpace or ConstrainSwapSpace must be set to yes in order for this
parameter to be applied.

MinKmemSpace=<number>
Set a lower bound (in MB) on the memory limits defined by AllowedKmemSpace. The
default limit is 30M.

MinRAMSpace=<number>
Set a lower bound (in MB) on the memory limits defined by AllowedRAMSpace and
AllowedSwapSpace. This prevents accidentally creating a memory cgroup with such a
low limit that slurmstepd is immediately killed due to lack of RAM. The default
limit is 30M.

TaskAffinity=<yes|no>
If configured to "yes" then set a default task affinity to bind each step task to a
subset of the allocated cores using sched_setaffinity. The default value is "no".
Note: This feature requires the Portable Hardware Locality (hwloc) library to be
installed.

DISTRIBUTION-SPECIFIC NOTES

       Debian and derivatives (e.g. Ubuntu) usually exclude the memory and memsw  (swap)  cgroups
       by  default.  To  include  them,  add the following parameters to the kernel command line:
       cgroup_enable=memory swapaccount=1

       This can usually be placed in /etc/default/grub inside the GRUB_CMDLINE_LINUX variable.  A
       command such as update-grub must be run after updating the file.

EXAMPLE

       ###
       # Slurm cgroup support configuration file
       ###
       CgroupAutomount=yes
       ConstrainCores=yes
       #

COPYING