Provided by:
slurm-llnl_2.3.2-1ubuntu1_i386 
NAME
cgroup.conf - Slurm configuration file for the cgroup support
DESCRIPTION
cgroup.conf is an ASCII file which defines parameters used by Slurm's
Linux cgroup related plugins. The file location can be modified at
system build time using the DEFAULT_SLURM_CONF parameter or at
execution time by setting the SLURM_CONF environment variable. The file
will always be located in the same directory as the slurm.conf file.
Parameter names are case insensitive. Any text following a "#" in the
configuration file is treated as a comment through the end of that
line. The size of each line in the file is limited to 1024 characters.
Changes to the configuration file take effect upon restart of SLURM
daemons, daemon receipt of the SIGHUP signal, or execution of the
command "scontrol reconfigure" unless otherwise noted.
Two cgroup plugins are currently available in SLURM. The first one is a
proctrack plugin, the second one a task plugin.
The following cgroup.conf parameters are defined to control the general
behavior of Slurm cgroup plugins.
CgroupMountpoint=PATH
Specify the PATH under which cgroups should be mounted. This
should be a writable directory which will contain cgroups
mounted one per subsystem. The default PATH is /cgroup.
CgroupAutomount=<yes|no>
Slurm cgroup plugins require valid and functional cgroup
subsystem to be mounted under /cgroup/<subsystem_name>. When
launched, plugins check their subsystem availability. If not
available, the plugin launch fails unless CgroupAutomount is set
to yes. In that case, the plugin will first try to mount the
required subsystems.
CgroupReleaseAgentDir=<path_to_release_agent_directory>
Used to tune the cgroup system behavior. This parameter
identifies the location of the directory containing Slurm cgroup
release_agent files. A release_agent file is required for each
mounted subsystem. The release_agent file name must have the
following format: release_<subsystem_name>. For instance, the
release_agent file for the cpuset subsystem must be named
release_cpuset. See also CLEANUP OF CGROUPS below.
PROCTRACK/CGROUP PLUGIN
Slurm proctrack/cgroup plugin is used to track processes using the
freezer control group subsystem. It creates a hierarchical set of
directories for each step, putting the step tasks into the leaf.
This directory structure is like the following:
/cgroup/freezer/uid_%uid/job_%jobid/step_%stepid
Slurm cgroup proctrack plugin is enabled with the following parameter
in slurm.conf:
ProctrackType=proctrack/cgroup
No particular cgroup.conf parameter is defined to control the behavior
of this particular plugin.
TASK/CGROUP PLUGIN
Slurm task/cgroup plugin is used to enforce allocated resources
constraints, thus avoiding tasks to use unallocated resources. It
currently only uses cpuset subsystem but could use memory and devices
subsystems in a near future too.
It creates a hierarchical set of directories for each task and
subsystem. The directory structure is like the following:
/cgroup/%subsys/uid_%uid/job_%jobid/step_%stepid/task_%taskid
Slurm cgroup task plugin is enabled with the following parameter in
slurm.conf:
TaskPlugin=task/cgroup
The following cgroup.conf parameters are defined to control the
behavior of this particular plugin:
ConstrainCores=<yes|no>
If configured to "yes" then constrain allowed cores to the
subset of allocated resources. It uses the cpuset subsystem.
The default value is "no".
TaskAffinity=<yes|no>
If configured to "yes" then set a default task affinity to bind
each step task to a subset of the allocated cores using
sched_setaffinity. The default value is "no".
The following cgroup.conf parameters could be defined to control the
behavior of this particular plugin in a next version where memory and
devices support would be added :
AllowedRAMSpace=<number>
Constrain the job cgroup RAM to this percentage of the allocated
memory. The default value is 100. If SLURM is not allocating
memory to jobs, The percentage supplied may be expressed as
floating point number, e.g. 98.5. If the AllowedRAMSpace limit
is exceeded, the job steps will be killed and a warning message
will be written to standard error. Also see ConstrainRAMSpace.
AllowedSwapSpace=<number>
Constrain the job cgroup swap space to this percentage of the
allocated memory. The default value is 0, which means that
RAM+Swap will be limited to AllowedRAMSpace. The supplied
percentage may be expressed as a floating point number, e.g.
50.5. If the limit is exceeded, the job steps will be killed
and a warning message will be written to standard error. Also
see ConstrainSwapSpace.
ConstrainRAMSpace=<yes|no>
If configured to "yes" then constrain the job's RAM usage. The
default value is "no". Also see AllowedRAMSpace.
ConstrainSwapSpace=<yes|no>
If configured to "yes" then constrain the job's swap space
usage. The default value is "no". Also see AllowedSwapSpace.
MaxRAMPercent=PERCENT
Set an upper bound in percent of total RAM on the RAM constraint
for a job. This will be the memory constraint applied to jobs
that are not explicitly allocated memory by SLURM. The PERCENT
may be an arbitrary floating point number. The default value is
100.
MaxSwapPercent=PERCENT
Set an upper bound (in percent of total RAM) on the amount of
RAM+Swap that may be used for a job. This will be the swap limit
applied to jobs on systems where memory is not being explicitly
allocated to job. The PERCENT may be an arbitrary floating point
number between 0 and 100. The default value is 100.
MinRAMSpace=<number>
Set a lower bound (in MB) on the memory limits defined by
AllowedRAMSpace and AllowedSwapSpace. This prevents accidentally
creating a memory cgroup with such a low limit that slurmstepd
is immediately killed due to lack of RAM. The default limit is
30M.
ConstrainDevices=<yes|no>
If configured to "yes" then constrain the job's allowed devices
based on GRES allocated resources. It uses the devices subsystem
for that. The default value is "no".
AllowedDevicesFile=<path_to_allowed_devices_file>
If the ConstrainDevices field is set to "yes" then this file has
to be used to declare the devices that need to be allowed by
default for all the jobs. The current implementation of cgroup
devices subsystem works as a whitelist of entries, which means
that in order to isolate the access of a job upon particular
devices we need to allow the access on all the devices,
supported by default and then deny on those that the job does
not have the permission to use. The default value is
"/etc/slurm/cgroup_allowed_devices_file.conf". The syntax of the
file accepts one device per line and it permits lines like
/dev/sda* or /dev/cpu/*/*. See also an example of this file in
etc/allowed_devices_file.conf.example.
EXAMPLE
###
# Slurm cgroup support configuration file
###
CgroupAutomount=yes
CgroupReleaseAgentDir="/etc/slurm/cgroup"
ConstrainCores=yes
#
NOTES
Only one instance of a cgroup subsystem is valid at a time in the
kernel. If you try to mount another cgroup hierarchy that uses the
same cpuset subsystem it will fail. However you can mount another
cgroup hierarchy for a different cpuset subsystem.
CLEANUP OF CGROUPS
To allow cgroups to be removed automatically when they are no longer in
use the notify_on_release flag is set in each cgroup when the cgroup is
instantiated. The release_agent file for each subsystem is set up when
the subsystem is mounted. The name of each release_agent file is
release_<subsystem name>. The directory is specified via the
CgroupReleaseAgentDir parameter in cgroup.conf. A simple release agent
mechanism to remove slurm cgroups when they become empty may be set up
by creating the release agent files for each required subsystem as
symbolic links to a common release agent script, as shown in the
example below:
[sulu] (slurm) etc> cat cgroup.conf | grep CgroupReleaseAgentDir
CgroupReleaseAgentDir="/etc/slurm/cgroup"
[sulu] (slurm) etc> ls -al /etc/slurm/cgroup
total 12
drwxr-xr-x 2 root root 4096 2010-04-23 14:55 .
drwxr-xr-x 4 root root 4096 2010-07-22 14:48 ..
-rwxrwxrwx 1 root root 234 2010-04-23 14:52 release_common
lrwxrwxrwx 1 root root 32 2010-04-23 11:04 release_cpuset ->
/etc/slurm/cgroup/release_common
lrwxrwxrwx 1 root root 32 2010-04-23 11:03 release_freezer ->
/etc/slurm/cgroup/release_common
[sulu] (slurm) etc> cat /etc/slurm/cgroup/release_common
#!/bin/bash
base_path=/cgroup
progname=$(basename $0)
subsystem=${progname##*_}
rmcg=${base_path}/${subsystem}$@
uidcg=${rmcg%/job*}
if [[ -d ${base_path}/${subsystem} ]]
then
flock -x ${uidcg} -c "rmdir ${rmcg}"
fi
[sulu] (slurm) etc>
COPYING
Copyright (C) 2010 Lawrence Livermore National Security. Produced at
Lawrence Livermore National Laboratory (cf, DISCLAIMER).
CODE-OCEC-09-009. All rights reserved.
This file is part of SLURM, a resource management program. For
details, see <http://www.schedmd.com/slurmdocs/>.
SLURM is free software; you can redistribute it and/or modify it under
the terms of the GNU General Public License as published by the Free
Software Foundation; either version 2 of the License, or (at your
option) any later version.
SLURM is distributed in the hope that it will be useful, but WITHOUT
ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
for more details.
SEE ALSO
slurm.conf(5)