Provided by: slurm-llnl_2.3.2-1ubuntu1_amd64 bug

NAME

       cgroup.conf - Slurm configuration file for the cgroup support

DESCRIPTION

       cgroup.conf is an ASCII file which defines parameters used by Slurm's Linux cgroup related
       plugins.   The  file  location  can  be  modified  at  system   build   time   using   the
       DEFAULT_SLURM_CONF  parameter  or  at execution time by setting the SLURM_CONF environment
       variable. The file will always be located in the same directory as the slurm.conf file.

       Parameter names are case insensitive.  Any text following a "#" in the configuration  file
       is  treated  as a comment through the end of that line.  The size of each line in the file
       is limited to 1024 characters.  Changes to the configuration file take effect upon restart
       of  SLURM  daemons,  daemon  receipt  of  the  SIGHUP  signal, or execution of the command
       "scontrol reconfigure" unless otherwise noted.

       Two cgroup plugins are currently available in SLURM. The first one is a proctrack  plugin,
       the second one a task plugin.

       The  following cgroup.conf parameters are defined to control the general behavior of Slurm
       cgroup plugins.

       CgroupMountpoint=PATH
              Specify the PATH under which cgroups should be mounted. This should be  a  writable
              directory which will contain cgroups mounted one per subsystem. The default PATH is
              /cgroup.

       CgroupAutomount=<yes|no>
              Slurm cgroup plugins require valid and functional cgroup subsystem  to  be  mounted
              under  /cgroup/<subsystem_name>.   When  launched,  plugins  check  their subsystem
              availability. If not available, the plugin launch fails unless  CgroupAutomount  is
              set  to  yes.  In  that  case,  the  plugin  will  first  try to mount the required
              subsystems.

       CgroupReleaseAgentDir=<path_to_release_agent_directory>
              Used to tune the cgroup system behavior. This parameter identifies the location  of
              the  directory containing Slurm cgroup release_agent files. A release_agent file is
              required for each mounted subsystem. The release_agent  file  name  must  have  the
              following  format:  release_<subsystem_name>.  For instance, the release_agent file
              for the cpuset subsystem must be named release_cpuset.  See also CLEANUP OF CGROUPS
              below.

PROCTRACK/CGROUP PLUGIN

       Slurm  proctrack/cgroup  plugin is used to track processes using the freezer control group
       subsystem. It creates a hierarchical set of directories for each step,  putting  the  step
       tasks into the leaf.

       This directory structure is like the following:
       /cgroup/freezer/uid_%uid/job_%jobid/step_%stepid

       Slurm cgroup proctrack plugin is enabled with the following parameter in slurm.conf:
       ProctrackType=proctrack/cgroup

       No  particular cgroup.conf parameter is defined to control the behavior of this particular
       plugin.

TASK/CGROUP PLUGIN

       Slurm task/cgroup plugin is used to enforce allocated resources constraints, thus avoiding
       tasks  to use unallocated resources. It currently only uses cpuset subsystem but could use
       memory and devices subsystems in a near future too.

       It creates a hierarchical set of directories for each task and subsystem.   The  directory
       structure is like the following:
       /cgroup/%subsys/uid_%uid/job_%jobid/step_%stepid/task_%taskid

       Slurm cgroup task plugin is enabled with the following parameter in slurm.conf:
       TaskPlugin=task/cgroup

       The  following  cgroup.conf  parameters  are  defined  to  control  the  behavior  of this
       particular plugin:

       ConstrainCores=<yes|no>
              If configured to "yes" then constrain allowed cores  to  the  subset  of  allocated
              resources. It uses the cpuset subsystem.  The default value is "no".

       TaskAffinity=<yes|no>
              If configured to "yes" then set a default task affinity to bind each step task to a
              subset of the allocated cores using sched_setaffinity.  The default value is "no".

       The following cgroup.conf parameters could be defined to  control  the  behavior  of  this
       particular plugin in a next version where memory and devices support would be added :

       AllowedRAMSpace=<number>
              Constrain  the  job  cgroup  RAM  to  this percentage of the allocated memory.  The
              default value is 100. If SLURM is not allocating memory  to  jobs,  The  percentage
              supplied   may   be   expressed  as  floating  point  number,  e.g.  98.5.  If  the
              AllowedRAMSpace limit is exceeded, the job steps  will  be  killed  and  a  warning
              message will be written to standard error.  Also see ConstrainRAMSpace.

       AllowedSwapSpace=<number>
              Constrain  the  job  cgroup  swap space to this percentage of the allocated memory.
              The  default  value  is  0,  which  means  that  RAM+Swap  will   be   limited   to
              AllowedRAMSpace.  The  supplied  percentage  may  be  expressed as a floating point
              number, e.g. 50.5.  If the limit is exceeded, the job steps will be  killed  and  a
              warning message will be written to standard error.  Also see ConstrainSwapSpace.

       ConstrainRAMSpace=<yes|no>
              If  configured  to  "yes" then constrain the job's RAM usage.  The default value is
              "no".  Also see AllowedRAMSpace.

       ConstrainSwapSpace=<yes|no>
              If configured to "yes" then constrain the job's  swap  space  usage.   The  default
              value is "no".  Also see AllowedSwapSpace.

       MaxRAMPercent=PERCENT
              Set  an  upper bound in percent of total RAM on the RAM constraint for a job.  This
              will be the memory constraint applied to jobs that  are  not  explicitly  allocated
              memory by SLURM. The PERCENT may be an arbitrary floating point number. The default
              value is 100.

       MaxSwapPercent=PERCENT
              Set an upper bound (in percent of total RAM) on the amount of RAM+Swap that may  be
              used for a job. This will be the swap limit applied to jobs on systems where memory
              is not being explicitly allocated to job. The PERCENT may be an arbitrary  floating
              point number between 0 and 100.  The default value is 100.

       MinRAMSpace=<number>
              Set  a  lower  bound  (in  MB)  on the memory limits defined by AllowedRAMSpace and
              AllowedSwapSpace. This prevents accidentally creating a memory cgroup with  such  a
              low  limit  that  slurmstepd  is immediately killed due to lack of RAM. The default
              limit is 30M.

       ConstrainDevices=<yes|no>
              If configured to "yes" then constrain the  job's  allowed  devices  based  on  GRES
              allocated  resources. It uses the devices subsystem for that.  The default value is
              "no".

       AllowedDevicesFile=<path_to_allowed_devices_file>
              If the ConstrainDevices field is set to "yes" then this file  has  to  be  used  to
              declare  the  devices  that  need  to  be  allowed by default for all the jobs. The
              current implementation of cgroup devices subsystem works as a whitelist of entries,
              which means that in order to isolate the access of a job upon particular devices we
              need to allow the access on all the devices, supported by default and then deny  on
              those  that  the  job  does  not  have  the permission to use. The default value is
              "/etc/slurm/cgroup_allowed_devices_file.conf". The syntax of the file  accepts  one
              device  per  line and it permits lines like /dev/sda* or /dev/cpu/*/*.  See also an
              example of this file in etc/allowed_devices_file.conf.example.

EXAMPLE

       ###
       # Slurm cgroup support configuration file
       ###
       CgroupAutomount=yes
       CgroupReleaseAgentDir="/etc/slurm/cgroup"
       ConstrainCores=yes
       #

NOTES

       Only one instance of a cgroup subsystem is valid at a time in the kernel.  If you  try  to
       mount  another cgroup hierarchy that uses the same cpuset subsystem it will fail.  However
       you can mount another cgroup hierarchy for a different cpuset subsystem.

CLEANUP OF CGROUPS

       To allow cgroups to  be  removed  automatically  when  they  are  no  longer  in  use  the
       notify_on_release  flag  is  set  in  each  cgroup  when  the  cgroup is instantiated. The
       release_agent file for each subsystem is set up when the subsystem is mounted.   The  name
       of each release_agent file is release_<subsystem name>. The directory is specified via the
       CgroupReleaseAgentDir parameter in cgroup.conf. A simple release agent mechanism to remove
       slurm cgroups when they become empty may be set up by creating the release agent files for
       each required subsystem as symbolic links to a common release agent script,  as  shown  in
       the example below:

       [sulu] (slurm) etc> cat cgroup.conf | grep CgroupReleaseAgentDir
       CgroupReleaseAgentDir="/etc/slurm/cgroup"

       [sulu] (slurm) etc> ls -al /etc/slurm/cgroup
       total 12
       drwxr-xr-x 2 root root 4096 2010-04-23 14:55 .
       drwxr-xr-x 4 root root 4096 2010-07-22 14:48 ..
       -rwxrwxrwx 1 root root  234 2010-04-23 14:52 release_common
       lrwxrwxrwx     1     root     root       32    2010-04-23    11:04    release_cpuset    ->
       /etc/slurm/cgroup/release_common
       lrwxrwxrwx    1    root    root      32    2010-04-23     11:03     release_freezer     ->
       /etc/slurm/cgroup/release_common

       [sulu] (slurm) etc> cat /etc/slurm/cgroup/release_common
       #!/bin/bash
       base_path=/cgroup
       progname=$(basename $0)
       subsystem=${progname##*_}
       rmcg=${base_path}/${subsystem}$@
       uidcg=${rmcg%/job*}
       if [[ -d ${base_path}/${subsystem} ]]
       then
            flock -x ${uidcg} -c "rmdir ${rmcg}"
       fi
       [sulu] (slurm) etc>

COPYING

       Copyright  (C)  2010 Lawrence Livermore National Security.  Produced at Lawrence Livermore
       National Laboratory (cf, DISCLAIMER).  CODE-OCEC-09-009. All rights reserved.

       This  file  is  part  of  SLURM,  a  resource  management  program.   For   details,   see
       <http://www.schedmd.com/slurmdocs/>.

       SLURM  is  free  software; you can redistribute it and/or modify it under the terms of the
       GNU General Public License as published by the Free Software Foundation; either version  2
       of the License, or (at your option) any later version.

       SLURM is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without
       even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
       GNU General Public License for more details.

SEE ALSO

       slurm.conf(5)