Provided by: slurm-client_21.08.5-2ubuntu1_amd64 bug

NAME

       scontrol - view or modify Slurm configuration and state.

SYNOPSIS

       scontrol [OPTIONS...] [COMMAND...]

DESCRIPTION

       scontrol  is  used  to  view or modify Slurm configuration including: job, job step, node,
       partition, reservation, and overall system configuration. Most of the commands can only be
       executed  by  user root or an Administrator. If an attempt to view or modify configuration
       information is made by an unauthorized user, an error message  will  be  printed  and  the
       requested  action  will not occur.  If no command is entered on the execute line, scontrol
       will operate in an interactive mode and prompt for input. It will continue  prompting  for
       input  and  executing commands until explicitly terminated. If a command is entered on the
       execute line, scontrol will execute that command and terminate. All commands  and  options
       are  case-insensitive,  although  node  names,  partition names, and reservation names are
       case-sensitive (node names "LX" and "lx" are distinct). All commands and  options  can  be
       abbreviated to the extent that the specification is unique. A modified Slurm configuration
       can be written to a file using the scontrol write config command. The resulting file  will
       be named using the convention "slurm.conf.<datetime>" and located in the same directory as
       the original "slurm.conf" file. The directory containing the original slurm.conf  must  be
       writable for this to occur.

OPTIONS

       -a, --all
              When  the  show  command  is used, then display all partitions, their jobs and jobs
              steps. This causes information to be displayed about partitions that are configured
              as hidden and partitions that are unavailable to user's group.

       -M, --clusters=<string>
              The  cluster  to  issue  commands to. Only one cluster name may be specified.  Note
              that the SlurmDBD must be up  for  this  option  to  work  properly.   This  option
              implicitly sets the --local option.

       -d, --details
              Causes the show command to provide additional details where available.

       --federation
              Report jobs from federation if a member of one.

       -F, --future
              Report nodes in FUTURE state.

       -h, --help
              Print a help message describing the usage of scontrol.

       --hide Do  not  display information about hidden partitions, their jobs and job steps.  By
              default, neither partitions that are configured  as  hidden  nor  those  partitions
              unavailable to user's group will be displayed (i.e. this is the default behavior).

       --local
              Show only information local to this cluster. Ignore other clusters in the federated
              if a member of one. Overrides --federation.

       -o, --oneliner
              Print information one line per record.

       -Q, --quiet
              Print no warning or informational messages, only fatal error messages.

       --sibling
              Show all sibling jobs on a federated cluster. Implies --federation.

       -u, --uid=<uid>
              Attempt to update a job as user <uid> instead of the invoking user id.

       -v, --verbose
              Print detailed event logging. Multiple -v's will further increase the verbosity  of
              logging. By default only errors will be displayed.

       -V , --version
              Print version information and exit.

COMMANDS

       abort  Instruct  the  Slurm  controller to terminate immediately and generate a core file.
              See "man slurmctld" for information about where the core file will be written.

       cancel_reboot <NodeList>
              Cancel pending reboots on nodes. The node will be undrain'ed and the reason cleared
              if the node was drained by an ASAP reboot.

       cluster <CLUSTER_NAME>
              The cluster to issue commands to. Only one cluster name may be specified.

       create <SPECIFICATION>
              Create  a  new  partition  or  reservation.  See the full list of parameters below.
              Include the tag "res" to create a  reservation  without  specifying  a  reservation
              name.

       completing
              Display  all  jobs  in  a  COMPLETING state along with associated nodes in either a
              COMPLETING or DOWN state.

       delete <SPECIFICATION>
              Delete the entry with the specified SPECIFICATION.  The two  SPECIFICATION  choices
              are PartitionName=<name> and Reservation=<name>. Reservations and partitions should
              have no associated jobs at the time of their deletion (modify the jobs first).   If
              the specified partition is in use, the request is denied.

       errnumstr <ERRNO>
              Given a Slurm error number, return a descriptive string.

       fsdampeningfactor <FACTOR>
              Set the FairShareDampeningFactor in slurmctld.

       help   Display a description of scontrol options and commands.

       hold <job_list>
              Prevent a pending job from being started (sets its priority to 0).  Use the release
              command to permit the job to be  scheduled.   The  job_list  argument  is  a  comma
              separated  list of job IDs OR "jobname=" with the job's name, which will attempt to
              hold all jobs having that name.   Note  that  when  a  job  is  held  by  a  system
              administrator  using  the hold command, only a system administrator may release the
              job for execution (also see the uhold command). When the job is held by its  owner,
              it  may  also  be  released by the job's owner.  Additionally, attempting to hold a
              running job will have not suspend or cancel it. But, it will set the  job  priority
              to  0  and update the job reason field, which would hold the job if it was requeued
              at a later time.

       notify <job_id> <message>
              Send a message to standard error of  the  salloc  or  srun  command  or  batch  job
              associated with the specified job_id.

       pidinfo <proc_id>
              Print the Slurm job id and scheduled termination time corresponding to the supplied
              process id, proc_id, on the current node.  This will work only  with  processes  on
              node  on  which  scontrol is run, and only for those processes spawned by Slurm and
              their descendants.

       listpids [<job_id>[.<step_id>]] [<NodeName>]
              Print a listing of the process IDs in a job step (if JOBID.STEPID is provided),  or
              all  of  the job steps in a job (if job_id is provided), or all of the job steps in
              all of the jobs on the local node (if job_id is not provided  or  job_id  is  "*").
              This  will  work only with processes on the node on which scontrol is run, and only
              for those processes spawned by Slurm and their descendants. Note  that  some  Slurm
              configurations  (ProctrackType  value of pgid) are unable to identify all processes
              associated with a job or job step.

              Note that the NodeName option is only really useful when you have  multiple  slurmd
              daemons running on the same host machine.  Multiple slurmd daemons on one host are,
              in general, only used by Slurm developers.

       ping   Ping the primary and secondary slurmctld daemon and report if they are responding.

       reboot [ASAP] [nextstate={RESUME|DOWN}] [reason=<reason>] {ALL|<NodeList>}
              Reboot the nodes in the system when they become idle  using  the  RebootProgram  as
              configured in Slurm's slurm.conf file.  Each node will have the "REBOOT" flag added
              to its node state.  After a node reboots and the slurmd daemon starts up again, the
              HealthCheckProgram will run once. Then, the slurmd daemon will register itself with
              the slurmctld daemon and the "REBOOT" flag will be  cleared.   The  node's  "DRAIN"
              state flag will be cleared if the reboot was "ASAP", nextstate=resume or down.  The
              "ASAP" option adds the "DRAIN" flag to each  node's  state,  preventing  additional
              jobs  from  running  on  the node so it can be rebooted and returned to service "As
              Soon As Possible" (i.e. ASAP).  "ASAP" will also set the  node  reason  to  "Reboot
              ASAP"  if  the  "reason"  option  isn't  specified.   If  the "nextstate" option is
              specified as "DOWN", then the node will remain in a down state after rebooting.  If
              "nextstate"  is specified as "RESUME", then the nodes will resume as normal and the
              node's reason and "DRAIN" state will be cleared.  Resuming nodes will be considered
              as available in backfill future scheduling and won't be replaced by idle nodes in a
              reservation.  The "reason"  option  sets  each  node's  reason  to  a  user-defined
              message.   A  default reason of "reboot requested" is set if no other reason is set
              on the node.  The reason will be appended with "reboot issued" when the  reboot  is
              issued  and  "reboot  complete"  when  the  node registers and has a "nextstate" of
              "DOWN".  You must specify either a list of nodes  or  that  ALL  nodes  are  to  be
              rebooted.   NOTE:  By  default,  this command does not prevent additional jobs from
              being scheduled on any nodes before reboot.  To do this, you  can  either  use  the
              "ASAP" option or explicitly drain the nodes beforehand.  You can alternately create
              an advanced reservation to prevent additional jobs from being initiated on nodes to
              be  rebooted.   Pending  reboots  can be cancelled by using "scontrol cancel_reboot
              <node>" or setting the node state to "CANCEL_REBOOT".  A node will be marked "DOWN"
              if it doesn't reboot within ResumeTimeout.

       reconfigure
              Instruct  all  Slurm  daemons to re-read the configuration file.  This command does
              not restart the daemons.  This mechanism would  be  used  to  modify  configuration
              parameters  (Epilog,  Prolog,  SlurmctldLogFile,  SlurmdLogFile,  etc.).  The Slurm
              controller (slurmctld) forwards the request to all other daemons (slurmd daemon  on
              each compute node). Running jobs continue execution.  Most configuration parameters
              can be changed by just running this  command,  however,  Slurm  daemons  should  be
              shutdown  and  restarted  if  any  of these parameters are to be changed: AuthType,
              ControlMach,  PluginDir,  StateSaveLocation,   SlurmctldHost,   SlurmctldPort,   or
              SlurmdPort.  The slurmctld daemon and all slurmd daemons must be restarted if nodes
              are added to or removed from the cluster.

       release <job_list>
              Release a previously held job to begin execution.  The job_list argument is a comma
              separated  list of job IDs OR "jobname=" with the job's name, which will attempt to
              hold all jobs having that name.  Also see hold.

       requeue  [<option>] <job_list>
              Requeue a running, suspended or finished Slurm batch job into pending  state.   The
              job_list  argument  is  a comma separated list of job IDs.  The command accepts the
              following option:

              Incomplete
                     Operate only on jobs (or tasks of a job array)  which  have  not  completed.
                     Specifically   only   jobs   in  the  following  states  will  be  requeued:
                     CONFIGURING, RUNNING, STOPPED or SUSPENDED.

       requeuehold [<option>] <job_list>
              Requeue a running, suspended or  finished  Slurm  batch  job  into  pending  state,
              moreover  the job is put in held state (priority zero).  The job_list argument is a
              comma separated list of job IDs.  A held job can  be  released  using  scontrol  to
              reset  its  priority  (e.g.   "scontrol release <job_id>"). The command accepts the
              following options:

              Incomplete
                     Operate only on jobs (or tasks of a job array)  which  have  not  completed.
                     Specifically   only   jobs   in  the  following  states  will  be  requeued:
                     CONFIGURING, RUNNING, STOPPED or SUSPENDED.

              State=SpecialExit
                     The "SpecialExit" keyword specifies that the job has to be put in a  special
                     state  JOB_SPECIAL_EXIT.   The  "scontrol show job" command will display the
                     JobState as SPECIAL_EXIT, while the "squeue" command as SE.

       resume <job_list>
              Resume a previously suspended job.  The job_list argument is a comma separated list
              of job IDs.  Also see suspend.

              NOTE:  A  suspended job releases its CPUs for allocation to other jobs.  Resuming a
              previously suspended job may result in multiple jobs being allocated the same CPUs,
              which  could trigger gang scheduling with some configurations or severe degradation
              in performance with other configurations.  Use  of  the  scancel  command  to  send
              SIGSTOP  and  SIGCONT  signals  would  stop  a  job  without releasing its CPUs for
              allocation to other jobs and would be a preferable mechanism  in  many  cases.   If
              performing  system  maintenance you may want to use suspend/resume in the following
              way. Before suspending set all nodes to draining or set all partitions to  down  so
              that  no  new  jobs  can  be scheduled. Then suspend jobs. Once maintenance is done
              resume jobs then resume nodes and/or set all  partitions  back  to  up.   Use  with
              caution.

       schedloglevel <LEVEL>
              Enable or disable scheduler logging.  LEVEL may be "0", "1", "disable" or "enable".
              "0" has the same effect as "disable". "1" has the same effect  as  "enable".   This
              value  is  temporary  and  will  be overwritten when the slurmctld daemon reads the
              slurm.conf configuration file (e.g.  when  the  daemon  is  restarted  or  scontrol
              reconfigure is executed) if the SlurmSchedLogLevel parameter is present.

       setdebug <LEVEL>
              Change  the debug level of the slurmctld daemon for all active logging channels not
              originally configured off (quiet).  LEVEL may be an integer value between zero  and
              nine  (using  the same values as SlurmctldDebug in the slurm.conf file) or the name
              of the most detailed message type to be printed: "quiet", "fatal", "error", "info",
              "verbose",  "debug",  "debug2",  "debug3",  "debug4",  or  "debug5".  This value is
              temporary  and  will  be  overwritten  whenever  the  slurmctld  daemon  reads  the
              slurm.conf  configuration  file  (e.g.  when  the  daemon  is restarted or scontrol
              reconfigure is executed).

       setdebugflags [+|-]<FLAG>
              Add or remove DebugFlags of the slurmctld daemon.  See "man slurm.conf" for a  list
              of  supported DebugFlags.  NOTE: Changing the value of some DebugFlags will have no
              effect without restarting the slurmctld daemon, which would  set  DebugFlags  based
              upon the contents of the slurm.conf configuration file.

       show <ENTITY>[=<ID>] or <ENTITY> [<ID>]
              Display the state of the specified entity with the specified identification.

              aliases
                     Returns  all NodeName values associated with a given NodeHostname (useful to
                     get the list of virtual nodes associated with a real node in a configuration
                     where multiple slurmd daemons execute on a single compute node).

              assoc_mgr
                     Displays  the  current contents of the slurmctld's internal cache for users,
                     associations and/or qos. The output can  be  filtered  by  different  record
                     types:

                     users=<user1>[...,<userN>]
                            Limit  the  User  Records  displayed to those with the specified user
                            name(s).

                     accounts=<acct1>[...,<acctN>]
                            Limit the Association Records displayed to those with  the  specified
                            account name(s).

                     qos=<qos1>[...,<qosN>]
                            Limit  the  QOS  Records  displayed  to  those with the specified QOS
                            name(s).

                     flags={users|assoc|qos}
                            Specify the desired record type to be  displayed.  If  no  flags  are
                            specified, all record types are displayed.

              bbstat Displays  output  from  Cray's  burst  buffer status tool. Options following
                     bbstat are passed directly to the dwstat command by the slurmctld daemon and
                     the response returned to the user. Equivalent to dwstat.

              burstbuffer
                     Displays the current status of the BurstBuffer plugin.

              config Displays  parameter  names  from the configuration files in mixed case (e.g.
                     SlurmdPort=7003) while derived parameters names are in upper case only (e.g.
                     SLURM_VERSION).

              daemons
                     Reports which daemons should be running on this node.

              dwstat Displays  output  from  Cray's  burst  buffer status tool. Options following
                     dwstat are passed directly to the dwstat command by the slurmctld daemon and
                     the response returned to the user. Equivalent to bbstat.

              federation
                     The  federation name that the controller is part of and the sibling clusters
                     part of the federation will be listed.

              frontend
                     Shows configured frontend nodes.

              hostlist
                     Takes a list of host names and prints the hostlist expression for them  (the
                     inverse  of  hostnames).   hostlist can also take the absolute pathname of a
                     file (beginning with the character '/')  containing  a  list  of  hostnames.
                     Multiple  node  names  may  be specified using simple node range expressions
                     (e.g. "lx[10-20]"). All other ID values must identify a single element.  The
                     job  step  ID  is  of  the  form  "job_id.step_id", (e.g. "1234.1").  slurmd
                     reports the current status of the slurmd daemon executing on the  same  node
                     from  which  the  scontrol  command  is executed (the local host). It can be
                     useful to diagnose problems.  By default hostlist does  not  sort  the  node
                     list  or make it unique (e.g. tux2,tux1,tux2 = tux[2,1-2]).  If you wanted a
                     sorted list use  hostlistsorted  (e.g.  tux2,tux1,tux2  =  tux[1-2,2]).   By
                     default, all elements of the entity type specified are printed.

              hostlistsorted
                     Takes  a  list of host names and prints a sorted, unique hostlist expression
                     for them. See hostlist.

              hostnames
                     Takes an optional  hostlist  expression  as  input  and  writes  a  list  of
                     individual  host  names  to  standard  output (one per line). If no hostlist
                     expression is supplied, the contents of the  SLURM_JOB_NODELIST  environment
                     variable  is  used.  For  example  "tux[1-3]" is mapped to "tux1","tux2" and
                     "tux3" (one hostname per line).

              job    Displays statistics about all jobs by  default.  If  an  optional  jobid  is
                     specified, details for just that job will be displayed.  If the job does not
                     specify socket-per-node, cores-per-socket or threads-per-core then  it  will
                     display '*' in the ReqS:C:T=*:*:* field.

              node   Displays  statistics  about all nodes by default. If an optional nodename is
                     specified, details for just that node will be displayed.

              partition
                     Displays  statistics  about  all  partitions  by  default.  If  an  optional
                     partition  name  is  specified,  details  for  just  that  partition will be
                     displayed.

              reservation
                     Displays statistics about  all  reservations  by  default.  If  an  optional
                     reservation  name  is  specified,  details for just that reservation will be
                     displayed.

              slurmd Displays statistics for the slurmd running on the current node.

              step   Displays statistics about all job steps by default. If an optional jobid  is
                     specified,  details  about  steps for just that job will be displayed.  If a
                     jobid.stepid is specified, details for just that step will be displayed.

              topology
                     Displays information about the defined  topology  layout.  If  a  switch  is
                     specified, information about that switch will be shown.  If one node name is
                     specified, all switches connected to that node (and their  parent  switches)
                     will  be shown.  If more than one node name is specified, only switches that
                     connect to all named nodes will be shown.

       shutdown <OPTION>
              Instruct Slurm daemons to save current state and terminate.  By default, the  Slurm
              controller  (slurmctld)  forwards  the  request all other daemons (slurmd daemon on
              each compute node).  An OPTION of slurmctld  or  controller  results  in  only  the
              slurmctld daemon being shutdown and the slurmd daemons remaining active.

       suspend <job_list>
              Suspend a running job.  The job_list argument is a comma separated list of job IDs.
              Use the resume command to resume  its  execution.   User  processes  must  stop  on
              receipt  of SIGSTOP signal and resume upon receipt of SIGCONT for this operation to
              be effective.  Not all architectures and configurations support job suspension.  If
              a  suspended job is requeued, it will be placed in a held state.  The time a job is
              suspended  will  not  count  against  a  job's  time  limit.   Only  an   operator,
              administrator, SlurmUser, or root can suspend jobs.

       takeover [<INDEX>]
              Instruct one of Slurm's backup controllers (slurmctld) to take over system control.
              By default the first backup controller (INDEX=1) requests control from the  primary
              and  waits  for  its  termination.  After  that,  it  switches  from backup mode to
              controller mode. If primary controller can not be contacted, it  directly  switches
              to  controller  mode.  This  can be used to speed up the Slurm controller fail-over
              mechanism when the primary node is down.  This can be used to  minimize  disruption
              if  the  computer executing the primary Slurm controller is scheduled down.  (Note:
              Slurm's primary controller will take the control back at startup.)

       top <job_list>
              Move the specified job IDs to the top  of  the  queue  of  jobs  belonging  to  the
              identical  user  ID,  partition name, account, and QOS.  The job_list argument is a
              comma separated ordered list of job IDs.  Any job not matching all of those  fields
              will  not be effected.  Only jobs submitted to a single partition will be effected.
              This operation changes the order of jobs by adjusting job  nice  values.   The  net
              effect  on  that  user's  throughput will be negligible to slightly negative.  This
              operation  is  disabled  by  default  for  non-privileged   (non-operator,   admin,
              SlurmUser,  or  root) users. This operation may be enabled for non-privileged users
              by the system administrator  by  including  the  option  "enable_user_top"  in  the
              SchedulerParameters configuration parameter.

       token [lifespan=<lifespan>] [username=<username>]
              Return  an  auth  token  which  can  be  used  to  support  JWT  authentication  if
              AuthAltTypes=auth/jwt has been  enabled  on  the  system.   Supports  two  optional
              arguments.  lifespan=  may  be  used  to  specify  the token's lifespan in seconds.
              username (only available to SlurmUser/root) may be used to request a  token  for  a
              different username.

       uhold <job_list>
              Prevent  a  pending  job from being started (sets its priority to 0).  The job_list
              argument is a space separated list of job  IDs  or  job  names.   Use  the  release
              command  to  permit the job to be scheduled.  This command is designed for a system
              administrator to hold a job so that the  job  owner  may  release  it  rather  than
              requiring the intervention of a system administrator (also see the hold command).

       update <SPECIFICATION>
              Update  job,  step,  node, partition, or reservation configuration per the supplied
              specification. SPECIFICATION is in the same format as the Slurm configuration  file
              and  the output of the show command described above. It may be desirable to execute
              the show command (described above) on the specific entity you want to update,  then
              use  cut-and-paste  tools to enter updated configuration values to the update. Note
              that while most configuration values can be changed using this command, not all can
              be  changed  using  this  mechanism. In particular, the hardware configuration of a
              node or the physical addition or removal of nodes from  the  cluster  may  only  be
              accomplished  through  editing  the  Slurm  configuration  file  and  executing the
              reconfigure command (described above).

       version
              Display the version number of scontrol being executed.

       wait_job <job_id>
              Wait until a job and all of its nodes are ready for use or the job has entered some
              termination state. This option is particularly useful in the Slurm Prolog or in the
              batch script itself if nodes  are  powered  down  and  restarted  automatically  as
              needed.

              NOTE:   Don't   use   scontrol   wait_job   in   PrologSlurmctld   or  Prolog  with
              PrologFlags=Alloc as this will result in a deadlock.

       write batch_script <job_id> [<optional_filename>]
              Write the batch script for a given job_id to a file or to  stdout.  The  file  will
              default  to  slurm-<job_id>.sh  if the optional filename argument is not given. The
              script will be written to stdout if - is given instead of a  filename.   The  batch
              script can only be retrieved by an admin or operator, or by the owner of the job.

       write config <optional_filename>
              Write   the  current  configuration  to  a  file  with  the  naming  convention  of
              "slurm.conf.<datetime>" in the same directory as the original slurm.conf file.   If
              a filename is given that file location with a .<datetime> suffix is created.

INTERACTIVE COMMANDS

       NOTE:  All  commands  listed  below  can  be  used in the interactive mode, but NOT on the
       initial command line.

       all    Show all partitions, their jobs and jobs  steps.  This  causes  information  to  be
              displayed  about  partitions  that are configured as hidden and partitions that are
              unavailable to user's group.

       details
              Causes the show  command  to  provide  additional  details  where  available.   Job
              information will include CPUs and NUMA memory allocated on each node.  Note that on
              computers with hyperthreading enabled and Slurm configured to allocate cores,  each
              listed  CPU  represents  one  physical  core.  Each hyperthread on that core can be
              allocated a separate task, so a job's CPU count and task count may differ.  See the
              --cpu-bind   and  --mem-bind  option  descriptions  in  srun  man  pages  for  more
              information.  The details option is currently  only  supported  for  the  show  job
              command.

       exit   Terminate scontrol interactive session.

       hide   Do  not  display  partition,  job  or jobs step information for partitions that are
              configured as hidden or partitions that are unavailable to the user's group.   This
              is the default behavior.

       oneliner
              Print information one line per record.

       quiet  Print no warning or informational messages, only fatal error messages.

       quit   Terminate the execution of scontrol.

       verbose
              Print detailed event logging.  This includes time-stamps on data structures, record
              counts, etc.

       !!     Repeat the last command executed.

JOBS - SPECIFICATIONS FOR UPDATE COMMAND

       Note that update requests done by either root, SlurmUser or Administrators are not subject
       to  certain  restrictions.  For instance, if an Administrator changes the QOS on a pending
       job, certain limits such as the TimeLimit will not be  changed  automatically  as  changes
       made by the Administrators are allowed to violate these restrictions.

       Account=<account>
              Account  name to be changed for this job's resource use.  Value may be cleared with
              blank data value, "Account=".

       AdminComment=<spec>
              Arbitrary descriptive string. Can only be set by a Slurm administrator.

       ArrayTaskThrottle=<count>
              Specify the maximum number of tasks in a job array that can  execute  at  the  same
              time.   Set  the  count to zero in order to eliminate any limit.  The task throttle
              count for a job array is reported as part of its ArrayTaskId field, preceded with a
              percent  sign.   For  example  "ArrayTaskId=1-10%2" indicates the maximum number of
              running tasks is limited to 2.

       BurstBuffer=<spec>
              Burst buffer specification to be changed for this job's resource use.  Value may be
              cleared  with  blank  data  value,  "BurstBuffer=".   Format is burst buffer plugin
              specific.

       Clusters=<spec>
              Specifies the clusters that the federated job can run on.

       ClusterFeatures=<spec>
              Specifies features that a federated  cluster  must  have  to  have  a  sibling  job
              submitted  to it. Slurm will attempt to submit a sibling job to a cluster if it has
              at least one of the specified features.

       Comment=<spec>
              Arbitrary descriptive string.

       Contiguous={yes|no}
              Set the job's requirement for  contiguous  (consecutive)  nodes  to  be  allocated.
              Possible  values  are  "YES"  and  "NO".   Only the Slurm administrator or root can
              change this parameter.

       CoreSpec=<count>
              Number of cores to reserve per node for system use.  The job will  be  charged  for
              these  cores,  but  be  unable  to  use  them.   Will  be  reported  as  "*" if not
              constrained.

       CPUsPerTask=<count>
              Change the CPUsPerTask job's value.

       Deadline=<time_spec>
              It accepts times of the form HH:MM:SS to specify a deadline to a job at a  specific
              time  of  day (seconds are optional).  You may also specify midnight, noon, fika (3
              PM) or teatime (4 PM) and you can have a time-of-day suffixed with AM or PM  for  a
              deadline  in  the  morning  or the evening.  You can specify a deadline for the job
              with a date of the form MMDDYY or MM/DD/YY or MM.DD.YY,  or  a  date  and  time  as
              YYYY-MM-DD[THH:MM[:SS]].   You  can  also  give  times like now + count time-units,
              where the time-units can be seconds (default), minutes, hours, days, or  weeks  and
              you  can  tell Slurm to put a deadline for tomorrow with the keyword tomorrow.  The
              specified deadline must be later than the current time.  Only pending jobs can have
              the  deadline  updated.   Only  the  Slurm  administrator  or  root can change this
              parameter.

       DelayBoot=<time_spec>
              Change the time to decide whether to reboot nodes in order to satisfy job's feature
              specification  if  the job has been eligible to run for less than this time period.
              See salloc/sbatch man pages option --delay-boot.

       Dependency=<dependency_list>
              Defer job's initiation until specified job dependency specification  is  satisfied.
              Cancel   dependency   with   an   empty   dependency_list   (e.g.   "Dependency=").
              <dependency_list> is  of  the  form  <type:job_id[:job_id][,type:job_id[:job_id]]>.
              Many jobs can share the same dependency and these jobs may even belong to different
              users.

              after:job_id[:jobid...]
                     This job can begin execution after the specified jobs have  begun  execution
                     or been canceled.

              afterany:job_id[:jobid...]
                     This job can begin execution after the specified jobs have terminated.

              afternotok:job_id[:jobid...]
                     This  job  can  begin  execution after the specified jobs have terminated in
                     some failed state (non-zero exit code, node failure, timed out, etc).

              afterok:job_id[:jobid...]
                     This job can begin execution after  the  specified  jobs  have  successfully
                     executed (ran to completion with an exit code of zero).

              singleton
                     This  job can begin execution after any previously launched jobs sharing the
                     same job name and user have terminated.  In other words,  only  one  job  by
                     that name and owned by that user can be running or suspended at any point in
                     time.

       EligibleTime=<time_spec>
              See StartTime.

       EndTime
              The time the job is expected to terminate based on the job's time limit.  When  the
              job ends sooner, this field will be updated with the actual end time.

       ExcNodeList=<nodes>
              Set  the  job's  list  of excluded node. Multiple node names may be specified using
              simple node range expressions (e.g. "lx[10-20]").  Value may be cleared with  blank
              data value, "ExcNodeList=".

       Features=<features>
              Set  the  job's  required node features.  The list of features may include multiple
              feature names separated by ampersand (AND) and/or vertical bar (OR) operators.  For
              example: Features="opteron&video" or Features="fast|faster".  In the first example,
              only nodes having both the feature "opteron" AND the feature "video" will be  used.
              There  is no mechanism to specify that you want one node with feature "opteron" and
              another node with feature "video" in case no node has both features.  If  only  one
              of  a  set of possible options should be used for all allocated nodes, then use the
              OR  operator  and  enclose  the  options  within  square  brackets.   For  example:
              "Features=[rack1|rack2|rack3|rack4]"  might  be used to specify that all nodes must
              be allocated on a single rack of the cluster, but any of those four  racks  can  be
              used.   A  request can also specify the number of nodes needed with some feature by
              appending  an  asterisk  and  count  after   the   feature   name.    For   example
              "Features=graphics*4"  indicates  that  at least four allocated nodes must have the
              feature "graphics."  Parenthesis are  also  supported  for  features  to  be  ANDed
              together.    For   example  "Features=[(knl&a2a&flat)*4&haswell*2]"  indicates  the
              resource allocation should include 4 nodes with ALL of the features  "knl",  "a2a",
              and  "flat"  plus 2 nodes with the feature "haswell".  Constraints with node counts
              may only be combined with AND operators.  Value may  be  cleared  with  blank  data
              value, for example "Features=".

       Gres=<list>
              Specifies  a  comma-delimited  list of generic consumable resources.  The format of
              each entry on the list is "name[:count[*cpu]]".  The name is that of the consumable
              resource.   The  count  is the number of those resources with a default value of 1.
              The specified resources will be allocated to the job on each node allocated  unless
              "*cpu"  is  appended,  in  which  case the resources will be allocated on a per cpu
              basis.  The available generic consumable resources is configurable  by  the  system
              administrator.   A  list  of available generic consumable resources will be printed
              and the command will exit if the  option  argument  is  "help".   Examples  of  use
              include "Gres=gpus:2*cpu,disk=40G" and "Gres=help".

       JobId=<job_list>
              Identify  the  job(s) to be updated.  The job_list may be a comma separated list of
              job IDs.  Either JobId or JobName is required.

       Licenses=<name>
              Specification of licenses (or  other  resources  available  on  all  nodes  of  the
              cluster) as described in salloc/sbatch/srun man pages.

       MailType=<types>
              Set  the  mail  event types. Valid type values are NONE, BEGIN, END, FAIL, REQUEUE,
              ALL (equivalent to BEGIN, END, FAIL,  REQUEUE,  and  STAGE_OUT),  STAGE_OUT  (burst
              buffer  stage  out  and  teardown completed), TIME_LIMIT, TIME_LIMIT_90 (reached 90
              percent  of  time  limit),  TIME_LIMIT_80  (reached  80  percent  of  time  limit),
              TIME_LIMIT_50  (reached  50 percent of time limit) and ARRAY_TASKS (send emails for
              each array task). Multiple type values may be specified in a comma separated  list.
              Unless  the  ARRAY_TASKS  option is specified, mail notifications on job BEGIN, END
              and FAIL apply to a job array as a whole rather than  generating  individual  email
              messages for each task in the job array.

       MailUser=<name>
              Set  the  user  to receive email notification of state changes. A blank string will
              set the mail user to the default which is the submitting user.

       MinCPUsNode=<count>
              Set the job's minimum number of CPUs per node to the specified value.

       MinMemoryCPU=<megabytes>
              Set the job's minimum real memory required  per  allocated  CPU  to  the  specified
              value. Either MinMemoryCPU or MinMemoryNode may be set, but not both.

       MinMemoryNode=<megabytes>
              Set the job's minimum real memory required per node to the specified value.  Either
              MinMemoryCPU or MinMemoryNode may be set, but not both.

       MinTmpDiskNode=<megabytes>
              Set the job's minimum temporary disk space  required  per  node  to  the  specified
              value.  Only the Slurm administrator or root can change this parameter.

       TimeMin=<timespec>
              Change TimeMin value which specifies the minimum time limit minutes of the job.

       JobName=<name>
              Identify  the  name  of  jobs to be modified or set the job's name to the specified
              value.  When used to identify jobs to be modified, all jobs belonging to all  users
              are  modified unless the UserID option is used to identify a specific user.  Either
              JobId or JobName is required.

       Name[=<name>]
              See JobName.

       Nice[=<adjustment>]
              Update the  job  with  an  adjusted  scheduling  priority  within  Slurm.  With  no
              adjustment value the scheduling priority is decreased by 100. A negative nice value
              increases the priority,  otherwise  decreases  it.  The  adjustment  range  is  +/-
              2147483645. Only privileged users can specify a negative adjustment.

       NodeList=<nodes>
              Change the nodes allocated to a running job to shrink its size.  The specified list
              of nodes must be a subset of the nodes currently allocated  to  the  job.  Multiple
              node names may be specified using simple node range expressions (e.g. "lx[10-20]").
              After a job's allocation is  reduced,  subsequent  srun  commands  must  explicitly
              specify node and task counts which are valid for the new allocation.

       NumCPUs=<min_count>[-<max_count>]
              Set the job's minimum and optionally maximum count of CPUs to be allocated.

       NumNodes=<min_count>[-<max_count>]
              Set  the  job's  minimum and optionally maximum count of nodes to be allocated.  If
              the job is already running, use this to specify a node count  less  than  currently
              allocated and resources previously allocated to the job will be relinquished. After
              a job's allocation is reduced, subsequent srun  commands  must  explicitly  specify
              node  and task counts which are valid for the new allocation. Also see the NodeList
              parameter above. This is the same as ReqNodes.

       NumTasks=<count>
              Set the job's count of required tasks to the specified value. This is the  same  as
              ReqProcs.

       OverSubscribe={yes|no}
              Set  the job's ability to share compute resources (i.e. individual CPUs) with other
              jobs. Possible values are "YES" and "NO".  This option  can  only  be  changed  for
              pending jobs.

       Partition=<name>
              Set the job's partition to the specified value.

       Priority=<number>
              Set  the  job's  priority to the specified value.  Note that a job priority of zero
              prevents the job from ever being scheduled.  By setting a job's priority to zero it
              is  held.   Set  the  priority to a non-zero value to permit it to run.  Explicitly
              setting a job's priority clears any previously  set  nice  value  and  removes  the
              priority/multifactor  plugin's  ability  to  manage  a job's priority.  In order to
              restore the priority/multifactor plugin's ability to manage a job's priority,  hold
              and  then release the job.  Only the Slurm administrator or root can increase job's
              priority.

       QOS=<name>
              Set the job's QOS (Quality Of Service)  to  the  specified  value.   Value  may  be
              cleared with blank data value, "QOS=".

       Reboot={yes|no}
              Set  the  job's  flag that specifies whether to force the allocated nodes to reboot
              before starting the job. This is only supported with some system configurations and
              therefore it could be silently ignored.

       ReqCores=<count>
              Change the job's requested Cores count.

       ReqNodeList=<nodes>
              Set  the  job's  list  of required node. Multiple node names may be specified using
              simple node range expressions (e.g. "lx[10-20]").  Value may be cleared with  blank
              data value, "ReqNodeList=".

       ReqNodes=<min_count>[-<max_count>]
              See NumNodes.

       ReqProcs=<count>
              See NumTasks.

       ReqSockets=<count>
              Change the job's requested socket count.

       ReqThreads=<count>
              Change the job's requested threads count.

       IRequeue={0|1}
              Stipulates  whether  a job should be requeued after a node failure: 0 for no, 1 for
              yes.

       ReservationName=<name>
              Set the job's reservation to the specified value.  Value may be cleared with  blank
              data value, "ReservationName=".

       ResetAccrueTime
              Reset  the  job's  accrue  time value to 0 meaning it will lose any time previously
              accrued for priority.  Helpful if you have a large queue of  jobs  already  in  the
              queue  and want to start limiting how many jobs can accrue time without waiting for
              the queue to flush out.

       SiteFactor=<account>
              Specify the job's admin priority  factor  in  the  range  of  +/-2147483645.   Only
              privileged users can modify the value.

       StdOut=<filepath>
              Set the batch job's stdout file path.

       Shared={yes|no}
              See OverSubscribe option above.

       StartTime=<time_spec>
              Set  the  job's earliest initiation time.  It accepts times of the form HH:MM:SS to
              run a job at a specific time of day (seconds  are  optional).   (If  that  time  is
              already  past, the next day is assumed.)  You may also specify midnight, noon, fika
              (3 PM) or teatime (4 PM) and you can have a time-of-day suffixed with AM or PM  for
              running  in  the morning or the evening.  You can also say what day the job will be
              run, by specifying a date of the form MMDDYY or MM/DD/YY or MM.DD.YY, or a date and
              time  as  YYYY-MM-DD[THH:MM[:SS]].   You  can  also  give  times  like  now + count
              time-units, where the time-units can be seconds (default), minutes, hours, days, or
              weeks and you can tell Slurm to run the job today with the keyword today and to run
              the job tomorrow with the keyword tomorrow.

              Notes on date/time specifications:
               - although the 'seconds' field of the HH:MM:SS time specification  is  allowed  by
              the  code,  note that the poll time of the Slurm scheduler is not precise enough to
              guarantee dispatch of the job on the exact second.  The job  will  be  eligible  to
              start  on  the  next  poll  following  the  specified time. The exact poll interval
              depends on the Slurm scheduler (e.g., 60 seconds with the default sched/builtin).
               - if no time (HH:MM:SS) is specified, the default is (00:00:00).
               - if a date is specified without a year (e.g., MM/DD) then  the  current  year  is
              assumed,  unless  the combination of MM/DD and HH:MM:SS has already passed for that
              year, in which case the next year is used.

       Switches=<count>[@<max-time-to-wait>]
              When a tree topology is used, this defines the maximum count  of  switches  desired
              for  the job allocation. If Slurm finds an allocation containing more switches than
              the count specified, the job remain pending until it  either  finds  an  allocation
              with  desired switch count or the time limit expires. By default there is no switch
              count limit and no time limit delay. Set the count to zero in order  to  clean  any
              previously  set  count  (disabling the limit).  The job's maximum time delay may be
              limited by the system administrator  using  the  SchedulerParameters  configuration
              parameter with the max_switch_wait parameter option.  Also see wait-for-switch.

       wait-for-switch=<seconds>
              Change max time to wait for a switch <seconds> secs.

       TasksPerNode=<count>
              Change the job's requested TasksPerNode.

       ThreadSpec=<count>
              Number  of threads to reserve per node for system use.  The job will be charged for
              these threads, but be unable  to  use  them.   Will  be  reported  as  "*"  if  not
              constrained.

       TimeLimit=<time>
              The   job's   time   limit.    Output  format  is  [days-]hours:minutes:seconds  or
              "UNLIMITED".  Input format (for update command) set  is  minutes,  minutes:seconds,
              hours:minutes:seconds,          days-hours,          days-hours:minutes          or
              days-hours:minutes:seconds.  Time resolution is one minute and  second  values  are
              rounded up to the next minute.  If changing the time limit of a job, either specify
              a new time limit value or precede the time and equal sign with  a  "+"  or  "-"  to
              increment  or  decrement the current time limit (e.g. "TimeLimit+=30"). In order to
              increment or decrement the current time limit, the JobId specification must precede
              the TimeLimit specification.  Note that incrementing or decrementing the time limit
              for a job array is only allowed before the job array has been split into more  than
              one job record.  Only the Slurm administrator or root can increase job's TimeLimit.

       UserID=<UID or name>
              Used  with  the JobName option to identify jobs to be modified.  Either a user name
              or numeric ID (UID), may be specified.

       WCKey=<key>
              Set the job's workload characterization key to the specified value.

       WorkDir=<directory_name>
              Set the job's working directory to the specified value. Note that this may only  be
              set for jobs in the PENDING state, and that jobs may fail to launch if they rely on
              relative paths to the originally submitted WorkDir.

JOBS - SPECIFICATIONS FOR SHOW COMMAND

       The "show" command, when used with the "job" or "job  <jobid>"  entity  displays  detailed
       information  about  a  job  or  jobs.   Much of this information may be modified using the
       "update job" command as described above.  However, the following fields displayed  by  the
       show job command are read-only and cannot be modified:

       AllocNode:Sid
              Local node and system id making the resource allocation.

       BatchFlag
              Jobs  submitted  using  the sbatch command have BatchFlag set to 1.  Jobs submitted
              using other commands have BatchFlag set to 0.

       ExitCode=<exit>:<sig>
              Exit status reported for the job by the wait() function.  The first number  is  the
              exit  code,  typically  as  set  by  the exit() function.  The second number of the
              signal that caused the process to terminate if it was terminated by a signal.

       GroupId
              The group under which the job was submitted.

       JobState
              The current state of the job.

       NodeListIndices
              The NodeIndices expose the internal indices into the node table associated with the
              node(s) allocated to the job.

       NtasksPerN:B:S:C=<tasks_per_node>:<tasks_per_baseboard>:<tasks_per_socket>:<tasks_per_core>
              Specifies the  number  of  tasks  to  be  started  per  hardware  component  (node,
              baseboard, socket and core).  Unconstrained values may be shown as "0" or "*".

       PreemptEligibleTime
              Time the job becomes eligible for preemption. Modified by PreemptExemptTime, either
              from the global option in slurm.conf or the job QOS. This is hidden if the job  has
              not started or if PreemptMode=OFF.

       PreemptTime
              Time  at  which  job was signaled that it was selected for preemption.  (Meaningful
              only for PreemptMode=CANCEL and  the  partition  or  QOS  with  which  the  job  is
              associated  has  a  GraceTime  value designated.) This is hidden if the job has not
              started or if PreemptMode=OFF.

       PreSusTime
              Time the job ran prior to last suspend.

       Reason The reason job is not running: e.g., waiting "Resources".

       ReqB:S:C:T=<baseboard_count>:<socket_per_baseboard_count>:<core_per_socket_count>:<thread_per_core_count>
              Specifies   the  count  of  various  hardware  components  requested  by  the  job.
              Unconstrained values may be shown as "0" or "*".

       SecsPreSuspend=<seconds>
              If the job is suspended, this is the run time accumulated by the job  (in  seconds)
              prior to being suspended.

       Socks/Node=<count>
              Count of desired sockets per node

       SubmitTime
              The  time  and  date stamp (in localtime) the job was submitted.  The format of the
              output is identical to that of the EndTime field.

              NOTE: If a job is requeued, the submit time  is  reset.   To  obtain  the  original
              submit  time it is necessary to use the "sacct -j <job_id[.<step_id>]" command also
              designating the -D or --duplicate option to display all  duplicate  entries  for  a
              job.

       SuspendTime
              Time the job was last suspended or resumed.

              NOTE on information displayed for various job states: When you submit a request for
              the "show job" function the scontrol process makes an RPC request call to slurmctld
              with  a REQUEST_JOB_INFO message type.  If the state of the job is PENDING, then it
              returns some detail information such as: min_nodes, min_procs, cpus_per_task,  etc.
              If  the  state is other than PENDING the code assumes that it is in a further state
              such as RUNNING, COMPLETE, etc. In these cases the code explicitly returns zero for
              these  values.  These  values  are  meaningless  once  the  job resources have been
              allocated and the job has started.

STEPS - SPECIFICATIONS FOR UPDATE COMMAND

       StepId=<job_id>[.<step_id>]
              Identify the step to be updated.  If  the  job_id  is  given,  but  no  step_id  is
              specified   then   all  steps  of  the  identified  job  will  be  modified.   This
              specification is required.

       TimeLimit=<time>
              The  job's  time  limit.   Output   format   is   [days-]hours:minutes:seconds   or
              "UNLIMITED".   Input  format  (for update command) set is minutes, minutes:seconds,
              hours:minutes:seconds,          days-hours,          days-hours:minutes          or
              days-hours:minutes:seconds.   Time  resolution  is one minute and second values are
              rounded up to the next minute.  If changing  the  time  limit  of  a  step,  either
              specify  a  new time limit value or precede the time with a "+" or "-" to increment
              or decrement the current time limit (e.g. "TimeLimit=+30"). In order  to  increment
              or  decrement  the  current  time  limit, the StepId specification must precede the
              TimeLimit specification.

NODES - SPECIFICATIONS FOR UPDATE COMMAND

       NodeName=<name>
              Identify the node(s) to be updated. Multiple node  names  may  be  specified  using
              simple node range expressions (e.g. "lx[10-20]"). This specification is required.

       ActiveFeatures=<features>
              Identify  the  feature(s)  currently  active on the specified node.  Any previously
              active feature specification will be overwritten with  the  new  value.   Also  see
              AvailableFeatures.     Typically    ActiveFeatures    will    be    identical    to
              AvailableFeatures; however ActiveFeatures may be configured  as  a  subset  of  the
              AvailableFeatures. For example, a node may be booted in multiple configurations. In
              that case, all possible configurations  may  be  identified  as  AvailableFeatures,
              while ActiveFeatures would identify the current node configuration.

       AvailableFeatures=<features>
              Identify  the  feature(s)  available on the specified node.  Any previously defined
              available  feature  specification  will  be  overwritten  with   the   new   value.
              AvailableFeatures assigned via scontrol will only persist across the restart of the
              slurmctld daemon with the -R  option  and  state  files  preserved  or  slurmctld's
              receipt  of  a  SIGHUP.   Update slurm.conf with any changes meant to be persistent
              across normal restarts of slurmctld or the execution of scontrol reconfig.

              Note: Available features being removed via scontrol must not be active (i.e. remove
              them from ActiveFeatures first).

       Comment=<comment>
              Arbitrary descriptive string.  Use quotes to enclose a comment having more than one
              word

       CpuBind=<node>
              Specify the task binding mode to be used  by  default  for  this  node.   Supported
              options  include:  "none",  "board",  "socket", "ldom" (NUMA), "core", "thread" and
              "off" (remove previous binding mode).

       Extra=<comment>
              Arbitrary string on the node. Use quotes to enclose a string having more  than  one
              word.

       Gres=<gres>
              Identify  generic  resources  to  be  associated  with  the  specified  node.   Any
              previously defined generic resources  will  be  overwritten  with  the  new  value.
              Specifications  for  multiple  generic  resources  should be comma separated.  Each
              resource specification consists of a name followed by  an  optional  colon  with  a
              numeric  value  (default value is one) (e.g. "Gres=bandwidth:10000").  Modification
              of GRES count associated with specific files (e.g. GPUs) is not allowed other  than
              to set their count on a node to zero.  In order to change the GRES count to another
              value, modify your slurm.conf and gres.conf files and restart daemons.  If GRES are
              associated  with specific sockets, that information will be reported For example if
              all 4 GPUs on a node are all associated with socket zero,  then  "Gres=gpu:4(S:0)".
              If  associated  with  sockets 0 and 1 then "Gres=gpu:4(S:0-1)".  The information of
              which specific GPUs are associated with specific GPUs is  not  reported,  but  only
              available  by  parsing the gres.conf file.  Generic resources assigned via scontrol
              will only persist across the restart of the slurmctld daemon with the -R option and
              state  files  preserved or slurmctld's receipt of a SIGHUP.  Update slurm.conf with
              any changes meant to be persistent across  normal  restarts  of  slurmctld  or  the
              execution of scontrol reconfig.

       NodeAddr=<node address>
              Name that a node should be referred to in establishing a communications path.  This
              name will be used as an argument to the getaddrinfo() function for  identification.
              If  a  node range expression is used to designate multiple nodes, they must exactly
              match the entries in  the  NodeName  (e.g.  "NodeName=lx[0-7]  NodeAddr=elx[0-7]").
              NodeAddr may also contain IP addresses.

       NodeHostname=<node hostname>
              Typically this would be the string that "/bin/hostname -s" returns.  It may also be
              the  fully  qualified  domain  name  as  returned  by  "/bin/hostname   -f"   (e.g.
              "foo1.bar.com"), or any valid domain name associated with the host through the host
              database (/etc/hosts) or DNS, depending on the resolver settings. Note that if  the
              short  form of the hostname is not used, it may prevent use of hostlist expressions
              (the numeric portion in brackets must be at the end of the string).  A  node  range
              expression  can  be  used  to specify a set of nodes. If an expression is used, the
              number of nodes identified by NodeHostname must be identical to the number of nodes
              identified by NodeName.

       Reason=<reason>
              Identify  the  reason  the node is in a "DOWN", "DRAINED", "DRAINING", "FAILING" or
              "FAIL" state.  Use quotes to enclose a reason having more than one word.

       State=<state>
              Assign one of the following states/actions to the node(s) specified by  the  update
              command.

              CANCEL_REBOOT
                     Cancels  a  pending  reboot  on  the  node  (same  as scontrol cancel_reboot
                     <node>).

              DOWN   Stop all running and suspended jobs and make the node  unavailable  for  new
                     jobs.

              DRAIN  Indicates  that  no  new jobs may be started on this node. Existing jobs are
                     allowed to run to completion, leaving the node in a DRAINED state  once  all
                     the jobs have completed.

              FAIL   Similar to DRAIN except that some applications will seek to relinquish those
                     nodes before the job completes.

              FUTURE Indicates the node is not fully configured, but is expected to be  available
                     at some point in the future.

              NoResp This  will  set  the  "Not  Responding" flag for a node without changing its
                     underlying state.

              POWER_DOWN
                     Will use the configured SuspendProgram program to explicitly place a node in
                     power  saving  mode.  If  a  node is already in the process of being powered
                     down, the command will only change the state of the node but won't have  any
                     effect  until the configured SuspendTimeout is reached.  Use of this command
                     can be useful in situations  where  a  ResumeProgram,  like  capmc  in  Cray
                     machines,  is  stalled and one wants to restore the node to "IDLE" manually.
                     In this case rebooting the node and setting the state to  "POWER_DOWN"  will
                     cancel the previous "POWER_UP" state and the node will become "IDLE".

              POWER_DOWN_ASAP
                     Will  drain the node and mark it for power down. Currently running jobs will
                     complete first and no additional jobs will be allocated to the node.

              POWER_DOWN_FORCE
                     Will cancel all jobs on the node, power it down,  and  reset  its  state  to
                     "IDLE".

              POWER_UP
                     Will  use the configured ResumeProgram program to explicitly move a node out
                     of power saving mode. If a node is already in the process of  being  powered
                     up,  the  command  will only change the state of the node but won't have any
                     effect until the configured ResumeTimeout is reached.

              RESUME Not an actual node state, but will change a node state from DRAIN, DRAINING,
                     DOWN  or  REBOOT to IDLE and NoResp.  slurmctld will then attempt to contact
                     slurmd to request that the node register itself. Once registered,  the  node
                     state will then remove the NoResp flag and will resume normal operations. It
                     will also clear the POWERING_DOWN state of a node and make it eligible to be
                     allocted.

              UNDRAIN
                     Clears  the  node  from being drained (like RESUME), but will not change the
                     node's base state (e.g. DOWN). UNDRAIN requires a  valid  node  registration
                     before  new  jobs  can  be  scheduled on the node.  Setting a node DOWN will
                     cause all running and suspended jobs on that node to be terminated.

              While all of the above states are valid, some of them are not valid new node states
              given their prior state.

              NOTE: The scontrol command should not be used to change node state on Cray systems.
              Use Cray tools such as xtprocadmin instead.

       Weight=<weight>
              Identify weight to be associated with specified nodes. This allows dynamic  changes
              to  weight  associated  with  nodes,  which  will  be  used for the subsequent node
              allocation decisions.  Weight assigned via scontrol will only  persist  across  the
              restart  of  the  slurmctld  daemon with the -R option and state files preserved or
              slurmctld's receipt of a SIGHUP.  Update slurm.conf with any changes  meant  to  be
              persistent  across  normal  restarts  of  slurmctld  or  the  execution of scontrol
              reconfig.

NODES - SPECIFICATIONS FOR SHOW COMMAND

       AllocMem
              The total memory, in MB, currently allocated by jobs on the node.

       CPUSpecList
              The list of Slurm abstract CPU IDs on this node reserved for exclusive use  by  the
              Slurm compute node daemons (slurmd, slurmstepd).

       FreeMem
              The total memory, in MB, currently free on the node as reported by the OS.

       LastBusyTime
              The last time the node was busy (i.e. last time the node had jobs on it). This time
              is used in PowerSave to determine when to suspend nodes  (e.g.  now  -  LastBusy  >
              SuspendTime).

       MemSpecLimit
              The  combined  memory  limit, in megabytes, on this node for the Slurm compute node
              daemons (slurmd, slurmstepd).

       RealMemory
              The total memory, in MB, on the node.

       State  Identify the state(s) assigned to the node with '+' delimited state flags.

              States:

              ALLOCATED
                     Indicates that the node has all CPUs allocated  to  job(s)  running  on  the
                     node.

              DOWN   The node does not have any running jobs and is unavailable for new work.

              ERROR  The  node  is in an error state. Consult the logs for more information about
                     what caused this state.

              FUTURE The node is currently not fully configured, but expected to be available  at
                     some point in the indefinite future for use.

              IDLE   Indicates  that  the  node is available for work but does not currently have
                     any jobs assigned to it.

              MIXED  Indicates that the node is in multiple states.  For instance if only part of
                     the  node  is  ALLOCATED  and the rest of the node is IDLE the state will be
                     MIXED.

              UNKNOWN
                     The node has not yet registered with the controller and  its  state  is  not
                     known.

              Flags:

              CLOUD  Indicates  that  the node is configured as a cloud node, to be brought up on
                     demand, but not currently running.

              COMPLETING
                     Indicates that the only job on the node or that all jobs on the node are  in
                     the process of completing.

              DRAIN  The  node  is not accepting any new jobs and any currently running jobs will
                     complete.

              DYNAMIC
                     Slurm allows you to define multiple types of nodes in a FUTURE state.   When
                     starting slurmd on a node you can specify the -F flag to have the node match
                     and use an existing definition in your slurm.conf file.  The  DYNAMIC  state
                     indicates that the node was started as a Dynamic Future node.

              INVALID_REG
                     The node did not register correctly with the controller.

              MAINTENANCE
                     The node is currently in a reservation that includes the maintenance flag.

              NOT_RESPONDING
                     Node is not responding.

              PERFCTRS
                     Indicates that Network Performance Counters associated with this node are in
                     use, rendering this node as not usable for any other jobs.

              POWER_DOWN
                     Node is pending power down.

              POWERED_DOWN
                     Node is currently powered down and not capable of running any jobs.

              POWERING_DOWN
                     Node is in the process of powering down.

              POWERING_UP
                     Node is in the process of powering up.

              PLANNED
                     The node is earmarked for a job that will start in the future.

              REBOOT_ISSUED
                     A reboot request has been sent  to  the  agent  configured  to  handle  this
                     request.

              REBOOT_REQUESTED
                     A request to reboot this node has been made, but hasn't been handled yet.

              RESERVED
                     Indicates  the  node  is  in  an  advanced  reservation  and  not  generally
                     available.

       The meaning of the energy information is as follows:

              CurrentWatts
                     The instantaneous power consumption of the node at the time of the last node
                     energy accounting sample, in watts.

              LowestJoules
                     The  energy consumed by the node between the last time it was powered on and
                     the last time it was registered by slurmd, in joules.

              ConsumedJoules
                     The energy consumed by the node between the last time it was  registered  by
                     the slurmd daemon and the last node energy accounting sample, in joules.

              If  the  reported  value  is  "n/s"  (not supported), the node does not support the
              configured AcctGatherEnergyType plugin. If  the  reported  value  is  zero,  energy
              accounting for nodes is disabled.

       The meaning of the external sensors information is as follows:

              ExtSensorsJoules
                     The  energy consumed by the node between the last time it was powered on and
                     the last external sensors plugin node sample, in joules.

              ExtSensorsWatts
                     The instantaneous power consumption of the node at  the  time  of  the  last
                     external sensors plugin node sample, in watts.

              ExtSensorsTemp
                     The  temperature of the node at the time of the last external sensors plugin
                     node sample, in celsius.

              If the reported value is "n/s" (not supported),  the  node  does  not  support  the
              configured ExtSensorsType plugin.

FRONTEND - SPECIFICATIONS FOR UPDATE COMMAND

       FrontendName=<name>
              Identify the front end node to be updated. This specification is required.

       Reason=<reason>
              Identify  the  reason  the  node  is  in  a "DOWN" or "DRAIN" state.  Use quotes to
              enclose a reason having more than one word.

       State=<state>
              Identify the state to be assigned to  the  front  end  node.  Possible  values  are
              "DOWN",  "DRAIN" or "RESUME".  If you want to remove a front end node from service,
              you typically want to set its state to "DRAIN".  "RESUME" is  not  an  actual  node
              state,  but  will  return  a  "DRAINED",  "DRAINING",  or  "DOWN" front end node to
              service, either "IDLE" or "ALLOCATED" state as appropriate.  Setting  a  front  end
              node  "DOWN"  will  cause  all  running  and  suspended  jobs  on  that  node to be
              terminated.

PARTITIONS - SPECIFICATIONS FOR CREATE, UPDATE, AND DELETE COMMANDS

       AllowGroups=<name>
              Identify the user groups which may use this  partition.   Multiple  groups  may  be
              specified  in  a  comma  separated list.  To permit all groups to use the partition
              specify "AllowGroups=ALL".

       AllocNodes=<name>
              Comma separated list of nodes from which users can execute jobs in  the  partition.
              Node names may be specified using the node range expression syntax described above.
              The default value is "ALL".

       Alternate=<partition name>
              Alternate partition to be used if  the  state  of  this  partition  is  "DRAIN"  or
              "INACTIVE."  The value "NONE" will clear a previously set alternate partition.

       CpuBind=<node>
              Specify  the task binding mode to be used by default for this partition.  Supported
              options include: "none", "board", "socket", "ldom"  (NUMA),  "core",  "thread"  and
              "off" (remove previous binding mode).

       Default={yes|no}
              Specify  if this partition is to be used by jobs which do not explicitly identify a
              partition to use.  Possible output values are "YES" and "NO".  In order  to  change
              the  default partition of a running system, use the scontrol update command and set
              Default=yes for the partition that you want to become the new default.

       DefaultTime=<time>
              Run time limit used for jobs that don't specify a value. If not  set  then  MaxTime
              will be used.  Format is the same as for MaxTime.

       DefMemPerCPU=<MB>
              Set  the  default  memory  to be allocated per CPU for jobs in this partition.  The
              memory size is specified in megabytes.

       DefMemPerNode=<MB>
              Set the default memory to be allocated per node for jobs in  this  partition.   The
              memory size is specified in megabytes.

       DisableRootJobs={yes|no}
              Specify if jobs can be executed as user root.  Possible values are "YES" and "NO".

       GraceTime=<seconds>
              Specifies,  in  units of seconds, the preemption grace time to be extended to a job
              which has been selected for preemption.  The default value is zero,  no  preemption
              grace   time   is   allowed  on  this  partition  or  qos.   (Meaningful  only  for
              PreemptMode=CANCEL)

       Hidden={yes|no}
              Specify if the  partition  and  its  jobs  should  be  hidden  from  view.   Hidden
              partitions  will  by  default  not be reported by Slurm APIs or commands.  Possible
              values are "YES" and "NO".

       JobDefaults=<specs>
              Specify job default values using  a  comma-delimited  list  of  "key=value"  pairs.
              Supported keys include

              DefCpuPerGPU  Default number of CPUs per allocated GPU.

              DefMemPerGPU  Default memory limit (in megabytes) per allocated GPU.

       MaxMemPerCPU=<MB>
              Set  the  maximum  memory  to be allocated per CPU for jobs in this partition.  The
              memory size is specified in megabytes.

       MaxMemPerCNode=<MB>
              Set the maximum memory to be allocated per node for jobs in  this  partition.   The
              memory size is specified in megabytes.

       MaxNodes=<count>
              Set  the  maximum  number of nodes which will be allocated to any single job in the
              partition. Specify a number, "INFINITE" or "UNLIMITED".  Changing the MaxNodes of a
              partition has no effect upon jobs that have already begun execution.

       MaxTime=<time>
              The  maximum  run  time for jobs.  Output format is [days-]hours:minutes:seconds or
              "UNLIMITED".  Input  format  (for  update  command)  is  minutes,  minutes:seconds,
              hours:minutes:seconds,          days-hours,          days-hours:minutes          or
              days-hours:minutes:seconds.  Time resolution is one minute and  second  values  are
              rounded  up  to the next minute.  Changing the MaxTime of a partition has no effect
              upon jobs that have already begun execution.

       MinNodes=<count>
              Set the minimum number of nodes which will be allocated to any single  job  in  the
              partition.   Changing the MinNodes of a partition has no effect upon jobs that have
              already begun execution. Increasing  this  value  may  prevent  pending  jobs  from
              starting,  even if they were submitted without -N/--nodes specification.  If you do
              get in that situation, updating the MinNodes value  of  a  pending  job  using  the
              scontrol command will allow that job to be scheduled.

       Nodes=<name>
              Identify  the node(s) to be associated with this partition. Multiple node names may
              be specified using simple node range expressions  (e.g.  "lx[10-20]").   Note  that
              jobs  may  only be associated with one partition at any time.  Specify a blank data
              value to remove all nodes from a partition: "Nodes=".   Changing  the  Nodes  in  a
              partition has no effect upon jobs that have already begun execution.

       OverTimeLimit=<count>
              Number  of  minutes by which a job can exceed its time limit before being canceled.
              The configured job time limit is treated as a soft limit.  Adding OverTimeLimit  to
              the  soft limit provides a hard limit, at which point the job is canceled.  This is
              particularly useful for backfill scheduling, which bases upon each job's soft  time
              limit.   A  partition-specific OverTimeLimit will override any global OverTimeLimit
              value.  If not specified, the global OverTimeLimit value will take precedence.  May
              not  exceed 65533 minutes.  An input value of "UNLIMITED" will clear any previously
              configured partition-specific OverTimeLimit value.

       OverSubscribe={yes|no|exclusive|force}[:<job_count>]
              Specify if compute resources (i.e. individual CPUs) in this partition can be shared
              by  multiple  jobs.   Possible values are "YES", "NO", "EXCLUSIVE" and "FORCE".  An
              optional job count specifies how many jobs can be allocated to use each resource.

       PartitionName=<name>
              Identify the partition to be updated. This specification is required.

       PreemptMode=<mode>
              Reset the mechanism used to preempt  jobs  in  this  partition  if  PreemptType  is
              configured to preempt/partition_prio. The default preemption mechanism is specified
              by the cluster-wide  PreemptMode  configuration  parameter.   Possible  values  are
              "OFF", "CANCEL", "REQUEUE" and "SUSPEND".

       Priority=<count>
              Jobs  submitted  to  a  higher priority partition will be dispatched before pending
              jobs in lower priority partitions and if possible they will  preempt  running  jobs
              from  lower priority partitions.  Note that a partition's priority takes precedence
              over a job's priority.  The value may not exceed 65533.

       PriorityJobFactor=<count>
              Partition factor used by priority/multifactor plugin in calculating  job  priority.
              The value may not exceed 65533.  Also see PriorityTier.

       PriorityTier=<count>
              Jobs  submitted to a partition with a higher priority tier value will be dispatched
              before pending jobs in partition with lower priority tier value and,  if  possible,
              they   will  preempt  running jobs from partitions with lower priority tier values.
              Note that a partition's priority tier takes precedence over a job's priority.   The
              value may not exceed 65533.  Also see PriorityJobFactor.

       QOS=<QOSname|blank to remove>
              Set  the  partition  QOS  with  a QOS name or to remove the Partition QOS leave the
              option blank.

       RootOnly={yes|no}
              Specify if only allocation requests initiated by user root will be satisfied.  This
              can  be used to restrict control of the partition to some meta-scheduler.  Possible
              values are "YES" and "NO".

       ReqResv={yes|no}
              Specify if only allocation requests designating a reservation  will  be  satisfied.
              This  is  used to restrict partition usage to be allowed only within a reservation.
              Possible values are "YES" and "NO".

       Shared={yes|no|exclusive|force}[:<job_count>]
              Renamed to OverSubscribe, see option descriptions above.

       State={up|down|drain|inactive}
              Specify if jobs can be allocated nodes  or  queued  in  this  partition.   Possible
              values are "UP", "DOWN", "DRAIN" and "INACTIVE".

              UP        Designates  that  new jobs may queued on the partition, and that jobs may
                        be allocated nodes and run from the partition.

              DOWN      Designates that new jobs may be queued on the partition, but queued  jobs
                        may  not  be  allocated  nodes  and  run from the partition. Jobs already
                        running on the partition continue to run. The  jobs  must  be  explicitly
                        canceled to force their termination.

              DRAIN     Designates  that  no  new  jobs  may  be  queued  on  the  partition (job
                        submission requests will be denied  with  an  error  message),  but  jobs
                        already queued on the partition may be allocated nodes and run.  See also
                        the "Alternate" partition specification.

              INACTIVE  Designates that no new jobs may be queued  on  the  partition,  and  jobs
                        already  queued  may  not  be  allocated  nodes  and  run.   See also the
                        "Alternate" partition specification.

       TRESBillingWeights=<TRES Billing Weights>
              TRESBillingWeights is used to define the billing weights of  each  TRES  type  that
              will  be  used in calculating the usage of a job. The calculated usage is used when
              calculating fairshare and when enforcing the TRES billing limit on  jobs.   Updates
              affect  new  jobs  and  not  existing  jobs.   See the slurm.conf man page for more
              information.

RESERVATIONS - SPECIFICATIONS FOR CREATE, UPDATE, AND DELETE COMMANDS

       Reservation=<name>
              Identify the name of the reservation to be  created,  updated,  or  deleted.   This
              parameter is required for update and is the only parameter for delete.  For create,
              if you do not want to give a reservation name, use "scontrol create res ..." and  a
              name will be created automatically.

       Accounts=<account list>
              List   of   accounts   permitted   to   use   the   reserved   nodes,  for  example
              "Accounts=physcode1,physcode2".  A user in any of the accounts may use the reserved
              nodes.   A  new  reservation must specify Users or Groups and/or Accounts.  If both
              Users/Groups and Accounts are specified, a job must match both in order to use  the
              reservation.   Accounts  can also be denied access to reservations by preceding all
              of the account names with '-'. Alternately precede the equal sign  with  '-'.   For
              example,  "Accounts=-physcode1,-physcode2"  or "Accounts-=physcode1,physcode2" will
              permit any account except physcode1 and physcode2 to use the reservation.  You  can
              add  or remove individual accounts from an existing reservation by using the update
              command and adding a '+' or '-' sign before the '=' sign.  If accounts  are  denied
              access  to  a reservation (account name preceded by a '-'), then all other accounts
              are implicitly allowed to use the reservation  and  it  is  not  possible  to  also
              explicitly specify allowed accounts.

       BurstBuffer=<buffer_spec>[,<buffer_spec>,...]
              Specification  of  burst  buffer resources which are to be reserved.  "buffer_spec"
              consists of four elements: [plugin:][type:]#[units] "plugin" is  the  burst  buffer
              plugin  name, currently either "datawarp" or "generic".  If no plugin is specified,
              the reservation applies to all configured burst buffer plugins.  "type" specifies a
              Cray  generic  burst  buffer  resource,  for  example  "nodes".   if  "type" is not
              specified, the number is a measure of  storage  space.   The  "units"  may  be  "N"
              (nodes), "K|KiB", "M|MiB", "G|GiB", "T|TiB", "P|PiB" (for powers of 1024) and "KB",
              "MB", "GB", "TB", "PB" (for powers of 1000).   The  default  units  are  bytes  for
              reservations of storage space.  For example "BurstBuffer=datawarp:2TB" (reserve 2TB
              of storage plus 3 nodes from the Cray plugin) or "BurstBuffer=100GB"  (reserve  100
              GB  of  storage  from  all  configured  burst  buffer  plugins).   Jobs  using this
              reservation are not restricted to these burst buffer resources, but may  use  these
              reserved  resources  plus  any  which are generally available.  NOTE: Usually Slurm
              interprets KB, MB, GB, TB, PB, TB units as powers of 1024, but  for  Burst  Buffers
              size  specifications  Slurm supports both IEC/SI formats.  This is because the CRAY
              API for managing DataWarps supports both formats.

       CoreCnt=<num>
              This option is only supported when SelectType=select/cons_res or  select/cons_tres.
              Identify  number  of  cores  to  be  reserved.   If  NodeCnt  is  used  without the
              FIRST_CORES flag, this is the total number of cores to reserve where cores per node
              is  CoreCnt/NodeCnt.   If  a  nodelist  is  used,  or  if  NodeCnt is used with the
              FIRST_CORES flag, this should be an array of core numbers by node:  Nodes=node[1-5]
              CoreCnt=2,2,3,3,4 or flags=FIRST_CORES NodeCnt=5 CoreCnt=1,2,1,3,2.

       Licenses=<license>
              Specification  of  licenses  (or  other  resources  available  on  all nodes of the
              cluster) which are to be reserved.  License names can be followed by  a  colon  and
              count (the default count is one).  Multiple license names should be comma separated
              (e.g. "Licenses=foo:4,bar").  A new reservation must specify one or  more  resource
              to  be  included:  NodeCnt,  Nodes  and/or  Licenses.   If  a  reservation includes
              Licenses, but no NodeCnt or Nodes, then the option Flags=LICENSE_ONLY must also  be
              specified.   Jobs  using this reservation are not restricted to these licenses, but
              may use these reserved licenses plus any which are generally available.

       MaxStartDelay[=<timespec>]
              Change MaxStartDelay value which specifies the maximum time  an  eligible  job  not
              requesting  this reservation can delay a job requesting it. Default is none.  Valid
              formats   are   minutes,   minutes:seconds,   hours:minutes:seconds,    days-hours,
              days-hours:minutes,  days-hours:minutes:seconds.  Time resolution is one minute and
              second values  are  rounded  up  to  the  next  minute.  Output  format  is  always
              [days-]hours:minutes:seconds.

       NodeCnt=<num>[,<num>,...]
              Identify  number of nodes to be reserved. The number can include a suffix of "k" or
              "K", in which case the number specified is multiplied by 1024.  A  new  reservation
              must specify one or more resource to be included: NodeCnt, Nodes and/or Licenses.

       Nodes=<name>
              Identify  the  node(s)  to  be reserved. Multiple node names may be specified using
              simple node range expressions (e.g. "Nodes=lx[10-20]").  Specify a blank data value
              to  remove  all nodes from a reservation: "Nodes=".  A new reservation must specify
              one  or  more  resource  to  be  included:  NodeCnt,  Nodes  and/or   Licenses.   A
              specification   of   "ALL"   will  reserve  all  nodes.  Set  Flags=PART_NODES  and
              PartitionName= in order for changes in the nodes associated  with  a  partition  to
              also be reflected in the nodes associated with a reservation.

       StartTime=<time_spec>
              The  start  time for the reservation.  A new reservation must specify a start time.
              It accepts times of the form HH:MM:SS for a  specific  time  of  day  (seconds  are
              optional).   (If that time is already past, the next day is assumed.)  You may also
              specify midnight, noon, fika  (3  PM)  or  teatime  (4  PM)  and  you  can  have  a
              time-of-day  suffixed with AM or PM for running in the morning or the evening.  You
              can also say what day the job will be run, by specifying a date of the form  MMDDYY
              or  MM/DD/YY  or  MM.DD.YY, or a date and time as YYYY-MM-DD[THH:MM[:SS]].  You can
              also give times like now + count time-units, where the time-units  can  be  seconds
              (default),  minutes,  hours,  days,  or weeks and you can tell Slurm to run the job
              today with the keyword today and to run the job tomorrow with the keyword tomorrow.
              You cannot update the StartTime of a reservation in ACTIVE state.

       EndTime=<time_spec>
              The  end time for the reservation.  A new reservation must specify an end time or a
              duration.  Valid formats are the same as for StartTime.

       Duration=<time>
              The length of a reservation.  A new reservation must  specify  an  end  time  or  a
              duration.   Valid  formats  are  minutes,  minutes:seconds,  hours:minutes:seconds,
              days-hours, days-hours:minutes,  days-hours:minutes:seconds,  or  UNLIMITED.   Time
              resolution  is  one  minute  and  second  values are rounded up to the next minute.
              Output format is always [days-]hours:minutes:seconds.

       PartitionName=<name>
              Partition used to reserve nodes from. This will attempt to allocate  all  nodes  in
              the  specified partition unless you request fewer resources than are available with
              CoreCnt, NodeCnt or TRES. Jobs will be allowed to  use  this  reservation  even  if
              running  in  a  different  partition. There only needs to be overlapping nodes from
              that different partition and the nodes used in the reservation.

       Flags=<flags>
              Flags associated with the reservation.  You can add or remove individual flags from
              an  existing  reservation  by  adding  a  '+' or '-' sign before the '=' sign.  For
              example: Flags-=DAILY (NOTE:  this  shortcut  is  not  supported  for  all  flags).
              Currently supported flags include:

              ANY_NODES     This  is a reservation for burst buffers and/or licenses only and not
                            compute nodes.  If this flag is set, a job using this reservation may
                            use  the  associated  burst  buffers and/or licenses plus any compute
                            nodes.  If this flag is not set, a job using this reservation may use
                            only the nodes and licenses associated with the reservation.

              DAILY         Repeat the reservation at the same time every day.

              FLEX          Permit  jobs  requesting  the  reservation  to  begin  prior  to  the
                            reservation's start time, end after the reservation's end  time,  and
                            use any resources inside and/or outside of the reservation regardless
                            of any constraints possibly set in the  reservation.  A  typical  use
                            case  is  to  prevent  jobs not explicitly requesting the reservation
                            from  using  those  reserved  resources  rather  than  forcing   jobs
                            requesting  the  reservation to use those resources in the time frame
                            reserved. Another use case could  be  to  always  have  a  particular
                            number  of  nodes  with  a  specific  feature reserved for a specific
                            account so users in this account may use  this  nodes  plus  possibly
                            other nodes without this feature.

              FIRST_CORES   Use  the lowest numbered cores on a node only. Flag removal with '-='
                            is not supported.

              IGNORE_JOBS   Ignore currently running jobs when creating  the  reservation.   This
                            can  be  especially useful when reserving all nodes in the system for
                            maintenance.

              LICENSE_ONLY  See ANY_NODES.

              MAINT         Maintenance  mode,  receives  special  accounting  treatment.    This
                            reservation is permitted to use resources that are already in another
                            reservation.

              MAGNETIC      This flag allows jobs to be considered for this reservation  even  if
                            they didn't request it.

              NO_HOLD_JOBS_AFTER
                            By  default,  when a reservation ends the reservation request will be
                            removed from any pending jobs submitted to the reservation  and  will
                            be  put  into a held state.  Use this flag to let jobs run outside of
                            the reservation after the reservation is gone. Flag removal with '-='
                            is not supported.

              OVERLAP       This  reservation  can  be  allocated  resources  that are already in
                            another reservation. Flag removal with '-=' is not supported.

              PART_NODES    This flag can be used to  reserve  all  nodes  within  the  specified
                            partition.   PartitionName  and Nodes=ALL must be specified with this
                            flag.

              PURGE_COMP[=<timespec>]
                            Purge the reservation if it  is  ever  idle  for  timespec  (no  jobs
                            associated  with  it).  If timespec isn't given then 5 minutes is the
                            default.   Valid  timespec  formats  are  minutes,   minutes:seconds,
                            hours:minutes:seconds,         days-hours,        days-hours:minutes,
                            days-hours:minutes:seconds.  Time resolution is one minute and second
                            values  are  rounded  up  to the next minute. Output format is always
                            [days-]hours:minutes:seconds.

              REPLACE       Nodes which are DOWN, DRAINED, or allocated to jobs are automatically
                            replenished  using  idle  resources.   This  option  can  be  used to
                            maintain a constant number of idle resources  available  for  pending
                            jobs  (subject  to  availability  of idle resources).  This should be
                            used with the NodeCnt reservation option; do  not  identify  specific
                            nodes  to  be  included in the reservation. Flag removal with '-=' is
                            not supported.

                            NOTE: Removing a node from the cluster while in  a  reservation  with
                            the REPLACE flag will not cause it to be replaced.

              REPLACE_DOWN  Nodes  which  are DOWN or DRAINED are automatically replenished using
                            idle resources.  This option can be used to maintain a constant sized
                            pool of resources available for pending jobs (subject to availability
                            of idle resources).  This should be used with the NodeCnt reservation
                            option;  do  not  identify  specific  nodes  to  be  included  in the
                            reservation. Flag removal with '-=' is not supported.

                            NOTE: Removing a node from the cluster while in  a  reservation  with
                            the REPLACE_DOWN flag will not cause it to be replaced.

              SPEC_NODES    Reservation is for specific nodes (output only).

              STATIC_ALLOC  Make  it so after the nodes are selected for a reservation they don't
                            change.   Without  this  option  when  nodes  are  selected   for   a
                            reservation  and one goes down the reservation will select a new node
                            to fill the spot.

              TIME_FLOAT    The reservation start time is relative to the current time and  moves
                            forward  through  time (e.g. a StartTime=now+10minutes will always be
                            10  minutes  in  the  future).  Repeating   (e.g.   DAILY)   floating
                            reservations  are  not  supported. Flag cannot be added to or removed
                            from an existing reservation.

              WEEKDAY       Repeat the reservation at the same time  on  every  weekday  (Monday,
                            Tuesday, Wednesday, Thursday and Friday).

              WEEKEND       Repeat  the  reservation  at  the  same  time  on  every  weekend day
                            (Saturday and Sunday).

              WEEKLY        Repeat the reservation at the same time every week.

       Features=<features>
              Set the reservation's required node features. Multiple values may be "&"  separated
              if  all  features  are  required  (AND operation) or separated by "|" if any of the
              specified features are required (OR operation).  Parenthesis are also supported for
              features  to  be ANDed together with counts of nodes having the specified features.
              For  example   "Features=[(knl&a2a&flat)*4&haswell*2]"   indicates   the   advanced
              reservation  should  include  4  nodes  with  ALL of the features "knl", "a2a", and
              "flat" plus 2 nodes with the feature "haswell".

              Value may be cleared with blank data value, "Features=".

       Groups=<group list>
              List of groups permitted to use the reserved nodes, for  example  "Group=bio,chem".
              A  new  reservation  must  specify  Users  or  Groups  and/or  Accounts.   If  both
              Users/Groups and Accounts are specified, a job must match both in order to use  the
              reservation.   Unlike users groups do not allow denied access to reservations.  You
              can add or remove individual groups from  an  existing  reservation  by  using  the
              update  command and adding a '+' or '-' sign before the '=' sign.  NOTE: Groups and
              Users are mutually exclusive in reservations, if you want to switch between  the  2
              you must update the reservation with a group='' or user='' and fill in the opposite
              with the appropriate setting.

       Skip   Used on a reoccurring reservation, skip to the next reservation  iteration.   NOTE:
              Only available for update.

       Users=<user list>
              List    of   users   permitted   to   use   the   reserved   nodes,   for   example
              "User=jones1,smith2".  A new  reservation  must  specify  Users  or  Groups  and/or
              Accounts.   If  both Users/Groups and Accounts are specified, a job must match both
              in order to use the reservation.  Users can also be denied access  to  reservations
              by  preceding  all  of  the user names with '-'. Alternately precede the equal sign
              with '-'.  For example, "User=-jones1,-smith2" or "User-=jones1,smith2" will permit
              any  user  except  jones1 and smith2 to use the reservation.  You can add or remove
              individual users from an existing reservation  by  using  the  update  command  and
              adding  a  '+'  or  '-'  sign before the '=' sign.  If users are denied access to a
              reservation (user name preceded by a '-'), then  all  other  users  are  implicitly
              allowed  to  use  the reservation and it is not possible to also explicitly specify
              allowed users.  NOTE: Groups and Users are mutually exclusive in  reservations,  if
              you want to switch between the 2 you must update the reservation with a group='' or
              user='' and fill in the opposite with the appropriate setting.

       TRES=<tres_spec>
              Comma-separated list of TRES required for the reservation. Current  supported  TRES
              types  with  reservations  are:  CPU, Node, License and BB. CPU and Node follow the
              same format as CoreCnt and NodeCnt parameters respectively.  License names  can  be
              followed by an equal '=' and a count:

              License/<name1>=<count1>[,License/<name2>=<count2>,...]

              BurstBuffer  can  be  specified in a similar way as BurstBuffer parameter. The only
              difference is that colon symbol ':' should be replaced by an equal '=' in order  to
              follow the TRES format.

              Some examples of TRES valid specifications:

              TRES=cpu=5,bb/cray=4,license/iop1=1,license/iop2=3

              TRES=node=5k,license/iop1=2

              As  specified  in  CoreCnt, if a nodelist is specified, cpu can be an array of core
              numbers by node: nodes=compute[1-3] TRES=cpu=2,2,1,bb/cray=4,license/iop1=2

              Please note that CPU, Node, License and BB can override CoreCnt, NodeCnt,  Licenses
              and  BurstBuffer  parameters  respectively.   Also  CPU  represents  CoreCnt,  in a
              reservation and will be adjusted if you have threads per core on your nodes.

              Note that a reservation that  contains  nodes  or  cores  is  associated  with  one
              partition,  and  can't span resources over multiple partitions.  The only exception
              from this is when the reservation is created with explicitly requested nodes.

PERFORMANCE

       Executing scontrol sends a remote procedure  call  to  slurmctld.  If  enough  calls  from
       scontrol  or other Slurm client commands that send remote procedure calls to the slurmctld
       daemon come in at once, it can result in a degradation of  performance  of  the  slurmctld
       daemon, possibly resulting in a denial of service.

       Do  not  run  scontrol  or other Slurm client commands that send remote procedure calls to
       slurmctld from loops in shell scripts or other programs. Ensure that programs limit  calls
       to scontrol to the minimum necessary for the information you are trying to gather.

ENVIRONMENT VARIABLES

       Some  scontrol  options may be set via environment variables. These environment variables,
       along with their corresponding options, are listed below. (Note: Command line options will
       always override these settings.)

       SCONTROL_ALL        -a, --all

       SCONTROL_FEDERATION --federation

       SCONTROL_FUTURE     -F, --future

       SCONTROL_LOCAL      --local

       SCONTROL_SIBLING    --sibling

       SLURM_BITSTR_LEN    Specifies  the string length to be used for holding a job array's task
                           ID expression.  The default value is 64 bytes.   A  value  of  0  will
                           print the full expression with any length required.  Larger values may
                           adversely impact the application performance.

       SLURM_CLUSTERS      Same as --clusters

       SLURM_CONF          The location of the Slurm configuration file.

       SLURM_CONF_OUT      When running 'write config', the location of the  Slurm  configuration
                           file to be written.

       SLURM_TIME_FORMAT   Specify  the  format  used to report time stamps. A value of standard,
                           the    default    value,    generates    output    in     the     form
                           "year-month-dateThour:minute:second".   A  value  of  relative returns
                           only "hour:minute:second" if the current day.  For other dates in  the
                           current   year  it  prints  the  "hour:minute"  preceded  by  "Tomorr"
                           (tomorrow), "Ystday" (yesterday), the name of the day for  the  coming
                           week  (e.g.  "Mon",  "Tue", etc.), otherwise the date (e.g. "25 Apr").
                           For other years it returns a date month and year without a time  (e.g.
                           "6 Jun 2012"). All of the time stamps use a 24 hour format.

                           A  valid strftime() format can also be specified. For example, a value
                           of "%a %T" will report the day of the week and a time stamp (e.g. "Mon
                           12:34:56").

       SLURM_TOPO_LEN      Specify  the  maximum  size of the line when printing Topology. If not
                           set, the default value is unlimited.

AUTHORIZATION

       When using SlurmDBD, users who have an AdminLevel defined (Operator or  Admin)  and  users
       who   are  account  coordinators  are  given  the  authority  to  view  and  modify  jobs,
       reservations, nodes, etc., as defined in the following table -  regardless  of  whether  a
       PrivateData restriction has been defined in the slurm.conf file.

       scontrol show job(s):        Admin, Operator, Coordinator
       scontrol update job:         Admin, Operator, Coordinator
       scontrol requeue:            Admin, Operator, Coordinator
       scontrol show step(s):       Admin, Operator, Coordinator
       scontrol update step:        Admin, Operator, Coordinator

       scontrol show node:          Admin, Operator
       scontrol update node:        Admin

       scontrol create partition:   Admin
       scontrol show partition:     Admin, Operator
       scontrol update partition:   Admin
       scontrol delete partition:   Admin

       scontrol create reservation: Admin, Operator
       scontrol show reservation:   Admin, Operator
       scontrol update reservation: Admin, Operator
       scontrol delete reservation: Admin, Operator

       scontrol reconfig:           Admin
       scontrol shutdown:           Admin
       scontrol takeover:           Admin

EXAMPLES

       $ scontrol
       scontrol: show part debug
       PartitionName=debug
          AllocNodes=ALL AllowGroups=ALL Default=YES
          DefaultTime=NONE DisableRootJobs=NO Hidden=NO
          MaxNodes=UNLIMITED MaxTime=UNLIMITED MinNodes=1
          Nodes=snowflake[0-48]
          Priority=1 RootOnly=NO OverSubscribe=YES:4
          State=UP TotalCPUs=694 TotalNodes=49
       scontrol: update PartitionName=debug MaxTime=60:00 MaxNodes=4
       scontrol: show job 71701
       JobId=71701 Name=hostname
          UserId=da(1000) GroupId=da(1000)
          Priority=66264 Account=none QOS=normal WCKey=*123
          JobState=COMPLETED Reason=None Dependency=(null)
          TimeLimit=UNLIMITED Requeue=1 Restarts=0 BatchFlag=0 ExitCode=0:0
          SubmitTime=2010-01-05T10:58:40 EligibleTime=2010-01-05T10:58:40
          StartTime=2010-01-05T10:58:40 EndTime=2010-01-05T10:58:40
          SuspendTime=None SecsPreSuspend=0
          Partition=debug AllocNode:Sid=snowflake:4702
          ReqNodeList=(null) ExcNodeList=(null)
          NodeList=snowflake0
          NumNodes=1 NumCPUs=10 CPUs/Task=2 ReqS:C:T=1:1:1
          MinCPUsNode=2 MinMemoryNode=0 MinTmpDiskNode=0
          Features=(null) Reservation=(null)
          OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null)
       scontrol: update JobId=71701 TimeLimit=30:00 Priority=500
       scontrol: show hostnames tux[1-3]
       tux1
       tux2
       tux3
       scontrol: create res StartTime=2009-04-01T08:00:00 Duration=5:00:00 Users=dbremer NodeCnt=10
       Reservation created: dbremer_1
       scontrol: update Reservation=dbremer_1 Flags=Maint NodeCnt=20
       scontrol: delete Reservation=dbremer_1
       scontrol: quit

COPYING

       Copyright (C) 2002-2007 The Regents of the University of California.  Produced at Lawrence
       Livermore National Laboratory (cf, DISCLAIMER).
       Copyright (C) 2008-2010 Lawrence Livermore National Security.
       Copyright (C) 2010-2021 SchedMD LLC.

       This  file  is  part  of  Slurm,  a  resource  management  program.   For   details,   see
       <https://slurm.schedmd.com/>.

       Slurm  is  free  software; you can redistribute it and/or modify it under the terms of the
       GNU General Public License as published by the Free Software Foundation; either version  2
       of the License, or (at your option) any later version.

       Slurm is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without
       even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
       GNU General Public License for more details.

FILES

       /etc/slurm.conf

SEE ALSO

       scancel(1),  sinfo(1),  squeue(1), slurm_create_partition (3), slurm_delete_partition (3),
       slurm_load_ctl_conf (3), slurm_load_jobs (3), slurm_load_node  (3),  slurm_load_partitions
       (3),  slurm_reconfigure  (3),   slurm_requeue  (3),  slurm_resume (3), slurm_shutdown (3),
       slurm_suspend  (3),  slurm_takeover  (3),  slurm_update_job  (3),  slurm_update_node  (3),
       slurm_update_partition (3), slurm.conf(5), slurmctld(8)