Provided by: slurm-client_23.11.4-1.2ubuntu5_amd64 
NAME
scontrol - view or modify Slurm configuration and state.
SYNOPSIS
scontrol [OPTIONS...] [COMMAND...]
DESCRIPTION
scontrol is used to view or modify Slurm configuration including: job, job step, node,
partition, reservation, and overall system configuration. Most of the commands can only be
executed by user root or an Administrator. If an attempt to view or modify configuration
information is made by an unauthorized user, an error message will be printed and the
requested action will not occur. If no command is entered on the execute line, scontrol
will operate in an interactive mode and prompt for input. It will continue prompting for
input and executing commands until explicitly terminated. If a command is entered on the
execute line, scontrol will execute that command and terminate. All commands and options
are case-insensitive, although node names, partition names, and reservation names are
case-sensitive (node names "LX" and "lx" are distinct). All commands and options can be
abbreviated to the extent that the specification is unique. A modified Slurm configuration
can be written to a file using the scontrol write config command. The resulting file will
be named using the convention "slurm.conf.<datetime>" and located in the same directory as
the original "slurm.conf" file. The directory containing the original slurm.conf must be
writable for this to occur.
OPTIONS
-a, --all
When the show command is used, then display all partitions, their jobs and jobs
steps. This causes information to be displayed about partitions that are configured
as hidden and partitions that are unavailable to user's group.
-M, --clusters=<string>
The cluster to issue commands to. Only one cluster name may be specified. Note
that the SlurmDBD must be up for this option to work properly. This option
implicitly sets the --local option.
-d, --details
Causes the show command to provide additional details where available.
--federation
Report jobs from federation if a member of one.
-F, --future
Report nodes in FUTURE state.
-h, --help
Print a help message describing the usage of scontrol.
--hide Do not display information about hidden partitions, their jobs and job steps. By
default, neither partitions that are configured as hidden nor those partitions
unavailable to user's group will be displayed (i.e. this is the default behavior).
--json, --json=list, --json=<data_parser>
Dump information as JSON using the default data_parser plugin or explicit
data_parser with parameters. All information is dumped, even if it would normally
not be. Sorting and formatting arguments passed to other options are ignored;
however, filtering arguments are still used. This option implicitly sets the
--details option.
--local
Show only information local to this cluster. Ignore other clusters in the federated
if a member of one. Overrides --federation.
-o, --oneliner
Print information one line per record.
-Q, --quiet
Print no warning or informational messages, only fatal error messages.
--sibling
Show all sibling jobs on a federated cluster. Implies --federation.
-u, --uid=<uid>
Attempt to update a job as user <uid> instead of the invoking user id.
-v, --verbose
Print detailed event logging. Multiple '-v's will further increase the verbosity of
logging. By default only errors will be displayed.
-V , --version
Print version information and exit.
Dump information as YAML using the default data_parser plugin or explicit
data_parser with parameters. All information is dumped, even if it would normally
not be. Sorting and formatting arguments passed to other options are ignored;
however, filtering arguments are still used. This option implicitly sets the
--details option.
COMMANDS
cancel_reboot <NodeList>
Cancel pending reboots on nodes. The node will be undrain'ed and the reason cleared
if the node was drained by an ASAP reboot.
create <SPECIFICATION>
Create a new node, partition, or reservation. See the full list of parameters
below.
completing
Display all jobs in a COMPLETING state along with associated nodes in either a
COMPLETING or DOWN state.
delete <SPECIFICATION>
Delete the entry with the specified SPECIFICATION. The three SPECIFICATION choices
are NodeName=<nodelist>, PartitionName=<name> and Reservation=<name>. Only dynamic
nodes that have no running jobs and that are not part of a reservation can be
deleted. Reservations and partitions should have no associated jobs at the time of
their deletion (modify the jobs first). If the specified partition is in use, the
request is denied.
errnumstr <ERRNO>
Given a Slurm error number, return a descriptive string.
fsdampeningfactor <FACTOR>
Set the FairShareDampeningFactor in slurmctld.
getaddrs <NODES>
Get IP addresses of <NODES> from slurmctld.
help Display a description of scontrol options and commands.
hold <job_list>
Prevent a pending job from being started (sets its priority to 0). Use the release
command to permit the job to be scheduled. The job_list argument is a comma
separated list of job IDs OR "jobname=" with the job's name, which will attempt to
hold all jobs having that name. Note that when a job is held by a system
administrator using the hold command, only a system administrator may release the
job for execution (also see the uhold command). When the job is held by its owner,
it may also be released by the job's owner. Additionally, attempting to hold a
running job will have not suspend or cancel it. But, it will set the job priority
to 0 and update the job reason field, which would hold the job if it was requeued
at a later time.
notify <job_id> <message>
Send a message to standard error of the salloc or srun command or batch job
associated with the specified job_id.
pidinfo <proc_id>
Print the Slurm job id and scheduled termination time corresponding to the supplied
process id, proc_id, on the current node. This will work only with processes on
node on which scontrol is run, and only for those processes spawned by Slurm and
their descendants.
listpids [<job_id>[.<step_id>]] [<NodeName>]
Print a listing of the process IDs in a job step (if JOBID.STEPID is provided), or
all of the job steps in a job (if job_id is provided), or all of the job steps in
all of the jobs on the local node (if job_id is not provided or job_id is "*").
This will work only with processes on the node on which scontrol is run, and only
for those processes spawned by Slurm and their descendants. Note that some Slurm
configurations (ProctrackType value of pgid) are unable to identify all processes
associated with a job or job step.
Note that the NodeName option is only really useful when you have multiple slurmd
daemons running on the same host machine. Multiple slurmd daemons on one host are,
in general, only used by Slurm developers.
ping Ping the primary and secondary slurmctld daemon and report if they are responding.
reboot [ASAP] [nextstate={RESUME|DOWN}] [reason=<reason>] {ALL|<NodeList>|<NodeSet>}
Reboot the nodes in the system when they become idle using the RebootProgram as
configured in Slurm's slurm.conf file. Each node will have the "REBOOT" flag added
to its node state. After a node reboots and the slurmd daemon starts up again, the
HealthCheckProgram will run once. Then, the slurmd daemon will register itself with
the slurmctld daemon and the "REBOOT" flag will be cleared. The "ASAP" option adds
the "DRAIN" flag to each node's state, preventing additional jobs from running on
the node so it can be rebooted and returned to service "As Soon As Possible" (i.e.
ASAP). "ASAP" will also set the node reason to "Reboot ASAP" if the "reason"
option isn't specified. If the "nextstate" option is specified as "DOWN", then the
node will remain in a down state after rebooting. If "nextstate" is specified as
"RESUME", then the nodes will resume as normal and the node's reason and "DRAIN"
state will be cleared. Resuming nodes will be considered as available in backfill
future scheduling and won't be replaced by idle nodes in a reservation. The
"reason" option sets each node's reason to a user-defined message. A default
reason of "reboot requested" is set if no other reason is set on the node. The
reason will be appended with: "reboot issued" when the reboot is issued; "reboot
complete" when the node registers and has a "nextstate" of "DOWN"; or "reboot timed
out" when the node fails to register within ResumeTimeout. You must specify either
a list of nodes or that ALL nodes are to be rebooted. NOTE: By default, this
command does not prevent additional jobs from being scheduled on any nodes before
reboot. To do this, you can either use the "ASAP" option or explicitly drain the
nodes beforehand. You can alternately create an advanced reservation to prevent
additional jobs from being initiated on nodes to be rebooted. Pending reboots can
be cancelled by using "scontrol cancel_reboot <node>" or setting the node state to
"CANCEL_REBOOT". A node will be marked "DOWN" if it doesn't reboot within
ResumeTimeout.
reconfigure
Instruct all slurmctld and slurmd daemons to re-read the configuration file. This
mechanism can be used to modify configuration parameters set in slurm.conf(5)
without interrupting running jobs. Starting in 23.11, this command operates by
creating new processes for the daemons, then passing control to the new processes
when or if they start up successfully. This allows it to gracefully catch
configuration problems and keep running with the previous configuration if there is
a problem. This will not be able to change the daemons' listening TCP port
settings or authentication mechanism.
release <job_list>
Release a previously held job to begin execution. The job_list argument is a comma
separated list of job IDs OR "jobname=" with the job's name, which will attempt to
hold all jobs having that name. Also see hold.
requeue [<option>] <job_list>
Requeue a running, suspended or finished Slurm batch job into pending state. The
job_list argument is a comma separated list of job IDs. The command accepts the
following option:
Incomplete
Operate only on jobs (or tasks of a job array) which have not completed.
Specifically only jobs in the following states will be requeued:
CONFIGURING, RUNNING, STOPPED or SUSPENDED.
requeuehold [<option>] <job_list>
Requeue a running, suspended or finished Slurm batch job into pending state,
moreover the job is put in held state (priority zero). The job_list argument is a
comma separated list of job IDs. A held job can be released using scontrol to
reset its priority (e.g. "scontrol release <job_id>"). The command accepts the
following options:
Incomplete
Operate only on jobs (or tasks of a job array) which have not completed.
Specifically only jobs in the following states will be requeued:
CONFIGURING, RUNNING, STOPPED or SUSPENDED.
State=SpecialExit
The "SpecialExit" keyword specifies that the job has to be put in a special
state JOB_SPECIAL_EXIT. The "scontrol show job" command will display the
JobState as SPECIAL_EXIT, while the "squeue" command as SE.
resume <job_list>
Resume a previously suspended job. The job_list argument is a comma separated list
of job IDs. Also see suspend.
NOTE: A suspended job releases its CPUs for allocation to other jobs. Resuming a
previously suspended job may result in multiple jobs being allocated the same CPUs,
which could trigger gang scheduling with some configurations or severe degradation
in performance with other configurations. Use of the scancel command to send
SIGSTOP and SIGCONT signals would stop a job without releasing its CPUs for
allocation to other jobs and would be a preferable mechanism in many cases. If
performing system maintenance you may want to use suspend/resume in the following
way. Before suspending set all nodes to draining or set all partitions to down so
that no new jobs can be scheduled. Then suspend jobs. Once maintenance is done
resume jobs then resume nodes and/or set all partitions back to up. Use with
caution.
schedloglevel <LEVEL>
Enable or disable scheduler logging. LEVEL may be "0", "1", "disable" or "enable".
"0" has the same effect as "disable". "1" has the same effect as "enable". This
value is temporary and will be overwritten when the slurmctld daemon reads the
slurm.conf configuration file (e.g. when the daemon is restarted or scontrol
reconfigure is executed) if the SlurmSchedLogLevel parameter is present.
setdebug <LEVEL> [nodes=<NODES>]
Change the debug level of the slurmctld daemon for all active logging channels not
originally configured off (quiet). LEVEL may be an integer value between zero and
nine (using the same values as SlurmctldDebug in the slurm.conf file) or the name
of the most detailed message type to be printed: "quiet", "fatal", "error", "info",
"verbose", "debug", "debug2", "debug3", "debug4", or "debug5". This value is
temporary, and will be overwritten whenever the daemon reads the slurm.conf
configuration file. (Such as when the daemon is restarted or scontrol reconfigure
is executed).
nodes=<NodeName>
If set, the request to change debug level is sent to the slurmd processes on
the nodes instead of to the slurmctld. A node range expression may be used
for NodeName.
setdebugflags {+|-}<FLAG> [{+|-}<FLAG>] [nodes=<NODES>]
Add or remove DebugFlags of the slurmctld daemon. See "man slurm.conf" for a list
of supported DebugFlags.
NOTE: Changing the value of some DebugFlags will have no effect without restarting
the slurmctld daemon, which would set DebugFlags based upon the contents of the
slurm.conf configuration file or the SLURM_DEBUG_FLAGS environment variable. The
environment variable takes precedence over the setting in the slurm.conf.
nodes=<NodeName>
The argument is optional and if used the request to change debug level is
sent to slurmd instead of slurmctld. A node range expression may be used for
NodeName.
show <ENTITY>[=<ID>] or <ENTITY> [<ID>]
Display the state of the specified entity with the specified identification.
aliases
Returns all NodeName values associated with a given NodeHostname (useful to
get the list of virtual nodes associated with a real node in a configuration
where multiple slurmd daemons execute on a single compute node).
assoc_mgr
Displays the current contents of the slurmctld's internal cache for users,
associations and/or qos. The output can be filtered by different record
types:
users=<user1>[...,<userN>]
Limit the User Records displayed to those with the specified user
name(s).
accounts=<acct1>[...,<acctN>]
Limit the Association Records displayed to those with the specified
account name(s).
qos=<qos1>[...,<qosN>]
Limit the QOS Records displayed to those with the specified QOS
name(s).
flags={users|assoc|qos}
Specify the desired record type to be displayed. If no flags are
specified, all record types are displayed.
bbstat Displays output from Cray's burst buffer status tool. Options following
bbstat are passed directly to the dwstat command by the slurmctld daemon and
the response returned to the user. Equivalent to dwstat.
burstbuffer
Displays the current status of the BurstBuffer plugin.
config Displays parameter names from the configuration files in mixed case (e.g.
SlurmdPort=7003) while derived parameters names are in upper case only (e.g.
SLURM_VERSION).
daemons
Reports which daemons should be running on this node.
dwstat Displays output from Cray's burst buffer status tool. Options following
dwstat are passed directly to the dwstat command by the slurmctld daemon and
the response returned to the user. Equivalent to bbstat.
federation
The federation name that the controller is part of and the sibling clusters
part of the federation will be listed.
frontend
Shows configured frontend nodes.
hostlist
Takes a list of host names and prints the hostlist expression for them (the
inverse of hostnames). hostlist can also take the absolute pathname of a
file (beginning with the character '/') containing a list of hostnames, or a
single '-' to have it read from stdin directly. Multiple node names may be
specified using simple node range expressions (e.g. "lx[10-20]"). By
default hostlist does not sort the node list or make it unique (e.g.
tux2,tux1,tux2 = tux[2,1-2]). If you wanted a sorted list use hostlistsorted
(e.g. tux2,tux1,tux2 = tux[1-2,2]).
hostlistsorted
Takes a list of host names and prints a sorted (but not unique) hostlist
expression for them. See hostlist.
hostnames
Takes an optional hostlist expression as input and writes a list of
individual host names to standard output (one per line). If no hostlist
expression is supplied, the contents of the SLURM_JOB_NODELIST environment
variable is used. For example "tux[1-3]" is mapped to "tux1","tux2" and
"tux3" (one hostname per line).
job Displays statistics about all jobs by default. If an optional jobid is
specified, details for just that job will be displayed. If the job does not
specify socket-per-node, cores-per-socket or threads-per-core then it will
display '*' in the ReqS:C:T=*:*:* field.
licenses
Displays statistics about all configured licenses (local and remote) by
default. If an optional license name is specified, details for just that
license will be displayed.
node Displays statistics about all nodes by default. If an optional nodename is
specified, details for just that node will be displayed.
partition
Displays statistics about all partitions by default. If an optional
partition name is specified, details for just that partition will be
displayed.
reservation
Displays statistics about all reservations by default. If an optional
reservation name is specified, details for just that reservation will be
displayed.
slurmd Displays statistics for the slurmd running on the current node.
step Displays statistics about all job steps by default. If an optional jobid is
specified, details about steps for just that job will be displayed. If a
jobid.stepid is specified, details for just that step will be displayed. If
a container-id=<id> is specified, then the first matching step with the
given container-id will be displayed.
topology
Displays information about the defined topology layout. If a switch is
specified, information about that switch will be shown. If one node name is
specified, all switches connected to that node (and their parent switches)
will be shown. If more than one node name is specified, only switches that
connect to all named nodes will be shown.
shutdown <OPTION>
Instruct Slurm daemons to save current state and terminate. By default, the Slurm
controller (slurmctld) forwards the request all other daemons (slurmd daemon on
each compute node). An OPTION of slurmctld or controller results in only the
slurmctld daemon being shutdown and the slurmd daemons remaining active.
suspend <job_list>
Suspend a running job. The job_list argument is a comma separated list of job IDs.
Use the resume command to resume its execution. User processes must stop on
receipt of SIGSTOP signal and resume upon receipt of SIGCONT for this operation to
be effective. Not all architectures and configurations support job suspension. If
a suspended job is requeued, it will be placed in a held state. The time a job is
suspended will not count against a job's time limit. Only an operator,
administrator, SlurmUser, or root can suspend jobs.
takeover [<INDEX>]
Instruct one of Slurm's backup controllers (slurmctld) to take over system control.
By default the first backup controller (INDEX=1) requests control from the primary
and waits for its termination. After that, it switches from backup mode to
controller mode. If primary controller can not be contacted, it directly switches
to controller mode. This can be used to speed up the Slurm controller fail-over
mechanism when the primary node is down. This can be used to minimize disruption
if the computer executing the primary Slurm controller is scheduled down. (Note:
Slurm's primary controller will take the control back at startup.)
top <job_list>
Move the specified job IDs to the top of the queue of jobs belonging to the
identical user ID, partition name, account, and QOS. The job_list argument is a
comma separated ordered list of job IDs. Any job not matching all of those fields
will not be effected. Only jobs submitted to a single partition will be effected.
This operation changes the order of jobs by adjusting job nice values. The net
effect on that user's throughput will be negligible to slightly negative. This
operation is disabled by default for non-privileged (non-operator, admin,
SlurmUser, or root) users. This operation may be enabled for non-privileged users
by the system administrator by including the option "enable_user_top" in the
SchedulerParameters configuration parameter.
token [lifespan=<lifespan>] [username=<username>]
Return an auth token which can be used to support JWT authentication if
AuthAltTypes=auth/jwt has been enabled on the system. Supports two optional
arguments. lifespan= may be used to specify the token's lifespan in seconds.
username (only available to SlurmUser/root) may be used to request a token for a
different username. Lifespan can be set to "infinite" to have a token that will not
expire. Sites are suggested to use the smallest lifespan required and generate
tokens more often instead of using longer lived tokens.
uhold <job_list>
Prevent a pending job from being started (sets its priority to 0). The job_list
argument is a space separated list of job IDs or job names. Use the release
command to permit the job to be scheduled. This command is designed for a system
administrator to hold a job so that the job owner may release it rather than
requiring the intervention of a system administrator (also see the hold command).
update <SPECIFICATION>
Update job, step, node, partition, or reservation configuration per the supplied
specification. SPECIFICATION is in the same format as the Slurm configuration file
and the output of the show command described above. It may be desirable to execute
the show command (described above) on the specific entity you want to update, then
use cut-and-paste tools to enter updated configuration values to the update. Note
that while most configuration values can be changed using this command, not all can
be changed using this mechanism. In particular, the hardware configuration of a
node or the physical addition or removal of nodes from the cluster may only be
accomplished through editing the Slurm configuration file and executing the
reconfigure command (described above).
update <SuspendExc*>[=|+=|-=]<LIST>
Update SuspendExcNodes, SuspendExcParts, or SuspendExcStates. <LIST> is either a
NodeList, list of partitions, or list of node states respectively. Use +=/-= to
add/remove nodes, partitions, or states to/from the currently configured list. Use
= to replace the current list. SuspendExcNodes does not support "+="/"-=" when the
":" option is used, however, direct assignment "=" is always supported. Consider
using "scontrol show config | grep SuspendExc" to see current state of these
settings.
version
Display the version number of scontrol being executed.
wait_job <job_id>
Wait until a job and all of its nodes are ready for use or the job has entered some
termination state. This option is particularly useful in the Slurm Prolog or in the
batch script itself if nodes are powered down and restarted automatically as
needed.
NOTE: Don't use scontrol wait_job in PrologSlurmctld or Prolog with
PrologFlags=Alloc as this will result in a deadlock.
NOTE: When using wait_job for an array job, use the SLURM_JOB_ID environment
variable to reference the job rather than the SLURM_ARRAY_JOB_ID variable.
write batch_script <job_id> [<optional_filename>]
Write the batch script for a given job_id to a file or to stdout. The file will
default to slurm-<job_id>.sh if the optional filename argument is not given. The
script will be written to stdout if - is given instead of a filename. The batch
script can only be retrieved by an admin or operator, or by the owner of the job.
write config <optional_filename>
Write the current configuration to a file with the naming convention of
"slurm.conf.<datetime>" in the same directory as the original slurm.conf file. If
a filename is given that file location with a .<datetime> suffix is created.
INTERACTIVE COMMANDS
NOTE: All commands listed below can be used in the interactive mode, but NOT on the
initial command line.
all Show all partitions, their jobs and jobs steps. This causes information to be
displayed about partitions that are configured as hidden and partitions that are
unavailable to user's group.
cluster <CLUSTER_NAME>
The cluster to issue commands to. Only one cluster name may be specified.
details
Causes the show command to provide additional details where available. Job
information will include CPUs and NUMA memory allocated on each node. Note that on
computers with hyperthreading enabled and Slurm configured to allocate cores, each
listed CPU represents one physical core. Each hyperthread on that core can be
allocated a separate task, so a job's CPU count and task count may differ. See the
--cpu-bind and --mem-bind option descriptions in srun man pages for more
information. The details option is currently only supported for the show job
command.
exit Terminate scontrol interactive session.
hide Do not display partition, job or jobs step information for partitions that are
configured as hidden or partitions that are unavailable to the user's group. This
is the default behavior.
oneliner
Print information one line per record.
quiet Print no warning or informational messages, only fatal error messages.
quit Terminate the execution of scontrol.
verbose
Print detailed event logging. This includes time-stamps on data structures, record
counts, etc.
!! Repeat the last command executed.
JOBS - SPECIFICATIONS FOR UPDATE COMMAND
Note that update requests done by either root, SlurmUser or Administrators are not subject
to certain restrictions. For instance, if an Administrator changes the QOS on a pending
job, certain limits such as the TimeLimit will not be changed automatically as changes
made by the Administrators are allowed to violate these restrictions.
Account=<account>
Account name to be changed for this job's resource use. Value may be cleared with
blank data value, "Account=".
AdminComment=<spec>
Arbitrary descriptive string. Can only be set by a Slurm administrator.
ArrayTaskThrottle=<count>
Specify the maximum number of tasks in a job array that can execute at the same
time. Set the count to zero in order to eliminate any limit. The task throttle
count for a job array is reported as part of its ArrayTaskId field, preceded with a
percent sign. For example "ArrayTaskId=1-10%2" indicates the maximum number of
running tasks is limited to 2.
BurstBuffer=<spec>
Burst buffer specification to be changed for this job's resource use. Value may be
cleared with blank data value, "BurstBuffer=". Format is burst buffer plugin
specific.
Clusters=<spec>
Specifies the clusters that the federated job can run on.
ClusterFeatures=<spec>
Specifies features that a federated cluster must have to have a sibling job
submitted to it. Slurm will attempt to submit a sibling job to a cluster if it has
at least one of the specified features.
Comment=<spec>
Arbitrary descriptive string.
Contiguous={yes|no}
Set the job's requirement for contiguous (consecutive) nodes to be allocated.
Possible values are "YES" and "NO". Only the Slurm administrator or root can
change this parameter.
CoreSpec=<count>
Number of cores to reserve per node for system use. The job will be charged for
these cores, but be unable to use them. Will be reported as "*" if not
constrained.
CPUsPerTask=<count>
Change the CPUsPerTask job's value.
Deadline=<time_spec>
It accepts times of the form HH:MM:SS to specify a deadline to a job at a specific
time of day (seconds are optional). You may also specify midnight, noon, fika (3
PM) or teatime (4 PM) and you can have a time-of-day suffixed with AM or PM for a
deadline in the morning or the evening. You can specify a deadline for the job
with a date of the form MMDDYY or MM/DD/YY or MM.DD.YY, or a date and time as
YYYY-MM-DD[THH:MM[:SS]]. You can also give times like now + count time-units,
where the time-units can be seconds (default), minutes, hours, days, or weeks and
you can tell Slurm to put a deadline for tomorrow with the keyword tomorrow. The
specified deadline must be later than the current time. Only pending jobs can have
the deadline updated. Only the Slurm administrator or root can change this
parameter.
DelayBoot=<time_spec>
Change the time to decide whether to reboot nodes in order to satisfy job's feature
specification if the job has been eligible to run for less than this time period.
See salloc/sbatch man pages option --delay-boot.
Dependency=<dependency_list>
Defer job's initiation until specified job dependency specification is satisfied.
Cancel dependency with an empty dependency_list (e.g. "Dependency=").
<dependency_list> is of the form <type:job_id[:job_id][,type:job_id[:job_id]]>.
Many jobs can share the same dependency and these jobs may even belong to different
users.
after:job_id[:jobid...]
This job can begin execution after the specified jobs have begun execution
or been canceled.
afterany:job_id[:jobid...]
This job can begin execution after the specified jobs have terminated.
afternotok:job_id[:jobid...]
This job can begin execution after the specified jobs have terminated in
some failed state (non-zero exit code, node failure, timed out, etc). This
dependency must be added while the specified job is still active or within
MinJobAge seconds after the specified job has ended.
afterok:job_id[:jobid...]
This job can begin execution after the specified jobs have successfully
executed (ran to completion with an exit code of zero). This dependency
must be added while the specified job is still active or within MinJobAge
seconds after the specified job has ended.
singleton
This job can begin execution after any previously launched jobs sharing the
same job name and user have terminated. In other words, only one job by
that name and owned by that user can be running or suspended at any point in
time.
EligibleTime=<time_spec>
See StartTime.
EndTime
The time the job is expected to terminate based on the job's time limit. When the
job ends sooner, this field will be updated with the actual end time.
ExcNodeList=<nodes>
Set the job's list of excluded node. Multiple node names may be specified using
simple node range expressions (e.g. "lx[10-20]"). Value may be cleared with blank
data value, "ExcNodeList=".
Extra=<spec>
An arbitrary string enclosed in single or double quotes if using spaces or some
special characters. See <https://slurm.schedmd.com/extra_constraints.html> for more
details.
Features=<features>
Set the job's required node features. The list of features may include multiple
feature names separated by ampersand (AND) and/or vertical bar (OR) operators. For
example: Features="opteron&video" or Features="fast|faster". In the first example,
only nodes having both the feature "opteron" AND the feature "video" will be used.
There is no mechanism to specify that you want one node with feature "opteron" and
another node with feature "video" in case no node has both features. If only one
of a set of possible options should be used for all allocated nodes, then use the
OR operator and enclose the options within square brackets. For example:
"Features=[rack1|rack2|rack3|rack4]" might be used to specify that all nodes must
be allocated on a single rack of the cluster, but any of those four racks can be
used. A request can also specify the number of nodes needed with some feature by
appending an asterisk and count after the feature name. For example
"Features=graphics*4" indicates that at least four allocated nodes must have the
feature "graphics." Parenthesis are also supported for features to be ANDed
together. For example "Features=[(knl&a2a&flat)*4&haswell*2]" indicates the
resource allocation should include 4 nodes with ALL of the features "knl", "a2a",
and "flat" plus 2 nodes with the feature "haswell". Constraints with node counts
may only be combined with AND operators. Value may be cleared with blank data
value, for example "Features=".
Gres=<list>
Specifies a comma-delimited list of generic consumable resources. The format of
each entry on the list is "name[:count[*cpu]]". The name is that of the consumable
resource. The count is the number of those resources with a default value of 1.
The specified resources will be allocated to the job on each node allocated unless
"*cpu" is appended, in which case the resources will be allocated on a per cpu
basis. The available generic consumable resources is configurable by the system
administrator. A list of available generic consumable resources will be printed
and the command will exit if the option argument is "help". Examples of use
include "Gres=gpus:2*cpu,disk=40G" and "Gres=help".
JobId=<job_list>
Identify the job(s) to be updated. The job_list may be a comma separated list of
job IDs. Either JobId or JobName is required. If the JobId is equal to the
ArrayJobID then the update will affect all of the individual jobs of the array. In
that case, to update the specific individual job, the form
<ArrayJobID>_<ArrayTaskId> must be used.
Licenses=<name>
Specification of licenses (or other resources available on all nodes of the
cluster) as described in salloc/sbatch/srun man pages.
MailType=<types>
Set the mail event types. Valid type values are NONE, BEGIN, END, FAIL, REQUEUE,
ALL (equivalent to BEGIN, END, FAIL, REQUEUE, and STAGE_OUT), STAGE_OUT (burst
buffer stage out and teardown completed), TIME_LIMIT, TIME_LIMIT_90 (reached 90
percent of time limit), TIME_LIMIT_80 (reached 80 percent of time limit),
TIME_LIMIT_50 (reached 50 percent of time limit) and ARRAY_TASKS (send emails for
each array task). Multiple type values may be specified in a comma separated list.
Unless the ARRAY_TASKS option is specified, mail notifications on job BEGIN, END
and FAIL apply to a job array as a whole rather than generating individual email
messages for each task in the job array.
MailUser=<name>
Set the user to receive email notification of state changes. A blank string will
set the mail user to the default which is the submitting user.
MinCPUsNode=<count>
Set the job's minimum number of CPUs per node to the specified value.
MinMemoryCPU=<megabytes>
Set the job's minimum real memory required per allocated CPU to the specified
value. Either MinMemoryCPU or MinMemoryNode may be set, but not both.
MinMemoryNode=<megabytes>
Set the job's minimum real memory required per node to the specified value. Either
MinMemoryCPU or MinMemoryNode may be set, but not both.
MinTmpDiskNode=<megabytes>
Set the job's minimum temporary disk space required per node to the specified
value. Only the Slurm administrator or root can change this parameter.
TimeMin=<timespec>
Change TimeMin value which specifies the minimum time limit minutes of the job.
JobName=<name>
Identify the name of jobs to be modified or set the job's name to the specified
value. When used to identify jobs to be modified, all jobs belonging to all users
are modified unless the UserID option is used to identify a specific user. Either
JobId or JobName is required.
Name[=<name>]
See JobName.
Nice[=<adjustment>]
Update the job with an adjusted scheduling priority within Slurm. With no
adjustment value the scheduling priority is decreased by 100. A negative nice value
increases the priority, otherwise decreases it. The adjustment range is +/-
2147483645. Only privileged users can specify a negative adjustment.
NodeList=<nodes>
Change the nodes allocated to a running job to shrink its size. The specified list
of nodes must be a subset of the nodes currently allocated to the job. Multiple
node names may be specified using simple node range expressions (e.g. "lx[10-20]").
After a job's allocation is reduced, subsequent srun commands must explicitly
specify node and task counts which are valid for the new allocation.
NOTE: The allocated nodes of jobs with arbitrary distribution can not be updated.
NumCPUs=<min_count>[-<max_count>]
Set the job's minimum and optionally maximum count of CPUs to be allocated.
NumNodes=<min_count>[-<max_count>]
Set the job's minimum and optionally maximum count of nodes to be allocated. If
the job is already running, use this to specify a node count less than currently
allocated and resources previously allocated to the job will be relinquished. After
a job's allocation is reduced, subsequent srun commands must explicitly specify
node and task counts which are valid for the new allocation. Also see the NodeList
parameter above. This is the same as ReqNodes.
NOTE: The node count of jobs with arbitrary distribution can not be updated.
NumTasks=<count>
Set the job's count of requested tasks to the specified value. The number of tasks
started in a specific step inside the allocation may differ from this value, for
instance when a different number of tasks is requested on step creation. This is
the same as ReqProcs.
OverSubscribe={yes|no}
Set the job's ability to share compute resources (i.e. individual CPUs) with other
jobs. Possible values are "YES" and "NO". This option can only be changed for
pending jobs.
Partition=<name>
Set the job's partition to the specified value.
Prefer=<features>
Set the job's preferred node features. This list is only preferred, not required
like Features is. This list will override what is requested in Features. See
Features option above.
Priority=<number>
Set the job's priority to the specified value. Note that a job priority of zero
prevents the job from ever being scheduled. By setting a job's priority to zero it
is held. Set the priority to a non-zero value to permit it to run. Explicitly
setting a job's priority clears any previously set nice value and removes the
priority/multifactor plugin's ability to manage a job's priority. In order to
restore the priority/multifactor plugin's ability to manage a job's priority, hold
and then release the job. Only the Slurm administrator or root can increase job's
priority.
QOS=<name>
Set the job's QOS (Quality Of Service) to the specified value. Value may be
cleared with blank data value, "QOS=".
Reboot={yes|no}
Set the job's flag that specifies whether to force the allocated nodes to reboot
before starting the job. This is only supported with some system configurations and
therefore it could be silently ignored.
ReqCores=<count>
Change the job's requested Cores count.
ReqNodeList=<nodes>
Set the job's list of required node. Multiple node names may be specified using
simple node range expressions (e.g. "lx[10-20]"). Value may be cleared with blank
data value, "ReqNodeList=".
ReqNodes=<min_count>[-<max_count>]
See NumNodes.
ReqProcs=<count>
See NumTasks.
ReqSockets=<count>
Change the job's requested socket count.
ReqThreads=<count>
Change the job's requested threads count.
Requeue={0|1}
Stipulates whether a job should be requeued after a node failure: 0 for no, 1 for
yes.
ReservationName=<name>
Set the job's reservation to the specified value. Value may be cleared with blank
data value, "ReservationName=".
ResetAccrueTime
Set the job's accrue time value to 'now' meaning it will lose any time previously
accrued for priority. Helpful if you have a large queue of jobs already in the
queue and want to start limiting how many jobs can accrue time without waiting for
the queue to flush out.
SiteFactor=<account>
Specify the job's admin priority factor in the range of +/-2147483645. Only
privileged users can modify the value.
StdErr=<filepath>
Set the batch job's stderr file path. Value may be reset to job default with blank
data value, "StdErr=".
NOTE: By default, StdErr will be merged into StdOut.
StdIn=<filepath>
Set the batch job's stdin file path. Value may be reset to job default with blank
data value, "StdIn=".
NOTE: By default, StdIn will be '/dev/null'.
StdOut=<filepath>
Set the batch job's stdout file path. Value may be reset to job default with blank
data value, "StdOut=".
NOTE: By default, StdOut will be based on the JobId.
Shared={yes|no}
See OverSubscribe option above.
StartTime=<time_spec>
Set the job's earliest initiation time. It accepts times of the form HH:MM:SS to
run a job at a specific time of day (seconds are optional). (If that time is
already past, the next day is assumed.) You may also specify midnight, noon, fika
(3 PM) or teatime (4 PM) and you can have a time-of-day suffixed with AM or PM for
running in the morning or the evening. You can also say what day the job will be
run, by specifying a date of the form MMDDYY or MM/DD/YY or MM.DD.YY, or a date and
time as YYYY-MM-DD[THH:MM[:SS]]. You can also give times like now + count
time-units, where the time-units can be seconds (default), minutes, hours, days, or
weeks and you can tell Slurm to run the job today with the keyword today and to run
the job tomorrow with the keyword tomorrow.
Notes on date/time specifications:
- although the 'seconds' field of the HH:MM:SS time specification is allowed by
the code, note that the poll time of the Slurm scheduler is not precise enough to
guarantee dispatch of the job on the exact second. The job will be eligible to
start on the next poll following the specified time. The exact poll interval
depends on the Slurm scheduler (e.g., 60 seconds with the default sched/builtin).
- if no time (HH:MM:SS) is specified, the default is (00:00:00).
- if a date is specified without a year (e.g., MM/DD) then the current year is
assumed, unless the combination of MM/DD and HH:MM:SS has already passed for that
year, in which case the next year is used.
Switches=<count>[@<max-time-to-wait>]
When a tree topology is used, this defines the maximum count of switches desired
for the job allocation. If Slurm finds an allocation containing more switches than
the count specified, the job remain pending until it either finds an allocation
with desired switch count or the time limit expires. By default there is no switch
count limit and no time limit delay. Set the count to zero in order to clean any
previously set count (disabling the limit). The job's maximum time delay may be
limited by the system administrator using the SchedulerParameters configuration
parameter with the max_switch_wait parameter option. Also see wait-for-switch.
wait-for-switch=<seconds>
Change max time to wait for a switch <seconds> secs.
TasksPerNode=<count>
Change the job's requested TasksPerNode.
ThreadSpec=<count>
Number of threads to reserve per node for system use. The job will be charged for
these threads, but be unable to use them. Will be reported as "*" if not
constrained.
TimeLimit=<time>
The job's time limit. Output format is [days-]hours:minutes:seconds or
"UNLIMITED". Input format (for update command) set is minutes, minutes:seconds,
hours:minutes:seconds, days-hours, days-hours:minutes or
days-hours:minutes:seconds. Time resolution is one minute and second values are
rounded up to the next minute. If changing the time limit of a job, either specify
a new time limit value or precede the time and equal sign with a "+" or "-" to
increment or decrement the current time limit (e.g. "TimeLimit+=30"). In order to
increment or decrement the current time limit, the JobId specification must precede
the TimeLimit specification. Note that incrementing or decrementing the time limit
for a job array is only allowed before the job array has been split into more than
one job record. Only the Slurm administrator or root can increase job's TimeLimit.
UserID=<UID or name>
Used with the JobName option to identify jobs to be modified. Either a user name
or numeric ID (UID), may be specified.
WCKey=<key>
Set the job's workload characterization key to the specified value.
WorkDir=<directory_name>
Set the job's working directory to the specified value. Note that this may only be
set for jobs in the PENDING state, and that jobs may fail to launch if they rely on
relative paths to the originally submitted WorkDir.
JOBS - SPECIFICATIONS FOR SHOW COMMAND
The "show" command, when used with the "job" or "job <jobid>" entity displays detailed
information about a job or jobs. Much of this information may be modified using the
"update job" command as described above. However, the following fields displayed by the
show job command are read-only and cannot be modified:
AllocNode:Sid
Local node and system id making the resource allocation.
BatchFlag
Jobs submitted using the sbatch command have BatchFlag set to 1. Jobs submitted
using other commands have BatchFlag set to 0.
ExitCode=<exit>:<sig>
Exit status reported for the job by the wait() function. The first number is the
exit code, typically as set by the exit() function. The second number of the
signal that caused the process to terminate if it was terminated by a signal.
GroupId
The group under which the job was submitted.
JobState
The current state of the job.
NodeListIndices
The NodeIndices expose the internal indices into the node table associated with the
node(s) allocated to the job.
NtasksPerN:B:S:C=<tasks_per_node>:<tasks_per_baseboard>:<tasks_per_socket>:<tasks_per_core>
Specifies the number of tasks to be started per hardware component (node,
baseboard, socket and core). Unconstrained values may be shown as "0" or "*".
PreemptEligibleTime
Time the job becomes eligible for preemption. Modified by PreemptExemptTime, either
from the global option in slurm.conf or the job QOS. This is hidden if the job has
not started or if PreemptMode=OFF.
PreemptTime
Time at which job was signaled that it was selected for preemption. This value is
only meaningful for PreemptMode=CANCEL and PreemptMode=REQUEUE and for jobs in a
partition or QOS that has a GraceTime value designated. This is hidden if the job
has not started or if PreemptMode=OFF.
PreSusTime
Time the job ran prior to last suspend.
Reason The reason job is not running: e.g., waiting "Resources".
ReqB:S:C:T=<baseboard_count>:<socket_per_baseboard_count>:<core_per_socket_count>:<thread_per_core_count>
Specifies the count of various hardware components requested by the job.
Unconstrained values may be shown as "0" or "*".
SecsPreSuspend=<seconds>
If the job is suspended, this is the run time accumulated by the job (in seconds)
prior to being suspended.
Socks/Node=<count>
Count of desired sockets per node
SubmitTime
The time and date stamp (in localtime) the job was submitted. The format of the
output is identical to that of the EndTime field.
NOTE: If a job is requeued, the submit time is reset. To obtain the original
submit time it is necessary to use the "sacct -j <job_id[.<step_id>]" command also
designating the -D or --duplicate option to display all duplicate entries for a
job.
SuspendTime
Time the job was last suspended or resumed.
NOTE on information displayed for various job states: When you submit a request for
the "show job" function the scontrol process makes an RPC request call to slurmctld
with a REQUEST_JOB_INFO message type. If the state of the job is PENDING, then it
returns some detail information such as: min_nodes, min_procs, cpus_per_task, etc.
If the state is other than PENDING the code assumes that it is in a further state
such as RUNNING, COMPLETE, etc. In these cases the code explicitly returns zero for
these values. These values are meaningless once the job resources have been
allocated and the job has started.
STEPS - SPECIFICATIONS FOR UPDATE COMMAND
StepId=<job_id>[.<step_id>]
Identify the step to be updated. If the job_id is given, but no step_id is
specified then all steps of the identified job will be modified. This
specification is required.
TimeLimit=<time>
The job's time limit. Output format is [days-]hours:minutes:seconds or
"UNLIMITED". Input format (for update command) set is minutes, minutes:seconds,
hours:minutes:seconds, days-hours, days-hours:minutes or
days-hours:minutes:seconds. Time resolution is one minute and second values are
rounded up to the next minute. If changing the time limit of a step, either
specify a new time limit value or precede the time with a "+" or "-" to increment
or decrement the current time limit (e.g. "TimeLimit=+30"). In order to increment
or decrement the current time limit, the StepId specification must precede the
TimeLimit specification.
NODES - SPECIFICATIONS FOR CREATE COMMAND
Provide the same NodeName configuration as found in the slurm.conf. See slurm.conf man
page for details. Only State=CLOUD and State=FUTURE nodes are allowed.
NODES - SPECIFICATIONS FOR UPDATE COMMAND
NodeName=<name>
Identify the node(s) to be updated. Multiple node names may be specified using
simple node range expressions (e.g. "lx[10-20]"). Nodesets can also be specified by
themselves or mixed with node range expressions, using a comma as a list separator.
If the keyword "ALL" is specified alone, then the update will be attempted against
all the nodes in the local cluster. This specification is required.
ActiveFeatures=<features>
Identify the feature(s) currently active on the specified node. Any previously
active feature specification will be overwritten with the new value. Also see
AvailableFeatures. Typically ActiveFeatures will be identical to
AvailableFeatures; however ActiveFeatures may be configured as a subset of the
AvailableFeatures. For example, a node may be booted in multiple configurations. In
that case, all possible configurations may be identified as AvailableFeatures,
while ActiveFeatures would identify the current node configuration. When updating
the ActiveFeatures with scontrol, the change is only made in slurmctld. When using
a node_features plugin the state/features of the node must be updated on the node
such that a new node start will report the updated state/features.
AvailableFeatures=<features>
Identify the feature(s) available on the specified node. Any previously defined
available feature specification will be overwritten with the new value.
AvailableFeatures assigned via scontrol will only persist across the restart of the
slurmctld daemon with the -R option and state files preserved or slurmctld's
receipt of a SIGHUP. Update slurm.conf with any changes meant to be persistent
across normal restarts of slurmctld or the execution of scontrol reconfig.
Note: Available features being removed via scontrol must not be active (i.e. remove
them from ActiveFeatures first).
Comment=<comment>
Arbitrary descriptive string. Use quotes to enclose a comment having more than one
word
CpuBind=<node>
Specify the task binding mode to be used by default for this node. Supported
options include: "none", "socket", "ldom" (NUMA), "core", "thread" and "off"
(remove previous binding mode).
Extra=<comment>
Arbitrary string on the node. Use quotes to enclose a string having more than one
word. See <https://slurm.schedmd.com/extra_constraints.html> for more details.
Gres=<gres>
Identify generic resources to be associated with the specified node. Any
previously defined generic resources will be overwritten with the new value.
Specifications for multiple generic resources should be comma separated. Each
resource specification consists of a name followed by an optional colon with a
numeric value (default value is one) (e.g. "Gres=bandwidth:10000"). Modification
of GRES count associated with specific files (e.g. GPUs) is not allowed other than
to set their count on a node to zero. In order to change the GRES count to another
value, modify your slurm.conf and gres.conf files and restart daemons. If GRES are
associated with specific sockets, that information will be reported For example if
all 4 GPUs on a node are all associated with socket zero, then "Gres=gpu:4(S:0)".
If associated with sockets 0 and 1 then "Gres=gpu:4(S:0-1)". The information of
which specific GPUs are associated with specific GPUs is not reported, but only
available by parsing the gres.conf file. Generic resources assigned via scontrol
will only persist across the restart of the slurmctld daemon with the -R option and
state files preserved or slurmctld's receipt of a SIGHUP. Update slurm.conf with
any changes meant to be persistent across normal restarts of slurmctld or the
execution of scontrol reconfig.
InstanceId=<instance_id>
Cloud instance ID.
InstanceType=<instance_type>
Cloud instance type.
NodeAddr=<node address>
Name that a node should be referred to in establishing a communications path. This
name will be used as an argument to the getaddrinfo() function for identification.
If a node range expression is used to designate multiple nodes, they must exactly
match the entries in the NodeName (e.g. "NodeName=lx[0-7] NodeAddr=elx[0-7]").
NodeAddr may also contain IP addresses.
NodeHostname=<node hostname>
Typically this would be the string that "/bin/hostname -s" returns. It may also be
the fully qualified domain name as returned by "/bin/hostname -f" (e.g.
"foo1.bar.com"), or any valid domain name associated with the host through the host
database (/etc/hosts) or DNS, depending on the resolver settings. Note that if the
short form of the hostname is not used, it may prevent use of hostlist expressions
(the numeric portion in brackets must be at the end of the string). A node range
expression can be used to specify a set of nodes. If an expression is used, the
number of nodes identified by NodeHostname must be identical to the number of nodes
identified by NodeName.
Reason=<reason>
Identify the reason the node is in a "DOWN", "DRAINED", "DRAINING", "FAILING" or
"FAIL" state. Use quotes to enclose a reason having more than one word.
ResumeAfter=<seconds>
Schedule a node state resume after this amount of seconds, when its state is
updated to "DOWN" or "DRAIN". Upon state resume, the node's state will be changed
from DRAIN, DRAINING, DOWN or REBOOT to IDLE and NoResp. slurmctld will then
attempt to contact slurmd to request that the node register itself.
NOTE: A value of -1 will unschedule the node state resume.
State=<state>
Assign one of the following states/actions to the node(s) specified by the update
command.
CANCEL_REBOOT
Cancels a pending reboot on the node (same as scontrol cancel_reboot
<node>).
DOWN Stop all running and suspended jobs and make the node unavailable for new
jobs.
DRAIN Indicates that no new jobs may be started on this node. Existing jobs are
allowed to run to completion, leaving the node in a DRAINED state once all
the jobs have completed.
FAIL Similar to DRAIN except that some applications will seek to relinquish those
nodes before the job completes.
FUTURE Indicates the node is not fully configured, but is expected to be available
at some point in the future.
IDLE Will clear DOWN, DRAIN and FAIL states. Will set the state to IDLE and
NoResp. slurmctld will then attempt to contact slurmd to request that the
node register itself. Once registered, the node state will then remove the
NoResp flag and will resume normal operations.
NoResp This will set the "Not Responding" flag for a node without changing its
underlying state.
POWER_DOWN
Will use the configured SuspendProgram program to explicitly place a node in
power saving mode. If a node is already in the process of being powered
down, the command will only change the state of the node but won't have any
effect until the configured SuspendTimeout is reached. Use of this command
can be useful in situations where a ResumeProgram, like capmc in Cray
machines, is stalled and one wants to restore the node to "IDLE" manually.
In this case rebooting the node and setting the state to "POWER_DOWN" will
cancel the previous "POWER_UP" state and the node will become "IDLE".
POWER_DOWN_ASAP
Will drain the node and mark it for power down. Currently running jobs will
complete first and no additional jobs will be allocated to the node.
POWER_DOWN_FORCE
Will cancel all jobs on the node, power it down, and reset its state to
"IDLE".
POWER_UP
Will use the configured ResumeProgram program to explicitly move a node out
of power saving mode. If a node is already in the process of being powered
up, the command will only change the state of the node but won't have any
effect until the configured ResumeTimeout is reached.
RESUME Not an actual node state, but will change a node state from DRAIN, DRAINING,
DOWN or REBOOT to IDLE and NoResp. slurmctld will then attempt to contact
slurmd to request that the node register itself. Once registered, the node
state will then remove the NoResp flag and will resume normal operations. It
will also clear the POWERING_DOWN state of a node and make it eligible to be
allocated.
UNDRAIN
Clears the node from being drained (like RESUME), but will not change the
node's base state (e.g. DOWN). UNDRAIN requires a valid node registration
before new jobs can be scheduled on the node. Setting a node DOWN will
cause all running and suspended jobs on that node to be terminated.
While all of the above states are valid, some of them are not valid new node states
given their prior state.
NOTE: The scontrol command should not be used to change node state on Cray systems.
Use Cray tools such as xtprocadmin instead.
Weight=<weight>
Identify weight to be associated with specified nodes. This allows dynamic changes
to weight associated with nodes, which will be used for the subsequent node
allocation decisions. Weight assigned via scontrol will only persist across the
restart of the slurmctld daemon with the -R option and state files preserved or
slurmctld's receipt of a SIGHUP. Update slurm.conf with any changes meant to be
persistent across normal restarts of slurmctld or the execution of scontrol
reconfig.
NODES - SPECIFICATIONS FOR DELETE COMMAND
NodeName=<nodes>
Identify the node(s) to be deleted. Multiple node names may be specified using
simple node range expressions (e.g. "lx[10-20]"). Nodesets can also be specified by
themselves or mixed with node range expressions, using a comma as a list separator.
If the keyword "ALL" is specified alone, then the update will be attempted against
all the nodes in the local cluster. This specification is required.
NODES - SPECIFICATIONS FOR SHOW COMMAND
AllocMem
The total memory, in MB, currently allocated by jobs on the node.
CPULoad
CPU load of a node as reported by the OS.
CPUSpecList
The list of Slurm abstract CPU IDs on this node reserved for exclusive use by the
Slurm compute node daemons (slurmd, slurmstepd).
FreeMem
The total memory, in MB, currently free on the node as reported by the OS.
LastBusyTime
The last time the node was busy (i.e. last time the node had jobs on it). This time
is used in PowerSave to determine when to suspend nodes (e.g. now - LastBusy >
SuspendTime).
MemSpecLimit
The combined memory limit, in megabytes, on this node for the Slurm compute node
daemons (slurmd, slurmstepd).
RealMemory
The total memory, in MB, on the node.
State Identify the state(s) assigned to the node with '+' delimited state flags.
States:
ALLOCATED
Indicates that the node has all CPUs allocated to job(s) running on the
node.
DOWN The node does not have any running jobs and is unavailable for new work.
ERROR The node is in an error state. Consult the logs for more information about
what caused this state.
FUTURE The node is currently not fully configured, but expected to be available at
some point in the indefinite future for use.
IDLE Indicates that the node is available for work but does not currently have
any jobs assigned to it.
MIXED Indicates that the node is in multiple states. For instance if only part of
the node is ALLOCATED and the rest of the node is IDLE the state will be
MIXED.
UNKNOWN
The node has not yet registered with the controller and its state is not
known.
Flags:
CLOUD Indicates that the node is configured as a cloud node, to be brought up on
demand, but not currently running.
COMPLETING
Indicates that the only job on the node or that all jobs on the node are in
the process of completing.
DRAIN The node is not accepting any new jobs and any currently running jobs will
complete.
DYNAMIC
Slurm allows you to define multiple types of nodes in a FUTURE state. When
starting slurmd on a node you can specify the -F flag to have the node match
and use an existing definition in your slurm.conf file. The DYNAMIC state
indicates that the node was started as a Dynamic Future node.
INVALID_REG
The node did not register correctly with the controller. This happens when a
node registers with less resources than configured in the slurm.conf file.
The node will clear from this state with a valid registration (i.e. a slurmd
restart is required).
MAINTENANCE
The node is currently in a reservation that includes the maintenance flag.
NOT_RESPONDING
Node is not responding.
PERFCTRS
Indicates that Network Performance Counters associated with this node are in
use, rendering this node as not usable for any other jobs.
POWER_DOWN
Node is pending power down.
POWERED_DOWN
Node is currently powered down and not capable of running any jobs.
POWERING_DOWN
Node is in the process of powering down.
POWERING_UP
Node is in the process of powering up.
PLANNED
The node is earmarked for a job that will start in the future.
REBOOT_ISSUED
A reboot request has been sent to the agent configured to handle this
request.
REBOOT_REQUESTED
A request to reboot this node has been made, but hasn't been handled yet.
RESERVED
Indicates the node is in an advanced reservation and not generally
available.
The meaning of the energy information is as follows:
CurrentWatts
The instantaneous power consumption of the node at the time of the last node
energy accounting sample, in watts.
LowestJoules
The energy consumed by the node between the last time it was powered on and
the last time it was registered by slurmd, in joules.
ConsumedJoules
The energy consumed by the node between the last time it was registered by
the slurmd daemon and the last node energy accounting sample, in joules.
If the reported value is "n/s" (not supported), the node does not support the
configured AcctGatherEnergyType plugin. If the reported value is zero, energy
accounting for nodes is disabled.
The meaning of the external sensors information is as follows:
ExtSensorsJoules
The energy consumed by the node between the last time it was powered on and
the last external sensors plugin node sample, in joules.
ExtSensorsWatts
The instantaneous power consumption of the node at the time of the last
external sensors plugin node sample, in watts.
ExtSensorsTemp
The temperature of the node at the time of the last external sensors plugin
node sample, in celsius.
If the reported value is "n/s" (not supported), the node does not support the
configured ExtSensorsType plugin.
FRONTEND - SPECIFICATIONS FOR UPDATE COMMAND
FrontendName=<name>
Identify the front end node to be updated. This specification is required.
Reason=<reason>
Identify the reason the node is in a "DOWN" or "DRAIN" state. Use quotes to
enclose a reason having more than one word.
State=<state>
Identify the state to be assigned to the front end node. Possible values are
"DOWN", "DRAIN" or "RESUME". If you want to remove a front end node from service,
you typically want to set its state to "DRAIN". "RESUME" is not an actual node
state, but will return a "DRAINED", "DRAINING", or "DOWN" front end node to
service, either "IDLE" or "ALLOCATED" state as appropriate. Setting a front end
node "DOWN" will cause all running and suspended jobs on that node to be
terminated.
PARTITIONS - SPECIFICATIONS FOR CREATE, UPDATE, AND DELETE COMMANDS
AllocNodes=<name>
Comma separated list of nodes from which users can execute jobs in the partition.
Node names may be specified using the node range expression syntax described above.
The default value is "ALL".
AllowAccounts=<name>
Identify the Accounts which may use this partition. Multiple Accounts may be
specified in a comma separated list. To permit all Accounts to use the partition
specify "AllowAccounts=ALL".
AllowGroups=<name>
Identify the user groups which may use this partition. Multiple groups may be
specified in a comma separated list. To permit all groups to use the partition
specify "AllowGroups=ALL".
AllowQOS=<name>
Identify the QOSs which may use this partition. Multiple QOSs may be specified in
a comma separated list. To permit all QOSs to use the partition specify
"AllowQOS=ALL".
Alternate=<partition name>
Alternate partition to be used if the state of this partition is "DRAIN" or
"INACTIVE." The value "NONE" will clear a previously set alternate partition.
CpuBind=<node>
Specify the task binding mode to be used by default for this partition. Supported
options include: "none", "socket", "ldom" (NUMA), "core", "thread" and "off"
(remove previous binding mode).
Default={yes|no}
Specify if this partition is to be used by jobs which do not explicitly identify a
partition to use. Possible output values are "YES" and "NO". In order to change
the default partition of a running system, use the scontrol update command and set
Default=yes for the partition that you want to become the new default.
DefaultTime=<time>
Run time limit used for jobs that don't specify a value. If not set then MaxTime
will be used. Format is the same as for MaxTime.
DefMemPerCPU=<MB>
Set the default memory to be allocated per CPU for jobs in this partition. The
memory size is specified in megabytes.
DefMemPerNode=<MB>
Set the default memory to be allocated per node for jobs in this partition. The
memory size is specified in megabytes.
DenyAccounts=<name>
Identify the Accounts which should be denied access to this partition. Multiple
Accounts may be specified in a comma separated list.
DenyQOS=<name>
Identify the QOSs which should be denied access to this partition. Multiple QOSs
may be specified in a comma separated list.
DisableRootJobs={yes|no}
Specify if jobs can be executed as user root. Possible values are "YES" and "NO".
ExclusiveUser={yes|no}
When enabled nodes will be exclusively allocated to users. Multiple jobs can be run
simultaneously, but those jobs must be from a single user.
GraceTime=<seconds>
Specifies, in units of seconds, the preemption grace time to be extended to a job
which has been selected for preemption. The default value is zero, no preemption
grace time is allowed on this partition or qos. (Meaningful only for
PreemptMode=CANCEL and PreemptMode=REQUEUE)
Hidden={yes|no}
Specify if the partition and its jobs should be hidden from view. Hidden
partitions will by default not be reported by Slurm APIs or commands. Possible
values are "YES" and "NO".
JobDefaults=<specs>
Specify job default values using a comma-delimited list of "key=value" pairs.
Supported keys include
DefCpuPerGPU Default number of CPUs per allocated GPU.
DefMemPerGPU Default memory limit (in megabytes) per allocated GPU.
MaxCPUsPerNode=<count>
Set the maximum number of CPUs that can be allocated per node to all jobs in this
partition.
LLN={yes|no}
Schedule jobs on the least loaded nodes (based on the number of idle CPUs).
MaxMemPerCPU=<MB>
Set the maximum memory to be allocated per CPU for jobs in this partition. The
memory size is specified in megabytes.
MaxMemPerNode=<MB>
Set the maximum memory to be allocated per node for jobs in this partition. The
memory size is specified in megabytes.
MaxNodes=<count>
Set the maximum number of nodes which will be allocated to any single job in the
partition. Specify a number, "INFINITE" or "UNLIMITED". Changing the MaxNodes of a
partition has no effect upon jobs that have already begun execution.
MaxTime=<time>
The maximum run time for jobs. Output format is [days-]hours:minutes:seconds or
"UNLIMITED". Input format (for update command) is minutes, minutes:seconds,
hours:minutes:seconds, days-hours, days-hours:minutes or
days-hours:minutes:seconds. Time resolution is one minute and second values are
rounded up to the next minute. Changing the MaxTime of a partition has no effect
upon jobs that have already begun execution.
MinNodes=<count>
Set the minimum number of nodes which will be allocated to any single job in the
partition. Changing the MinNodes of a partition has no effect upon jobs that have
already begun execution. Increasing this value may prevent pending jobs from
starting, even if they were submitted without -N/--nodes specification. If you do
get in that situation, updating the MinNodes value of a pending job using the
scontrol command will allow that job to be scheduled.
MaxCPUsPerSocket=<count>
Set the maximum number of CPUs that can be allocated per socket to all jobs in this
partition.
Nodes=<name>
Identify the node(s) to be associated with this partition. Multiple node names may
be specified using simple node range expressions (e.g. "lx[10-20]"). A
specification of "ALL" will associate all nodes. You can add or remove nodes by
adding a '+' or '-' sign before the '=' sign. It also works to do
nodes=+lx0,-lx[2-4] if you want to remove lx[2-4] and add lx0 at the same time.
Note that jobs may only be associated with one partition at any time. Specify a
blank data value to remove all nodes from a partition: "Nodes=". Changing the
Nodes in a partition has no effect upon jobs that have already begun execution.
OverSubscribe={yes|no|exclusive|force}[:<job_count>]
Specify if compute resources (i.e. individual CPUs) in this partition can be shared
by multiple jobs. Possible values are "YES", "NO", "EXCLUSIVE" and "FORCE". An
optional job count specifies how many jobs can be allocated to use each resource.
OverTimeLimit=<count>
Number of minutes by which a job can exceed its time limit before being canceled.
The configured job time limit is treated as a soft limit. Adding OverTimeLimit to
the soft limit provides a hard limit, at which point the job is canceled. This is
particularly useful for backfill scheduling, which bases upon each job's soft time
limit. A partition-specific OverTimeLimit will override any global OverTimeLimit
value. If not specified, the global OverTimeLimit value will take precedence. May
not exceed 65533 minutes. An input value of "UNLIMITED" will clear any previously
configured partition-specific OverTimeLimit value.
PartitionName=<name>
Identify the partition to be updated. This specification is required.
PowerDownOnIdle
If set to "YES", then nodes allocated from this partition will immediately be
requested to power down upon becoming IDLE. A power down request prevents further
scheduling to the node until it has been put into power save mode by
SuspendProgram.
Also see SuspendTime.
PreemptMode=<mode>
Reset the mechanism used to preempt jobs in this partition if PreemptType is
configured to preempt/partition_prio. The default preemption mechanism is specified
by the cluster-wide PreemptMode configuration parameter. Possible values are
"OFF", "CANCEL", "REQUEUE" and "SUSPEND".
Priority=<count>
Jobs submitted to a higher priority partition will be dispatched before pending
jobs in lower priority partitions and if possible they will preempt running jobs
from lower priority partitions. Note that a partition's priority takes precedence
over a job's priority. The value may not exceed 65533.
PriorityJobFactor=<count>
Partition factor used by priority/multifactor plugin in calculating job priority.
The value may not exceed 65533. Also see PriorityTier.
PriorityTier=<count>
Jobs submitted to a partition with a higher priority tier value will be dispatched
before pending jobs in partition with lower priority tier value and, if possible,
they will preempt running jobs from partitions with lower priority tier values.
Note that a partition's priority tier takes precedence over a job's priority. The
value may not exceed 65533. Also see PriorityJobFactor.
QOS=<QOSname|blank to remove>
Set the partition QOS with a QOS name or to remove the Partition QOS leave the
option blank.
ReqResv={yes|no}
Specify if only allocation requests designating a reservation will be satisfied.
This is used to restrict partition usage to be allowed only within a reservation.
Possible values are "YES" and "NO".
RootOnly={yes|no}
Specify if only allocation requests initiated by user root will be satisfied. This
can be used to restrict control of the partition to some meta-scheduler. Possible
values are "YES" and "NO".
Shared={yes|no|exclusive|force}[:<job_count>]
Renamed to OverSubscribe, see option descriptions above.
State={up|down|drain|inactive}
Specify if jobs can be allocated nodes or queued in this partition. Possible
values are "UP", "DOWN", "DRAIN" and "INACTIVE".
UP Designates that new jobs may queued on the partition, and that jobs may
be allocated nodes and run from the partition.
DOWN Designates that new jobs may be queued on the partition, but queued jobs
may not be allocated nodes and run from the partition. Jobs already
running on the partition continue to run. The jobs must be explicitly
canceled to force their termination.
DRAIN Designates that no new jobs may be queued on the partition (job
submission requests will be denied with an error message), but jobs
already queued on the partition may be allocated nodes and run. See also
the "Alternate" partition specification.
INACTIVE Designates that no new jobs may be queued on the partition, and jobs
already queued may not be allocated nodes and run. See also the
"Alternate" partition specification.
TRESBillingWeights=<TRES Billing Weights>
TRESBillingWeights is used to define the billing weights of each TRES type that
will be used in calculating the usage of a job. The calculated usage is used when
calculating fairshare and when enforcing the TRES billing limit on jobs. Updates
affect new jobs and not existing jobs. See the slurm.conf man page for more
information.
RESERVATIONS - SPECIFICATIONS FOR CREATE, UPDATE, AND DELETE COMMANDS
Reservation=<name>
Identify the name of the reservation to be created, updated, or deleted. This
parameter is required for update and is the only parameter for delete. For create,
if you do not want to give a reservation name, use "scontrol create reservation
..." and a name will be created automatically.
Accounts=<account list>
List of accounts permitted to use the reserved nodes, for example
"Accounts=physcode1,physcode2". A user in any of the accounts may use the reserved
nodes. A new reservation must specify Users or Groups and/or Accounts. If both
Users/Groups and Accounts are specified, a job must match both in order to use the
reservation. Accounts can also be denied access to reservations by preceding all
of the account names with '-'. Alternately precede the equal sign with '-'. For
example, "Accounts=-physcode1,-physcode2" or "Accounts-=physcode1,physcode2" will
permit any account except physcode1 and physcode2 to use the reservation. You can
add or remove individual accounts from an existing reservation by using the update
command and adding a '+' or '-' sign before the '=' sign. If accounts are denied
access to a reservation (account name preceded by a '-'), then all other accounts
are implicitly allowed to use the reservation and it is not possible to also
explicitly specify allowed accounts.
BurstBuffer=<buffer_spec>[,<buffer_spec>,...]
Specification of burst buffer resources which are to be reserved. "buffer_spec"
consists of four elements: [plugin:][type:]#[units] "plugin" is the burst buffer
plugin name, currently either "datawarp" or "generic". If no plugin is specified,
the reservation applies to all configured burst buffer plugins. "type" specifies a
Cray generic burst buffer resource, for example "nodes". if "type" is not
specified, the number is a measure of storage space. The "units" may be "N"
(nodes), "K|KiB", "M|MiB", "G|GiB", "T|TiB", "P|PiB" (for powers of 1024) and "KB",
"MB", "GB", "TB", "PB" (for powers of 1000). The default units are bytes for
reservations of storage space. For example "BurstBuffer=datawarp:2TB" (reserve 2TB
of storage plus 3 nodes from the Cray plugin) or "BurstBuffer=100GB" (reserve 100
GB of storage from all configured burst buffer plugins). Jobs using this
reservation are not restricted to these burst buffer resources, but may use these
reserved resources plus any which are generally available. NOTE: Usually Slurm
interprets KB, MB, GB, TB, PB units as powers of 1024, but for Burst Buffers size
specifications Slurm supports both IEC/SI formats. This is because the CRAY API
for managing DataWarps supports both formats.
CoreCnt=<num>
This option is only supported when select/cons_tres is used. Identify number of
cores to be reserved. If NodeCnt or Nodelist is used this is the total number of
cores to reserve where cores per node is CoreCnt/NodeCnt.
Licenses=<license>
Specification of licenses (or other resources available on all nodes of the
cluster) which are to be reserved. License names can be followed by a colon and
count (the default count is one). Multiple license names should be comma separated
(e.g. "Licenses=foo:4,bar"). A new reservation must specify one or more resource
to be included: NodeCnt, Nodes and/or Licenses. If a reservation includes
Licenses, but no NodeCnt or Nodes, then the option Flags=LICENSE_ONLY must also be
specified. Jobs using this reservation are not restricted to these licenses, but
may use these reserved licenses plus any which are generally available.
MaxStartDelay[=<timespec>]
Change MaxStartDelay value which specifies the maximum time an eligible job not
requesting this reservation can delay a job requesting it. Default is none. Valid
formats are minutes, minutes:seconds, hours:minutes:seconds, days-hours,
days-hours:minutes, days-hours:minutes:seconds. Time resolution is one minute and
second values are rounded up to the next minute. Output format is always
[days-]hours:minutes:seconds.
NodeCnt=<num>[,<num>,...]
Identify number of nodes to be reserved. The number can include a suffix of "k" or
"K", in which case the number specified is multiplied by 1024. A new reservation
must specify one or more resource to be included: NodeCnt, Nodes and/or Licenses.
Nodes=<name>
Identify the node(s) to be reserved. Multiple node names may be specified using
simple node range expressions (e.g. "Nodes=lx[10-20]"). When using Nodes to
specify more or fewer nodes, NodeCnt will be updated to honor the new number of
nodes. However, when setting an empty list ("Nodes="), the nodelist will be filled
with random nodes to fulfill the previous nodecnt and the SPEC_NODES flag will be
removed. A new reservation must specify one or more resource to be included:
NodeCnt, Nodes and/or Licenses. A specification of "ALL" will reserve all nodes.
Set Flags=PART_NODES and PartitionName= in order for changes in the nodes
associated with a partition to also be reflected in the nodes associated with a
reservation. You can add or remove nodes from an existing reservation by adding a
'+' or '-' sign before the '=' sign. It also works to do nodes=+lx0,-lx[2-4] if you
want to remove lx[2-4] and add lx0 at the same time.
NOTE: When updating a reservation, if Nodes and Nodecnt are set simultaneously,
nodecnt will always be honored. The reservation will get a subset of nodes if nodes
> nodecnt, or it will add extra nodes to the list when nodes < nodecnt.
StartTime=<time_spec>
The start time for the reservation. A new reservation must specify a start time.
It accepts times of the form HH:MM:SS for a specific time of day (seconds are
optional). (If that time is already past, the next day is assumed.) You may also
specify midnight, noon, fika (3 PM) or teatime (4 PM) and you can have a
time-of-day suffixed with AM or PM for running in the morning or the evening. You
can also say what day the job will be run, by specifying a date of the form MMDDYY
or MM/DD/YY or MM.DD.YY, or a date and time as YYYY-MM-DD[THH:MM[:SS]]. You can
also give times like now + count time-units, where the time-units can be seconds
(default), minutes, hours, days, or weeks and you can tell Slurm to run the job
today with the keyword today and to run the job tomorrow with the keyword tomorrow.
You cannot update the StartTime of a reservation in ACTIVE state.
EndTime=<time_spec>
The end time for the reservation. A new reservation must specify an end time or a
duration. Valid formats are the same as for StartTime.
Duration=<time>
The length of a reservation. A new reservation must specify an end time or a
duration. Valid formats are minutes, minutes:seconds, hours:minutes:seconds,
days-hours, days-hours:minutes, days-hours:minutes:seconds, or UNLIMITED. Time
resolution is one minute and second values are rounded up to the next minute.
Output format is always [days-]hours:minutes:seconds.
PartitionName=<name>
Partition used to reserve nodes from. This will attempt to allocate all nodes in
the specified partition unless you request fewer resources than are available with
CoreCnt, NodeCnt or TRES. If no partition is specified at submit time, this
partition will override the job's default partition. Jobs explicitly requesting a
different partition will still be allowed to use this reservation as long as there
are enough overlapping nodes between both partitions to allocate the job.
Flags=<flags>
Flags associated with the reservation. You can add or remove individual flags from
an existing reservation by adding a '+' or '-' sign before the '=' sign. For
example: Flags-=DAILY (NOTE: This shortcut is not supported for all flags).
Currently supported flags include:
ANY_NODES This is a reservation for burst buffers and/or licenses only and not
compute nodes. If this flag is set, a job using this reservation may
use the associated burst buffers and/or licenses plus any compute
nodes. If this flag is not set, a job using this reservation may use
only the nodes and licenses associated with the reservation.
DAILY Repeat the reservation at the same time every day.
FLEX Permit jobs requesting the reservation to begin prior to the
reservation's start time, end after the reservation's end time, and
use any resources inside and/or outside of the reservation regardless
of any constraints possibly set in the reservation. A typical use
case is to prevent jobs not explicitly requesting the reservation
from using those reserved resources rather than forcing jobs
requesting the reservation to use those resources in the time frame
reserved. Another use case could be to always have a particular
number of nodes with a specific feature reserved for a specific
account so users in this account may use this nodes plus possibly
other nodes without this feature.
IGNORE_JOBS Ignore currently running jobs when creating the reservation. This
can be especially useful when reserving all nodes in the system for
maintenance.
HOURLY Repeat the reservation at the same time every hour.
LICENSE_ONLY See ANY_NODES.
MAINT Maintenance mode, receives special accounting treatment. This
reservation is permitted to use resources that are already in another
reservation. The MAINT flag behaves similar to STATIC_ALLOC (see
below) in that it will not replace nodes once they are set.
MAGNETIC This flag allows jobs to be considered for this reservation even if
they didn't request it.
NO_HOLD_JOBS_AFTER
By default, when a reservation ends the reservation request will be
removed from any pending jobs submitted to the reservation and will
be put into a held state. Use this flag to let jobs run outside of
the reservation after the reservation is gone. Flag removal with '-='
is not supported.
OVERLAP This reservation can be allocated resources that are already in
another reservation. Flag removal with '-=' is not supported.
PART_NODES This flag can be used to reserve all nodes within the specified
partition. PartitionName and Nodes=ALL must be specified with this
flag.
PURGE_COMP[=<timespec>]
Purge the reservation if it is ever idle for timespec (no jobs
associated with it). If timespec isn't given then 5 minutes is the
default. Valid timespec formats are minutes, minutes:seconds,
hours:minutes:seconds, days-hours, days-hours:minutes,
days-hours:minutes:seconds. Time resolution is one minute and second
values are rounded up to the next minute. Output format is always
[days-]hours:minutes:seconds.
REPLACE Nodes which are DOWN, DRAINED, or allocated to jobs are automatically
replenished using idle resources. This option can be used to
maintain a constant number of idle resources available for pending
jobs (subject to availability of idle resources). This should be
used with the NodeCnt reservation option; do not identify specific
nodes to be included in the reservation. Flag removal with '-=' is
not supported.
NOTE: Removing a node from the cluster while in a reservation with
the REPLACE flag will not cause it to be replaced.
REPLACE_DOWN Nodes which are DOWN or DRAINED are automatically replenished using
idle resources. This option can be used to maintain a constant sized
pool of resources available for pending jobs (subject to availability
of idle resources). This should be used with the NodeCnt reservation
option; do not identify specific nodes to be included in the
reservation. Flag removal with '-=' is not supported.
NOTE: Removing a node from the cluster while in a reservation with
the REPLACE_DOWN flag will not cause it to be replaced.
SPEC_NODES Reservation is for specific nodes (output only).
STATIC_ALLOC Make it so after the nodes are selected for a reservation they don't
change. Without this option when nodes are selected for a
reservation and one goes down the reservation will select a new node
to fill the spot.
TIME_FLOAT The reservation start time is relative to the current time and moves
forward through time (e.g. a StartTime=now+10minutes will always be
10 minutes in the future). Repeating (e.g. DAILY) floating
reservations are not supported. Flag cannot be added to or removed
from an existing reservation.
WEEKDAY Repeat the reservation at the same time on every weekday (Monday,
Tuesday, Wednesday, Thursday and Friday).
WEEKEND Repeat the reservation at the same time on every weekend day
(Saturday and Sunday).
WEEKLY Repeat the reservation at the same time every week.
Features=<features>
Set the reservation's required node features. Multiple values may be "&" separated
if all features are required (AND operation) or separated by "|" if any of the
specified features are required (OR operation). Parenthesis are also supported for
features to be ANDed together with counts of nodes having the specified features.
For example "Features=[(knl&a2a&flat)*4&haswell*2]" indicates the advanced
reservation should include 4 nodes with ALL of the features "knl", "a2a", and
"flat" plus 2 nodes with the feature "haswell".
Value may be cleared with blank data value, "Features=".
Groups=<group list>
List of groups permitted to use the reserved nodes, for example "Group=bio,chem".
A new reservation must specify Users or Groups and/or Accounts. If both
Users/Groups and Accounts are specified, a job must match both in order to use the
reservation. Unlike users groups do not allow denied access to reservations. You
can add or remove individual groups from an existing reservation by using the
update command and adding a '+' or '-' sign before the '=' sign. NOTE: Groups and
Users are mutually exclusive in reservations, if you want to switch between the 2
you must update the reservation with a group='' or user='' and fill in the opposite
with the appropriate setting.
Skip Used on a reoccurring reservation, skip to the next reservation iteration. NOTE:
Only available for update.
Users=<user list>
List of users permitted to use the reserved nodes, for example
"User=jones1,smith2". A new reservation must specify Users or Groups and/or
Accounts. If both Users/Groups and Accounts are specified, a job must match both
in order to use the reservation. Users can also be denied access to reservations
by preceding all of the user names with '-'. Alternately precede the equal sign
with '-'. For example, "User=-jones1,-smith2" or "User-=jones1,smith2" will permit
any user except jones1 and smith2 to use the reservation. You can add or remove
individual users from an existing reservation by using the update command and
adding a '+' or '-' sign before the '=' sign. If users are denied access to a
reservation (user name preceded by a '-'), then all other users are implicitly
allowed to use the reservation and it is not possible to also explicitly specify
allowed users. NOTE: Groups and Users are mutually exclusive in reservations, if
you want to switch between the 2 you must update the reservation with a group='' or
user='' and fill in the opposite with the appropriate setting.
TRES=<tres_spec>
Comma-separated list of TRES required for the reservation. Current supported TRES
types with reservations are: CPU, GRES, Node, License and BB. CPU and Node follow
the same format as CoreCnt and NodeCnt parameters respectively. License names can
be followed by an equal '=' and a count:
License/<name1>=<count1>[,License/<name2>=<count2>,...]
BurstBuffer can be specified in a similar way as BurstBuffer parameter. The only
difference is that colon symbol ':' should be replaced by an equal '=' in order to
follow the TRES format.
Some examples of TRES valid specifications:
TRES=cpu=5,bb/cray=4,license/iop1=1,license/iop2=3
TRES=node=5k,license/iop1=2
TRES=gres/gpu:a100=2
Please note that CPU, Node, License and BB can override CoreCnt, NodeCnt, Licenses
and BurstBuffer parameters respectively. Also CPU represents CoreCnt, in a
reservation and will be adjusted if you have threads per core on your nodes.
Note that a reservation that contains nodes or cores is associated with one
partition, and can't span resources over multiple partitions. The only exception
from this is when the reservation is created with explicitly requested nodes.
Note that if GRES is requested those GRES must be TRES listed in
AccountingStorageTRES to be valid.
TRESPerNode=<tres_spec>
Comma-separated list of TRES required per node for the reservation. See TRES above
for supported types.
TRESPerNode=gres/gpu:a100=2
Above will allocate 2 gpu:a100 per node as specified in nodecnt.
PERFORMANCE
Executing scontrol sends a remote procedure call to slurmctld. If enough calls from
scontrol or other Slurm client commands that send remote procedure calls to the slurmctld
daemon come in at once, it can result in a degradation of performance of the slurmctld
daemon, possibly resulting in a denial of service.
Do not run scontrol or other Slurm client commands that send remote procedure calls to
slurmctld from loops in shell scripts or other programs. Ensure that programs limit calls
to scontrol to the minimum necessary for the information you are trying to gather.
ENVIRONMENT VARIABLES
Some scontrol options may be set via environment variables. These environment variables,
along with their corresponding options, are listed below. (Note: Command line options will
always override these settings.)
SCONTROL_ALL -a, --all
SCONTROL_FEDERATION --federation
SCONTROL_FUTURE -F, --future
SCONTROL_LOCAL --local
SCONTROL_SIBLING --sibling
SLURM_BITSTR_LEN Specifies the string length to be used for holding a job array's task
ID expression. The default value is 64 bytes. A value of 0 will
print the full expression with any length required. Larger values may
adversely impact the application performance.
SLURM_CLUSTERS Same as --clusters
SLURM_CONF The location of the Slurm configuration file.
SLURM_CONF_OUT When running 'write config', the location of the Slurm configuration
file to be written.
SLURM_DEBUG_FLAGS Specify debug flags for scontrol to use. See DebugFlags in the
slurm.conf(5) man page for a full list of flags. The environment
variable takes precedence over the setting in the slurm.conf.
SLURM_TIME_FORMAT Specify the format used to report time stamps. A value of standard,
the default value, generates output in the form
"year-month-dateThour:minute:second". A value of relative returns
only "hour:minute:second" if the current day. For other dates in the
current year it prints the "hour:minute" preceded by "Tomorr"
(tomorrow), "Ystday" (yesterday), the name of the day for the coming
week (e.g. "Mon", "Tue", etc.), otherwise the date (e.g. "25 Apr").
For other years it returns a date month and year without a time (e.g.
"6 Jun 2012"). All of the time stamps use a 24 hour format.
A valid strftime() format can also be specified. For example, a value
of "%a %T" will report the day of the week and a time stamp (e.g. "Mon
12:34:56").
SLURM_TOPO_LEN Specify the maximum size of the line when printing Topology. If not
set, the default value is unlimited.
AUTHORIZATION
When using SlurmDBD, users who have an AdminLevel defined (Operator or Admin) and users
who are account coordinators are given the authority to view and modify jobs,
reservations, nodes, etc., as defined in the following table - regardless of whether a
PrivateData restriction has been defined in the slurm.conf file.
scontrol show job(s): Admin, Operator, Coordinator
scontrol update job: Admin, Operator, Coordinator
scontrol requeue: Admin, Operator, Coordinator
scontrol show step(s): Admin, Operator, Coordinator
scontrol update step: Admin, Operator, Coordinator
scontrol show node: Admin, Operator
scontrol update node: Admin
scontrol create partition: Admin
scontrol show partition: Admin, Operator
scontrol update partition: Admin
scontrol delete partition: Admin
scontrol create reservation: Admin, Operator
scontrol show reservation: Admin, Operator
scontrol update reservation: Admin, Operator
scontrol delete reservation: Admin, Operator
scontrol reconfig: Admin
scontrol shutdown: Admin
scontrol takeover: Admin
EXAMPLES
$ scontrol
scontrol: show part debug
PartitionName=debug
AllocNodes=ALL AllowGroups=ALL Default=YES
DefaultTime=NONE DisableRootJobs=NO Hidden=NO
MaxNodes=UNLIMITED MaxTime=UNLIMITED MinNodes=1
Nodes=snowflake[0-48]
Priority=1 RootOnly=NO OverSubscribe=YES:4
State=UP TotalCPUs=694 TotalNodes=49
scontrol: update PartitionName=debug MaxTime=60:00 MaxNodes=4
scontrol: show job 71701
JobId=71701 Name=hostname
UserId=da(1000) GroupId=da(1000)
Priority=66264 Account=none QOS=normal WCKey=*123
JobState=COMPLETED Reason=None Dependency=(null)
TimeLimit=UNLIMITED Requeue=1 Restarts=0 BatchFlag=0 ExitCode=0:0
SubmitTime=2010-01-05T10:58:40 EligibleTime=2010-01-05T10:58:40
StartTime=2010-01-05T10:58:40 EndTime=2010-01-05T10:58:40
SuspendTime=None SecsPreSuspend=0
Partition=debug AllocNode:Sid=snowflake:4702
ReqNodeList=(null) ExcNodeList=(null)
NodeList=snowflake0
NumNodes=1 NumCPUs=10 CPUs/Task=2 ReqS:C:T=1:1:1
MinCPUsNode=2 MinMemoryNode=0 MinTmpDiskNode=0
Features=(null) Reservation=(null)
OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null)
scontrol: update JobId=71701 TimeLimit=30:00 Priority=500
scontrol: show hostnames tux[1-3]
tux1
tux2
tux3
scontrol: create res StartTime=2009-04-01T08:00:00 Duration=5:00:00 Users=dbremer NodeCnt=10
Reservation created: dbremer_1
scontrol: update Reservation=dbremer_1 Flags=Maint NodeCnt=20
scontrol: delete Reservation=dbremer_1
scontrol: quit
COPYING
Copyright (C) 2002-2007 The Regents of the University of California. Produced at Lawrence
Livermore National Laboratory (cf, DISCLAIMER).
Copyright (C) 2008-2010 Lawrence Livermore National Security.
Copyright (C) 2010-2022 SchedMD LLC.
This file is part of Slurm, a resource management program. For details, see
<https://slurm.schedmd.com/>.
Slurm is free software; you can redistribute it and/or modify it under the terms of the
GNU General Public License as published by the Free Software Foundation; either version 2
of the License, or (at your option) any later version.
Slurm is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without
even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
FILES
/etc/slurm.conf
SEE ALSO
scancel(1), sinfo(1), squeue(1), slurm_create_partition (3), slurm_delete_partition (3),
slurm_load_ctl_conf (3), slurm_load_jobs (3), slurm_load_node (3), slurm_load_partitions
(3), slurm_reconfigure (3), slurm_requeue (3), slurm_resume (3), slurm_shutdown (3),
slurm_suspend (3), slurm_takeover (3), slurm_update_job (3), slurm_update_node (3),
slurm_update_partition (3), slurm.conf(5), slurmctld(8)