Provided by: slurm-llnl_2.6.5-1_amd64 
NAME
strigger - Used set, get or clear Slurm trigger information.
SYNOPSIS
strigger --set [OPTIONS...]
strigger --get [OPTIONS...]
strigger --clear [OPTIONS...]
DESCRIPTION
strigger is used to set, get or clear Slurm trigger information. Triggers include events
such as a node failing, a job reaching its time limit or a job terminating. These events
can cause actions such as the execution of an arbitrary script. Typical uses include
notifying system administrators of node failures and gracefully terminating a job when
it's time limit is approaching. A hostlist expression for the nodelist or job ID is
passed as an argument to the program.
Trigger events are not processed instantly, but a check is performed for trigger events on
a periodic basis (currently every 15 seconds). Any trigger events which occur within that
interval will be compared against the trigger programs set at the end of the time
interval. The trigger program will be executed once for any event occurring in that
interval. The record of those events (e.g. nodes which went DOWN in the previous 15
seconds) will then be cleared. The trigger program must set a new trigger before the end
of the next interval to insure that no trigger events are missed OR the trigger must be
created with an argument of "--flags=PERM". If desired, multiple trigger programs can be
set for the same event.
IMPORTANT NOTE: This command can only set triggers if run by the user SlurmUser unless
SlurmUser is configured as user root. This is required for the slurmctld daemon to set
the appropriate user and group IDs for the executed program. Also note that the trigger
program is executed on the same node that the slurmctld daemon uses rather than some
allocated compute node. To check the value of SlurmUser, run the command:
scontrol show config | grep SlurmUser
ARGUMENTS
-a, --primary_slurmctld_failure
Trigger an event when the primary slurmctld fails.
-A, --primary_slurmctld_resumed_operation
Trigger an event when the primary slurmctld resuming operation after failure.
-b, --primary_slurmctld_resumed_control
Trigger an event when primary slurmctld resumes control.
--block_err
Trigger an event when a BlueGene block enters an ERROR state.
-B, --backup_slurmctld_failure
Trigger an event when the backup slurmctld fails.
-c, --backup_slurmctld_resumed_operation
Trigger an event when the backup slurmctld resumes operation after failure.
-C, --backup_slurmctld_assumed_control
Trigger event when backup slurmctld assumes control.
--clear
Clear or delete a previously defined event trigger. The --id, --jobid or --user
option must be specified to identify the trigger(s) to be cleared. Only user root
or the trigger's creator can delete a trigger.
-d, --down
Trigger an event if the specified node goes into a DOWN state.
-D, --drained
Trigger an event if the specified node goes into a DRAINED state.
-e, --primary_slurmctld_acct_buffer_full
Trigger an event when primary slurmctld accounting buffer is full.
-F, --fail
Trigger an event if the specified node goes into a FAILING state.
-f, --fini
Trigger an event when the specified job completes execution.
--flags=type
Associate flags with the reservation. Multiple flags should be comma separated.
Valid flags include:
PERM Make the trigger permanent. Do not purge it after the event occurs.
--front_end
Trigger events based upon changes in state of front end nodes rather than compute
nodes. Applies to BlueGene and Cray architectures only, where the slurmd daemon
executes on front end nodes rather than the compute nodes. Use this option with
either the --up or --down option.
-g, --primary_slurmdbd_failure
Trigger an event when the primary slurmdbd fails.
-G, --primary_slurmdbd_resumed_operation
Trigger an event when the primary slurmdbd resumes operation after failure.
--get Show registered event triggers. Options can be used for filtering purposes.
-h, --primary_database_failure
Trigger an event when the primary database fails.
-H, --primary_database_resumed_operation
Trigger an event when the primary database resumes operation after failure.
-i, --id=id
Trigger ID number.
-I, --idle
Trigger an event if the specified node remains in an IDLE state for at least the
time period specified by the --offset option. This can be useful to hibernate a
node that remains idle, thus reducing power consumption.
-j, --jobid=id
Job ID of interest. NOTE: The --jobid option can not be used in conjunction with
the --node option. When the --jobid option is used in conjunction with the --up or
--down option, all nodes allocated to that job will considered the nodes used as a
trigger event.
-n, --node[=host]
Host name(s) of interest. By default, all nodes associated with the job (if
--jobid is specified) or on the system are considered for event triggers. NOTE:
The --node option can not be used in conjunction with the --jobid option. When the
--jobid option is used in conjunction with the --up, --down or --drained option,
all nodes allocated to that job will considered the nodes used as a trigger
event.Since this option's argument is optional, for proper parsing the single
letter option must be followed immediately with the value and not include a space
between them. For example "-ntux" and not "-n tux".
-M, --clusters=<string>
Clusters to issue commands to.
-o, --offset=seconds
The specified action should follow the event by this time interval. Specify a
negative value if action should preceded the event. The default value is zero if
no --offset option is specified. The resolution of this time is about 20 seconds,
so to execute a script not less than five minutes prior to a job reaching its time
limit, specify --offset=320 (5 minutes plus 20 seconds).
-p, --program=path
Execute the program at the specified fully qualified pathname when the event
occurs. The program will be executed as the user who sets the trigger. If the
program fails to terminate within 5 minutes, it will be killed along with any
spawned processes.
-Q, --quiet
Do not report non-fatal errors. This can be useful to clear triggers which may
have already been purged.
-r, --reconfig
Trigger an event when the system configuration changes.
--set Register an event trigger based upon the supplied options. NOTE: An event is only
triggered once. A new event trigger must be set established for future events of
the same type to be processed. Triggers can only be set if the command is run by
the user SlurmUser unless SlurmUser is configured as user root.
-t, --time
Trigger an event when the specified job's time limit is reached. This must be used
in conjunction with the --jobid option.
-u, --up
Trigger an event if the specified node is returned to service from a DOWN state.
--user=user_name_or_id
Clear or get triggers created by the specified user. For example, a trigger
created by user root for a job created by user adam could be cleared with an option
--user=root. Specify either a user name or user ID.
-v, --verbose
Print detailed event logging. This includes time-stamps on data structures, record
counts, etc.
-V , --version
Print version information and exit.
OUTPUT FIELD DESCRIPTIONS
TRIG_ID
Trigger ID number.
RES_TYPE
Resource type: job or node
RES_ID Resource ID: job ID or host names or "*" for any host
TYPE Trigger type: time or fini (for jobs only), down or up (for jobs or nodes), or
drained, idle or reconfig (for nodes only)
OFFSET Time offset in seconds. Negative numbers indicated the action should occur before
the event (if possible)
USER Name of the user requesting the action
PROGRAM
Pathname of the program to execute when the event occurs
EXAMPLES
Execute the program "/usr/sbin/primary_slurmctld_failure" whenever the primary slurmctld
fails.
> cat /usr/sbin/primary_slurmctld_failure
#!/bin/bash
# Submit trigger for next primary slurmctld failure event
strigger --set --primary_slurmctld_failure \
--program=/usr/sbin/primary_slurmctld_failure
# Notify the administrator of the failure using by e-mail
/usr/bin/mail slurm_admin@site.com -s Primary_SLURMCTLD_FAILURE
> strigger --set --primary_slurmctld_failure \
--program=/usr/sbin/primary_slurmctld_failure
Execute the program "/usr/sbin/slurm_admin_notify" whenever any node in the cluster goes
down. The subject line will include the node names which have entered the down state
(passed as an argument to the script by SLURM).
> cat /usr/sbin/slurm_admin_notify
#!/bin/bash
# Submit trigger for next event
strigger --set --node --down \
--program=/usr/sbin/slurm_admin_notify
# Notify administrator using by e-mail
/usr/bin/mail slurm_admin@site.com -s NodesDown:$*
> strigger --set --node --down \
--program=/usr/sbin/slurm_admin_notify
Execute the program "/usr/sbin/slurm_suspend_node" whenever any node in the cluster
remains in the idle state for at least 600 seconds.
> strigger --set --node --idle --offset=600 \
--program=/usr/sbin/slurm_suspend_node
Execute the program "/home/joe/clean_up" when job 1234 is within 10 minutes of reaching
its time limit.
> strigger --set --jobid=1234 --time --offset=-600 \
--program=/home/joe/clean_up
Execute the program "/home/joe/node_died" when any node allocated to job 1234 enters the
DOWN state.
> strigger --set --jobid=1234 --down \
--program=/home/joe/node_died
Show all triggers associated with job 1235.
> strigger --get --jobid=1235
TRIG_ID RES_TYPE RES_ID TYPE OFFSET USER PROGRAM
123 job 1235 time -600 joe /home/bob/clean_up
125 job 1235 down 0 joe /home/bob/node_died
Delete event trigger 125.
> strigger --clear --id=125
Execute /home/joe/job_fini upon completion of job 1237.
> strigger --set --jobid=1237 --fini --program=/home/joe/job_fini
COPYING
Copyright (C) 2007 The Regents of the University of California. Produced at Lawrence
Livermore National Laboratory (cf, DISCLAIMER).
Copyright (C) 2008-2010 Lawrence Livermore National Security.
Copyright (C) 2010-2013 SchedMD LLC.
This file is part of SLURM, a resource management program. For details, see
<http://slurm.schedmd.com/>.
SLURM is free software; you can redistribute it and/or modify it under the terms of the
GNU General Public License as published by the Free Software Foundation; either version 2
of the License, or (at your option) any later version.
SLURM is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without
even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
SEE ALSO
scontrol(1), sinfo(1), squeue(1)