NAME

       sdiag - Diagnostic tool for SLURM

SYNOPSIS

       sview

DESCRIPTION

       sdiag  shows  information  related  to  slurmctld  execution about: threads, agents, jobs, and scheduling
       algorithms. The goal is  to  obtain  data  from  slurmctld  behaviour  helping  to  adjust  configuration
       parameters  or  queues  policies.  The main reason behind is to know SLURM behaviour under systems with a
       high throughput.

       It has two execution modes. The default mode --all shows several counters and statistics explained later,
       and there is another execution option --reset for resetting those values.

       Values are reset at midnight UTC time by default.

       The first block of information is related to global slurmctld execution:

       Server thread count
              The number of current active slurmctld threads. A high number would mean a  high  load  processing
              events  like  job  submissions,  jobs dispatching, jobs completing, etc. If this is often close to
              MAX_SERVER_THREADS it could point to a potential bottleneck.

       Agent queue size
              SLURM design has scalability in mind and sending messages to thousands of nodes is not  a  trivial
              task.  The  agent  mechanism  helps  to  control  communication  between the slurm daemons and the
              controller for a best effort. If this values is close to MAX_AGENT_CNT there could be some  delays
              affecting jobs management.

       Jobs submitted
              Number of jobs submitted since last reset

       Jobs started
              Number of jobs started since last reset. This includes backfilled jobs.

       Jobs completed
              Number of jobs completed since last reset.

       Jobs canceled
              Number of jobs canceled since last reset.

       Jobs failed
              Number of jobs failed since last reset.

       The  second  block  of  information  is  related to main scheduling algorithm based on jobs priorities. A
       scheduling cycle implies to get the job_write_lock lock, then trying to get resources for  jobs  pending,
       starting  from  the most priority one and going in descendent order. Once a job can not get the resources
       the loop keeps going but just for jobs requesting other partitions. Jobs with  dependencies  or  affected
       by accounts limits are not processed.

       Last cycle
              Time in microseconds for last scheduling cycle.

       Max cycle
              Time in microseconds for the maximum scheduling cycle since last reset.

       Total cycles
              Number of scheduling cycles since last reset. Scheduling is done in periodically and when a job is
              submitted or a job is completed.

       Mean cycle
              Mean of scheduling cycles since last reset

       Mean depth cycle
              Mean of cycle depth. Depth means number of jobs processed in a scheduling cycle.

       Cycles per minute
              Counter of scheduling executions per minute

       Last queue length
              Length of jobs pending queue.

       The  third block of information is related to backfilling scheduling algorithm.  A backfilling scheduling
       cycle implies to get locks for jobs, nodes and partitions objects then trying to get resources  for  jobs
       pending.  Jobs are processed based on priorities. If a job can not get resources the algorithm calculates
       when it could get them obtaining a future start time for the job.  Then next job  is  processed  and  the
       algorithm  tries  to  get  resources  for that job but avoiding to affect the previous ones, and again it
       calculates the future start time if not current resources available. The backfilling algorithm takes more
       time for each new job to process since more priority jobs can not be affected. The algorithm itself takes
       measures for avoiding a long execution cycle and for taking all the locks for too long.

       Total backfilled jobs (since last slurm start)
              Number of jobs started thanks to backfilling since last slurm start.

       Total backfilled jobs (since last stats cycle start)
              Number of jobs started thanks to backfilling since last time stats where reset.  By default  these
              values are reset at midnight UTC time.

       Total cycles
              Number of scheduling cycles since last reset

       Last cycle when
              Time  when  last  execution  cycle  happened in format "weekday Month MonthDay hour:minute.seconds
              year"

       Last cycle
              Time in microseconds of last backfilling cycle.  It counts only execution time removing sleep time
              inside a scheduling cycle when it takes too much time.  Note that locks are  released  during  the
              sleep time so that other work can proceed.

       Max cycle
              Time  in  microseconds  of  maximum  backfilling cycle execution since last reset.  It counts only
              execution time removing sleep time inside a scheduling cycle when it takes too  much  time.   Note
              that locks are released during the sleep time so that other work can proceed.

       Mean cycle
              Mean of backfilling scheduling cycles in microseconds since last reset

       Last depth cycle
              Number of processed jobs during last backfilling scheduling cycle. It counts every process even if
              it has no option to execute due to dependencies or limits.

       Last depth cycle (try sched)
              Number of processed jobs during last backfilling scheduling cycle. It counts only processes with a
              chance  to  run  waiting  for  available  resources.  These  jobs  are which makes the backfilling
              algorithm heavier.

       Depth Mean
              Mean of processed jobs during backfilling scheduling cycles since last reset.

       Depth Mean (try sched)
              Mean of processed jobs during backfilling scheduling cycles since  last  reset.   It  counts  only
              processes  with  a  chance to run waiting for available resources.  These jobs are which makes the
              backfilling algorithm heavier.

       Last queue length
              Number of jobs pending to be processed by backfilling algorithm. A job appears as  much  times  as
              partitions it requested.

       Queue length Mean
              Mean of jobs pending to be processed by backfilling algorithm.

OPTIONS

       -a, --all
              Get and report information. This is the default mode of operation.

       -h, --help
              Print description of options and exit.

       -r, --reset
              Reset counters. Only used by user SlurmUser or root.

       --usage
              Print list of options and exit.

       -V, --version
              Print current version number and exit.

COPYING

       Copyright (C) 2010-2011 Barcelona Supercomputing Center.
       Copyright (C) 2010-2013 SchedMD LLC.

       SLURM  is  free  software;  you  can  redistribute it and/or modify it under the terms of the GNU General
       Public License as published by the Free Software Foundation; either version 2 of the License, or (at your
       option) any later version.

       SLURM is distributed in the hope that it will be useful, but  WITHOUT  ANY  WARRANTY;  without  even  the
       implied  warranty  of  MERCHANTABILITY  or  FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public
       License for more details.

SEE ALSO

       sinfo(1), squeue(1), scontrol(1), slurm.conf(5),

December 2011                                       SLURM 2.4                                           sdiag(1)