Provided by: torque-mom_2.4.16+dfsg-1.3ubuntu1.1_amd64 bug

NAME

       pbs_mom - start a pbs batch execution mini-server

SYNOPSIS

       pbs_mom [-a alarm] [-C chkdirectory] [-c config] [-d directory] [-H hostname] [-L logfile]
       [-M MOMport] [-R RPPport] [-p|-q|-r] [-x]

DESCRIPTION

       The pbs_mom command starts the operation of a batch Machine Oriented Mini-server, MOM,  on
       the  local  host.   Typically,  this  command  will  be  in  a  local  boot  file  such as
       /etc/rc.local .  To insure that the pbs_mom command is not runnable by  the  general  user
       community, the server will only execute if its real and effective uid is zero.

       One  function  of  pbs_mom  is  to  place  jobs  into execution as directed by the server,
       establish resource usage limits, monitor the job's usage, and notify the server  when  the
       job  completes.   If they exist, pbs_mom will execute a prologue script before executing a
       job and an epilogue script after executing the job.  The next function of  pbs_mom  is  to
       respond  to  resource  monitor  requests.  This was done by a separate process in previous
       versions of PBS but has now been combined into one process.  The resource monitor function
       is  provided  mainly  for  the PBS scheduler.  It provides information about the status of
       running jobs, memory available etc.  The next function of pbs_mom is to  respond  to  task
       manager  requests.   This  involves  communicating with running tasks over a tcp socket as
       well as communicating with other MOMs within a job (aka a "sisterhood").

       Pbs_mom will record a diagnostic message in a log file for any error occurrence.  The  log
       files are maintained in the mom_logs directory below the home directory of the server.  If
       the log file cannot be opened, the diagnostic message is written to the system console.

OPTIONS

       -a alarm        Used to specify the alarm timeout in seconds  for  computing  a  resource.
                       Every  time a resource request is processed, an alarm is set for the given
                       amount of time.  If the request has not completed before the  given  time,
                       an alarm signal is generated.  The default is 5 seconds.

       -C chkdirectory Specifieds  the  path  of  the  directory  used  to hold checkpoint files.
                       [Currently this is only valid on Cray systems.]  The default directory  is
                       PBS_HOME/spool/checkpoint,  see  the  -d  option.  The directory specified
                       with the -C option must be owned by root and accessible (rwx) only by root
                       to protect the security of the checkpoint files.

       -c config       Specify  a alternative configuration file, see description below.  If this
                       is a relative file name it will be relative to PBS_HOME/mom_priv, see  the
                       -d  option.   If  the specified file cannot be opened, pbs_mom will abort.
                       If the -c option is not supplied, pbs_mom will attempt to open the default
                        configuration file "config" in PBS_HOME/mom_priv.  If this  file  is  not
                       present, pbs_mom will log the fact and continue.

       -H hostname     Set MOM's hostname.  This can be useful on multi-homed networks.

       -d directory    Specifies  the  path  of  the  directory  which is the home of the servers
                       working files, PBS_HOME.  This option is typically used along with -M when
                       debugging  MOM.   The default directory is given by $PBS_SERVER_HOME which
                       is typically /usr/spool/PBS.

       -L logfile      Specify an absolute path name for use as the log file.  If not  specified,
                       MOM  will  open a file named for the current date in the PBS_HOME/mom_logs
                       directory, see the -d option.

       -M port         Specifies the port number on which the mini-server (MOM) will  listen  for
                       batch requests.

       -R port         Specifies  the  port number on which the mini-server (MOM) will listen for
                       resource monitor requests, task manager requests and  inter-MOM  messages.
                       Both a UDP and a TCP port of this number will be used.

       -p              (Default  after  version  2.4.0)  (Preserve running jobs) -- Specifies the
                       impact on jobs which were in execution when the     mini-server shut-down.
                       The  -p  option  tries to preserve any running jobs when the MOM restarts.
                       The new mini-server will not be the parent of any running  jobs,  MOM  has
                       lost control of her     offspring (not a new situation for a mother).  The
                       MOM will allow the jobs to continue to run and monitor them indirectly via
                       polling.  All  recovered  jobs will report an exit code of 0 when they are
                       complete. The -p option is mutually exclusive  with  the  -r,  -P  and  -q
                       options.

       -P              (Terminate  all  jobs  and  remove  them  from the queue) -- Specifies the
                       impact on jobs which were in execution  when  the  mini-server  shut-down.
                       With  the  -P option, it is assumed that either the entire system has been
                       restarted or the MOM has been down so long that it can no longer guarantee
                       that  the  pid  of  any  running  process  is the same as the recorded job
                       process pid of a recovering job. Unlike the -p option no attempt  is  made
                       to  try  and preserve or recover running jobs. All jobs are terminated and
                       removed from the queue.  The -q option is mutually exclusive with the  -p,
                       -q and -r options.

       -q              (Requeue  all  jobs  -  This  is the default behavior in versions prior to
                       2.4.0) -- Specifies the impact on jobs which were in  execution  when  the
                       mini-servershut-down.  Do  not  terminate  running processes.  With the -q
                       option, it is assumed that either the entire system has been restarted  or
                       the MOM has been down so long that it can no longer guarantee that the pid
                       of any running process is the same as the recorded job process  pid  of  a
                       recovering  job.  No  attempt is made to kill job processes.  The MOM will
                       mark the jobs as terminated and notify the batch  server  which  owns  the
                       job.  Re-runnable  jobs  will  be  requeued.   The  -q  option is mutually
                       exclusive with the -p, -P and -r options.

       -r              (Terminate running processes and requeue all jobs) -- Specifies the impact
                       on  jobs  which were in execution when the mini-server shut-down. With the
                       -r option, MOM will kill any processes belonging to running jobs, mark the
                       jobs  as  terminated  and  notify  the batch server that owns the job. Re-
                       runnable jobs are reset to a queued state so they can be run  again.   The
                       -r option is mutually exclusive with the -p, -P and -q options.

                       If  the  -r  option  is used following a reboot, process IDs (pids) may be
                       reused and MOM may kill a process that is not a batch session.

       -S port         Specifies the port  number  on  which  the  pbs_server  is  listening  for
                       requests.  If pbs_server is started with a -p option, pbs_mom will need to
                       use the -S option and match  the  port  value  which  was  used  to  start
                       pbs_server.

       -x              Disables the check for privileged port resource monitor connections.  This
                       is used mainly for testing since the privileged port is the only mechanism
                       used to prevent any ordinary user from connecting.

CONFIGURATION FILE

       The  configuration  file may be specified on the command line at program start with the -c
       flag.  The use of this file is to  provide  several  types  of  run  time  information  to
       pbs_mom:  static resource names and values, external resources provided by a program to be
       run on request via a shell escape, and values to pass to  internal  set  up  functions  at
       initialization (and re-initialization).

       Each  item type is on a single line with the component parts separated by white space.  If
       the line starts with a hash mark (pound sign, #), the line is considered to be  a  comment
       and is skipped.

       Static Resources
              For  static  resource  names  and values, the configuration file contains a list of
              resource names/values pairs, one pair per line and separated by white  space.    An
              Example  of  static resource names and values could be the number of tape drives of
              different types and could be specified by

              tape3480      4
              tape3420      2
              tapedat       1
              tape8mm       1

       Shell Commands
              If the first character of the value is an exclamation mark (!), the entire rest  of
              the  line  is  saved  to be executed through the services of the system(3) standard
              library routine.

              The shell escape provides a means for  the  resource  monitor  to  yield  arbitrary
              information  to  the scheduler.  Parameter substitution is done such that the value
              of any qualifier sent with the query, as explained below, replaces a token  with  a
              percent  sign  (%)  followed  by the name of the qualifier.  For example, here is a
              configuration file line which gives a resource name of "escape":

              escape     !echo %xxx %yyy

              If a query for "escape" is sent with no qualifiers, the command executed  would  be
              "echo  %xxx  %yyy".   If one qualifier is sent, "escape[xxx=hi there]", the command
              executed  would  be  "echo  hi  there  %yyy".   If   two   qualifiers   are   sent,
              "escape[xxx=hi][yyy=there]",  the  command executed would be "echo hi there".  If a
              qualifier is sent with no matching token in the command line,  "escape[zzz=snafu]",
              an error is reported.

       size[fs=<FS>]
              Specifies that the available and configured disk space in the <FS> filesystem is to
              be reported to the pbs_server and scheduler.  NOTE: To request disk space on a  per
              job  basis,  specify  the  file  resource  as in 'qsub -l nodes=1,file=1000kb'  For
              example, the available and configured disk space in  the  /localscratch  filesystem
              will be reported:

              size[fs=/localscratch]

       Initialization Value
              An  initialization  value  directive has a name which starts with a dollar sign ($)
              and must be known to MOM via an internal table.  The entries in this table now are:

              pbsserver
                     which defines hostnames running pbs_server that will be  allowed  to  submit
                     jobs,  issue  Resource  Monitor  (RM) requests, and get status updates.  MOM
                     will continually attempt to contact all server hosts  for  node  status  and
                     state  updates.   Like  $PBS_SERVER_HOME/server_name,  the  hostname  may be
                     followed by a colon and a port number.  This  parameter  replaces  the  oft-
                     confused  $clienthost  parameter from TORQUE 2.0.0p0 and earlier.  Note that
                     the hostname  in  $PBS_SERVER_HOME/server_name  is  used  if  no  $pbsserver
                     parameters are found

              pbsclient
                     which  causes  a  host  name  to be added to the list of hosts which will be
                     allowed to connect to MOM as long as they are using a  privilaged  port  for
                     the  purposes  of  resource  monitor  requests.   For  example, here are two
                     configuration file lines which will allow the hosts "fred"  and  "wilma"  to
                     connect:

                     $pbsclient      fred
                     $pbsclient      wilma

                     Two  host  name are always allowed to connection to pbs_mom, "localhost" and
                     the name returned to pbs_mom by the system call gethostname().  These  names
                     need  not  be  specified  in  the  configuration  file.  The hosts listed as
                     "clients" can issue Resource Monitor (RM) requests.   Other  MOM  nodes  and
                     servers do not need to be listed as clients.

              restricted
                     which  causes  a  host  name  to be added to the list of hosts which will be
                     allowed to connect to MOM without needing to use a privilaged  port.   These
                     names  allow  for  wildcard  matching.  For example, here is a configuration
                     file line which will allow queries from any host from the domain "ibm.com".

                     $restricted      *.ibm.com

                     The restriction which applies to these connections  is  that  only  internal
                     queries  may  be made.  No resources from a config file will be found.  This
                     is to prevent any shell commands from being run by a non-root process.
                     This parameter is generally not required except for some versions of OSX.

              logevent
                     which sets the mask that determines which event types are logged by pbs_mom.
                     For example:

                     $logevent 0x1fff
                     $logevent 255

                     The  first example would set the log event mask to 0x1ff (511) which enables
                     logging of all events including debug events.  The second example would  set
                     the mask to 0x0ff (255) which enables all events except debug events.

              cputmult
                     which sets a factor used to adjust cpu time used by a job.  This is provided
                     to allow adjustment of time charged and limits enforced where the job  might
                     run  on  systems  with different cpu performance.  If Mom's system is faster
                     than the reference system, set cputmult to a decimal value greater than 1.0.
                     If Mom's system is slower, set cputmult to a value between 1.0 and 0.0.  For
                     example:

                     $cputmult 1.5
                     $cputmult 0.75

              usecp  specifies which directories should be staged with cp instead of rcp/scp.  If
                     a  shared  filesystem is available on all hosts in a cluster, this directive
                     is used to make these filesystems known to MOM.  For example,  if  /home  is
                     NFS mounted on all nodes in a cluster:

                     $usecp *:/home  /home

              wallmult
                     which  sets  a  factor  used to adjust wall time usage by to job to a common
                     reference system.  The factor is used for walltime calculations  and  limits
                     the same as cputmult is used for cpu time.

              configversion
                     specifies the version of the config file data, a string.

              check_poll_time
                     specifies  the  MOM  interval  in  seconds.  MOM checks each job for updated
                     resource usages, exited processes,  over-limit  conditions,  etc.  once  per
                     interval.    This   value   should   be   equal  or  lower  to  pbs_server's
                     job_stat_rate.   High  values  result  in  stale  information  reported   to
                     pbs_server.  Low values result in increased system usage by MOM.  Default is
                     45 seconds.

              down_on_error
                     causes MOM to report itself as state "down" to pbs_server in the event of  a
                     failed  health check.  This feature is EXPERIMENTAL and likely to be removed
                     in the future.  See HEALTH CHECK below.

              ideal_load
                     ideal processor load.  Represents a low water mark  for  the  load  average.
                     Nodes  that are currently busy will consider itself free after falling below
                     ideal_load.

              auto_ideal_load
                     if jobs are running, sets idea_load  based  on  a  simple  expression.   The
                     expressions  start  with  the  variable  't'  (total  assigned  CPUs) or 'c'
                     (existing CPUs), an operator (+ - / *), and followed by a float constant.

                     $auto_ideal_load t-0.2

              loglevel
                     specifies the verbosity of  logging  with  higher  numbers  specifying  more
                     verbose logging.  Values may range between 0 and 7.

              log_file_max_size
                     If   this  is set to a value > 0 then pbs_mom will roll the current log file
                     to log-file-name.1 when its size is  greater  than or  equal  to  the  value
                     of log_file_max_size. This value is interpreted as kilobytes.

              log_file_roll_depth
                     If  this is set to a value >=1 and  log_file_max_size  is  set then  pbs_mom
                     will continue rolling the log files to log-file-name.log_file_roll_depth.

              max_load
                     maximum processor load.  Nodes over this load average  are  considered  busy
                     (see ideal_load above).

              auto_max_load
                     if  jobs  are  running,  sets  max_load  based  on a simple expression.  The
                     expressions start with  the  variable  't'  (total  assigned  CPUs)  or  'c'
                     (existing CPUs), an operator (+ - / *), and followed by a float constant.

              enablemomrestart
                     enable  automatic restarts of MOM.  If enabled, MOM will check if its binary
                     has been updated and restart itself  at  a  safe  point  when  no  jobs  are
                     running;  thus  making  upgrades easier.  The check is made by comparing the
                     mtime of the pbs_mom executable.  Command-line args, the process  name,  and
                     the PATH env variable are preserved across restarts.  It is recommended that
                     this not be enabled in the config file, but enabled when desired with momctl
                     (see RESOURCES for more information.)

              node_check_script
                     specifies  the  fully  qualified  pathname of the health check script to run
                     (see HEALTH CHECK for more information).

              node_check_interval
                     specifies when to run the  MOM  health  check.   The  check  can  be  either
                     periodic,   event-driver,  or  both.   The  value  starts  with  an  integer
                     specifying the number of MOM intervals between subsequent executions of  the
                     specified  health  check.   After the integer is an optional comma-separated
                     list of event names.  Currently supported are "jobstart" and "jobend".  This
                     value  defaults  to  1  with no events indicating the check is run every MOM
                     interval. (see HEALTH CHECK for more information)

                     $node_check_interval 0Disabled.
                     $node_check_interval 0,jobstartOnly
                     $node_check_interval 10,jobstart,jobend

              prologalarm
                     Specifies maximum duration (in seconds) which the MOM will wait for the  job
                     prolog or job job epilog to complete.  This parameter default to 300 seconds
                     (5 minutes)

              rcpcmd Specify the the full path and argument to be used for  remote  file  copies.
                     This  overrides  the  compile-time  default  found  in configure.  This must
                     contain 2 words: the full path to the command and the  switches.   The  copy
                     command must be able to recursively copy files to the remote host and accept
                     arguments of the form "user@host:files"  For example:

                     $rcpcmd /usr/bin/rcp -rp
                     $rcpcmd /usr/bin/scp -rpB

              remote_checkpoint_dirs
                     Specifies what server checkpoint directories  are  remotely  mounted.   This
                     directive  is  used  to  tell  the MOM which directories are shared with the
                     server.  Using remote checkpoint directories eliminates the need to copy the
                     checkpoint  files  back  and  forth  between  the  MOM  and the server. This
                     parameter is available in 2.4.1 and later.

                     $remote_checkpoint_dirs /var/spool/torque/checkpoint

              remote_reconfig
                     Enables the ability to remotely reconfigure pbs_mom with a new config  file.
                     Default is disabled.  This parameter accepts various forms of true, yes, and
                     1.

              timeout
                     Specifies the number of seconds before TCP  messages  will  time  out.   TCP
                     messages  include  job  obituaries,  and  TM  requests  if  RPP is disabled.
                     Default is 60 seconds.

              tmpdir Sets the directory basename for a per-job temporary directory.   Before  job
                     launch,  MOM  will  append  the  jobid to the tmpdir basename and create the
                     directory.  After the job exit, MOM will recursively  delete  it.   The  env
                     variable  TMPDIR will be set for all pro/epilog scripts, the job script, and
                     TM tasks.
                     Directory creation and removal is done as the job owner and  group,  so  the
                     owner  must have write permission to create the directory.  If the directory
                     already exists and is owned by the job owner, it will not be  deleted  after
                     the job.  If the directory already exists and is NOT owned by the job owner,
                     the job start will be rejected.

              status_update_time
                     Specifies (in seconds) how often  MOM  updates  its  status  information  to
                     pbs_server.   This  value  should  correlate  with  the  server's scheduling
                     interval.  High values increase the load of pbs_server and the network.  Low
                     values cause pbs_server to report stale information.  Default is 45 seconds.

              varattr
                     This  is  similar  to a shell escape above, but includes a TTL.  The command
                     will only be run every TTL seconds.  A TTL of -1 will cause the  command  to
                     be  executed  only  once.   A  TTL  of  0  will  cause the command to be run
                     everytime varattr is requested.  This parameter may be used multiple  times,
                     but  all  output  will  be  grouped into a single "varattr" attribute in the
                     request and status output.  The command should output data in  the  form  of
                     varattrname=va1ue1[+value2]...

                     $varattr 3600 /path/to/script [<ARGS>]...

       The configuration file must be executable and "secure".  It must be owned by a user id and
       group id less than 10 and not be world writable.  Output from this file  must  be  in  the
       format $VAR=$VAL, i.e.,

              dataset13=20070104
              dataset22=20070202
              viraltest=abdd3

       xauthpath
              Specifies the path to the xauth binary to enable X11 fowarding.

       ignvmem
              If set to true, then pbs_mom will ignore vmem/pvmem limit enforcement.

       ignwalltime
              If set to true, then pbs_mom will ignore walltime limit enforcement.

       mom_host
              Sets the local hostname as used by pbs_mom.

RESOURCES

       Resource  Monitor  queries can be made with momctl's -q option to retrieve and set pbs_mom
       options.  Any configured static resource may be retrieved with a request of the same name.
       These are resource requests not otherwise documented in the PBS ERS.

       cycle  forces an immediate MOM cycle

       status_update_time
              retrieve or set the $status_update_time parameter

       check_poll_time
              retrieve or set the $check_poll_time parameter

       configversion
              retrieve the config version

       jobstartblocktime
              retrieve or set the $jobstartblocktime parameter

       enablemomrestart
              retrieve or set the $enablemomrestart parameter

       loglevel
              retrieve or set the $loglevel parameter

       down_on_error
              retrieve or set the EXPERIMENTAL $down_on_error parameter

       diag0 - diag4
              retrieves various diagnostic information

       rcpcmd retrieve or set the $rcpcmd parameter

       version
              retrieves the pbs_mom version

HEALTH CHECK

       The health check script is executed directly by the pbs_mom daemon under the root user id.
       It must be accessible from the compute node and may be a  script  or  compiled  executable
       program.   It  may  make  any  needed  system  calls and execute any combination of system
       utilities but should not execute resource manager client commands.   Also,  as  of  TORQUE
       1.0.1,  the pbs_mom daemon blocks until the health check is completed and does not possess
       a built-in timeout.  Consequently, it is advisable to keep  the  launch  script  execution
       time short and verify that the script will not block even under failure conditions.

       If  the  script detects a failure, it should return the keyword 'ERROR' to stdout followed
       by an error message.  The message (up to 256 characters) immediately following  the  ERROR
       string will be assigned to the node attribute 'message' of the associated node.

       If  the  script detects a failure when run from "jobstart", then the job will be rejected.
       This should probably only be used with advanced schedulers like Moab so that the  job  can
       be routed to another node.

       TORQUE  currently ignores ERROR messages by default, but advanced schedulers like moab can
       be configured to react appropriately.

       If the experimental $down_on_error MOM setting is enabled, MOM will set  itself  to  state
       down   and  report  to  pbs_server;  and  pbs_server  will  report  the  node  as  "down".
       Additionally, the experimental "down_on_error" server attribute can be enabled  which  has
       the  same  effect  but  moves  the  decision to pbs_server.  It is redundant to have MOM's
       $down_on_error and pbs_server's down_on_error features enabled.   See  "down_on_error"  in
       pbs_server_attributes(7B).

FILES

       $PBS_SERVER_HOME/server_name
              contains the hostname running pbs_server.

       $PBS_SERVER_HOME/mom_priv
                 the     default     directory     for     configuration     files,     typically
                 (/usr/spool/pbs)/mom_priv.

       $PBS_SERVER_HOME/mom_logs
                 directory for log files recorded by the server.

       $PBS_SERVER_HOME/mom_priv/prologue
                 the administrative script to be run before job execution.

       $PBS_SERVER_HOME/mom_priv/epilogue
                 the administrative script to be run after job execution.

SIGNAL HANDLING

       pbs_mom handles the following signals:

       SIGHUP causes pbs_mom to re-read its configuration file, close and reopen  the  log  file,
              and reinitialize resource structures.

       SIGALRM
              results  in a log file entry. The signal is used to limit the time taken by certain
              children processes, such as the prologue and epilogue.

       SIGINT and SIGTERM
              results in pbs_mom exiting without terminating  any  running  jobs.   This  is  the
              action  for  the  following  signals  as  well:  SIGXCPU,  SIGXFSZ,  SIGCPULIM, and
              SIGSHUTDN.

       SIGUSR1, SIGUSR2
              causes MOM to increase and decrease logging levels, respectively.

       SIGPIPE, SIGINFO
               are ignored.

       SIGBUS, SIGFPE, SIGILL, SIGTRAP, and SIGSYS
              cause a core dump if the PBSCOREDUMP environmental variable is defined.

       All other signals have their default behavior installed.

EXIT STATUS

       If the mini-server command fails to begin operation, the server exits with a value greater
       than zero.

SEE ALSO

       pbs_server(8B),  pbs_scheduler_basl(8B), pbs_scheduler_tcl(8B), the PBS External Reference
       Specification, and the PBS Administrator's Guide.