Ubuntu Manpage: pdsh - issue commands to groups of hosts in parallel

NAME

       pdsh - issue commands to groups of hosts in parallel

SYNOPSIS

       pdsh [options]... command

DESCRIPTION

pdsh is a variant of the rsh(1) command. Unlike rsh(1), which runs commands on a single remote host, pdsh
can run multiple remote commands in parallel. pdsh uses a "sliding window" (or fanout) of threads to
conserve resources on the initiating host while allowing some connections to time out.

When pdsh receives SIGINT (ctrl-C), it lists the status of current threads. A second SIGINT within one
second terminates the program. Pending threads may be canceled by issuing ctrl-Z within one second of
ctrl-C. Pending threads are those that have not yet been initiated, or are still in the process of
connecting to the remote host.

If a remote command is not specified on the command line, pdsh runs interactively, prompting for commands
and executing them when terminated with a carriage return. In interactive mode, target nodes that time
out on the first command are not contacted for subsequent commands, and commands prefixed with an
exclamation point will be executed on the local system.

The core functionality of pdsh may be supplemented by dynamically loadable modules. The modules may
provide a new connection protocol (replacing the standard rcmd(3) protocol used by rsh(1)), filtering
options (e.g. removing hosts that are "down" from the target list), and/or host selection options (e.g.,
-a selects all hosts from a configuration file.). By default, pdsh must have at least one "rcmd" module
loaded. See the RCMD MODULES section for more information.

RCMD MODULES

The method by which pdsh runs commands on remote hosts may be selected at runtime using the -R option
(See OPTIONS below). This functionality is ultimately implemented via dynamically loadable modules, and
so the list of available options may be different from installation to installation. A list of currently
available rcmd modules is printed when using any of the -h, -V, or -L options. The default rcmd module
will also be displayed with the -h and -V options.

A list of rcmd modules currently distributed with pdsh follows.

rsh Uses an internal, thread-safe implementation of BSD rcmd(3) to run commands using the standard
rsh(1) protocol.

exec Executes an arbitrary command for each target host. The first of the pdsh remote arguments is the
local command to execute, followed by any further arguments. Some simple parameters are
substitued on the command line, including %h for the target hostname, %u for the remote username,
and %n for the remote rank [0-n] (To get a literal % use %%). For example, the following would
duplicate using the ssh module to run hostname(1) across the hosts foo[0-10]:

pdsh -R exec -w foo[0-10] ssh -x -l %u %h hostname

and this command line would run grep(1) in parallel across the files console.foo[0-10]:

pdsh -R exec -w foo[0-10] grep BUG console.%h

ssh Uses a variant of popen(3) to run multiple copies of the ssh(1) command.

mrsh This module uses the mrsh(1) protocol to execute jobs on remote hosts. The mrsh protocol uses a
credential based authentication, forgoing the need to allocate reserved ports. In other aspects,
it acts just like rsh. Remote nodes must be running mrshd(8) in order for the mrsh module to
work.

qsh Allows pdsh to execute MPI jobs over QsNet. Qshell propagates the current working directory, pdsh
environment, and Elan capabilities to the remote process. The following environment variable are
also appended to the environment: RMS_RANK, RMS_NODEID, RMS_PROCID, RMS_NNODES, and RMS_NPROCS.
Since pdsh needs to run setuid root for qshell support, qshell does not directly support
propagation of LD_LIBRARY_PATH and LD_PREOPEN. Instead the QSHELL_REMOTE_LD_LIBRARY_PATH and
QSHELL_REMOTE_LD_PREOPEN environment variables will may be used and will be remapped to
LD_LIBRARY_PATH and LD_PREOPEN by the qshell daemon if set.

mqsh Similar to qshell, but uses the mrsh protocol instead of the rsh protocol.

krb4 The krb4 module allows users to execute remote commands after authenticating with kerberos. Of
course, the remote rshd daemons must be kerberized.

xcpu The xcpu module uses the xcpu service to execute remote commands.

OPTIONS

       The list of available options is determined at runtime by supplementing the list of standard pdsh options
       with  any  options  provided by loaded rcmd and misc modules.  In some cases, options provided by modules
       may conflict with each other. In these cases, the modules are incompatible and the  first  module  loaded
       wins.

Standard target nodelist options

-w TARGETS,...
Target and or filter the specified list of hosts. Do not use with any other node selection options
(e.g. -a, -g, if they are available). No spaces are allowed in the comma-separated list.
Arguments in the TARGETS list may include normal host names, a range of hosts in hostlist format
(See HOSTLIST EXPRESSIONS), or a single `-' character to read the list of hosts on stdin.

If a host or hostlist is preceded by a `-' character, this causes those hosts to be explicitly
excluded. If the argument is preceded by a single `^' character, it is taken to be the path to
file containing a list of hosts, one per line. If the item begins with a `/' character, it is
taken as a regular expression on which to filter the list of hosts (a regex argument may also be
optionally trailed by another '/', e.g. /node.*/). A regex or file name argument may also be
preceeded by a minus `-' to exclude instead of include thoses hosts.

A list of hosts may also be preceded by "user@" to specify a remote username other than the
default, or "rcmd_type:" to specify an alternate rcmd connection type for these hosts. When used
together, the rcmd type must be specified first, e.g. "ssh:user1@host0" would use ssh to connect
to host0 as user "user1."

-x host,host,...
Exclude the specified hosts. May be specified in conjunction with other target node list options
such as -a and -g (when available). Hostlists may also be specified to the -x option (see the
HOSTLIST EXPRESSIONS section below). Arguments to -x may also be preceeded by the filename (`^')
and regex ('/') characters as described above, in which case the resulting hosts are excluded as
if they had been given to -w and preceeded with the minus `-' character.

Standard pdsh options

-S Return the largest of the remote command return values.

-h Output usage menu and quit. A list of available rcmd modules will also be printed at the end of
the usage message.

-s Only on AIX, separate remote command stderr and stdout into two sockets.

-q List option values and the target nodelist and exit without action.

-b Disable ctrl-C status feature so that a single ctrl-C kills parallel job. (Batch Mode)

-l user
This option may be used to run remote commands as another user, subject to authorization. For BSD
rcmd, this means the invoking user and system must be listed in the user´s .rhosts file (even for
root).

-t seconds
Set the connect timeout. Default is 10 seconds.

-u seconds
Set a limit on the amount of time a remote command is allowed to execute. Default is no limit.
See note in LIMITATIONS if using -u with ssh.

-f number
Set the maximum number of simultaneous remote commands to number. The default is 32.

-R name
Set rcmd module to name. This option may also be set via the PDSH_RCMD_TYPE environment variable.
A list of available rcmd modules may be obtained via the -h, -V, or -L options. The default will
be listed with -h or -V.

-M name,...
When multiple misc modules provide the same options to pdsh, the first module initialized "wins"
and subsequent modules are not loaded. The -M option allows a list of modules to be specified
that will be force-initialized before all others, in-effect ensuring that they load without
conflict (unless they conflict with eachother). This option may also be set via the
PDSH_MISC_MODULES environment variable.

-L List info on all loaded pdsh modules and quit.

-N Disable hostname: prefix on lines of output.

-d Include more complete thread status when SIGINT is received, and display connect and command time
statistics on stderr when done.

-V Output pdsh version information, along with list of currently loaded modules, and exit.

qsh/mqsh module options

       -n tasks_per_node
              Set the number of tasks spawned per node. Default is 1.

       -m block | cyclic
              Set block versus cyclic allocation of processes to nodes. Default is block.

       -r railmask
              Set the rail bitmask for a job on a multirail system. The default railmask is 1, which corresponds
              to rail 0 only. Each bit set in the argument to -r corresponds to a rail on the system, so a value
              of 2 would correspond to rail 1 only, and 3 would indicate to use both rail 1 and rail 0.

machines module options

       -a     Target all nodes from machines file.

genders module options

In addition to the genders options presented below, the genders attribute pdsh_rcmd_type may also be used
in the genders database to specify an alternate rcmd connect type than the pdsh default for hosts with
this attribute. For example, the following line in the genders file

host0 pdsh_rcmd_type=ssh

would cause pdsh to use ssh to connect to host0, even if rsh were the default. This can be overridden on
the commandline with the "rcmd_type:host0" syntax.

-A Target all nodes in genders database. The -A option will target every host listed in genders -- if
you want to omit some hosts by default, see the -a option below.

-a Target all nodes in genders database except those with the "pdsh_all_skip" attribute. This is
shorthand for running "pdsh -A -X pdsh_all_skip ..."

-g attr[=val][,attr[=val],...]
Target nodes that match any of the specified genders attributes (with optional values). Conflicts
with the -a option. If used in combination with other node selection options like -w, the -g
option will select from the supplied node list, instead of from the genders file as a whole.
Otherwise, This option targets the alternate hostnames in the genders database by default. The -i
option provided by the genders module may be used to translate these to the canonical genders
hostnames. If the installed version of genders supports it, attributes supplied to -g may also
take the form of genders queries. Genders queries will query the genders database for the union,
intersection, difference, or complement of genders attributes and values. The set operation union
is represented by two pipe symbols ('||'), intersection by two ampersand symbols ('&&'),
difference by two minus symbols ('--'), and complement by a tilde ('~'). Parentheses may be used
to change the order of operations. See the nodeattr(1) manpage for examples of genders queries.

-X attr[=val][,attr[=val],...]
Exclude nodes that match any of the specified genders attributes (optionally with values). This
option may be used in combination with any other of the node selection options (e.g. -w, -g, -a,
-X may also take the form of genders queries. Please see documentation for the genders -g option
for more information about genders queries.

-i Request translation between canonical and alternate hostnames.

-F filename
Read genders information from filename instead of the system default genders file. If filename
doesn't specify an absolute path then it is taken to be relative to the directory specified by the
PDSH_GENDERS_DIR environment variable (/etc by default). An alternate genders file may also be
specified via the PDSH_GENDERS_FILE environment variable.

nodeupdown module options

       -v     Eliminate target nodes that are considered "down" by libnodeupdown.

slurm module options

       The slurm module allows pdsh to target nodes based on currently running SLURM jobs. The slurm  module  is
       typically  called  after  all other node selection options have been processed, and if no nodes have been
       selected, the module will attempt to read a running  jobid  from  the  SLURM_JOBID  environment  variable
       (which  is  set when running under a SLURM allocation). If SLURM_JOBID references an invalid job, it will
       be silently ignored.

       -j jobid[,jobid,...]
              Target list of nodes allocated to the SLURM job jobid. This option may be used multiple  times  to
              target  multiple  SLURM  jobs.  The special argument "all" can be used to target all nodes running
              SLURM jobs, e.g.  -j all.

       -P partition[,partition,...]
              Target list of nodes containing in the  SLURM  partition  partition.   This  option  may  be  used
              multiple  times  to  target  multiple  SLURM partitions and/or partitions may be given in a comma-
              delimited list.

torque module options

       The torque module allows pdsh to target nodes based on currently running Torque/PBS jobs. Similar to  the
       slurm  module,  the  torque  module  is typically called after all other node selection options have been
       processed, and if no nodes have been selected, the module will attempt to read a running jobid  from  the
       PBS_JOBID environment variable (which is set when running under a Torque allocation).

       -j jobid[,jobid,...]
              Target  list of nodes allocated to the Torque job jobid. This option may be used multiple times to
              target multiple Torque jobs.

rms module options

       The rms module allows pdsh to target nodes based on an RMS resource. The rms module is  typically  called
       after  all  other node selection options, and if no nodes have been selected, the module will examine the
       RMS_RESOURCEID environment variable and attempt to set the target list of hosts to the nodes in  the  RMS
       resource. If an invalid resource is denoted, the variable is silently ignored.

SDR module options

       The SDR module supports targeting hosts via the System Data Repository on IBM SPs.

       -a     Target  all  nodes  in  the  SDR. The list is generated from the "reliable hostname" in the SDR by
              default.

       -i     Translate hostnames between reliable and initial in the SDR, when applicable.   If  the  a  target
              hostname  matches  either  the initial or reliable hostname in the SDR, the alternate name will be
              substitued. Thus a list composed of initial hostnames will instead be  replaced  with  a  list  of
              reliable  hostnames.   For  example, when used with -a above, all initial hostnames in the SDR are
              targeted.

       -v     Do not target nodes that are marked as not responding in the SDR on the targeted interface. (If  a
              hostname does not appear in the SDR, then that name will remain in the target hostlist.)

       -G     In combination with -a, include all partitions.

nodeattr module options

       The  nodeattr module supports access to the genders database via the nodeattr(1) command. See the genders
       section above for a list of support options with this module. The option usage with the  nodeattr  module
       is  the  same  as  genders,  above, with the exception that the -i option may only be used with -a or -g.
       NOTE: This module will only work with very old releases of genders where the nodeattr(1) command supports
       the  -r option, and before the libgenders API was available. Users running newer versions of genders will
       need to use the genders module instead.

dshgroup module options

       The dshgroup module allows pdsh to use dsh (or Dancer's shell) style group files from /etc/dsh/group/  or
       ~/.dsh/group/.  The  default search path may be overridden with the DSHGROUP_PATH environment variable, a
       colon-separated list of directories to search. The default value for DSHGROUP_PATH is /etc/dsh/group.

       -g groupname,...
              Target  nodes  in  dsh  group  file  "groupname"  found  in   either   ~/.dsh/group/groupname   or
              /etc/dsh/group/groupname.

       -X groupname,...
              Exclude nodes in dsh group file "groupname."

       As  an  enhancement  in  pdsh,  dshgroup  files may optionally include other dshgroup files via a special
       #include STRING syntax.  The argument to #include may be either a file path, or a group  name,  in  which
       case the path used to search for the group file is the same as if the group had been specified to -g.

netgroup module options

       The  netgroup  module  allows  pdsh  to  use  standard  netgroup  entries to build lists of target hosts.
       (/etc/netgroup or NIS)

       -g groupname,...
              Target nodes in netgroup "groupname."

       -X groupname,...
              Exclude nodes in netgroup "groupname."

ENVIRONMENT VARIABLES

PDSH_RCMD_TYPE
Equivalent to the -R option, the value of this environment variable will be used to set the
default rcmd module for pdsh to use (e.g. ssh, rsh).

PDSH_SSH_ARGS
Override the standard arguments that pdsh passes to the ssh(1) command ("-2 -a -x -l%u %h"). The
use of the parameters %u, %h, and %n (as documented in the rcmd/exec section above) is optional.
If these parameters are missing, pdsh will append them to the ssh commandline because it is
assumed they are mandatory.

PDSH_SSH_ARGS_APPEND
Append additional options to the ssh(1) command invoked by pdsh. For example,
PDSH_SSH_ARGS_APPEND="-q" would run ssh in quiet mode, or "-v" would increase the verbosity of
ssh. (Note: these arguments are actually prepended to the ssh commandline to ensure they appear
before any target hostname argument to ssh.)

WCOLL If no other node selection option is used, the WCOLL environment variable may be set to a filename
from which a list of target hosts will be read. The file should contain a list of hosts, one per
line (though each line may contain a hostlist expression. See HOSTLIST EXPRESSIONS section
below).

DSHPATH
If set, the path in DSHPATH will be used as the PATH for the remote processes.

FANOUT Set the pdsh fanout (See description of -f above).

HOSTLIST EXPRESSIONS

       As noted in sections above pdsh accepts lists of hosts the general form: prefix[n-m,l-k,...], where n < m
       and  l  <  k,  etc.,  as an alternative to explicit lists of hosts. This form should not be confused with
       regular expression character classes (also denoted by ``[]''). For example, foo[19] does not represent an
       expression matching foo1 or foo9, but rather represents the degenerate hostlist: foo19.

       The  hostlist  syntax is meant only as a convenience on clusters with a "prefixNNN" naming convention and
       specification of ranges should not be considered necessary -- this foo1,foo9 could be specified as  such,
       or by the hostlist foo[1,9].

       Some examples of usage follow:

       Run command on foo01,foo02,...,foo05
           pdsh -w foo[01-05] command

       Run command on foo7,foo9,foo10
            pdsh -w foo[7,9-10] command

       Run command on foo0,foo4,foo5
            pdsh -w foo[0-5] -x foo[1-3] command

       A suffix on the hostname is also supported:

       Run command on foo0-eth0,foo1-eth0,foo2-eth0,foo3-eth0
          pdsh -w foo[0-3]-eth0 command

       As  a  reminder  to  the  reader, some shells will interpret brackets ('[' and ']') for pattern matching.
       Depending on your shell, it may be necessary to enclose ranged lists  within  quotes.   For  example,  in
       tcsh, the first example above should be executed as:

            pdsh -w "foo[01-05]" command

ORIGIN

       Originally  a  rewrite of IBM dsh(1) by Jim Garlick <garlick@llnl.gov> on LLNL's ASCI Blue-Pacific IBM SP
       system. It is now used on Linux clusters at LLNL.

LIMITATIONS

       When using ssh for remote execution, expect the stderr of ssh to be folded in with  that  of  the  remote
       command.  When  invoked  by  pdsh, it is not possible for ssh to prompt for passwords if RSA/DSA keys are
       configured properly, etc..  For ssh implementations that suppport a connect timeout option, pdsh attempts
       to  use  that  option  to  enforce  the  timeout (e.g. -oConnectTimeout=T for OpenSSH), otherwise connect
       timeouts are not supported when using ssh.  Finally, there is no reliable way for  pdsh  to  ensure  that
       remote  commands  are  actually  terminated  when  using  a  command timeout. Thus if -u is used with ssh
       commands may be left running on remote hosts even after timeout has killed local ssh processes.

       Output from multiple processes per node may be interspersed when using qshell or mqshell rcmd modules.

       The number of nodes that pdsh can simultaneously execute remote jobs on is limited by the maximum  number
       of threads that can be created concurrently, as well as the availability of reserved ports in the rsh and
       qshell rcmd modules. On systems that implement Posix threads, the  limit  is  typically  defined  by  the
       constant PTHREADS_THREADS_MAX.