Provided by: clustershell_1.8.4-2_all bug


       clush - execute shell commands on a cluster


       clush -a | -g group | -w nodes  [OPTIONS]

       clush -a | -g group | -w nodes  [OPTIONS] command

       clush  -a  |  -g  group | -w nodes  [OPTIONS] --copy file | dir [ file | dir ...] [ --dest
       path ]

       clush -a | -g group | -w nodes  [OPTIONS] --rcopy file | dir [ file | dir  ...]  [  --dest
       path ]


       clush is a program for executing commands in parallel on a cluster and for gathering their
       results. clush executes commands interactively or can be used  within  shell  scripts  and
       other  applications.  It is a partial front-end to the ClusterShell library that ensures a
       light,  unified  and  robust  parallel  command  execution  framework.  Thus,  it   allows
       traditional  shell  scripts  to benefit from some of the library features. clush currently
       makes use of the Ssh worker  of  ClusterShell,  by  default,  that  only  requires  ssh(1)
       (OpenSSH SSH client).


       clush  can  be  started  non-interactively to run a shell command, or can be invoked as an
       interactive shell. To start a clush interactive session, invoke the clush command  without
       providing command.

       Non-interactive mode
              When  clush  is started non-interactively, the command is executed on the specified
              remote hosts in parallel. If option -b or --dshbak is specified,  clush  waits  for
              command completion and then displays gathered output results.

              The  -w  option  allows  you  to specify remote hosts by using ClusterShell NodeSet
              syntax, including the node groups @group special syntax and the  Extended  Patterns
              syntax  to  benefits  from  NodeSet  basic  arithmetic (like @Agroup\&@Bgroup). See
              EXTENDED PATTERNS in nodeset(1) and also groups.conf(5) for more information.

              Unless the option --nostdin (or -n) is specified, clush detects when  its  standard
              input  is  connected  to  a  terminal  (as  determined  by isatty(3)).  If actually
              connected to a terminal, clush listens to standard input when commands are running,
              waiting for an Enter key press.  Doing so will display the status of current nodes.
              If standard input is not connected to a terminal, and unless the  option  --nostdin
              is  specified,  clush  binds  the  standard input of the remote commands to its own
              standard input, allowing scripting methods like:
                 # echo foo | clush -w node[40-42] -b cat

              Please see some other great examples in the EXAMPLES section below.

       Interactive session
              If a command is not specified, and its standard input is connected to  a  terminal,
              clush runs interactively. In this mode, clush uses the GNU readline library to read
              command lines. Readline provides commands for searching through the command history
              for  lines containing a specified string. For instance, type Control-R to search in
              the history for the next entry matching the search string typed so far.  clush also
              recognizes special single-character prefixes that allows the user to see and modify
              the current nodeset (the nodes where the commands are executed).

              Single-character interactive commands are:

                     clush> ?
                            show current nodeset

                     clush> @<NODESET>
                            set current nodeset

                     clush> +<NODESET>
                            add nodes to current nodeset

                     clush> -<NODESET>
                            remove nodes from current nodeset

                     clush> !COMMAND
                            execute COMMAND on the local system

                     clush> =
                            toggle the output format (gathered or standard mode)

              To leave an interactive session, type quit or Control-D.

       Local execution ( --worker=exec or -R exec )
              Instead of running provided command on remote nodes, clush can  use  the  dedicated
              exec worker to launch the command locally, for each node.  Some parameters could be
              used in the command line to make a different command for each  node.  %h  or  %host
              will  be  replaced  by node name and %n or %rank by the remote rank [0-N] (to get a
              literal % use %%)

       File copying mode ( --copy )
              When clush is started with the -c  or  --copy  option,  it  will  attempt  to  copy
              specified  file  and/or  dir  to  the provided target cluster nodes.  If the --dest
              option is specified, it will put the copied files there.

       Reverse file copying mode ( --rcopy )
              When clush is started  with  the  --rcopy  option,  it  will  attempt  to  retrieve
              specified  file  and/or  dir  from  provided cluster nodes. If the --dest option is
              specified, it must be a directory path where the files will be  stored  with  their
              hostname appended. If the destination path is not specified, it will take the first
              file or dir basename directory as the local destination.


              show clush version number and exit

       -s GROUPSOURCE, --groupsource=GROUPSOURCE
              optional groups.conf(5) group source to use

       -n, --nostdin
              do not watch for possible input from stdin; this should be used when clush  is  run
              in the background (or in scripts).

              use alternate config file for groups.conf(5)

              use alternate config file for clush.conf(5)

       -O <KEY=VALUE>, --option=<KEY=VALUE>
              override any key=value clush.conf(5) options (repeat as needed)

       Selecting target nodes:

              -w NODES
                     nodes where to run the command

              -x NODES
                     exclude nodes from the node list

              -a, --all
                     run command on all nodes

              -g GROUP, --group=GROUP
                     run command on a group of nodes

              -X GROUP
                     exclude nodes from this group

              --hostfile=FILE, --machinefile=FILE
                     path  to a file containing a list of single hosts, node sets or node groups,
                     separated by spaces and lines (may be  specified  multiple  times,  one  per

                     topology configuration file to use for tree mode

                     pick N node(s) at random in nodeset

       Output behaviour:

              -q, --quiet
                     be quiet, print essential output only

              -v, --verbose
                     be verbose, print informative messages

              -d, --debug
                     output more messages for debugging purpose

              -G, --groupbase
                     do not display group source prefix

              -L     disable  header  block and order output by nodes; if -b/-B is not specified,
                     clush will wait for all commands  to  finish  and  then  display  aggregated
                     output   of   commands  with  same  return  codes,  ordered  by  node  name;
                     alternatively, when used in conjunction with -b/-B  (eg.  -bL),  clush  will
                     enable  a  "life  gathering"  of  results  by line, such as the next line is
                     displayed as soon as possible (eg. when all nodes have sent the line)

              -N     disable labeling of command line

              -P, --progress
                     show progress during command execution; if writing is performed to  standard
                     input, the live progress indicator will display the global bandwidth of data
                     written to the target nodes

              -b, --dshbak
                     display gathered results in a dshbak-like way (note: it  will  only  try  to
                     aggregate the output of commands with same return codes)

              -B     like -b but including standard error

              -r, --regroup
                     fold nodeset using node groups

              -S, --maxrc
                     return the largest of command return codes

                     clush  can  use NO_COLOR, CLICOLOR and CLICOLOR_FORCE environment variables.
                     NO_COLOR takes precedence over CLICOLOR_FORCE which  takes  precedence  over
                     CLICOLOR.   When  --color option is used these environment variables are not
                     taken into account. --color tells whether to use  ANSI  colors  to  surround
                     node or nodeset prefix/header with escape sequences to display them in color
                     on the terminal. WHENCOLOR is never, always or  auto  (which  use  color  if
                     standard  output/error  refer  to  a terminal). Colors are set to [34m (blue
                     foreground text) for stdout and [31m (red foreground text) for  stderr,  and
                     cannot be modified.

              --diff show diff between common outputs (find the best reference output by focusing
                     on largest nodeset and also smaller command return code)

       File copying:

              -c, --copy
                     copy local file or directory to remote nodes

                     copy file or directory from remote nodes

                     destination file or directory on the nodes (optional: use the  first  source
                     directory path when not specified)

              -p     preserve modification times and modes

       Connection options:

              -f FANOUT, --fanout=FANOUT
                     do  not  execute more than FANOUT commands at the same time, useful to limit
                     resource usage. In tree mode, the same fanout value is used on the head node
                     and on each gateway (the fanout value is propagated). That is, if the fanout
                     is 16, each gateway will initate up to 16 connections to their target  nodes
                     at the same time. Default fanout value is defined in clush.conf(5).

              -l USER, --user=USER
                     execute remote command as user

              -o OPTIONS, --options=OPTIONS
                     can  be  used to give ssh options, eg. -o "-p 2022 -i ~/.ssh/myidrsa"; these
                     options are added first to ssh and override default ones

              -t CONNECT_TIMEOUT, --connect_timeout=CONNECT_TIMEOUT
                     limit time to connect to a node

              -u COMMAND_TIMEOUT, --command_timeout=COMMAND_TIMEOUT
                     limit time for command to run on the node

              -R WORKER, --worker=WORKER
                     worker name to use for connection (exec, ssh, rsh, pdsh, or the  name  of  a
                     Python worker module), default is ssh

                     whether  to  enable remote execution: in tree mode, 'yes' forces connections
                     to the leaf nodes for execution, 'no' establishes connections up to the leaf
                     parent nodes for execution (default is 'yes')

       For a short explanation of these options, see -h, --help.


       By  default,  an  exit  status of zero indicates success of the clush command but gives no
       information about the remote  commands  exit  status.  However,  when  the  -S  option  is
       specified,  the  exit  status  of clush is the largest value of the remote commands return

       For failed remote commands whose exit status is non-zero, and unless  the  combination  of
       options -qS is specified, clush displays messages similar to:

       clush: node[40-42]: exited with exit code 1


   Remote parallel execution
       # clush -w node[3-5,62] uname -r
              Run command uname -r in parallel on nodes: node3, node4, node5 and node62

   Local parallel execution
       # clush -w node[1-3] --worker=exec ping -c1 %host
              Run  locally,  in  parallel, a ping command for nodes: node1, node2 and node3.  You
              may also use -R exec as the shorter and pdsh compatible option.

   Display features
       # clush -w node[3-5,62] -b uname -r
              Run  command  uname  -r  on  nodes[3-5,62]  and  display  gathered  output  results
              (integrated dshbak-like).

       # clush -w node[3-5,62] -bL uname -r
              Line  mode:  run  command  uname  -r  on  nodes[3-5,62] and display gathered output
              results without default header block.

       # ssh node32 find /etc/yum.repos.d -type f | clush -w node[40-42] -b xargs ls -l
              Search some files on node32 in /etc/yum.repos.d and use clush to list the  matching
              ones on node[40-42], and use -b to display gathered results.

       # clush -w node[3-5,62] --diff dmidecode -s bios-version
              Run  this  Linux  command  to  get  BIOS  version on nodes[3-5,62] and show version
              differences (if any).

   All nodes
       # clush -a uname -r
              Run command uname -r on all cluster nodes, see groups.conf(5) to setup all  cluster
              nodes (all: field).

       # clush -a -x node[5,7] uname -r
              Run command uname -r on all cluster nodes except on nodes node5 and node7.

       # clush -a --diff cat /some/file
              Run command cat /some/file on all cluster nodes and show differences (if any), line
              by line, between common outputs.

   Node groups
       # clush -w @oss modprobe lustre
              Run command modprobe lustre on nodes from node group named oss, see  groups.conf(5)
              to setup node groups (map: field).

       # clush -g oss modprobe lustre
              Same as previous example but using -g to avoid @ group prefix.

       # clush -w @mds,@oss modprobe lustre
              You  may  specify  several  node  groups by separating them with commas (please see
              EXTENDED PATTERNS in nodeset(1) and also groups.conf(5) for more information).

   Copy files
       # clush -w node[3-5,62] --copy /etc/motd
              Copy local file /etc/motd to remote nodes node[3-5,62].

       # clush -w node[3-5,62] --copy /etc/motd --dest /tmp/motd2
              Copy local file /etc/motd to remote nodes node[3-5,62] at path /tmp/motd2.

       # clush -w node[3-5,62] -c /usr/share/doc/clustershell
              Recursively copy local directory /usr/share/doc/clustershell to the  same  path  on
              remote nodes node[3-5,62].

       # clush -w node[3-5,62] --rcopy /etc/motd --dest /tmp
              Copy  /etc/motd  from  remote nodes node[3-5,62] to local /tmp directory, each file
              having their remote hostname appended, eg. /tmp/motd.node3.


              System-wide clush configuration file.

              User  configuration  file  for  clush.  If   $XDG_CONFIG_HOME   is   not   defined,
              $HOME/.config/clustershell/clush.conf is used instead.

              Local user configuration file for clush (default installation for pip --user)

              Deprecated per-user clush configuration file.

              File in which interactive clush command history is saved.


       clubak(1), cluset(1), nodeset(1), readline(3), clush.conf(5), groups.conf(5).


       Use the following URL to submit a bug report or feedback:


       Stephane Thiell <>


       GNU Lesser General Public License version 2.1 or later (LGPLv2.1+)