Provided by: clustershell_1.8-1_all bug


       clush - execute shell commands on a cluster


       clush -a | -g group | -w nodes  [OPTIONS]

       clush -a | -g group | -w nodes  [OPTIONS] command

       clush  -a  |  -g  group | -w nodes  [OPTIONS] --copy file | dir [ file | dir ...] [ --dest
       path ]

       clush -a | -g group | -w nodes  [OPTIONS] --rcopy file | dir [ file | dir  ...]  [  --dest
       path ]


       clush is a program for executing commands in parallel on a cluster and for gathering their
       results. clush executes commands interactively or can be used  within  shell  scripts  and
       other  applications.  It is a partial front-end to the ClusterShell library that ensures a
       light,  unified  and  robust  parallel  command  execution  framework.  Thus,  it   allows
       traditional  shell  scripts  to benefit from some of the library features. clush currently
       makes use of the Ssh worker  of  ClusterShell,  by  default,  that  only  requires  ssh(1)
       (OpenSSH SSH client).


       clush  can  be  started  non-interactively to run a shell command, or can be invoked as an
       interactive shell. To start a clush interactive session, invoke the clush command  without
       providing command.

       Non-interactive mode
              When  clush  is started non-interactively, the command is executed on the specified
              remote hosts in parallel. If option -b or --dshbak is specified,  clush  waits  for
              command completion and then displays gathered output results.

              The  -w  option  allows  you  to specify remote hosts by using ClusterShell NodeSet
              syntax, including the node groups @group special syntax and the  Extended  Patterns
              syntax  to  benefits  from  NodeSet  basic arithmetics (like @Agroup\&@Bgroup). See
              EXTENDED PATTERNS in nodeset(1) and also groups.conf(5) for more information.

              Unless the option --nostdin (or -n) is specified, clush detects when  its  standard
              input  is  connected  to  a  terminal  (as  determined  by isatty(3)).  If actually
              connected to a terminal, clush listens to standard input when commands are running,
              waiting for an Enter key press.  Doing so will display the status of current nodes.
              If standard input is not connected to a terminal, and unless the  option  --nostdin
              is  specified,  clush  binds  the  standard input of the remote commands to its own
              standard input, allowing scripting methods like:
                 # echo foo | clush -w node[40-42] -b cat

              Please see some other great examples in the EXAMPLES section below.

       Interactive session
              If a command is not specified, and its standard input is connected to  a  terminal,
              clush runs interactively. In this mode, clush uses the GNU readline library to read
              command lines. Readline provides commands for searching through the command history
              for  lines containing a specified string. For instance, type Control-R to search in
              the history for the next entry matching the search string typed so far.  clush also
              recognizes special single-character prefixes that allows the user to see and modify
              the current nodeset (the nodes where the commands are executed).

              Single-character interactive commands are:

                     clush> ?
                            show current nodeset

                     clush> @<NODESET>
                            set current nodeset

                     clush> +<NODESET>
                            add nodes to current nodeset

                     clush> -<NODESET>
                            remove nodes from current nodeset

                     clush> !COMMAND
                            execute COMMAND on the local system

                     clush> =
                            toggle the output format (gathered or standard mode)

              To leave an interactive session, type quit or Control-D.

       Local execution ( --worker=exec or -R exec )
              Instead of running provided command on remote nodes, clush can  use  the  dedicated
              exec worker to launch the command locally, for each node.  Some parameters could be
              used in the command line to make a different command for each  node.  %h  or  %host
              will  be  replaced  by node name and %r or %rank by the remote rank [0-N] (to get a
              literal % use %%)

       File copying mode ( --copy )
              When clush is started with the -c  or  --copy  option,  it  will  attempt  to  copy
              specified  file  and/or  dir  to  the provided target cluster nodes.  If the --dest
              option is specified, it will put the copied files there.

       Reverse file copying mode ( --rcopy )
              When clush is started  with  the  --rcopy  option,  it  will  attempt  to  retrieve
              specified  file  and/or  dir  from  provided cluster nodes. If the --dest option is
              specified, it must be a directory path where the files will be  stored  with  their
              hostname appended. If the destination path is not specified, it will take the first
              file or dir basename directory as the local destination.


              show clush version number and exit

       -s GROUPSOURCE, --groupsource=GROUPSOURCE
              optional groups.conf(5) group source to use

       -n, --nostdin
              do not watch for possible input from stdin; this should be used when clush  is  run
              in the background (or in scripts).

       -O <KEY=VALUE>, --option=<KEY=VALUE>
              override any key=value clush.conf(5) options (repeat as needed)

       Selecting target nodes:

              -w NODES
                     nodes where to run the command

              -x NODES
                     exclude nodes from the node list

              -a, --all
                     run command on all nodes

              -g GROUP, --group=GROUP
                     run command on a group of nodes

              -X GROUP
                     exclude nodes from this group

              --hostfile=FILE, --machinefile=FILE
                     path  to a file containing a list of single hosts, node sets or node groups,
                     separated by spaces and lines (may be  specified  multiple  times,  one  per

                     topology configuration file to use for tree mode

                     pick N node(s) at random in nodeset

       Output behaviour:

              -q, --quiet
                     be quiet, print essential output only

              -v, --verbose
                     be verbose, print informative messages

              -d, --debug
                     output more messages for debugging purpose

              -G, --groupbase
                     do not display group source prefix

              -L     disable  header  block and order output by nodes; if -b/-B is not specified,
                     clush will wait for all commands  to  finish  and  then  display  aggregated
                     output   of   commands  with  same  return  codes,  ordered  by  node  name;
                     alternatively, when used in conjunction with -b/-B  (eg.  -bL),  clush  will
                     enable  a  "life  gathering"  of  results  by line, such as the next line is
                     displayed as soon as possible (eg. when all nodes have sent the line)

              -N     disable labeling of command line

              -P, --progress
                     show progress during command execution; if writing is performed to  standard
                     input, the live progress indicator will display the global bandwidth of data
                     written to the target nodes

              -b, --dshbak
                     display gathered results in a dshbak-like way (note: it  will  only  try  to
                     aggregate the output of commands with same return codes)

              -B     like -b but including standard error

              -r, --regroup
                     fold nodeset using node groups

              -S     return the largest of command return codes

                     whether  to  use  ANSI colors to surround node or nodeset prefix/header with
                     escape sequences to display them in color  on  the  terminal.  WHENCOLOR  is
                     never,  always  or auto (which use color if standard output/error refer to a
                     terminal). Colors are set to [34m (blue foreground text) for stdout and [31m
                     (red foreground text) for stderr, and cannot be modified.

              --diff show diff between common outputs (find the best reference output by focusing
                     on largest nodeset and also smaller command return code)

       File copying:

              -c, --copy
                     copy local file or directory to remote nodes

                     copy file or directory from remote nodes

                     destination file or directory on the nodes (optional: use the  first  source
                     directory path when not specified)

              -p     preserve modification times and modes

       Connection options:

              -f FANOUT, --fanout=FANOUT
                     do  not  execute more than FANOUT commands at the same time, useful to limit
                     resource usage. In tree mode, the same fanout value is used on the head node
                     and on each gateway (the fanout value is propagated). That is, if the fanout
                     is 16, each gateway will initate up to 16 connections to their target  nodes
                     at the same time. Default fanout value is defined in clush.conf(5).

              -l USER, --user=USER
                     execute remote command as user

              -o OPTIONS, --options=OPTIONS
                     can  be  used to give ssh options, eg. -o "-p 2022 -i ~/.ssh/myidrsa"; these
                     options are added first to ssh and override default ones

              -t CONNECT_TIMEOUT, --connect_timeout=CONNECT_TIMEOUT
                     limit time to connect to a node

              -u COMMAND_TIMEOUT, --command_timeout=COMMAND_TIMEOUT
                     limit time for command to run on the node

              -R WORKER, --worker=WORKER
                     worker name to use for connection (exec, ssh, rsh, pdsh), default is ssh

                     whether to enable remote execution: in tree mode, 'yes'  forces  connections
                     to the leaf nodes for execution, 'no' establishes connections up to the leaf
                     parent nodes for execution (default is 'yes')

       For a short explanation of these options, see -h, --help.


       By default, an exit status of zero indicates success of the clush  command  but  gives  no
       information  about  the  remote  commands  exit  status.  However,  when  the -S option is
       specified, the exit status of clush is the largest value of  the  remote  commands  return

       For  failed  remote  commands whose exit status is non-zero, and unless the combination of
       options -qS is specified, clush displays messages similar to:

       clush: node[40-42]: exited with exit code 1


   Remote parallel execution
       # clush -w node[3-5,62] uname -r
              Run command uname -r in parallel on nodes: node3, node4, node5 and node62

   Local parallel execution
       # clush -w node[1-3] --worker=exec ping -c1 %host
              Run locally, in parallel, a ping command for nodes: node1, node2  and  node3.   You
              may also use -R exec as the shorter and pdsh compatible option.

   Display features
       # clush -w node[3-5,62] -b uname -r
              Run  command  uname  -r  on  nodes[3-5,62]  and  display  gathered  output  results
              (integrated dshbak-like).

       # clush -w node[3-5,62] -bL uname -r
              Line mode: run command uname  -r  on  nodes[3-5,62]  and  display  gathered  output
              results without default header block.

       # ssh node32 find /etc/yum.repos.d -type f | clush -w node[40-42] -b xargs ls -l
              Search  some files on node32 in /etc/yum.repos.d and use clush to list the matching
              ones on node[40-42], and use -b to display gathered results.

       # clush -w node[3-5,62] --diff dmidecode -s bios-version
              Run this Linux command to get  BIOS  version  on  nodes[3-5,62]  and  show  version
              differences (if any).

   All nodes
       # clush -a uname -r
              Run  command uname -r on all cluster nodes, see groups.conf(5) to setup all cluster
              nodes (all: field).

       # clush -a -x node[5,7] uname -r
              Run command uname -r on all cluster nodes except on nodes node5 and node7.

       # clush -a --diff cat /some/file
              Run command cat /some/file on all cluster nodes and show differences (if any), line
              by line, between common outputs.

   Node groups
       # clush -w @oss modprobe lustre
              Run  command modprobe lustre on nodes from node group named oss, see groups.conf(5)
              to setup node groups (map: field).

       # clush -g oss modprobe lustre
              Same as previous example but using -g to avoid @ group prefix.

       # clush -w @mds,@oss modprobe lustre
              You may specify several node groups by separating  them  with  commas  (please  see
              EXTENDED PATTERNS in nodeset(1) and also groups.conf(5) for more information).

   Copy files
       # clush -w node[3-5,62] --copy /etc/motd
              Copy local file /etc/motd to remote nodes node[3-5,62].

       # clush -w node[3-5,62] --copy /etc/motd --dest /tmp/motd2
              Copy local file /etc/motd to remote nodes node[3-5,62] at path /tmp/motd2.

       # clush -w node[3-5,62] -c /usr/share/doc/clustershell
              Recursively  copy  local  directory /usr/share/doc/clustershell to the same path on
              remote nodes node[3-5,62].

       # clush -w node[3-5,62] --rcopy /etc/motd --dest /tmp
              Copy /etc/motd from remote nodes node[3-5,62] to local /tmp  directory,  each  file
              having their remote hostname appended, eg. /tmp/motd.node3.


              System-wide clush configuration file.

              User   configuration   file   for   clush.  If  $XDG_CONFIG_HOME  is  not  defined,
              $HOME/.config/clustershell/clush.conf is used instead.

              Local user configuration file for clush (default installation for pip --user)

              Deprecated per-user clush configuration file.

              File in which interactive clush command history is saved.


       clubak(1), cluset(1), nodeset(1), readline(3), clush.conf(5), groups.conf(5).


       Use the following URL to submit a bug report or feedback:


       Stephane Thiell <>


       GNU Lesser General Public License version 2.1 or later (LGPLv2.1+)