Provided by: charliecloud_0.9.10-1_amd64 bug


       ch-run - Run a command in a Charliecloud container


          $ ch-run [OPTION...] NEWROOT CMD [ARG...]


       Run  command  CMD  in  a  Charliecloud  container  using  the flattened and unpacked image
       directory located at NEWROOT.

          -b, --bind=SRC[:DST]
                 mount SRC at guest DST (default /mnt/0, /mnt/1, etc.)

          -c, --cd=DIR
                 initial working directory in container

                 bind ch-ssh(1) into container at /usr/bin/ch-ssh

          -g, --gid=GID
                 run as group GID within container

          -j, --join
                 use the same container (namespaces) as peer ch-run invocations

                 join the namespaces of an existing process

                 number of ch-run peers (implies --join; default: see below)

                 label for ch-run peer group (implies --join; default: see below)

                 do not bind-mount your home  directory  (by  default,  your  home  directory  is
                 mounted at /home/$USER in the container)

          -t, --private-tmp
                 use container-private /tmp (by default, /tmp is shared with the host)

                 set environment variables as specified in host path FILE

          -u, --uid=UID
                 run as user UID within container

                 unset environment variables whose names match GLOB

          -v, --verbose
                 be more verbose (debug if repeated)

          -w, --write
                 mount image read-write (by default, the image is mounted read-only)

          -?, --help
                 print help and exit

                 print a short usage message and exit

          -V, --version
                 print version and exit


       In addition to any directories specified by the user with --bind, ch-run has standard host
       files and directories that are bind-mounted in as well.

       The following host files and directories are bind-mounted at  the  same  location  in  the
       container. These cannot be disabled.

          · /dev

          · /etc/passwd

          · /etc/group

          · /etc/hosts

          · /etc/resolv.conf

          · /proc

          · /sys

       Three additional bind mounts can be disabled by the user:

          · Your home directory (i.e., $HOME) is mounted at guest /home/$USER by default. This is
            accomplished by mounting a new tmpfs at /home, which hides any  image  content  under
            that path. If --no-home is specified, neither of these things happens and the image’s
            /home is exposed unaltered.

          · /tmp is shared with the host by default. If --private-tmp is specified, a  new  tmpfs
            is mounted on the guest’s /tmp instead.

          · If  file  /usr/bin/ch-ssh is present in the image, it is over-mounted with the ch-ssh
            binary in the same directory as ch-run.


       By default, different ch-run invocations use different user and  mount  namespaces  (i.e.,
       different  containers).  While  this  has  no  impact  on  sharing  most resources between
       invocations, there are a few important exceptions.  These include:

       1. ptrace(2), used by debuggers and related tools. One can attach a debugger to  processes
          in  descendant namespaces, but not sibling namespaces.  The practical effect of this is
          that (without --join), you can’t run a command with ch-run and then attach to it with a
          debugger also run with ch-run.

       2. Cross-memory  attach  (CMA)  is  used by cooperating processes to communicate by simply
          reading and writing one another’s memory. This is also not  permitted  between  sibling
          namespaces.  This  affects  various  MPI  implementations that use CMA to pass messages
          between ranks on the same node, because it’s faster than traditional shared memory.

       --join is designed to address this by placing related ch-run commands (the  “peer  group”)
       in  the  same  container.  This  is  done by one of the peers creating the namespaces with
       unshare(2) and the others joining with setns(2).

       To do so, we need to know the number of  peers  and  a  name  for  the  group.  These  are
       specified  by  additional arguments that can (hopefully) be left at default values in most

       · --join-ct sets the number of peers. The default  is  the  value  of  the  first  of  the
         following   environment   variables   that   is   defined:   OMPI_COMM_WORLD_LOCAL_SIZE,

       · --join-tag sets the tag that names the peer group. The default is  environment  variable
         SLURM_STEP_ID,  if  defined;  otherwise, the PID of ch-run’s parent. Tags can be re-used
         for peer groups that start at different times, i.e., once all peer ch-run have  replaced
         themselves with the user command, the tag can be re-used.


       · One  cannot  currently  add peers after the fact, for example, if one decides to start a
         debugger after the fact. (This is only required for  code  with  bugs  and  is  thus  an
         unusual use case.)

       · ch-run  instances  race.  The  winner of this race sets up the namespaces, and the other
         peers use the winner to find the namespaces to join. Therefore, if the user  command  of
         the  winner  exits, any remaining peers will not be able to join the namespaces, even if
         they are still active. There is currently no general way to specify which ch-run  should
         be the winner.

       · If --join-ct is too high, the winning ch-run’s user command exits before all peers join,
         or ch-run itself crashes, IPC resources such as semaphores and  shared  memory  segments
         will be leaked. These appear as files in /dev/shm/ and can be removed with rm(1).

       · Many  of the arguments given to the race losers, such as the image path and --bind, will
         be ignored in favor of what was given to the winner.


       ch-run leaves environment variables unchanged, i.e. the host environment is passed through
       unaltered, except:

       · limited tweaks to avoid significant guest breakage;

       · user-set variables via --set-env; and

       · user-unset variables via --unset-env.

       This section describes these features.

       The default tweaks happen first, and then --set-env and --unset-env in the order specified
       on the command line. The latter two can  be  repeated  arbitrarily  many  times,  e.g.  to
       add/remove multiple variable sets or add only some variables in a file.

   Default behavior
       By default, ch-run makes the following environment variable changes:

       · $HOME:  If  the  path  to  your  home  directory is not /home/$USER on the host, then an
         inherited $HOME will be incorrect inside the guest. This confuses some software, such as

         Thus, we change $HOME to /home/$USER, unless --no-home is specified, in which case it is
         left unchanged.

       · $PATH: Newer Linux distributions replace some root-level directories, such as /bin, with
         symlinks to their counterparts in /usr.

         Some  of  these  distributions (e.g., Fedora 24) have also dropped /bin from the default
         $PATH. This is a problem when the guest OS does not have a merged /usr (e.g.,  Debian  8
         “Jessie”). Thus, we add /bin to $PATH if it’s not already present.

         Further reading:

            · The case for the /usr Merge

            · Fedora

            · Debian

   Setting variables with --set-env
       The  purpose  of  --set-env=FILE  is to set environment variables that cannot be inherited
       from the host shell, e.g. Dockerfile ENV directives  or  other  build-time  configuration.
       FILE  is  a host path to provide the greatest flexibility; guest paths can be specified by
       prepending the image path.

       ch-docker2tar(1) lists variables specified at build time in Dockerfiles in  the  image  in
       file /environment. To set these variables: --set-env=$IMG/environment.

       Variable values in FILE replace any already set. If a variable is repeated, the last value

       The syntax of FILE is key-value pairs separated by the first equals  character  (=,  ASCII
       61),  one  per  line, with optional single straight quotes (', ASCII 39) around the value.
       Empty lines are ignored.  Newlines (ASCII 10) are not permitted in either key or value. No
       variable  expansion, comments, etc. are provided. The value may be empty, but not the key.
       (This syntax is designed to accept the output of printenv and be easily produced by  other
       simple mechanisms.) Examples of valid lines:

                          │Line              │ Key   │ Value                  │
                          │FOO=barFOObar                    │
                          │FOO=bar=bazFOObar=baz                │
                          │FLAGS=-march=fooFLAGS-march=foo -mtune=bar  │
                          │-mtune=bar        │       │                        │
                          │FLAGS='-march=fooFLAGS-march=foo -mtune=bar  │
                          │-mtune=bar'       │       │                        │
                          │FOO=FOO   │ (empty string)         │
                          │FOO=''FOO   │ (empty string)         │
                          │FOO=''''FOO'' (two single quotes) │

       Example invalid lines:

                                    │Line    │ Problem             │
                                    │FOO bar │ no separator        │
                                    │=bar    │ key cannot be empty │

       Example valid lines that are probably not what you want:

                  │Line                │ Key   │ Value          │ Problem             │
                  │FOO="bar"FOO"bar"          │ double       quotes │
                  │                    │       │                │ aren’t stripped     │
                  │FOO=bar # bazFOObar # baz      │ comments        not │
                  │                    │       │                │ supported           │
                  │PATH=$PATH:/opt/binPATH$PATH:/opt/bin │ variables       not │
                  │                    │       │                │ expanded            │
                  │ FOO=bar FOObar            │ leading  space   in │
                  │                    │       │                │ key                 │

                  │FOO= barFOO bar          │ leading   space  in │
                  │                    │       │                │ value               │

   Removing variables with --unset-env
       The purpose of --unset-env=GLOB is to remove unwanted environment variables. The  argument
       GLOB  is  a  glob  pattern (dialect fnmatch(3) with no flags); all variables with matching
       names are removed from the environment.

          Because the shell also interprets glob patterns, if  any  wildcard  characters  are  in
          GLOB, it is important to put it in single quotes to avoid surprises.

       GLOB must be a non-empty string.

       Example 1: Remove the single environment variable FOO:

          $ export FOO=bar
          $ env | fgrep FOO
          $ ch-run --unset-env=FOO $CH_TEST_IMGDIR/chtest -- env | fgrep FOO

       Example  2:  Hide  from  a  container the fact that it’s running in a Slurm allocation, by
       removing all variables beginning with SLURM. You might want to do  this  to  test  an  MPI
       program with one rank and no launcher:

          $ salloc -N1
          $ env | egrep '^SLURM' | wc
             44      44    1092
          $ ch-run $CH_TEST_IMGDIR/mpihello-openmpi -- /hello/hello
          [... long error message ...]
          $ ch-run --unset-env='SLURM*' $CH_TEST_IMGDIR/mpihello-openmpi -- /hello/hello
          0: MPI version:
          Open MPI v3.1.3, package: Open MPI root@c897a83f6f92 Distribution, ident: 3.1.3, repo rev: v3.1.3, Oct 29, 2018
          0: init ok cn001.localdomain, 1 ranks, userns 4026532530
          0: send/receive ok
          0: finalize ok

       Example 3: Clear the environment completely (remove all variables):

          $ ch-run --unset-env='*' $CH_TEST_IMGDIR/chtest -- env

       Note  that  some  programs, such as shells, set some environment variables even if started
       with no init files:

          $ ch-run --unset-env='*' $CH_TEST_IMGDIR/debian9 -- bash --noprofile --norc -c env


       Run the command echo hello inside a Charliecloud container using  the  unpacked  image  at

          $ ch-run /data/foo -- echo hello

       Run an MPI job that can use CMA to communicate:

          $ srun ch-run --join /data/foo -- bar


       If  Charliecloud  was  obtained  from your Linux distribution, use your distribution’s bug
       reporting procedures.

       Otherwise, report bugs to: <>



       Full documentation at: <>


       Reid Priedhorsky, Tim Randles, and others


       2014–2018, Los Alamos National Security, LLC

                           2019-09-08 18:23 Coordinated Universal Time                  CH-RUN(1)