Provided by: charliecloud_0.9.10-1_amd64 
NAME
ch-run - Run a command in a Charliecloud container
SYNOPSIS
$ ch-run [OPTION...] NEWROOT CMD [ARG...]
DESCRIPTION
Run command CMD in a Charliecloud container using the flattened and unpacked image directory located at
NEWROOT.
-b, --bind=SRC[:DST]
mount SRC at guest DST (default /mnt/0, /mnt/1, etc.)
-c, --cd=DIR
initial working directory in container
--ch-ssh
bind ch-ssh(1) into container at /usr/bin/ch-ssh
-g, --gid=GID
run as group GID within container
-j, --join
use the same container (namespaces) as peer ch-run invocations
--join-pid=PID
join the namespaces of an existing process
--join-ct=N
number of ch-run peers (implies --join; default: see below)
--join-tag=TAG
label for ch-run peer group (implies --join; default: see below)
--no-home
do not bind-mount your home directory (by default, your home directory is mounted at
/home/$USER in the container)
-t, --private-tmp
use container-private /tmp (by default, /tmp is shared with the host)
--set-env=FILE
set environment variables as specified in host path FILE
-u, --uid=UID
run as user UID within container
--unset-env=GLOB
unset environment variables whose names match GLOB
-v, --verbose
be more verbose (debug if repeated)
-w, --write
mount image read-write (by default, the image is mounted read-only)
-?, --help
print help and exit
--usage
print a short usage message and exit
-V, --version
print version and exit
HOST FILES AND DIRECTORIES AVAILABLE IN CONTAINER VIA BIND MOUNTS
In addition to any directories specified by the user with --bind, ch-run has standard host files and
directories that are bind-mounted in as well.
The following host files and directories are bind-mounted at the same location in the container. These
cannot be disabled.
• /dev
• /etc/passwd
• /etc/group
• /etc/hosts
• /etc/resolv.conf
• /proc
• /sys
Three additional bind mounts can be disabled by the user:
• Your home directory (i.e., $HOME) is mounted at guest /home/$USER by default. This is accomplished
by mounting a new tmpfs at /home, which hides any image content under that path. If --no-home is
specified, neither of these things happens and the image’s /home is exposed unaltered.
• /tmp is shared with the host by default. If --private-tmp is specified, a new tmpfs is mounted on
the guest’s /tmp instead.
• If file /usr/bin/ch-ssh is present in the image, it is over-mounted with the ch-ssh binary in the
same directory as ch-run.
MULTIPLE PROCESSES IN THE SAME CONTAINER WITH --JOIN
By default, different ch-run invocations use different user and mount namespaces (i.e., different
containers). While this has no impact on sharing most resources between invocations, there are a few
important exceptions. These include:
1. ptrace(2), used by debuggers and related tools. One can attach a debugger to processes in descendant
namespaces, but not sibling namespaces. The practical effect of this is that (without --join), you
can’t run a command with ch-run and then attach to it with a debugger also run with ch-run.
2. Cross-memory attach (CMA) is used by cooperating processes to communicate by simply reading and
writing one another’s memory. This is also not permitted between sibling namespaces. This affects
various MPI implementations that use CMA to pass messages between ranks on the same node, because it’s
faster than traditional shared memory.
--join is designed to address this by placing related ch-run commands (the “peer group”) in the same
container. This is done by one of the peers creating the namespaces with unshare(2) and the others
joining with setns(2).
To do so, we need to know the number of peers and a name for the group. These are specified by additional
arguments that can (hopefully) be left at default values in most cases:
• --join-ct sets the number of peers. The default is the value of the first of the following environment
variables that is defined: OMPI_COMM_WORLD_LOCAL_SIZE, SLURM_STEP_TASKS_PER_NODE, SLURM_CPUS_ON_NODE.
• --join-tag sets the tag that names the peer group. The default is environment variable SLURM_STEP_ID,
if defined; otherwise, the PID of ch-run’s parent. Tags can be re-used for peer groups that start at
different times, i.e., once all peer ch-run have replaced themselves with the user command, the tag can
be re-used.
Caveats:
• One cannot currently add peers after the fact, for example, if one decides to start a debugger after
the fact. (This is only required for code with bugs and is thus an unusual use case.)
• ch-run instances race. The winner of this race sets up the namespaces, and the other peers use the
winner to find the namespaces to join. Therefore, if the user command of the winner exits, any
remaining peers will not be able to join the namespaces, even if they are still active. There is
currently no general way to specify which ch-run should be the winner.
• If --join-ct is too high, the winning ch-run’s user command exits before all peers join, or ch-run
itself crashes, IPC resources such as semaphores and shared memory segments will be leaked. These
appear as files in /dev/shm/ and can be removed with rm(1).
• Many of the arguments given to the race losers, such as the image path and --bind, will be ignored in
favor of what was given to the winner.
ENVIRONMENT VARIABLES
ch-run leaves environment variables unchanged, i.e. the host environment is passed through unaltered,
except:
• limited tweaks to avoid significant guest breakage;
• user-set variables via --set-env; and
• user-unset variables via --unset-env.
This section describes these features.
The default tweaks happen first, and then --set-env and --unset-env in the order specified on the command
line. The latter two can be repeated arbitrarily many times, e.g. to add/remove multiple variable sets or
add only some variables in a file.
Default behavior
By default, ch-run makes the following environment variable changes:
• $HOME: If the path to your home directory is not /home/$USER on the host, then an inherited $HOME will
be incorrect inside the guest. This confuses some software, such as Spack.
Thus, we change $HOME to /home/$USER, unless --no-home is specified, in which case it is left
unchanged.
• $PATH: Newer Linux distributions replace some root-level directories, such as /bin, with symlinks to
their counterparts in /usr.
Some of these distributions (e.g., Fedora 24) have also dropped /bin from the default $PATH. This is a
problem when the guest OS does not have a merged /usr (e.g., Debian 8 “Jessie”). Thus, we add /bin to
$PATH if it’s not already present.
Further reading:
• The case for the /usr Merge
• Fedora
• Debian
Setting variables with --set-env
The purpose of --set-env=FILE is to set environment variables that cannot be inherited from the host
shell, e.g. Dockerfile ENV directives or other build-time configuration. FILE is a host path to provide
the greatest flexibility; guest paths can be specified by prepending the image path.
ch-docker2tar(1) lists variables specified at build time in Dockerfiles in the image in file
/environment. To set these variables: --set-env=$IMG/environment.
Variable values in FILE replace any already set. If a variable is repeated, the last value wins.
The syntax of FILE is key-value pairs separated by the first equals character (=, ASCII 61), one per
line, with optional single straight quotes (', ASCII 39) around the value. Empty lines are ignored.
Newlines (ASCII 10) are not permitted in either key or value. No variable expansion, comments, etc. are
provided. The value may be empty, but not the key. (This syntax is designed to accept the output of
printenv and be easily produced by other simple mechanisms.) Examples of valid lines:
┌────────────────────────────┬───────┬────────────────────────┐
│Line │ Key │ Value │
├────────────────────────────┼───────┼────────────────────────┤
│FOO=bar │ FOO │ bar │
├────────────────────────────┼───────┼────────────────────────┤
│FOO=bar=baz │ FOO │ bar=baz │
├────────────────────────────┼───────┼────────────────────────┤
│FLAGS=-march=foo -mtune=bar │ FLAGS │ -march=foo -mtune=bar │
├────────────────────────────┼───────┼────────────────────────┤
│FLAGS='-march=foo │ FLAGS │ -march=foo -mtune=bar │
│-mtune=bar' │ │ │
├────────────────────────────┼───────┼────────────────────────┤
│FOO= │ FOO │ (empty string) │
├────────────────────────────┼───────┼────────────────────────┤
│FOO='' │ FOO │ (empty string) │
├────────────────────────────┼───────┼────────────────────────┤
│FOO='''' │ FOO │ '' (two single quotes) │
└────────────────────────────┴───────┴────────────────────────┘
Example invalid lines:
┌────────┬─────────────────────┐
│Line │ Problem │
├────────┼─────────────────────┤
│FOO bar │ no separator │
├────────┼─────────────────────┤
│=bar │ key cannot be empty │
└────────┴─────────────────────┘
Example valid lines that are probably not what you want:
┌────────────────────┬───────┬────────────────┬────────────────────────┐
│Line │ Key │ Value │ Problem │
├────────────────────┼───────┼────────────────┼────────────────────────┤
│FOO="bar" │ FOO │ "bar" │ double quotes aren’t │
│ │ │ │ stripped │
├────────────────────┼───────┼────────────────┼────────────────────────┤
│FOO=bar # baz │ FOO │ bar # baz │ comments not supported │
├────────────────────┼───────┼────────────────┼────────────────────────┤
│PATH=$PATH:/opt/bin │ PATH │ $PATH:/opt/bin │ variables not expanded │
├────────────────────┼───────┼────────────────┼────────────────────────┤
│ FOO=bar │ FOO │ bar │ leading space in key │
├────────────────────┼───────┼────────────────┼────────────────────────┤
│FOO= bar │ FOO │ bar │ leading space in value │
└────────────────────┴───────┴────────────────┴────────────────────────┘
Removing variables with --unset-env
The purpose of --unset-env=GLOB is to remove unwanted environment variables. The argument GLOB is a glob
pattern (dialect fnmatch(3) with no flags); all variables with matching names are removed from the
environment.
WARNING:
Because the shell also interprets glob patterns, if any wildcard characters are in GLOB, it is
important to put it in single quotes to avoid surprises.
GLOB must be a non-empty string.
Example 1: Remove the single environment variable FOO:
$ export FOO=bar
$ env | fgrep FOO
FOO=bar
$ ch-run --unset-env=FOO $CH_TEST_IMGDIR/chtest -- env | fgrep FOO
$
Example 2: Hide from a container the fact that it’s running in a Slurm allocation, by removing all
variables beginning with SLURM. You might want to do this to test an MPI program with one rank and no
launcher:
$ salloc -N1
$ env | egrep '^SLURM' | wc
44 44 1092
$ ch-run $CH_TEST_IMGDIR/mpihello-openmpi -- /hello/hello
[... long error message ...]
$ ch-run --unset-env='SLURM*' $CH_TEST_IMGDIR/mpihello-openmpi -- /hello/hello
0: MPI version:
Open MPI v3.1.3, package: Open MPI root@c897a83f6f92 Distribution, ident: 3.1.3, repo rev: v3.1.3, Oct 29, 2018
0: init ok cn001.localdomain, 1 ranks, userns 4026532530
0: send/receive ok
0: finalize ok
Example 3: Clear the environment completely (remove all variables):
$ ch-run --unset-env='*' $CH_TEST_IMGDIR/chtest -- env
$
Note that some programs, such as shells, set some environment variables even if started with no init
files:
$ ch-run --unset-env='*' $CH_TEST_IMGDIR/debian9 -- bash --noprofile --norc -c env
SHLVL=1
PWD=/
_=/usr/bin/env
$
EXAMPLES
Run the command echo hello inside a Charliecloud container using the unpacked image at /data/foo:
$ ch-run /data/foo -- echo hello
hello
Run an MPI job that can use CMA to communicate:
$ srun ch-run --join /data/foo -- bar
REPORTING BUGS
If Charliecloud was obtained from your Linux distribution, use your distribution’s bug reporting
procedures.
Otherwise, report bugs to: <https://github.com/hpc/charliecloud/issues>
SEE ALSO
charliecloud(1)
Full documentation at: <https://hpc.github.io/charliecloud>
AUTHOR
Reid Priedhorsky, Tim Randles, and others
COPYRIGHT
2014–2018, Los Alamos National Security, LLC
2019-09-08 18:23 Coordinated Universal Time CH-RUN(1)