Provided by: charliecloud-builders_0.33-1_amd64 bug

NAME

       ch-image - Build and manage images; completely unprivileged

SYNOPSIS

          $ ch-image [...] build [-t TAG] [-f DOCKERFILE] [...] CONTEXT
          $ ch-image [...] build-cache [...]
          $ ch-image [...] delete IMAGE_GLOB [IMAGE_GLOB ...]
          $ ch-image [...] gestalt [SELECTOR]
          $ ch-image [...] import PATH IMAGE_REF
          $ ch-image [...] list [-l] [IMAGE_REF]
          $ ch-image [...] pull [...] IMAGE_REF [DEST_REF]
          $ ch-image [...] push [--image DIR] IMAGE_REF [DEST_REF]
          $ ch-image [...] reset
          $ ch-image [...] undelete IMAGE_REF
          $ ch-image { --help | --version | --dependencies }

DESCRIPTION

       ch-image  is  a  tool for building and manipulating container images, but not running them
       (for that you want ch-run). It is completely unprivileged,  with  no  setuid/setgid/setcap
       helpers.  Many  operations can use caching for speed. The action to take is specified by a
       sub-command.

       Options that print brief information and then exit:

          -h, --help
                 Print help and exit successfully. If specified  before  the  sub-command,  print
                 general  help  and  list  of  sub-commands; if after the sub-command, print help
                 specific to that sub-command.

          --dependencies
                 Report dependency problems on standard output, if any, and exit. If all is well,
                 there  is no output and the exit is successful; in case of problems, the exit is
                 unsuccessful.

          --version
                 Print version number and exit successfully.

       Common options placed before or after the sub-command:

          -a, --arch ARCH
                 Use ARCH for architecture-aware registry operations. (See section “Architecture”
                 below for details.)

          --always-download
                 Download  all  files  when pulling, even if they are already in builder storage.
                 Note that ch-image pull will always retrieve the  most  up-to-date  image;  this
                 option is mostly for debugging.

          --auth Authenticate   with  the  remote  repository,  then  (if  successful)  make  all
                 subsequent requests in authenticated mode. For most subcommands, the default  is
                 to  never  authenticate,  i.e.,  make all requests anonymously. The exception is
                 push, which implies --auth.

          --cache
                 Enable build cache. Default if a sufficiently new Git is available.

          --cache-large SIZE
                 Set the cache’s large file threshold to SIZE MiB, or 0 for no large files, which
                 is the default. This can speed up some builds.  Experimental. See section “Build
                 cache” for details.

          --no-cache
                 Disable build cache. Default if a sufficiently new Git is not  available.   This
                 option  turns  off  the cache completely; if you want to re-execute a Dockerfile
                 and store the new results in cache, use --rebuild instead.

          --no-lock
                 Disable storage directory locking. This lets you run as many concurrent ch-image
                 instances as you want against the same storage directory, which risks corruption
                 but may be OK for some workloads.

          --profile
                 Dump profile to files /tmp/chofile.p (cProfile dump format) and /tmp/chofile.txt
                 (text summary). You can convert the former to a PDF call graph with gprof2dot -f
                 pstats /tmp/chofile.p | dot -Tpdf -o /tmp/chofile.pdf. This excludes time  spend
                 in  subprocesses.  Profile data should still be written on fatal errors, but not
                 if the program crashes.

          --rebuild
                 Execute all instructions, even if they are build cache  hits,  except  for  FROM
                 which is retrieved from cache on hit.

          --password-many
                 Re-prompt the user every time a registry password is needed.

          -s, --storage DIR
                 Set the storage directory (see below for important details).

          --tls-no-verify
                 Don’t  verify TLS certificates of the repository. (Do not use this option unless
                 you understand the risks.)

          -v, --verbose
                 Print extra chatter; can be repeated.

ARCHITECTURE

       Charliecloud  provides  the  option  --arch  ARCH  to   specify   the   architecture   for
       architecture-aware  registry  operations.  The  argument  ARCH can be: (1) yolo, to bypass
       architecture-aware code and use the registry’s default architecture; (2) host, to use  the
       host’s  architecture,  obtained  with  the  equivalent  of uname -m (default if --arch not
       specified); or (3) an architecture name. If the specified architecture is  not  available,
       the error message will list which ones are.

       Notes:

       1. ch-image  is  limited  to  one  image per image reference in builder storage at a time,
          regardless of architecture. For example, if you say ch-image pull  --arch=foo  baz  and
          then ch-image pull --arch=bar baz, builder storage will contain one image called “baz”,
          with architecture “bar”.

       2. Images’ default architecture is usually amd64, so this is usually  what  you  get  with
          --arch=yolo.  Similarly,  if a registry image is architecture-unaware, it will still be
          pulled with --arch=amd64 and --arch=host on x86-64 hosts (other host architectures must
          specify --arch=yolo to pull architecture-unaware images).

       3. uname  -m and image registries often use different names for the same architecture. For
          example, what uname  -m  reports  as  “x86_64”  is  known  to  registries  as  “amd64”.
          --arch=host  should  translate  if  needed,  but it’s useful to know this is happening.
          Directly specified architecture names are passed to the registry without translation.

       4. Registries treat architecture as a pair of items, architecture  and  sometimes  variant
          (e.g.,  “arm”  and  “v7”).  Charliecloud  treats  architecture  as  a simple string and
          converts to/from the registry view transparently.

AUTHENTICATION

       Charliecloud does not have configuration files; thus, it has no separate login  subcommand
       to  store  secrets.  Instead,  Charliecloud  will  prompt for a username and password when
       authentication is needed. Note that some repositories refer to  the  secret  as  something
       other than a “password”; e.g., GitLab calls it a “personal access token (PAT)”, Quay calls
       it an “application token”, and nVidia NGC calls it an “API token”.

       For non-interactive authentication, you can use  environment  variables  CH_IMAGE_USERNAME
       and  CH_IMAGE_PASSWORD.  Only  do  this  if you fully understand the implications for your
       specific use case, because it is  difficult  to  securely  store  secrets  in  environment
       variables.

       By  default  for  most  subcommands,  all  registry  access  is  anonymous. To instead use
       authenticated access for everything,  specify  --auth  or  set  the  environment  variable
       $CH_IMAGE_AUTH=yes.  The  exception is push, which always runs in authenticated mode. Even
       for pulling public images, it can be useful  to  authenticate  for  registries  that  have
       per-user  rate  limits,  such  as Docker Hub. (Older versions of Charliecloud started with
       anonymous access, then tried to upgrade to authenticated if it seemed necessary.  However,
       this turned out to be brittle; see issue #1318.)

       The  username  and  password  are  remembered  for  the  life  of the process and silently
       re-offered to the registry if needed. One case when this happens is on push to  a  private
       registry:  many  registries  will  first  offer  a read-only token when ch-image checks if
       something exists, then re-authenticate when upgrading the token to read-write for  upload.
       If  your  site  uses  one-time  passwords  such  as provided by a security device, you can
       specify --password-many to provide a new secret each time.

       These values are not saved persistently, e.g. in a file. Note that we do use normal Python
       variables  for  this  information, without pinning them into physical RAM with mlock(2) or
       any other special treatment, so we cannot guarantee they  will  never  reach  non-volatile
       storage.

          Technical details

                 Most  registries  use  something  called Bearer authentication, where the client
                 (e.g., Charliecloud) includes a token in the headers of every HTTP request.

                 The authorization dance is different from the typical UNIX approach, where there
                 is  a  separate login sequence before any content requests are made.  The client
                 starts by simply making the HTTP  request  it  wants  (e.g.,  to  GET  an  image
                 manifest),  and  if the registry doesn’t like the client’s token (or if there is
                 no token because the client doesn’t have one yet),  it  replies  with  HTTP  401
                 Unauthorized, but crucially it also provides instructions in the response header
                 on how to get a token. The client then follows  those  instructions,  obtains  a
                 token,  re-tries  the  request,  and (hopefully) all is well. This approach also
                 allows a client to upgrade a token  if  needed,  e.g.  when  transitioning  from
                 asking if a layer exists to uploading its content.

                 The distinction between Charliecloud’s anonymous mode and authenticated modes is
                 that it will only ask for anonymous tokens in anonymous mode  and  authenticated
                 tokens   in  authenticated  mode.  That  is,  anonymous  mode  does  involve  an
                 authentication procedure to obtain a token, but this  “authentication”  is  done
                 anonymously. (Yes, it’s confusing.)

                 Registries  also  often reply HTTP 401 when an image does not exist, rather than
                 the seemingly more correct HTTP 404 Not Found.  This  is  to  avoid  information
                 leakage  about  the  existence  of images the client is not allowed to pull, and
                 it’s why Charliecloud never says an image simply does not exist.

STORAGE DIRECTORY

       ch-image maintains state using  normal  files  and  directories  located  in  its  storage
       directory; contents include various caches and temporary images used for building.

       In descending order of priority, this directory is located at:

          -s, --storage DIR
                 Command line option.

          $CH_IMAGE_STORAGE
                 Environment  variable. The path must be absolute, because the variable is likely
                 set in a very different context than when it’s used, which seems error-prone  on
                 what a relative path is relative to.

          /var/tmp/$USER.ch
                 Default.  (Previously,  the  default  was  /var/tmp/$USER/ch-image.  If  a valid
                 storage directory is found at the old default path, ch-image tries to move it to
                 the new default path.)

       Unlike  many  container  implementations,  there  is  no  notion of storage drivers, graph
       drivers, etc., to select and/or configure.

       The storage directory can reside on any single filesystem (i.e., it cannot be split across
       multiple  filesystems).  However, it contains lots of small files and metadata traffic can
       be intense. For example, the Charliecloud test suite uses approximately 400,000 files  and
       directories  in  the  storage  directory  as  of  this  writing.  Place it on a filesystem
       appropriate for this; tmpfs’es such as /var/tmp are a good choice if you have  enough  RAM
       (/tmp is not recommended because ch-run bind-mounts it into containers by default).

       While  you  can  currently  poke  around in the storage directory and find unpacked images
       runnable with ch-run, this is not a  supported  use  case.  The  supported  workflow  uses
       ch-convert to obtain a packed image; see the tutorial for details.

       The storage directory format changes on no particular schedule.  ch-image is normally able
       to upgrade directories produced by a given Charliecloud version up to one year after  that
       version’s release. Upgrades outside this window and downgrades are not supported. In these
       cases, ch-image will refuse  to  run  until  you  delete  and  re-initialize  the  storage
       directory with ch-image reset.

       WARNING:
          Network  filesystems,  especially  Lustre,  are  typically  bad choices for the storage
          directory. This is a site-specific question and your local  support  will  likely  have
          strong opinions.

BUILD CACHE

   Overview
       Subcommands  that  create  images,  such as build and pull, can use a build cache to speed
       repeated operations. That is, an image is created by starting from  the  empty  image  and
       executing a sequence of instructions, largely Dockerfile instructions but also some others
       like “pull” and “import”.  Some instructions are expensive  to  execute  (e.g.,  RUN  wget
       http://slow.example.com/bigfile  or  transferring  data billed by the byte), so it’s often
       cheaper to retrieve their results from cache instead.

       The build  cache  uses  a  relatively  new  Git  under  the  hood;  see  the  installation
       instructions  for  version  requirements.  Charliecloud  implements  workarounds for Git’s
       various storage limitations, so things like file metadata and Git repositories within  the
       image  should  work.  Important  exception: No files named .git* or other Git metadata are
       permitted in the image’s root directory.

       The cache has three modes, enabled, disabled, and a hybrid mode called rebuild  where  the
       cache  is  fully  enabled  for  FROM instructions, but all other operations re-execute and
       re-cache their results. The purpose of rebuild is to do a clean rebuild  of  a  Dockerfile
       atop a known-good base image.

       Enabled mode is selected with --cache or setting $CH_IMAGE_CACHE to enabled, disabled mode
       with --no-cache or disabled, and rebuild mode with --rebuild or rebuild. The default  mode
       is enabled if an appropriate Git is installed, otherwise disabled.

   Compared to other implementations
       Other  container  implementations  typically  use  build  caches  based  on  overlayfs, or
       fuse-overlayfs in unprivileged situations (configured via a “storage driver”). This  works
       by  creating  a  new  tmpfs  for each instruction, layered atop the previous instruction’s
       tmpfs using overlayfs. Each layer can then be tarred up separately  to  form  a  tar-based
       diff.

       The  Git-based  cache  has  two advantages over the overlayfs approach. First, kernel-mode
       overlayfs is only available unprivileged in Linux 5.11 and  higher,  forcing  the  use  of
       fuse-overlayfs  and its accompanying FUSE overhead for unprivileged use cases. Second, Git
       de-duplicates and compresses files in a fairly sophisticated way across the  entire  build
       cache,  not  just between image states with an ancestry relationship (detailed in the next
       section).

       A disadvantage is lowered performance in some cases. Preliminary experiments suggest  this
       performance  penalty  is  relatively modest, and sometimes Charliecloud is actually faster
       than alternatives. We have ongoing experiments to answer this performance question in more
       detail.

   De-duplication and garbage collection
       Charliecloud’s  build  cache  takes  advantage of Git’s file de-duplication features. This
       operates across the entire build cache, i.e., files are de-duplicated no matter  where  in
       the  cache  they  are  found or the relationship between their container images. Files are
       de-duplicated at different times  depending  on  whether  they  are  identical  or  merely
       similar.

       Identical  files  are  de-duplicated at git add time; in ch-image build terms, that’s upon
       committing a successful instruction.  That is, it’s impossible to store two files with the
       same  content  in  the  build  cache.  If you try — say with RUN yum install -y foo in one
       Dockerfile and RUN yum install -y foo bar in another, which are different instructions but
       both  install  RPM  foo’s  files  —  the content is stored once and each copy gets its own
       metadata and a pointer to the content, much like filesystem hard links.

       Similar files, however, are only de-duplicated during Git’s  garbage  collection  process.
       When  files are initially added to a Git repository (with git add), they are stored inside
       the repository as (possibly compressed) individual files, called objects  in  Git  jargon.
       Upon  garbage collection, which happens both automatically when certain parameters are met
       and explicitly with git gc, these files are archived and (re-)compressed together  into  a
       single  file  called  a  packfile. Also, existing packfiles may be re-written into the new
       one.

       During this process, similar files are identified, and each set of similar files is stored
       as one base file plus diffs to recover the others. (Similarity detection seems to be based
       primarily on file size.) This  delta  process  is  agnostic  to  alignment,  which  is  an
       advantage  over  alignment-sensitive  block-level  de-duplicating  filesystems. Exception:
       “Large” files are not compressed or de-duplicated. We use the Git default threshold of 512
       MiB (as of this writing).

       Charliecloud  runs  Git garbage collection at two different times. First, a lighter-weight
       garbage pass runs automatically when the number of loose files (objects)  grows  beyond  a
       limit.  This  limit  is  in  flux as we learn more about build cache performance, but it’s
       quite a bit higher than the Git default.  This garbage runs  in  the  background  and  can
       continue after the build completes; you may see Git processes using a lot of CPU.

       An  important  limitation of the automatic garbage is that large packfiles (again, this is
       in flux, but it’s several GiB) will not be re-packed, limiting the scope of  similar  file
       detection. To address this, a heavier garbage collection can be run manually with ch-image
       build-cache --gc. This will re-pack (and re-write) the entire build cache,  de-duplicating
       all similar files. In both cases, garbage uses all available cores.

       git  build-cache  prints  the specific garbage collection parameters in use, and -v can be
       added for more detail.

   Large file threshold
       Because Git uses content-addressed storage, upon commit, it must read in  full  all  files
       modified  by an instruction. This I/O cost can be a significant fraction of build time for
       some large images. Regular files larger than the experimental  large  file  threshold  are
       stored  outside  the  Git repository, somewhat like Git Large File Storage.  ch-image uses
       hard links to bring large files in and out of images as needed, which is a  fast  metadata
       operation that ignores file content.

       Option  --cache-large  sets  the  threshold  in  MiB;  if  not  set,  environment variable
       CH_IMAGE_CACHE_LARGE is used; if that is not set either, the  default  value  0  indicates
       that no files are considered large.

       There  are two trade-offs. First, large files in any image with the same path, mode, size,
       and mtime (to nanosecond precision if possible) are considered identical,  even  if  their
       content  is  not  actually  identical;  e.g.,  touch(1)  shenanigans can corrupt an image.
       Second, every version of a large file is stored verbatim and uncompressed (e.g.,  a  large
       file  with  a  one-byte  change  will  be  stored  in  full twice), and large files do not
       participate in the build cache’s de-duplication, so more  storage  space  will  likely  be
       used. Unused versions are deleted by ch-image build-cache --gc.

       (Note that Git has an unrelated setting called core.bigFileThreshold.)

   Example
       Suppose we have this Dockerfile:

          $ cat a.df
          FROM alpine:3.17
          RUN echo foo
          RUN echo bar

       On our first build, we get:

          $ ch-image build -t foo -f a.df .
            1. FROM alpine:3.17
          [ ... pull chatter omitted ... ]
            2. RUN echo foo
          copying image ...
          foo
            3. RUN echo bar
          bar
          grown in 3 instructions: foo

       Note  the  dot  after  each instruction’s line number. This means that the instruction was
       executed. You can also see this by the output of the two echo commands.

       But on our second build, we get:

          $ ch-image build -t foo -f a.df .
            1* FROM alpine:3.17
            2* RUN echo foo
            3* RUN echo bar
          copying image ...
          grown in 3 instructions: foo

       Here, instead of being executed, each instruction’s results  were  retrieved  from  cache.
       (Charliecloud uses lazy retrieval; nothing is actually retrieved until the end, as seen by
       the “copying image” message.) Cache hit for each instruction is indicated by  an  asterisk
       (*)  after  the  line  number.   Even for such a small and short Dockerfile, this build is
       noticeably faster than the first.

       We can also try a second,  slightly  different  Dockerfile.  Note  that  the  first  three
       instructions are the same, but the third is different:

          $ cat c.df
          FROM alpine:3.17
          RUN echo foo
          RUN echo qux
          $ ch-image build -t c -f c.df .
            1* FROM alpine:3.17
            2* RUN echo foo
            3. RUN echo qux
          copying image ...
          qux
          grown in 3 instructions: c

       Here,  the  first  two instructions are hits from the first Dockerfile, but the third is a
       miss, so Charliecloud retrieves that state and continues building.

       We can also inspect the cache:

          $ ch-image build-cache --tree
          *  (c) RUN echo qux
          | *  (a) RUN echo bar
          |/
          *  RUN echo foo
          *  (alpine+3.9) PULL alpine:3.17
          *  (root) ROOT

          named images:     4
          state IDs:        5
          commits:          5
          files:          317
          disk used:        3 MiB

       Here there are four named images: a and c  that  we  built,  the  base  image  alpine:3.17
       (written  as  alpine+3.9  because colon is not allowed in Git branch names), and the empty
       base of everything root. Also note how a and c diverge after the last  common  instruction
       RUN echo foo.

BUILD

       Build an image from a Dockerfile and put it in the storage directory.

   Synopsis
          $ ch-image [...] build [-t TAG] [-f DOCKERFILE] [...] CONTEXT

   Description
       Uses  ch-run  -w  -u0 -g0 --no-passwd --unsafe to execute RUN instructions. Note that FROM
       implicitly pulls the base image if needed,  so  you  may  want  to  read  about  the  pull
       subcommand below as well.

       Required argument:

          CONTEXT
                 Path  to  context  directory.  This  is  the  root  of  COPY instructions in the
                 Dockerfile. If a single hyphen (-) is specified: (a) read  the  Dockerfile  from
                 standard  input, (b) specifying --file is an error, and (c) there is no context,
                 so COPY will fail. (See --file for how to provide  the  Dockerfile  on  standard
                 input while also having a context.)

       Options:

          -b, --bind SRC[:DST]
                 For  RUN instructions only, bind-mount SRC at guest DST. The default destination
                 if not specified is to use the same path as  the  host;  i.e.,  the  default  is
                 equivalent  to  --bind=SRC:SRC.  If  DST  does not exist, try to create it as an
                 empty directory, though  images  do  have  ten  directories  /mnt/[0-9]  already
                 available as mount points. Can be repeated.

                 Note: See documentation for ch-run --bind for important caveats and gotchas.

                 Note:  Other instructions that modify the image filesystem, e.g.  COPY, can only
                 access host files from the context directory, regardless of this option.

          --build-arg KEY[=VALUE]
                 Set build-time variable KEY defined by ARG instruction to VALUE.  If  VALUE  not
                 specified, use the value of environment variable KEY.

          -f, --file DOCKERFILE
                 Use  DOCKERFILE  instead  of  CONTEXT/Dockerfile.  If  a  single  hyphen  (-) is
                 specified, read the Dockerfile from  standard  input;  like  docker  build,  the
                 context directory is still available in this case.

          --force[=MODE]
                 Use  unprivileged  build  workarounds  of  mode  MODE,  which can be fakeroot or
                 seccomp (the default). See section “Privilege model” below for details  on  what
                 this does and when you might need it.

          --force-cmd=CMD,ARG1[,ARG2...]
                 If  command  CMD  is found in a RUN instruction, add the comma-separated ARGs to
                 it. For example, --force-cmd=foo,-a,--bar=baz would transform RUN  foo  -c  into
                 RUN  foo  -a  --bar=baz -c. This is intended to suppress validation that defeats
                 --force=seccomp and  implies  that  option.   Can  be  repeated.  If  specified,
                 replaces  (does  not extend) the default suppression options. Literal commas can
                 be escaped with backslash;  importantly  however,  backslash  will  need  to  be
                 protected  from the shell also. Section “Privilege model” below explains why you
                 might need this.

          -n, --dry-run
                 Don’t actually execute any Dockerfile instructions.

          --parse-only
                 Stop after parsing the Dockerfile.

          -t, --tag TAG
                 Name of image to create. If not specified, infer the name:

                 1. If Dockerfile named Dockerfile with an  extension:  use  the  extension  with
                    invalid characters stripped, e.g.  Dockerfile.@FOO.barfoo.bar.

                 2. If  Dockerfile  has  extension  dockerfile:  use  the  basename with the same
                    transformation, e.g. baz.@QUX.dockerfile -> baz.qux.

                 3. If context directory is not /: use its name, i.e. the last component  of  the
                    absolute path to the context directory, with the same transformation,

                 4. Otherwise (context directory is /): use root.

                 If no colon present in the name, append :latest.

   Privilege model
   Overview
       ch-image  is  a  fully  unprivileged  image  builder. It does not use any setuid or setcap
       helper programs, and it does not use configuration files /etc/subuid or /etc/subgid.  This
       contrasts  with  the  “rootless”  or “fakeroot” modes of some competing builders, which do
       require privileged supporting code or utilities.

       Without workarounds provided by --force, this approach does confuse programs  that  expect
       to  have  real  root  privileges,  most  notably  distribution  package  installers.  This
       subsection describes why that happens and what you can do about it.

       ch-image executes all instructions as the normal user who invokes it.  For  RUN,  this  is
       accomplished  with  ch-run arguments including -w --uid=0 --gid=0. That is, your host EUID
       and EGID are both mapped to zero inside the container, and only one  UID  (zero)  and  GID
       (zero)  are  available  inside the container. Under this arrangement, processes running in
       the container for each RUN appear to be running as root, but many privileged system  calls
       will  fail  without  the workarounds described below.  This affects any fully unprivileged
       container build, not just Charliecloud.

       The most common time to see this is installing packages. For example, here is RPM  failing
       to chown(2) a file, which makes the package update fail:

            Updating   : 1:dbus-1.10.24-13.el7_6.x86_64                            2/4
          Error unpacking rpm package 1:dbus-1.10.24-13.el7_6.x86_64
          error: unpacking of archive failed on file /usr/libexec/dbus-1/dbus-daemon-launch-helper;5cffd726: cpio: chown
            Cleanup    : 1:dbus-libs-1.10.24-12.el7.x86_64                         3/4
          error: dbus-1:1.10.24-13.el7_6.x86_64: install failed

       This one is (ironically) apt-get failing to drop privileges:

          E: setgroups 65534 failed - setgroups (1: Operation not permitted)
          E: setegid 65534 failed - setegid (22: Invalid argument)
          E: seteuid 100 failed - seteuid (22: Invalid argument)
          E: setgroups 0 failed - setgroups (1: Operation not permitted)

       Charliecloud provides two different mechanisms to avoid these problems. Both involve lying
       to the containerized process about privileged system calls, but at very  different  levels
       of complexity.

   Workaround mode fakeroot
       This  mode  uses  fakeroot(1)  to  maintain  an elaborate web of deceit that is internally
       consistent. This program intercepts both privileged system calls (e.g., setuid(2)) as well
       as  other system calls whose return values depend on those calls (e.g., getuid(2)), faking
       success for privileged system calls (perhaps making no system call at  all)  and  altering
       return  values  to  be  consistent  with  earlier fake success. Charliecloud automatically
       installs the fakeroot(1) program inside the container  and  then  wraps  RUN  instructions
       having  known  privilege  needs  with  it.  Thus,  this mode is only available for certain
       distributions.

       The advantage of this mode is its consistency; e.g., careful programs that check  the  new
       UID  after  attempting  to  change  it will not notice anything amiss. Its disadvantage is
       complexity: detailed knowledge and procedures for multiple Linux distributions.

       This mode has three basic steps:

          1. After FROM, analyze the image to see what distribution it contains, which determines
             the specific workarounds.

          2. Before  the  user  command  in  the  first RUN instruction where the injection seems
             needed, install fakeroot(1) in the image, if one is not already installed,  as  well
             as  any  other  necessary  initialization commands. For example, we turn off the apt
             sandbox  (for  Debian  Buster)  and  configure  EPEL  but  leave  it  disabled  (for
             CentOS/RHEL).

          3. Prepend  fakeroot  to  RUN instructions that seem to need it, e.g. ones that contain
             apt, apt-get, dpkg for Debian  derivatives  and  dnf,  rpm,  or  yum  for  RPM-based
             distributions.

       RUN instructions that do not seem to need modification are unaffected by this mode.

       The  details  are  specific  to  each distribution. ch-image analyzes image content (e.g.,
       grepping /etc/debian_version) to select a configuration;  see  lib/force.py  for  details.
       ch-image prints exactly what it is doing.

       WARNING:
          Because  of  fakeroot  mode’s complexity, we plan to remove it if seccomp mode performs
          well enough. If you have a situation where fakeroot mode works and  seccomp  does  not,
          please let us know.

   Workaround mode seccomp (default)
       This  mode  uses  the  kernel’s  seccomp(2)  system  call  filtering  to intercept certain
       privileged system calls, do absolutely nothing, and return success to the program.

       The quashed system calls are: capset(2); chown(2) and friends;  mknod(2)  and  mknodat(2);
       and  setuid(2),  setgid(2), and setgroups(2) along with the other system calls that change
       user or group.

       The advantages of this approach is that it’s much simpler, it’s  faster,  it’s  completely
       agnostic  to libc, and it’s mostly agnostic to distribution. The disadvantage is that it’s
       a very lazy liar; even the most cursory consistency  checks  will  fail,  e.g.,  getuid(2)
       after setuid(2).

       While  this  mode  does  not  provide  consistency,  it  does offer a hook to help prevent
       programs asking for consistency. For  example,  apt-get  -o  APT::Sandbox::User=root  will
       prevent  apt-get  from  attempting  to  drop  privileges,  which it verifies, exiting with
       failure if the correct IDs are not found (which they won’t be under this  approach).  This
       can   be   expressed   with  --force-cmd=apt-get,-o,APT::Sandbox::User=root,  though  this
       particular case is  built-in  and  does  not  need  to  be  specified.  The  full  default
       configuration,  which  is applied regardless of the image distribution, can be examined in
       the source file force.py. If any --force-cmd are specified,  this  replaces  (rather  than
       extends) the default configuration.

       Note  that because the substitutions are a simple regex with no knowledge of shell syntax,
       they can cause unwanted modifications. For example, RUN apt-get install -y apt-get will be
       run   as   /bin/sh   -c   "apt-get   -o  APT::Sandbox::User=root  install  -y  apt-get  -o
       APT::Sandbox::User=root". One workaround is to add escape syntax transparent to the shell;
       e.g., RUN apt-get install -y apt-get.

       This mode executes all RUN instructions with the seccomp(2) filter and has no knowledge of
       which instructions actually used the intercepted  system  calls.  Therefore,  the  printed
       “instructions  modified”  number  is  only  a count of instructions with a hook applied as
       described above.

   Compatibility with other Dockerfile interpreters
       ch-image is an independent  implementation  and  shares  no  code  with  other  Dockerfile
       interpreters.  It  uses  a formal Dockerfile parsing grammar developed from the Dockerfile
       reference documentation and miscellaneous other sources, which  you  can  examine  in  the
       source code.

       We  believe  this  independence  is  valuable  for  several  reasons.  First, it helps the
       community examine Dockerfile syntax and semantics critically, think rigorously about  what
       is  really  needed,  and build a more robust standard.  Second, it yields disjoint sets of
       bugs (note that Podman, Buildah, and Docker all share the same Dockerfile parser).  Third,
       because  it is a much smaller code base, it illustrates how Dockerfiles work more clearly.
       Finally, it allows straightforward extensions if needed to support scientific computing.

       ch-image tries hard to be compatible with Docker and  other  interpreters,  though  as  an
       independent implementation, it is not bug-compatible.

       The  following  subsections  describe  differences  from  the Dockerfile reference that we
       expect to be approximately permanent. For not-yet-implemented features and  bugs  in  this
       area, see related issues on GitHub.

       None  of these are set in stone. We are very interested in feedback on our assessments and
       open questions. This helps us prioritize new features and revise our thinking  about  what
       is needed for HPC containers.

   Context directory
       The  context  directory  is  bind-mounted  into the build, rather than copied like Docker.
       Thus, the size of the context is immaterial, and the build  reads  directly  from  storage
       like  any  other local process would. However, you still can’t access anything outside the
       context directory.

   Variable substitution
       Variable substitution happens for all instructions,  not  just  the  ones  listed  in  the
       Dockerfile reference.

       ARG  and  ENV  cause  cache  misses  upon  definition, in contrast with Docker where these
       variables miss upon use, except for certain  cache-excluded  variables  that  never  cause
       misses, listed below.

       Note that ARG and ENV have different syntax despite very similar semantics.

       ch-image  passes  the  following  proxy  environment variables in to the build. Changes to
       these variables do not cause a cache miss. They do not  require  an  ARG  instruction,  as
       documented  in  the  Dockerfile  reference.  Unlike  Docker,  they  are  available  if the
       same-named environment variable is defined; --build-arg is not required.

          HTTP_PROXY
          http_proxy
          HTTPS_PROXY
          https_proxy
          FTP_PROXY
          ftp_proxy
          NO_PROXY
          no_proxy

       In addition to those listed in the Dockerfile reference, these environment  variables  are
       passed through in the same way:

          SSH_AUTH_SOCK
          USER

       Finally, these variables are also pre-defined but are unrelated to the host environment:

          PATH=/ch/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
          TAR_OPTIONS=--no-same-owner

   ARG
       Variables set with ARG are available anywhere in the Dockerfile, unlike Docker, where they
       only work in FROM instructions, and possibly in other ARG before the first FROM.

   FROM
       The FROM instruction accepts option --arg=NAME=VALUE, which serves the same purpose as the
       ARG instruction. It can be repeated.

   LABEL
       The LABEL instruction accepts key=value pairs to add metadata for an image. Unlike Docker,
       multiline values are not supported; see issue #1512.  Can be repeated.

   COPY
       Especially for people used to UNIX cp(1), the semantics of the Dockerfile COPY instruction
       can be confusing.

       Most  notably,  when  a source of the copy is a directory, the contents of that directory,
       not the directory itself, are copied. This is documented, but it’s a real  gotcha  because
       that’s  not what cp(1) does, and it means that many things you can do in one cp(1) command
       require multiple COPY instructions.

       Also, the reference documentation is incomplete. In our experience, Docker also behaves as
       follows; ch-image does the same in an attempt to be bug-compatible.

       1. You can use absolute paths in the source; the root is the context directory.

       2. Destination directories are created if they don’t exist in the following situations:

          1. If the destination path ends in slash. (Documented.)

          2. If  the  number  of  sources  is  greater  than 1, either by wildcard or explicitly,
             regardless of whether the destination ends in slash. (Not documented.)

          3. If there is a single source and it is a directory. (Not documented.)

       3. Symbolic links behave differently depending on how deep in the copied  tree  they  are.
          (Not documented.)

          1. Symlinks  at  the  top  level — i.e., named as the destination or the source, either
             explicitly or by wildcards — are dereferenced. They are followed, and whatever  they
             point to is used as the destination or source, respectively.

          2. Symlinks at deeper levels are not dereferenced, i.e., the symlink itself is copied.

       4. If  a  directory  appears at the same path in source and destination, and is at the 2nd
          level or deeper, the source directory’s metadata (e.g., permissions) are copied to  the
          destination directory. (Not documented.)

       5. If  an  object  appears  in both the source and destination, and is at the 2nd level or
          deeper, and is of different types in the source and destination, then the source object
          will  overwrite  the destination object. (Not documented.) For example, if /tmp/foo/bar
          is a regular file, and /tmp is the context directory,  then  the  following  Dockerfile
          snippet  will result in a file in the container at /foo/bar (copied from /tmp/foo/bar);
          the directory and all its contents will be lost.

                 RUN mkdir -p /foo/bar && touch /foo/bar/baz
                 COPY foo /foo

       We expect the following differences to be permanent:

       • Wildcards use Python glob semantics, not the Go semantics.

       • COPY --chown is ignored, because it doesn’t make sense in an unprivileged build.

   Features we do not plan to support
       • Parser directives are not supported. We have not identified a need for any of them.

       • EXPOSE: Charliecloud does not use the network namespace, so containerized processes  can
         simply listen on a host port like other unprivileged processes.

       • HEALTHCHECK: This instruction’s main use case is monitoring server processes rather than
         applications. Also, implementing it requires a container  supervisor  daemon,  which  we
         have no plans to add.

       • MAINTAINER is deprecated.

       • STOPSIGNAL  requires  a  container  supervisor daemon process, which we have no plans to
         add.

       • USER does not make sense for unprivileged builds.

       • VOLUME: This instruction is not currently supported. Charliecloud has good  support  for
         bind mounts; we anticipate that it will continue to focus on that and will not introduce
         the volume management features that Docker has.

   Examples
       Build image bar using ./foo/bar/Dockerfile and context directory ./foo/bar:

          $ ch-image build -t bar -f ./foo/bar/Dockerfile ./foo/bar
          [...]
          grown in 4 instructions: bar

       Same, but infer the image name and Dockerfile from the context directory path:

          $ ch-image build ./foo/bar
          [...]
          grown in 4 instructions: bar

       Build using humongous vendor compilers you want to bind-mount instead of  installing  into
       the image:

          $ ch-image build --bind /opt/bigvendor:/opt .
          $ cat Dockerfile
          FROM centos:7

          RUN /opt/bin/cc hello.c
          #COPY /opt/lib/*.so /usr/local/lib   # fail: COPY doesn’t bind mount
          RUN cp /opt/lib/*.so /usr/local/lib  # possible workaround
          RUN ldconfig

BUILD-CACHE

          $ ch-image [...] build-cache [...]

       Print  basic  information  about the cache. If -v is given, also print some Git statistics
       and the Git repository configuration.

       If any of the following options are given, do the corresponding operation before printing.
       Multiple options can be given, in which case they happen in this order.

          --dot  Create  a  DOT  export  of  the tree named ./build-cache.dot and a PDF rendering
                 ./build-cache.pdf. Requires graphviz and git2dot.

          --gc   Run Git garbage collection  on  the  cache,  including  full  de-duplication  of
                 similar  files.  This  will  immediately  remove all cache entries not currently
                 reachable from a named branch (which is likely to cause corruption if the  build
                 cache is being accessed concurrently by another process). The operation can take
                 a long time on large caches.

          --reset
                 Clear and re-initialize the build cache.

          --tree Print a text tree of the cache using Git’s git log --graph  feature.  If  -v  is
                 also given, the tree has more detail.

DELETE

          $ ch-image [...] delete IMAGE_GLOB [IMAGE_GLOB ... ]

       Delete the image(s) described by each IMAGE_GLOB from the storage directory (including all
       build stages).

       IMAGE_GLOB can be either  a  plain  image  reference  or  an  image  reference  with  glob
       characters  to  match multiple images. For example, ch-image delete 'foo*' will delete all
       images whose names start with foo.  Multiple images and/or globs can also be  given  in  a
       single command line.

       Importantly,  this  sub-command  does  not  also  remove  the  image from the build cache.
       Therefore, it can be used to reduce the size of the storage  directory,  trading  off  the
       time needed to retrieve an image from cache.

       WARNING:
          Glob  characters  must  be  quoted  or  otherwise  protected from the shell, which also
          desires to interpret them and will do so incorrectly.

GESTALT

          $ ch-image [...] gestalt [SELECTOR]

       Provide information about the configuration and available features of ch-image. End  users
       generally will not need this; it is intended for testing and debugging.

       SELECTOR is one of:

          • bucache.  Exit  successfully  if the build cache is available, unsuccessfully with an
            error message otherwise. With -v, also print version information about dependencies.

          • bucache-dot.  Exit  successfully  if  build  cache  DOT   trees   can   be   written,
            unsuccessfully  with  an  error  message  otherwise.  With  -v,  also  print  version
            information about dependencies.

          • python-path. Print the path to the Python interpreter in use and exit successfully.

          • storage-path. Print the storage directory path and exit successfully.

LIST

       Print information about images. If no argument given, list the images in builder storage.

   Synopsis
          $ ch-image [...] list [-l] [IMAGE_REF]

   Description
       Optional argument:

          -l, --long
                 Use long format (name, last change timestamp) when listing images.

          -u, --undeletable
                 List images that can be undeleted. Can also be spelled --undeleteable.

          IMAGE_REF
                 Print details of what’s known about IMAGE_REF, both locally and  in  the  remote
                 registry, if any.

   Examples
       List images in builder storage:

          $ ch-image list
          alpine:3.17 (amd64)
          alpine:latest (amd64)
          debian:buster (amd64)

       Print details about Debian Buster image:

          $ ch-image list debian:buster
          details of image:    debian:buster
          in local storage:    no
          full remote ref:     registry-1.docker.io:443/library/debian:buster
          available remotely:  yes
          remote arch-aware:   yes
          host architecture:   amd64
          archs available:     386       bae2738ed83
                               amd64     98285d32477
                               arm/v7    97247fd4822
                               arm64/v8  122a0342878

       For  remotely  available images like Debian Buster, the associated digest is listed beside
       each available architecture. Importantly, this feature does not provide the  hash  of  the
       local image, which is only calculated on push.

IMPORT

          $ ch-image [...] import PATH IMAGE_REF

       Copy the image at PATH into builder storage with name IMAGE_REF. PATH can be:

       • an image directory

       • a tarball with no top-level directory (a.k.a. a “tarbomb”)

       • a standard tarball with one top-level directory

       If  the  imported  image  contains Charliecloud metadata, that will be imported unchanged,
       i.e., images exported from ch-image builder storage will be  functionally  identical  when
       re-imported.

       WARNING:
          Descendant  images (i.e., FROM the imported IMAGE_REF) are linked using IMAGE_REF only.
          If a new image is imported under a new IMAGE_REF, all instructions descending from that
          IMAGE_REF will still hit, even if the new image is different.

PULL

       Pull  the  image described by the image reference IMAGE_REF from a repository to the local
       filesystem.

   Synopsis
          $ ch-image [...] pull [...] IMAGE_REF [DEST_REF]

       See the FAQ for the gory details on specifying image references.

   Description
       Destination:

          DEST_REF
                 If  specified,  use  this  as  the  destination  image  reference,  rather  than
                 IMAGE_REF.  This  lets  you  pull  an  image  with a complicated reference while
                 storing it locally with a simpler one.

       Options:

          --last-layer N
                 Unpack only N layers, leaving an incomplete image. This option is  intended  for
                 debugging.

          --parse-only
                 Parse  IMAGE_REF, print a parse report, and exit successfully without talking to
                 the internet or touching the storage directory.

       This script does a fair amount of validation and  fixing  of  the  layer  tarballs  before
       flattening  in  order to support unprivileged use despite image problems we frequently see
       in the wild. For example, device files are ignored, and file and directory permissions are
       increased  to  a  minimum  of  rwx------  and  rw------- respectively. Note, however, that
       symlinks pointing outside the image are permitted, because they  are  not  resolved  until
       runtime within a container.

       The  following  metadata  in the pulled image is retained; all other metadata is currently
       ignored. (If you have a need for additional metadata, please let us know!)

          • Current working directory set with WORKDIR is effective in downstream Dockerfiles.

          • Environment variables set with ENV are effective in downstream Dockerfiles  and  also
            written to /ch/environment for use in ch-run --set-env.

          • Mount  point directories specified with VOLUME are created in the image if they don’t
            exist, but no other action is taken.

       Note that some images (e.g., those with a “version 1 manifest”) do not contain metadata. A
       warning is printed in this case.

   Examples
       Download  the  Debian  Buster  image  matching the host’s architecture and place it in the
       storage directory:

          $ uname -m
          aarch32
          pulling image:    debian:buster
          requesting arch:  arm64/v8
          manifest list: downloading
          manifest: downloading
          config: downloading
          layer 1/1: c54d940: downloading
          flattening image
          layer 1/1: c54d940: listing
          validating tarball members
          resolving whiteouts
          layer 1/1: c54d940: extracting
          image arch: arm64
          done

       Same, specifying the architecture explicitly:

          $ ch-image --arch=arm/v7 pull debian:buster
          pulling image:    debian:buster
          requesting arch:  arm/v7
          manifest list: downloading
          manifest: downloading
          config: downloading
          layer 1/1: 8947560: downloading
          flattening image
          layer 1/1: 8947560: listing
          validating tarball members
          resolving whiteouts
          layer 1/1: 8947560: extracting
          image arch: arm (may not match host arm64/v8)

PUSH

       Push the image described by the image reference IMAGE_REF from the local filesystem  to  a
       repository.

   Synopsis
          $ ch-image [...] push [--image DIR] IMAGE_REF [DEST_REF]

       See the FAQ for the gory details on specifying image references.

   Description
       Destination:

          DEST_REF
                 If  specified,  use  this  as  the  destination  image  reference,  rather  than
                 IMAGE_REF. This lets you push to a repository without permanently adding  a  tag
                 to the image.

       Options:

          --image DIR
                 Use  the  unpacked  image  located  at  DIR  rather than an image in the storage
                 directory named IMAGE_REF.

       Because Charliecloud is fully unprivileged, the owner and group of files in its images are
       not  meaningful  in  the  broader ecosystem. Thus, when pushed, everything in the image is
       flattened to  user:group  root:root.  Also,  setuid/setgid  bits  are  removed,  to  avoid
       surprises if the image is pulled by a privileged container implementation.

   Examples
       Push a local image to the registry example.com:5000 at path /foo/bar with tag latest. Note
       that in this form, the local image must be named to match that remote reference.

          $ ch-image push example.com:5000/foo/bar:latest
          pushing image:   example.com:5000/foo/bar:latest
          layer 1/1: gathering
          layer 1/1: preparing
          preparing metadata
          starting upload
          layer 1/1: a1664c4: checking if already in repository
          layer 1/1: a1664c4: not present, uploading
          config: 89315a2: checking if already in repository
          config: 89315a2: not present, uploading
          manifest: uploading
          cleaning up
          done

       Same, except use local image alpine:3.17. In this form, the local image name does not have
       to match the destination reference.

          $ ch-image push alpine:3.17 example.com:5000/foo/bar:latest
          pushing image:   alpine:3.17
          destination:     example.com:5000/foo/bar:latest
          layer 1/1: gathering
          layer 1/1: preparing
          preparing metadata
          starting upload
          layer 1/1: a1664c4: checking if already in repository
          layer 1/1: a1664c4: not present, uploading
          config: 89315a2: checking if already in repository
          config: 89315a2: not present, uploading
          manifest: uploading
          cleaning up
          done

       Same, except use unpacked image located at /var/tmp/image rather than an image in ch-image
       storage. (Also, the sole layer is already present in the  remote  registry,  so  we  don’t
       upload it again.)

          $ ch-image push --image /var/tmp/image example.com:5000/foo/bar:latest
          pushing image:   example.com:5000/foo/bar:latest
          image path:      /var/tmp/image
          layer 1/1: gathering
          layer 1/1: preparing
          preparing metadata
          starting upload
          layer 1/1: 892e38d: checking if already in repository
          layer 1/1: 892e38d: already present
          config: 546f447: checking if already in repository
          config: 546f447: not present, uploading
          manifest: uploading
          cleaning up
          done

RESET

          $ ch-image [...] reset

       Delete all images and cache from ch-image builder storage.

UNDELETE

          $ ch-image [...] undelete IMAGE_REF

       If  IMAGE_REF  has been deleted but is in the build cache, recover it from the cache. Only
       available when the cache is enabled, and will not overwrite IMAGE_REF if it exists.

ENVIRONMENT VARIABLES

       CH_IMAGE_USERNAME, CH_IMAGE_PASSWORD
              Username and password for registry authentication. See important caveats in section
              “Authentication” above.

       CH_LOG_FILE
              If set, append log chatter to this file, rather than standard error. This is useful
              for debugging situations where standard error is consumed or lost.

              Also sets verbose mode if not already set (equivalent to --verbose).

       CH_LOG_FESTOON
              If set, prepend PID and timestamp to logged chatter.

REPORTING BUGS

       If Charliecloud was obtained from your Linux distribution,  use  your  distribution’s  bug
       reporting procedures.

       Otherwise, report bugs to: https://github.com/hpc/charliecloud/issues

SEE ALSO

       charliecloud(7)

       Full documentation at: <https://hpc.github.io/charliecloud>

COPYRIGHT

       2014–2022, Triad National Security, LLC and others