NAME

       dmtcp - Distributed MultiThreaded Checkpointing

SYNOPSIS

       dmtcp_coordinator [port]

       dmtcp_checkpoint command [args...]

       dmtcp_restart ckpt_FILE1.dmtcp [ckpt_FILE2.dmtcp...]

       dmtcp_command coordinatorCommand

       dmtcp_inspector [-o<ofile>] [-t<tool>] [-cdaznh] <ckpt1.dmtcp> [ckpt2.dmtcp...]

       mtcp_restart FILE.mtcp

DESCRIPTION

       DMTCP  is a tool to transparently checkpointing the state of an arbitrary group of programs spread across
       many machines and connected by sockets. It does not modify the user's program nor the  operating  system.
       MTCP is a standalone component of DMTCP available as a checkpointing library for a single process.

OPTIONS

       For  each  command,  the --help or -h flag will show the command-line options.  Most command line options
       can also be controlled through environment variables.  These can be set in bash with "export  NAME=value"
       or in tcsh with "setenv NAME value".

       DMTCP_CHECKPOINT_INTERVAL=integer
              Time  in  seconds  between  automatic  checkpoints.  Checkpoints can also be initiated manually by
              typing 'c' into the coordinator. (default: 0, disabled; dmtcp_coordinator only)

       DMTCP_HOST=string
              Hostname where the cluster-wide coordinator is  running.  (default:  localhost;  dmtcp_checkpoint,
              dmtcp_restart only)

       DMTCP_PORT=integer
              The port the cluster-wide coordinator listens on. (default: 7779)

       DMTCP_GZIP=(1|0)
              Set  to  "0"  to  disable  compression  of  checkpoint  images.  (default: 1, compression enabled;
              dmtcp_checkpoint only) WARNING:  gzip adds seconds.  Without gzip, ckpt/restart is often less than
              1 s

       DMTCP_CHECKPOINT_DIR=path
              Directory to store checkpoint images in. (default: ./)

       DMTCP_SIGCKPT=integer
              Internal signal number to use for checkpointing.  Must not be used by the user program.  (default:
              SIGUSR2; dmtcp_checkpoint only)

DMTCP_COORDINATOR

       Each computation to be checkpointed must include a DMTCP coordinator process.  One can explicitly start a
       coordinator  through  dmtcp_coordinator,  or  allow  one to be started implicitly in background by either
       dmtcp_checkpoint or dmtcp_restart to operate.  The address of the unique coordinator should be  specified
       by  dmtcp_checkpoint,  dmtcp_restart, and dmtcp_command either through the --host and --port command-line
       flags or through the the DMTCP_HOST and DMTCP_PORT environment variables.  If neither is given, the host-
       port  pair  defaults  to  localhost-7779.  The host-port pair associated with a particular coordinator is
       given by the command-line flags used in the dmtcp_coordinator command, or the environment variables  then
       in effect, or the default of localhost-7779.

       The  coordinator  is  stateless  and  is  not checkpointed.  On restart, one can use an existing or a new
       coordinator.  Multiple computations under DMTCP control can coexist by  providing  a  unique  coordinator
       (with a unique host-port pair) for each such computation.

       The  coordinator  initiates a checkpoint for all processes in its computation group.  Checkpoints can be:
       performed automatically on an interval (see DMTCP_CHECKPOINT_INTERVAL above); or  initiated  manually  on
       the  standard  input of the coordinator (see next paragraph); or initiated directly under program control
       by the comptuation through the dmtcpaware API (see below).

       The coordinator accepts the following commands on its standard input.  Each command should be followed by
       the <return> key.  The commands are:
         l : List connected nodes
         s : Print status message
         c : Checkpoint all nodes
         f : Force a restart even if there are missing nodes (debugging)
         k : Kill all nodes
         q : Kill all nodes and quit
         ? : Show this message

       Coordinator commands can also be issued remotely using dmtcp_command.

DMTCP_INSPECTOR

       dmtcp_inspector  is  a  tool  for  offline  checkpoint  analysis.  It  provides  information about socket
       connections and parent-child relations between processes of a  distributed  program.  The  output  is  in
       graphviz  package  format and can be rendered using graphviz tools like dot, neato, twopi, circo, fdp and
       sfdp.  Command line options are following:

       -o, --out <file>
              Write output to <file>

       -t, --tool
              Graphviz tool to use. By default no graphviz and output is in dot-like format

       -c, --cred
              Add information about parent-child relations on the graph

       -d, --no-sock
              Remove information about socket connections

       -a, --sock-all
              Add verbose information about socket connections

       -z, --sock-half
              Also represent half-connections (when some *.dmtcp files are missed

       -n, --node
              Verbose node names indication

       -h, --help
              Display this help

EXAMPLE USAGE

       1. In a separate terminal window, start the dmtcp_coodinator.
              (See previous section.)

               dmtcp_coordinator

       2. In separate terminal(s), replace each command(s) with "dmtcp_checkpoint
              [command]".  The checkpointed program will connect to the coordinator specified by DMTCP_HOST  and
              DMTCP_PORT.   New  threads  will  be  checkpointed  as  part of the process.  Child processes will
              automatically be checkpointed.  Remote processes started via ssh will automatically  checkpointed.
              (Internally, DMTCP modifies the ssh command line to call dmtcp_checkpoint on the remote host.)

               dmtcp_checkpoint ./myprogram

       3. To manually initiate a checkpoint, either run the command below
              or  type "c" followed by <return> into the coordinator.  Checkpoint files for each process will be
              written to DMTCP_CHECKPOINT_DIR. The dmtcp_coordinator will write "dmtcp_restart_script.sh" to its
              working  directory.   This  script  contains  the  necessary calls to dmtcp_restart to restart the
              entire computation, including remote processes created via ssh.

                   dmtcp_command -c
              OR:  dmtcp_command --checkpoint

       4. To restart, one should execute dmtcp_restart_script.sh, which is
              created by the dmtcp_coordinator in its working directory at  the  time  of  checkpoint.  One  can
              optionally  edit  this  script  to  migrate  processes  to  different hosts.  By default, only one
              restarted process will be restarted in the foreground and receive the standard input.  The  script
              may be edited to choose which process will be restarted in the foreground.

               ./dmtcp_restart_script.sh

DMTCPAWARE API

       DMTCP provides a programming interface to allow checkpointed applications to interact with dmtcp.  In the
       source distribution, see dmtcpaware/dmtcpaware.h for the functions available.  See test/dmtcpaware[123].c
       for three example applications.  For an example of its usage, try:

        cd test; rm dmtcpaware1; make dmtcpaware1; ./autotest -v dmtcpaware1

       The  user  application  should  link  with  libdmtcpaware.so  (-ldmtcpaware)  and  use  the  header  file
       dmtcp/dmtcpaware.h.

       The file utils/dmtcp.py in the source distribution provides an example python binding for the  dmtcpaware
       interface.

DMTCP PLUGIN MODULES

       The  source  distribution  includes  a top-level plugin directory, with examples of how to write a plugin
       module for DMTCP.  Further examples are in the test/plugin directory.  The plugin feature adds three  new
       user-programmable  capabilities.  A plugin may: add wrappers around system calls; take special actions at
       during certain events (e.g. pre-checkpoint, resume/post-checkpoint, restart); and  may  insert  key-value
       pairs  into a database at restart time that is then available to be queried by the restarted processes of
       a computation.  (The events available to the plugin feature form a superset of the events available  with
       the  dmtcpaware  interface.)   One  or  more  plugins  are invoked via a list of colon-separated absolute
       pathnames.

         dmtcp_checkpoint --with-plugin PLUGIN1[:PLUGIN2]...

RETURN CODE

       A target program under DMTCP control normally returns the same return code as if executed without  DMTCP.
       However, if DMTCP fails (as opposed to the target program failing), DMTCP returns a DMTCP-specific return
       code, rc (or rc+1, rc+2 for two special cases), where rc is the integer value of the environment variable
       DMTCP_FAIL_RC if set, or else the default value, 99.

SEE ALSO

       Full documentation is available from http://dmtcp.sourceforge.net/

AUTHORS

       DMTCP  and  its  standalone single-process compontent MTCP (MultiThreaded CheckPointing) were created and
       are maintained by Jason Ansel, Kapil Arya, Gene Cooperman, Artem  Y.  Polyakov,  Mike  Rieker,  Ana-Maria
       Visan,  and  a  series  of  newer contributors including Alex Brick, Tyler Denniston, Rohan Garg, Gregory
       Kerr, and others.