Ubuntu Manpage: pipexec - create a directed graph of processes and pipes

NAME

       pipexec - create a directed graph of processes and pipes

SYNOPSIS

       pipexec [OPTION]... [PROCESS DESCRIPTION]... [PIPE DESCRIPTION]...

DESCRIPTION

       pipexec  creates an arbitrary network (directed graph) of processes and pipes in between -
       even cycles are possible.  It overcomes the shortcomings of shells that are typically only
       able to create non cyclic trees.

       pipexec also monitors all it's child processes and is able to restart the whole network of
       processes and pipes if one crashes.  Therefore pipexec can be used in SYSV-init or systemd
       configuration to run a network of processes.

OPTIONS

       -h     print help and version information

       -l logfd
              use  the  given file descriptor for text logging.  If a 's' is specified, syslog is
              used.  Example: Specifying '2' means log to stderr.

       -j logfd
              use the given file descriptor for json logging.  If a 's' is specified,  syslog  is
              used.   Example: Specifying '2' means log to stderr.  As this is meant to be parsed
              by other programs, this is an official and supported interface which  is  described
              in the JSON LOGGING chapter.

       -p pidfile
              with  pipexec  it  is  possible  to handle pipes within SYSV-init scripts.  In some
              environments (e.g. RHEL6, Debian7) the start and stop routines need a pid file.  If
              this  option is given, pipexec writes its own pid into the file shortly after start
              of pipexec.

       -k     if one sub-process (child) gets killed and this options is given,  all  other  sub-
              processes are also killed.  Afterwards all processes are restarted.

       -s sleep_time
              the  time  interval in seconds before a restart.  This option makes only sense when
              also the '-k' option is specified.

BACKGROUND

       Inside a shell it is possible  to  start  processes  and  redirect  the  output  to  other
       processes.

       Example:
           cat Chap1.txt Chap2.txt | grep bird | wc -l

       Three  processes  are  created:  the standard output (file descriptor (fd) 1) of the 'cat'
       process is connected to the standard input (fd 0) of the 'grep' command, and the  standard
       output  of  the  'grep'  command  is  connected  to  the standard input (fd 0) of the 'wc'
       process.

       Please note that  the  assignment  between  names  and  file  descriptor  number  is  pure
       historical and has no technical background.

       Example:
           find / 1> >(grep .txt) 2> >(wc >/tmp/w.log)

       In  this  more  complex  example,  the  fd 1 of the 'find' process is connected to fd 0 of
       'grep' and fd 2 is connected to fd 0 of 'wc'.

       The limitation using this way of specifying  processes  and  pipes  is,  that  it  is  not
       possible  to have any cycles.  It is impossible to e.g. pass a fd of 'wc' either to 'grep'
       or to 'find'.

       pipexec overcomes these limitations.  It makes it possible to link any two arbitrary  file
       descriptors in a set of processes.

USAGE

When building up a network of processes and pipes, there is the need to specify each
element separately.

The processes will be the nodes in the network (directed graph), the connections of the
file descriptors between to processes are the edges. Each node (process) has a unique
name assigned to it. This makes it possible to differentiate between using the same
command more than once.

The format of specifying a process is
[ NAME /path/to/command arg1 arg2 ... argN ]

The first parameter 'NAME' must be a unique name. The second parameter must be the full
path of the command to execute. Please note that always the full path must be specified,
there is no PATH environment variable handling (execv(2) is used internally to span new
processes). The following parameters are the parameters passed to the command.

The whole definition must be enclosed in square brackets. The square brackets must be
given separately - before and after them must be a space.

The format of specifying a pipe between processes is
{NAME_1:FD1>NAME_2:FD2}

Example
{LS:1>GREP:0}

The names are the names of the processes, the numbers are the number of the file
descriptor that should be used to build the pipe in between. When using pipexec from a
shell (like bash) there is the need to escape the brackets or use quotation marks.

JSON LOGGING

pipexec can log in JSON format. This is an official supported interface which is defined
in this chapter. This can be seen as stable as no changes will be made during small
version upgrades. Of course additions will be made also within minor upgrades, but this
should be fine because of the underlaying JSON format.

Each JSON log object (e.g. line) will be in a separate line.

The following keys are always present:

timestamp
The timestamp is the time(0) (seconds since epoch) when the log in generated.

pipexec_pid
The pid of the pipexec process.

id An id that specifies defined log objects.

type An indicator in which internal module this log was generated.

serverity
One of debug, info, warning, error.

message
A short message describing the event.

Depending on the id there might be additional fields which are described in the next
section.

id = 0 id of 0 is a special case: this is used for internal logs only. The logs are not
documented and may change at any time. The information can be used to get an idea
what currently happens in pipexec.

id = 1 This gives information about the process ids (pids) of the commands. The field
command contains the command (i.e. the part in the command before the colon. The
field command_pid contains the pid of the command.

id = 2 This log message is emitted when a child (command) exits. It contains some
information about the exit status and termination reason of the command. The field
command_pid contains the pid of the command which just terminated. The field status
is the value which is set by waitpid(2). normal_exit, child_status and
child_signaled are WIFEXITED, WEXITSTATUS and WIFSIGNALED of the status
respectively.

RETURN

       pipexec returns 1 if any of the child processes fails else 0 is returned.

EXAMPLES

       The shell command
           cat Chap1.txt Chap2.txt | grep bird | wc -l

       is equivalent to
           pipexec [ CAT /bin/cat Chap1.txt Chap2.txt ] \
             [ GREP /usr/bin/grep bird ] [ WC /usr/bin/wc -l ] \
             "{CAT:1>GREP:0}" "{GREP:1>WC:0}"

       The pipexec equivalent is longer and more complex in this example.  But pipexec can  build
       cycles that are impossible within a shell:
           pipexec [ A /bin/cmd1 ] [ B /bin/cmd2 ] "{A:1>B:0}" "{B:1>A:0}"

       When using json log, you get output like:
       {"timestamp":1655706460,"pipexec_pid":42850,"id":1,"type":"exec","serverity":"info","message":"New child forked","command":"A","command_pid":"42851"}
       {"timestamp":1655706886,"pipexec_pid":42869,"id":2,"type":"tracing","serverity":"info","message":"child exit","command_pid":"42870","status":"1","normal_exit":"1","child_status":"1","child_signaled":"0"}

       For more examples see the ptee(1) and peet(1) man pages.

AUTHOR

       Written by Andreas Florath (andreas@florath.net)

COPYRIGHT

       Copyright  ©  2015,2022 by Andreas Florath (andreas@florath.net).  License GPLv2+: GNU GPL
       version 2 or later <http://gnu.org/licenses/gpl.html>.