xenial (1) include_server.1.gz

Provided by: distcc-pump_3.1-6.2_amd64 bug

NAME

       include_server.py - conservative approximation of include dependencies for C/C++

SYNOPSIS

       include_server --port INCLUDE_SERVER_PORT [OPTIONS]

DESCRIPTION

       include_server.py  starts an include server process.  This process answers queries from distcc(1) clients
       about what files to include in C/C++ compilations. The include_server.py  command  itself  terminates  as
       soon as the include server has been spawned.

       The  INCLUDE_SERVER_PORT  argument  is  the  name  of  a socket used for all communication between distcc
       clients and the include server.  The distcc-pump(1)  command  is  responsible  for  creating  the  socket
       location,  for  passing  it  to this script, and for passing it to all distcc clients via the environment
       variable named INCLUDE_SERVER_PORT.

       The protocol used by the include server uses distcc's RPC implementation. Each distcc request consists of
       (1) the current directory and (2) the list of arguments of the compilation command.

       If the include server is able to process the request, then it answers the distcc client by sending a list
       of filepaths.  The filepaths are those of the compressed source and header files found  to  be  necessary
       for  compilation  through  include analysis.  The list also comprises symbolic links and even dummy files
       needed for the compilation server to construct an accurate replica of the parts of the filesystem  needed
       for  compilation.   In this way, a needed header file like /path/foo.h is compressed, renamed, and stored
       in a temporary location, such  as  /dev/shm/tmpiAvfGv.include_server-9368-1/path/foo.h.lzo.   The  distcc
       client  will  pass  these  files  on to a compilation server, where they will be uncompressed and mounted
       temporarily.

       If the include server is not able to process the request, then it returns the empty list  to  the  distcc
       client.

       There are two kinds of failures that relate to the include server. The include server may fail to compute
       the includes or fail in other ways, see section INCLUDE SERVER SYMPTOMS.  Also, the  compilation  on  the
       remove server may fail due to inadequacy of the calculated include closure, but then succeed when locally
       retried, see section DISTCC DISCREPANCY SYMPTOMS.

OPTION SUMMARY

       The following options are understood by include_server.py.

       -dPAT, --debug_pattern=PAT
              Bit vector for turning on warnings and debugging
                  1 = warnings
                  2 = trace some functions
               other powers of two: see include_server/basics.py.

       -e, --email
              Send  email  to  'distcc-pump-errors'  or  if  defined,   the   value   of   enviroment   variable
              DISTCC_EMAILLOG_WHOM_TO_BLAME,  when  include  server gets in trouble.  The default is to not send
              email.

       --email_bound NUMBER
              Maximal number of emails to send (in addition to a final email). Default: 3.

       --no-email
              Do not send email. This is the default.

       --path_observation_re=RE
              Issue warning message whenever a filename is resolved to a realpath that is matched by  RE,  which
              is  a  regular  expression  in Python syntax.  This is useful for finding out where files included
              actually come from. Use RE="" to find them all.  Note: warnings must be enabled with at least -d1.

       --pid_file FILEPATH
              The pid of the include server is written to file FILEPATH. This allows a script such a distcc-pump
              to tear down the include server.

       -s, --statistics
              Print information to stdout about include analysis.

       --stat_reset_triggers=LIST
              Flush  stat  caches when the timestamp of any filepath in LIST changes or the filepath comes in or
              out of existence.  LIST is a colon separated string of filepaths, possibly containing simple globs
              (as  allowed by Python's glob module). Print a warning whenever such a change happens (if warnings
              are enabled). This option allows limited exceptions to distcc-pump's normal assumption that source
              files are not modified during the build.

       -t, --time
              Print elapsed, user, and system time to stderr.

       --unsafe_absolute_includes
              Do preprocessing on the compilation server even if includes of absolute filepaths are encountered.
              Normally the include-server will fall back on local  preprocessing  if  it  detects  any  absolute
              includes.   Thus, this flag is useful for preventing such fallbacks when the absolute includes are
              a false alarm, either because the absolute include is discarded during  preprocessing  or  because
              the absolutely included file exists on the compilation servers.

              More  precisely, with --unsafe_absolute_includes absolute includes are ignored for the purposes of
              gathering the include closure.  Using this option may lead to incorrect results  because  (1)  the
              header  may  actually  be  included on the compilation server and it may not be the same as on the
              client, (2) the include directives of the header are not further analyzed.

              The option is useful for compiling code that has such hardcoded absolute locations of header files
              inside  conditional  directives  (e.g.  "#ifdef")  that  render  the  includes  irrelevant.   More
              precisely, these includes must be eliminated during preprocessing for  the  actual  configuration.
              Then  the  question  of  existence of the header file is moot and the remote compilation is sound.
              This is often the case if such includes are meant for unusual configurations  different  from  the
              actual configuration.

       -v, --verify
              Verify that files in CPP closure are contained in closure calculated by include processor.

       -w, --write_include_closure
              Write  a  .d_approx file which lists all the included files calculated by the include server; with
              -x, additionally write the included files as calculated by CPP to a .d_exact file.

       -x, --exact_analysis
              Use CPP instead, do not omit system headers files.

INCLUDE SERVER SYMPTOMS AND ISSUES

       The most likely messages and warnings to come from the include processor are listed below.

       Preprocessing locally. Include server not covering: Couldn't determine default system include directories
              To determine the default system header directories, the include server runs the compiler once  for
              each  language  needed  during  its session. This message indicates that the compiler specified to
              distcc is not present on the client.

       Preprocessing locally. Include server not covering: Bailing out because include server  spent  more  than
       ...s user time handling request
              In  uncommon  situations,  the include server fails to analyze very complicated macro expressions.
              The distcc client will use plain distcc mode.

       Warning: Filepath must be relative but isn't
              The include server does not accept absolute filepaths, such as  /usr/include/stdio.h,  in  include
              directives,  because there is no guarantee that this header on the compilation server machine will
              be the same as that on the client.  The include server gives up analyzing the include closure. The
              distcc client cannot use pump-mode.

              To   overcome   this  problem  in  a  not  always  reliable  way,  set  the  environment  variable
              INCLUDE_SERVER_ARGS='--unsafe_absolute_includes' when invoking the distcc-pump script to pass  the
              --unsafe_absolute_includes option to the include server.

       Warning: Absolute filepath ... was IGNORED
              The  --unsafe_absolute_includes is in use.  This situation happens under the same circumstances as
              when "Filepath must be relative but isn't" is issued, but in this case the include will provide an
              answer to the distcc client.

       Warning: Path '/PATH/FILE' changed/came into existence/no longer exists
              These warnings are issued when using stat reset triggers.  Because /PATH/FILE changed, the include
              server clears its caches; the new version of the file (or the lack  of  it)  renders  the  include
              analysis  invalid.  This message can usually be ignored; it does signify a somewhat precarious use
              of files by the build system.  It is recommended to fix the build system so  that  files  are  not
              rewritten.

       Warning: For translation unit ..., lookup of file ... resolved to ... whose realpath is ...
              This  warning  occurs  with  --path_observation_re when a new realpath matching a source or header
              file is observed.

DISTCC DISCREPANCY SYMPTOMS

       The interactions between the build system, distcc, and the include server is  somewhat  complex.  When  a
       distcc commands receives a failing compilation from the remote server it retries the compilation locally.
       This section discusses the causes of discrepancies between  remote  and  local  compilation.   These  are
       flagged by the demotion message:

       __________Warning: ... pump-mode compilation(s) failed on server, but succeeded locally.
       __________Distcc-pump  was  demoted  to  plain  mode.  See the Distcc Discrepancy Symptoms section in the
       include_server(1) man page.

       The distcc-pump script issues this message at the end of the build. This means  that  for  at  least  one
       distcc  invocation  a  local  compilation  succeeded  after  the  remote  compilation failed. Each distcc
       invocation for which such a discrepancy occurred in turn also issues a message such as:

       Warning: remote compilation of '...' failed, retried locally and got a different result.

       The demotion makes subsequent distcc invocations use plain distcc mode.   Thus  preprocessing  will  take
       place  on  the  local  machine  for the remainder of the build.  This technique prevents very slow builds
       where all compilations end up on the local machine after failing remotely.

       Of course, if the local compilations fails after the remote failures, then the  distcc  invocation  exits
       with the non-zero status of the local compilation. The error messages printed are also those of the local
       compilation.

       The fallback behavior for  distcc-pump  mode  to  local  compilation  can  be  disabled  by  setting  the
       environment  variable  DISTCC_FALLBACK  to  1,  which makes the distcc command fail as soon as the remote
       compilation has failed. This setting is very useful for debugging why the remote compilation went  wrong,
       because now the output from the server will be printed.

       Next we discuss the possible causes of discrepancies.

       The user changed a source or header file during the build.
              This yields inconsistent results of course.

       A source or header file changed during the build.
              The    build   system   rewrites   a   file.    For   Linux   kernel   2.6,   this   happens   for
              'include/linux/compile.h' and 'include/asm/asm-offsets.h'. This condition is fixed by letting  the
              include  server  know  that  it  must  reset  its  caches when a stat of any of the files changes.
              Practically, this is done by gathering the files in a colon-separated list and  then  setting  the
              INCLUDE_SERVER_ARGS command as in:

                 include_server_args="--stat_reset_triggers=include/linux/compile.h:include/asm/asm-offsets.h"

       A header file is potentially included, but does not exist, and is then later included.

              This  occurs when some header foo.h includes another header file trick.h, but the trick.h file has
              not yet been generated and the inclusion is actually ignored because of preprocessing  directives.
              The  include  server  will  probe  for  the  existence of trick.h, because it overapproximates all
              possible ways directives actually evaluate. The file trick.h is determined not to exist. If it  is
              later  generated,  and then really included, then the include server will falsely believe that the
              file still does not exist.  The solution to this problem is to  make  the  build  system  generate
              trick.h  before  the  first  time  any header file is included that makes a syntactic reference to
              trick.h

       The include server was started with --unsafe_absolute_includes.
              This is a problem if there are header files locally that  do  not  exist  remotely  and  that  are
              actually  used.  Such includes are often protected by conditional directives that evaluate so that
              are actually used on only specific and often uncommon platforms. If you are not compiling for such
              a platform, then it may be correct to use --unsafe_absolute_include.

       The include server has calculated the wrong includes.
              We do not know of such a situation.

EXIT CODES

       The  exit  code  of  include_server.py is usually 0. That the include server has been started properly is
       communicated through the existence of the pid_file.

ENVIRONMENT VARIABLES

       DISTCC_EMAILLOG_WHOM_TO_BLAME The email address to use for include server automated emails.  The  default
       is 'distcc-pump-errors' (which is an email address that probably will not exist in your domain).

       Additionally, the invocation of the compiler may use additional environment variables.

BUGS

       If  you  think  you  have found a distcc bug, please see the file reporting-bugs.txt in the documentation
       directory for information on how to report it.

       In distcc-pump mode, the include server is unable to handle certain very complicated computed includes as
       found  in  parts  of  the Boost library. The include server will time out and distcc will revert to plain
       mode.

       Other known bugs may be documented on http://code.google.com/p/distcc/

AUTHOR

       The include server was written by Nils Klarlund, with assistance from Fergus Henderson,  Manos  Renieris,
       and Craig Silverstein. Please report bugs to <distcc@lists.samba.org>.

LICENCE

       You  are  free to use distcc.  distcc (including this manual) may be copied, modified or distributed only
       under the terms of the GNU General Public Licence version 2 or later.  distcc comes  with  absolutely  no
       warrany.  A copy of the GPL is included in the file COPYING.

SEE ALSO

       distcc(1),     distccd(1),     include_server(1),     and    gcc(1).     http://code.google.com/p/distcc/
       http://ccache.samba.org/

                                                   9 June 2008                                 include_server(1)