Provided by: dphys-config_20100216-1_all bug

NAME

       dphys-config - daily auto-install/update and/or remove config files

SYNOPSIS

       dphys-config [-f filter] [-cqvD]

       dphys-config -h

DESCRIPTION

       dphys-config installs/updates and/or removes config files. It also triggers commands after
       an new/updated config file is available or before an existing config file will  disappear.
       It can be run by hand, from cron and/or from init.d.

       Get  an  list  of  config  files  from  an configuration server. For each file in the list
       retrieve that file from the same server, and only install it  if  it  is  new  or  changed
       relative  to  what  is already here. If a file is newly installed (or changed) then run an
       postinstall script, which may trigger actions which are wanted to process the  new  config
       (such as inserting data from an config file into an database). Also remove unwanted files.
       If doing so first run an preremove script to tidy up stuff.

       This is part of  the  D-PHYS  (ETH  Zuerich,  Departement  of  Physics)  automatic  system
       operation and maintenance setup.

OPTIONS

       -c     configname: Use this set of config files instead of hostname set. Useful for chroot
              or vhost installs, or for tests.

       -f  filter
              filter: Only process lines which match the filter spec.

       -q     quiet: Don't produce an running report of activities.

       -v     verbose: Give large volume output, where sensible.

       -D     Debug: Activate an debug option. See source for how to use this.

       -h     help: Output help text, and then abort operation.

CONFIG

       The config files /etc/dphys-config (sitewide) and  ~/.dphys-config  (personal)  allow  the
       admin and users to set up the working environment for dphys-config.

       These  config  files  are  sh  script fragments full of assignments, which are sourced, in
       above row, later config files assignments overriding earlier  ones.   Standard  sh  syntax
       rules apply. Assignments are:

       CONF_TMP_DIR
              Sets  the  base  directory  in which all temporary files are stored. It defaults to
              /var/tmp (for enough size and safe operation). Some users may like to use /tmp  for
              higher  speed (tmpfs) or automatic deletion at boot time.  Standard sh syntax rules
              apply. Assignments are:

       CONF_BASEURL
              Sets the base URL to which all */<hostname>/<filename> combinations are added  when
              wget-ing  config  files. This can be an http: or ftp: or whatever other type of URL
              which wget understands and can fetch an file from. Additionally it can be an  file:
              (this  may be from an NFS server) URL, in which case wget is bypassed and the files
              fetched directly using cp. It defaults to the error message generating and aborting
              invalid                                  setting                                 of
              http://not-configured-server.example.net/not/configured/directory, as there  is  no
              sensible default possible. You must set this to where ever your config files should
              be taken from.

       CONF_CONFNAME
              Selects the name for which set of configuration files shall be used for this  host.
              Defaults to `hostname`.

       CONF_LINEFILTER
              Sets  an  regexp which selects which lines from the config file list are processed.
              Defaults to .* (all).

       CONF_LOG_DONE
              Log to syslog that dphys-config has run. Good to see if  cron  and/or  init.d  have
              done their job. Defaults to yes.

       The  config  file  list  dphys-config.list,  which  is  found  via  above settings, and is
       downloaded to /etc/dphys-config.list or ~/.dphys-config.list, then  allows  the  admin  to
       list  what  config  files  are  to  be  fetched and installed/updated or removed, and what
       scripts to run for them. These can be each given for the entire site (= all hosts)  and/or
       group and/or each host, or even merged from site+group+host subsections.

FILES

       /etc/dphys-config
              site admin config

       ~/.dphys-config
              users personal config

       /etc/dphys-config.list
              roots config file list gets stored here

       ~/.dphys-config.list
              users config file list gets stored here

       $CONF_BASEURL/`hostname`/dphys-config.list
              site-global  (all  hosts) common (usually, or group-global or host specific) config
              file list

       $CONF_BASEURL/`hostname`/dphys-config.list.*
              facultative host-specific (usually) or group-specific include-able subsection(s) to
              be added to above config file list dphys-config.list. We often use *.group (one per
              group of users) and *.host (per host), sometimes also *.base (all host  types)  and
              *.workstation          (only         workstation         hosts)         subsections
              $CONF_BASEURL/`hostname`/<file-name> actual config files referred to in config file
              list,       common       section       (usually       the       only       section)
              $CONF_BASEURL/`hostname`/<file-name>.*   facultative  host-specific  (usually)   or
              group-specific  include-able  subsection(s)  to  be  added  to  above  config  file
              <file-name>

CONFIG FILE LIST

       The config file list to be used for checking what files need to  be  installed/updated  or
       removed  and  its  subsections  included  by  #@include lines are merged to one list file,
       analog to cpp #include.

       These are all fetched via wget (or cp for file:), adding their names to  the  user-defined
       base  URL  in  CONF_BASEURL,  and  then  merged.  So CONF_BASEURL can be any URL that wget
       understands http: or ftp: or whatever else, or file:.

       The format of the resulting concatenated file must consist of lines, one per config  file,
       of following format:

       file-name:place-on-target:command-to-trigger

       Where the 3 fields have following meanings:

              file-name
                     Name  of the config file to be installed/updated. Must be only the base part
                     of the filename on the server, without  URL  and  hostname  before  it,  and
                     without  any  .*  subsection  endings  after it, as these are all auto-added
                     whenever they are needed. If this is set to - the line specifies  an  config
                     file to be removed

              place-on-target
                     Full directory or full filename (directory+filename) of where the file is to
                     be placed on the target system. If only an directory is given (any name that
                     ends  in /), then the above file-name (inclusive any directories in it) will
                     be automatically added to it. For removing this must be  the  full  filename
                     (or an directory name (without an /) if an entire directory and its contents
                     shall disapper). An directory name  ending  with  /  is  not  processed,  to
                     prevent  incomplete  edits  (filename  replaced  by  -,  but  not  added  to
                     directory) from killing entire directories (such as say all of /etc/ :-))

              command-to-trigger
                     Full command (directory+filename, with parameters) of an command to be  run,
                     after  this  config  file  has  been  newly installed or changed/updated, or
                     before this config file is removed.  This  can  also  be  multiple  commands
                     separated  by  ; separators. Useful for doing chown/chmod to files that need
                     it. If the marker {} appears in the command, this will be substituted by the
                     filename the config file is going to be installed as. This is analog to find
                     -exec filename substitution

       Lines which begin with an # are regarded as comments, and don't have any  effect  anything
       (Lines extended with one are chopped off at that point). The same applies for empty lines.

PREPROCESSOR

       If  the  first line of the config file list, or any config file fetched on its behalf, has
       the special format #@dphys-config-preprocess [action...] then this line will be  stripped,
       and  the  rest  of the file will be preprocessed. Depending on the list of actions present
       and their order (repeats are allowed) the file will be procesed. Valid actions are:

              backtick
                     Anything inside backticks (``) will be executed as a command, and its stdout
                     will  then  be  substituted  for  the  ``  expression.  This is analog to sh
                     backtick substitution

              if     For any line beginning with #@if the stuff between the #@if and the first  ;
                     character  will  be  executed as command, and if it returns true, everything
                     after the ; will be left, else the entire line  will  be  removed.  This  is
                     analog to shell if ... ; conditional execution

              include
                     For any line beginning with #@include the rest of the line is regarded as an
                     subsection name, which will be added to the  base  filename,  and  then  the
                     resulting  subsection  file fetched (also by wget or cp) and substituted for
                     the line. This is analog to an C  preprocessor  #include  oder  an  shell  .
                     include

EXAMPLES

       The  following  allows  you  to  fetch all your config file lists from an HTTP VirtualHost
       called www.admin.example.net under its subdirectory dphys-config.

       In file /etc/dphys-config, on every host, so it can find the config file server:

       # system will use ${CONF_BASEURL}/`hostname`/<file-name>*
       CONF_BASEURL=http://www.admin.example.net/dphys-config

       We advise  using  an  subdirectory  here,  because  other  /http://www.admin.example.net/*
       directories may already contain other admin stuff you put on the same VirtualHost. Such as
       software packages, site news, etc.

       For dphys-config to be useful you then need to make config file lists for it.  And provide
       the  actual  config files that can be installed, driven by the lists.  This is the largest
       job, as it basically amounts to extracting all your relevant config work from  your  site.
       Also known as reengineering your site.

       Assuming   your   VirtualHost   on   www.admin.example.net   has   as   its   DocumentRoot
       /vhost/www.admin, you would then begin  with  an  pseudo-host  Directory  for  site-global
       common stuff: /vhost/www.admin/dphys-config/SITE/.

       If  your  hosts  are  organised  in  groups  with  group-global  common  configs  (such as
       professors,  students,  staff),  make  an   pseudo-host   for   each   group,   such   as:
       /vhost/www.admin/dphys-config/PROFS/ and */STUDENTS/ and */STAFF/.

       Then   for   host   specific   stuff,   assuming   systems   called  prof1.example.com  to
       prof3.example.com, stud1 to stud20, staff1 to staff5, server1 and server2, make  for  each
       its own directory: /vhost/www.admin/dphys-config/prof1/ (and so on).

       Note  that we suggest using CAPITALS for pseudo-hosts and lowercase for actual hosts. This
       avoids name space collisions. You can also use loops like for host in [a-z]* ;  do  ...  ;
       done  to  work (say generating symlinks to an new config file in all hosts). Well at least
       you can do this so long no one goes and sets LANGUAGE= or  similar  junk,  then  bash  (or
       libc?)  will  hapily  screw  up  case  sensitivity and produce random lossage (yes, it was
       painfull).

       After this add to /vhost/www.admin/dphys-config/SITE/, the actual config files as  far  as
       they  are not host specific, or at least have an common section to all hosts. Example this
       would be /etc/hosts  for  all,  an  common  section  for  /etc/motd,  common  or  all  for
       sendmail.cf, common for inetd.conf, nothing for the ssh hostkeys.

       Then  add,  to an group, say /vhost/www.admin/dphys-config/STUDENTS/, whatever is specific
       to that group. Example this may be an entire special motd for the many changing users,  or
       just an motd.group to #@include into the common one.

       Then  for  each  host  in its /vhost/www.admin/dphys-config/prof1/ (or so) add all that is
       specific to it. Such as its ssh key files. And its own motd.host, it it  needs  one.  Same
       its  inetd.conf.host  if  it is going to offer special stuff. An configs for services only
       this host has such as httpd.conf.

       Then for each host add symlinks to the SITE or group versions that it is to use for common
       stuff, like on /vhost/www.admin/dphys-config/stud1/:

        .../dphys-config/stud1/dphys-config.list -> ../SITE/dphys-config.list
        .../dphys-config/stud1/hosts -> ../SITE/hosts
        .../dphys-config/stud1/inetd.conf -> ../SITE/inetd.conf
        .../dphys-config/stud1/motd -> ../SITE/motd
        .../dphys-config/stud1/motd.group -> GROUP/motd
        .../dphys-config/stud1/GROUP -> ../STUDENTS
        .../dphys-config/stud1/sendmail.cf -> ../SITE/sendmail.cf

       In   the   /vhost/www.admin/dphys-config/SITE/  directory  place  the  site-global  common
       dphys-config.list for all your hosts, containing stuff like this:

       # SITE dphys-config.list - just example stuff, for our exemplaric site
       # basics
       hosts:/etc/                      # simply works, no command
       motd:/etc/                       # this will be assembled group specific
       inetd.conf:/etc/:/etc/init.d/inetd restart  # needs an command to reload
       sendmail.cf:/etc/mail/:/etc/init.d/sendmail restart  # not in /etc
       # ssh restart only after last file, and ensure file modes for each file
       ssh_host_key:/etc/ssh/:chown root:root {}; chmod 600 {}
       ssh_host_rsa_key:/etc/ssh/:chown root:root {}; chmod 600 {}
       ssh_host_dsa_key:/etc/ssh/:chown root:root {}; chmod 600 {}; /etc/init.d/sshd restart
       # load stuff into an existing database file
       seed.debconf:/etc/:debconf-set-selections {}
       # other stuff
       daemon1-conf:/etc/daemon1/conf   # rename so names can differ on server
       daemon2-conf:/etc/daemon2/conf
       daemon1/conf:/etc/               # same as above, but with directories on server
       daemon2/conf:/etc/
       testing:/etc/                    # put something in there for an test
       # delete some stuff
       -:/etc/testing                   # change to above test to get rid of it again
       -:/etc/                          # you will get a warning if you leave this
       #-:/etc                          # you would reinstall your system after the resulting  rm -rf /etc  :-)
       # and some errors
       #only-an-name                    # you would get an error: no place on target
       #only-an-name:                   # you would get an error: no place on target
       #:only-an-place                  # you would get an error: no file to install

       For special services add an dphys-config.list.host on each host that  has  special  config
       files not present on others, such as on /vhost/www.admin/dphys-config/server2/:

       # server2 dphys-config.list.host - only used on our web server
       httpd.conf:/etc/apache/httpd.conf:/etc/init.d/apache restart

       You  can  also  use  dphys-config  to  run  arbitrary  commands, whenever config files are
       installed/updated or removed, to modify existing  config  files,  or  more  likely  modify
       complex  config  databases  which  can not be provided as files, but where one can provide
       edit info as files.

       dphys-config can even install scripts to use as  above  commands  (or  even  just  to  run
       scripts while installing), such as into /usr/local/sbin/.

       For  this  make  an  ../SITE/local/sbin/  directory,  place  the scripts in there (such as
       ../SITE/local/sbin/dphys-config-<whatever>), and symlink local to  ../SITE/local  on  each
       host,  and then add config lines for the scripts, with the command to trigger them, giving
       something like this:

       local/sbin/dphys-config-<whatever>:/usr/:chmod 755 {}; {}  # chmod and run

       It this script processes an config file your will want it to be run if either  the  script
       or the config file is updated, so add the script to the laters line as well:

       dphys-config-<whatever>:/etc/:if [ -x /usr/local/sbin/dphys-config-<whatever> ] ; then /usr/local/sbin/dphys-config-<whatever> ; fi  # run also here

       Finally,  new  hosts can then later simply be added, by making the new hosts directory and
       copying all files and symlinks from an existing host of the same group. Such as by doing:

       mkdir student21
       tar -cf - -C student1 . | tar xpf - -C student21

       To then run dphys-config by hand (say for tests), type on the host:

       dphys-config

       But usually you will want to run dphys-config automatically, every night (or if a  machine
       was/is switched off, at every boot), to keep your configs up to date.

       For  nightly updates the best thing is to use an cron job on every host. 03:00 to 03:59 is
       most likely idle time. Use an line like this one, with the cron option to  avoid  an  load
       peak  on  the  config file server, by random delaying the run by 0..3599 seconds, and with
       stdout and strerr thrown away to avoid getting an mail from every host, as error  messages
       are also allways sent to syslog:

       0 3 * * * root dphys-config cron > /dev/null 2>&1

       To  catch  machines  switched off over night, with no cron run on them, also run an init.d
       script. Use an script like this one, also with stdout and  stderr  thrown  away  to  avoid
       cluttering your boot console output:

       #!/bin/sh
       # /etc/init.d/dphys-config - boot time automatic config updates, if no cron
       case "$1" in
         start)
           dphys-config init > /dev/null 2>&1
           ;;
       esac
       exit 0

SECURITY

       If dphys-config is to be used to distribute all config files, this will also include files
       which are  security  relevant,  such  as  ssh  private  keys  (host  key  or  (root)  user
       authentification),  SSL certificates, passwd and shadow, lilo.conf, software license keys,
       etc.

       As all files are most likely fetched from an http: URL, measures must be taken  to  secure
       the  config website from other people downloading them. We here use an restriction to only
       IP addresses registered as hosts in our NIS server, and additionally  run  identd  on  all
       allowed  hosts,  and require the wget process opening the HTTP connection to be running by
       user root, and so also require dphys-config to run as root.

       To avoid sniffing it is recommended to give wget an https: URL.

GOTCHAS

       Config files are read by wget from an webserver, so they lose their owners and  modes.  So
       the commands triggered on their lines must be used to chown/chmod them to proper values.

       When  used  together  with dphys-admin, dphys-config should run as first (earlier cron and
       init.d entries). This is needed to provide configs before new packages are  installed,  so
       dphys-admin  can pretend that the packages were already once installed (and then non-purge
       removed), and so prevent questions on install, which is required for unattended  installs.
       [Note  that this pretending does not go as far as setting debconf up. Broken packages that
       ignore config files and only look at debconf will still ask questions.]

       As result of this, when installing for the first time on an new system (such as installing
       Debian  by  the  dphys3  end2stage  feature,  which  installs  first dphys-config and then
       dphys-admin), any scripts installed by packages by dphys-admin, to  be  called  on  config
       file  install/update  will  still  be  missing,  and  so  not  runnable. Either ignore the
       warnings, or better call the scripts by something like this:

       file:place:if [ -x script ]; then script; fi

       Note that in this case, trying to run dphys-config for a second time after dphys-admin has
       installed packages and scripts, will not automatically mend this, as the config files have
       not changed, and so dphys-config will  not  (re-)run  their  scripts.  Therefore  packages
       containing such scripts must also, as part of their postinst (or init.d which is called by
       postinst), check for existing config files and then run their scripts. This is the  normal
       behaviour  of  quite  a  few  packages  anyway.  Of course this requires the scripts to be
       idempotent, which is official Debian policy anyway.

AUTHOR

       neil@franklin.ch, http://neil.franklin.ch/