Provided by: pcp-manager_4.3.1-1_amd64 bug

NAME

       pmmgr - pcp daemon manager

SYNOPSIS

       pmmgr [-v] [-c config-directory] [-p polling-interval] [-l log-file]

DESCRIPTION

       pmmgr  manages  a collection of PCP daemons for a set of discovered local and remote hosts
       running the Performance Metrics Collection  Daemon  (PMCD),  according  to  zero  or  more
       configuration  directories.   It  keeps  a matching set of pmie, pmlogger, pmrep and other
       daemons running, and their archives/logs merged/rotated.  It provides  an  alternative  to
       the  default  pmlogger_daily  and  pmie_daily  scripts  that  administer pmlogger and pmie
       ``farms'' using cron(3).

       pmmgr is largely self-configuring and perseveres despite most run-time errors.  pmmgr runs
       in  the  foreground  until  interrupted.   When signaled, it will stop its running daemons
       before exiting.

ALGORITHM

       Each poll interval, pmmgr computes a list of possible targets for  a  pmcd  search.   This
       list  is  assembled  from  several  configuration files, and may include explicitly listed
       specifications, hosts discovered through several different mechanisms,  and/or  individual
       containers  running  within  them.   Once  the list is assembled, pmmgr attempts to make a
       brief pmNewContext connection to each target, in order to check for the  existence  of  an
       actual  running pmcd instance, and to extract a hostid.  The hostid is treated as a unique
       identifier for the instance, so that redundant connection paths to the same server can  be
       filtered  out.  Once the final list of live pmcd instances is identified, along with their
       unique hostids, pmmgr ensures that any  requested  pcp  client  daemons  are  started  (or
       restarted)  for  them.   If  any  pmcd  instances  disappear from the list, its pcp client
       daemons are stopped.  This entire cycle repeats every poll interval.

OPTIONS

       A description of the command line options specific to pmmgr follows:

       -c   directory adds a  given  configuration  directory  to  pmmgr.   pmmgr  can  supervise
            multiple  different  configurations at the same time, so this option may be repeated.
            Errors in the configuration may be noted to standard error, but pmmgr  will  fill  in
            missing   information   with   built-in   defaults.    The   default   directory   is
            $PCP_SYSCONF_DIR/pmmgr

       -p   polling-interval sets the host-discovery polling interval  to  the  given  number  of
            seconds.   The default is 60.  Daemons for a particular target host will be restarted
            no more frequently than this interval.  There may  be  a  short-lived  thread  inside
            pmmgr for startup and shutdown of each daemon for each target host.

       -l   log-file  redirects standard output and error to the given log file, which is created
            anew

       -v   adds more verbose tracing to standard output.

CONFIGURATION

   FILES
       A pmmgr configuration identifies which hosts should be monitored, which daemons should  be
       maintained  for  them,  and  what  options those daemons should be run with.  pmmgr uses a
       small number of files in a configuration directory, instead of  lines  in  a  single  text
       file.   The individual files carry zero or more lines of 100% pure configuration text, and
       no comments.  (If desired, a configuration may be commented upon within other files,  such
       as a free-form README.)

       Some  of  the  configuration  files  are  forked  into pairs: per-hostid and common.  This
       permits numerous almost-identical-configuration  targets  to  be  managed  from  the  same
       configuration  directory.   For these files, marked with * below, pmmgr will concatenate a
       per-hostid file (if it exists) and a common file (if it  exists)  in  order  to  form  the
       complete configuration item.

       For  example,  for pmie configuration for target hostid foo, pmmgr will search files named
       pmie.foo then pmie.  For single-line configuration items, the first file & line found will
       "win"; for multi-line configuration items, they all "win".

   TARGET SELECTION
       This  set  of configuration files identifies where pmmgr should search for pmcd instances,
       how to uniquely identify them, and where state such as log files should be kept for  each.
       Ideally,  a persistent and unique hostid string is computed for each potential target pmcd
       from specified metric values.  This hostid  is  also  used  as  a  subdirectory  name  for
       locating daemon data.  The rare empty hostid is mapped to "-".

       hostid-static
              This  file  contains  one or more lines specifying the static string that should be
              used as multiple distinct hostids for the  same  target  pmcd.   Treatment  of  the
              distinct   hostids   may   be  customized  using  per-hostid  configuration  files.
              Specifying values in this file overrides the hostid-metrics file  specified  below.
              It  should  be  noted  that  using  this  option  will cause all target pmcds to be
              assigned the same set of hostids.  Thus, this is useful in monitoring single  hosts
              or if each monitored host has its own configuration directory.

       hostid-metrics
              This  file  contains  one  or  more  lines  of  metric specifications in the format
              accepted by  pmParseMetricSpec.   Metrics  without  instance  specifiers  mean  all
              instances  of that metric.  These are used to generate the unique hostid string for
              each pmcd server that pmmgr discovers.  Upon discovery, all  the  metrics/instances
              named are queried, string values fetched, and normalized/concatenated into a single
              hyphenated printable string.  The default is the single metric pmcd.hostname, which
              is  sufficient if all the hosts discovered have unique hostname(2).  If they don't,
              you should add other pcp metric specifications to set them apart at your site.  The
              more  you  add,  the  longer the hostid string, but the more likely that accidental
              duplication is prevented.

       However, it may be desirable for a hostid to also be persistent, so  that  if  the  target
       host goes offline and later returns, the new hostid matches the previous one, because then
       old and new histories can be joined.  This argues against using metrics whose values  vary
       from boot to boot.

       Some       candidate       metrics       to      consider:      network.interface.hw_addr,
       network.interface.inet_addr["eth0"], network.interface.ipv6_addr, kernel.uname.nodename

       log-directory
              This  file  contains  the  path  of  a  directory  beneath  which  the   per-hostid
              subdirectories  are  to  be  created  by  pmmgr.   If  it is not a full path, it is
              implicitly  relative  to  the  configuration  directory  itself.   The  default  is
              $PCP_LOG_DIR/pmmgr/.

       target-host
              This  file  contains  one  or  more  lines  containing pmcd host specifications, as
              described on the PCPintro(1) manual page.  The default is to target pmcd at local:.

       target-discovery
              This  file  contains  one  or  more  lines  containing   specifications   for   the
              pmDiscoverServices  PMAPI  call,  each  of  which may map onto a fluctuating set of
              local or remote pmcd servers.  Each poll interval,  pmmgr  will  attempt  to  rerun
              discovery with all of the given specifications.  Again, it is not a problem if more
              than one specification matches the same actual pmcd: one confirmed access  path  is
              arbitrarily  selected.   The  default  is  to  do no discovery.  Consider including
              avahi,timeout=5 to rely in pmcd self-announcements on the local network  (searching
              for  up  to  five  seconds  each time).  Consider including probe=192.168.1.0/24 to
              quickly scan the given IP address range.

       subtarget-containers
              If this file  exists,  pmmgr  will  scan  each  host  that  is  found  for  running
              containers.   For each running container, it will create independent subtargets for
              running requested daemons.  The hostid string for these subtargets  is  the  host's
              hostid  string,  followed  by  a  double-hyphen,  then  the  full  unique container
              instance-name string.

       target-threads
              This file contains a limit  on  the  number  of  concurrent  threads  that  analyze
              potential  target  pmcds for their hostids and/or containers.  The default is a few
              dozen threads per CPU core, if known.  Set this to zero if remote pmcds  should  be
              analyzed  sequentially.   A  small  number  of  threads  is  not a good idea if any
              potential target pmcds are unreachable, since $PMCD_CONNECT_TIMEOUT may be  several
              seconds long each.

       log-subdirectory-gc
              This  file  may contain a time interval specification as per the PCPintro(1) manual
              page.  All subdirectories of the log-directory are presumed  to  contain  data  for
              pmmgr-monitored  servers.  Those that have not been modified in at least that long,
              and not associated with a currently monitored target, are deleted  entirely.   This
              value  should  be  longer  than  the longest interval that pmmgr normally recreates
              archives (such as due to pmmgr restarts, and pmlogmerge  intervals).   The  default
              value is 90days.

   PMLOGGER CONFIGURATION
       This  group  of  configuration options controls a pmlogger daemon for each host.  This may
       include generating its configuration, and managing its archives.

       pmlogger*
              If and only if this file exists, pmmgr will maintain a  pmlogger  daemon  for  each
              targeted  host.   This file contains one line of additional space-separated options
              for the pmlogger daemon.  (pmmgr already adds -h, -H, -f, -r, -l, and perhaps  -c.)
              The  default is to maintain no pmlogger (and no other configuration in this section
              is processed).

       pmlogger-timefmt*
              Specify a time format to use in the archive-* name for pmlogger generated archives.
              The default is "%Y%m%d.%H%M%S". Expected to be in strftime(3) format.

       pmlogconf*
              If  and  only  if  this  file  exists,  pmmgr  will  run  pmlogconf  to  generate a
              configuration file for each target pmcd.  The file  contains  one  line  of  space-
              separated  additional  options  for  the  pmlogconf program.  pmlogconf's generated
              output file will be stored under  the  log-directory/hostid  subdirectory.   (pmmgr
              already  adds  -c,  -r,  and  -h.)   The  default  is no pmlogconf, so instead, the
              pmlogger file above should probably  contain  a  -c  option,  to  specify  a  fixed
              pmlogger configuration.

   ARCHIVE LOG MANAGEMENT
       Default  pmlogger  configurations  can collect tens of megabytes of data per day (possibly
       split into multiple archives), per target host.  If your disk space is less than infinite,
       or  archive-splitting  unwieldy,  this should be managed.  In the default, unmanaged case,
       the system administrator is responsible for managing the individual archive-*  files  from
       the   per-host   logging   subdirectories.   pmmgr  offers  several  other  options,  each
       representing different performance and usability tradeoffs.

   ARCHIVE LOG MANAGEMENT - pmlogmerge
       This style of archive log management regularly creates a single merged archive from  prior
       archives  for  each  target host, in effect lopping off old data and appending the new.  A
       single merged archive can be relatively large (defaults to approximately  100-400  MB  per
       host), and puts a corresponding I/O load on storage, but is most convenient for a detailed
       long-timeframe analysis.  Once pmlogger is restarted, it always creates a new archive,  so
       in  the  steady state, there will be one merged archive of recent history, and one current
       archive being written-to by pmlogger.

       pmlogmerge*
              If this file exists, pmmgr will run pmlogextract  to  periodically  merge  together
              preexisting  log  archives for each target pmcd into a single large one.  Then, the
              preexisting log archives are deleted  (including  any  prior  merged  ones).   This
              configuration file may contain a time interval specification as per the PCPintro(1)
              manual page, representing the period after which  pmlogger  should  be  temporarily
              stopped,  and  archives  merged.  It represents the maximum amount of time that the
              merged archive lags the present time.  The default is 24hours.

       pmlogmerge-granular*
              If this file also exists, pmmgr  will  merge  only  a  subset  of  preexisting  log
              archives into the new one, instead of all of them, so as to approximate a granular,
              aligned set of merged archives.  The subset chosen corresponds to the previous time
              interval specified by the pmlogmerge control file.  The default is no granularity.

       pmlogcheck-corrupt-gc*
              Before  archives  are considered for merging, they are processed through pmlogcheck
              to check for corruption.  In the unlikely case of  a  problem,  such  archives  are
              renamed  out  of  the  way  (named "corrupt-*"), and retained up to a limited time.
              This file specifies how long.  If  this  file  exists,  it  the  time  interval  it
              contains  is  the  maximum  age.  The default is 90days.  To store corrupt archives
              indefinitely, set this to a large quantity like "99999weeks".

       pmlogmerge-rewrite*
              If this file exists, pmmgr will run pmlogrewrite -i (plus any other options  listed
              in this file) on each input archive before merging it.  This will naturally require
              more disk I/O.  The default is no rewriting.

       pmlogmerge-retain*
              pmmgr  reduces/deletes  any  original-resolution  archives  after  a  time   period
              specified  by  this  file,  as measured by the file mtime.  The period will also be
              passed to pmlogextract as a negative parameter to -S.  The default is  14days.   To
              store archives indefinitely, set this to a large quantity like "99999weeks".

       pmlogreduce*
              If  this  file  exists,  then  prior  to  removing  archives  that  expire past the
              pmlogmerge-retain period, they are processed with  pmlogreduce  to  create  reduced
              archives (named reduced-*).  If the file contains space-separated options, they are
              passed onto pmlogreduce.  (By default, pmlogreduce  down-samples  to  a  600-second
              interval.)

       pmlogreduce-retain*
              If  this  file  exists, then reduced archives (identified by the reduced-* pattern)
              are deleted after a time period specified by this file, as measured from  the  file
              mtime.   Since this time is likely that of the pmlogreduce run, the total retention
              time will be approximately the pmlogmerge-retain time plus  the  pmlogreduce-retain
              time.   The default is 90days.  To store reduced archives indefinitely, set this to
              a large quantity like "99999weeks".

       disk-full-threshold
              If this file exists, then pmmgr will track the disk space available where  pmlogger
              archives  are  kept.   If  that  partition fills up past the configured percentage,
              pmmgr will linearly reduce the duration logs are kept via  the  disk-full-retention
              variable.

              Values  must be greater than zero, and expressed either a value between 0 and 1, or
              decimal value between 1 and 100.

       disk-full-retention
              If expressed, this  variable  scales  the  rate  at  which  logs  are  culled  when
              disk-full-threshold  has  been  surpassed.   A lower percentage will cull logs more
              quickly (in favour of preserving disk space), while a higher percentage will opt to
              retain more pcp archives.

                     ┌──────────────────────────┬────────────────┬────────────────────────┐
                     │Normalized Full Threshold │ Full Retention │ Final Retention Factor │
                     ├──────────────────────────┼────────────────┼────────────────────────┤
                     │            1             │      0.0       │          0.0           │
                     ├──────────────────────────┼────────────────┼────────────────────────┤
                     │          0.75            │      0.0       │          0.25          │
                     ├──────────────────────────┼────────────────┼────────────────────────┤
                     │           0.5            │      0.0       │          0.5           │
                     ├──────────────────────────┼────────────────┼────────────────────────┤
                     │           0.0            │      0.0       │          1.0           │
                     ├──────────────────────────┼────────────────┼────────────────────────┤
                     │            1             │      0.5       │          0.5           │
                     ├──────────────────────────┼────────────────┼────────────────────────┤
                     │          0.75            │      0.5       │         0.625          │
                     ├──────────────────────────┼────────────────┼────────────────────────┤
                     │           0.5            │      0.5       │          0.75          │
                     ├──────────────────────────┼────────────────┼────────────────────────┤
                     │           0.0            │      0.5       │          1.0           │
                     ├──────────────────────────┼────────────────┼────────────────────────┤
                     │            1             │      1.0       │          1.0           │
                     ├──────────────────────────┼────────────────┼────────────────────────┤
                     │          0.75            │      1.0       │          1.0           │
                     ├──────────────────────────┼────────────────┼────────────────────────┤
                     │           0.5            │      1.0       │          1.0           │
                     ├──────────────────────────┼────────────────┼────────────────────────┤
                     │           0.0            │      1.0       │          1.0           │
                     └──────────────────────────┴────────────────┴────────────────────────┘
   PMIE CONFIGURATION
       This  group  of  configuration  options  controls  a  pmie daemon for each host.  This may
       include generating a custom configuration.

       pmie*  If and only if this file exists,  pmmgr  will  maintain  a  pmie  daemon  for  each
              targeted  pmcd.   This file contains one line of additional space-separated options
              for the pmie daemon.  (pmmgr already adds -h, -f, -l, and perhaps -c.)  The default
              is to maintain no pmie (and no other configuration in this section is processed).

       pmieconf*
              If  and  only  if  this  file  exists,  pmmgr  will  run  pmieconf  to  generate  a
              configuration file for each target pmcd.  The file  contains  one  line  of  space-
              separated  additional options for the pmieconf program.  pmieconf- generated output
              file will be stored under the log-directory/hostid  subdirectory.   (pmmgr  already
              adds  -F, -c, and -f.)  The default is no pmieconf, so instead, the pmie file above
              should probably contain a -c option, to specify a fixed pmie configuration.

   MONITOR DAEMON MANAGEMENT
       pmmgr may be used to invoke arbitrary PCP client programs for each target pmcd.  This  can
       enable automated invocation of reporting or relaying tools, such as pmrep, pcp2graphite or
       pcp2influxdb without needing a specialized system service.

       monitor*
              If this file exists, then for each line in this file, a new background process will
              be  invoked.   (It  is restarted if it exits.)  The line specifies the beginning of
              the command line (including the program name); pmmgr appends  a  -h  HOSTSPEC,  and
              arranges  to  collect the standard output and standard error into separate monitor-
              NN.out and monitor-NN.err files under the log directory.  Errors  messages  in  the
              latter are transcribed to pmmgr's own logs.

FILES

       $PCP_SYSCONF_DIR/pmmgr/
                 default configuration directory
       $PCP_LOG_DIR/pmmgr/
                 default logging directory

PCP ENVIRONMENT

       Environment  variables with the prefix PCP_ are used to parametrize the file and directory
       names used by PCP.  On each installation, the file /etc/pcp.conf contains the local values
       for  these  variables.   The  $PCP_CONF  variable  may  be  used to specify an alternative
       configuration file, as described in pcp.conf(5).

SEE ALSO

       PCPIntro(1), cron(1), pmcd(1), pmlogconf(1), pmlogger(1), pmlogger_daily(1),  pmieconf(1),
       pmie(1),   pmie_daily(1),   pmrep(1),  pcp2graphite(1),  pcp2influxdb(1),  pmlogreduce(1),
       pcp.conf(5) and pcp.env(5).