Provided by: pcp-manager_4.0.1-1_amd64 
NAME
pmmgr - pcp daemon manager
SYNOPSIS
pmmgr [-v] [-c config-directory] [-p polling-interval] [-l log-file]
DESCRIPTION
pmmgr manages a collection of PCP daemons for a set of discovered local and remote hosts running the
Performance Metrics Collection Daemon (PMCD), according to zero or more configuration directories. It
keeps a matching set of pmie, pmlogger, pmrep and other daemons running, and their archives/logs
merged/rotated. It provides an alternative to the default pmlogger_daily and pmie_daily scripts that
administer pmlogger and pmie ``farms'' using cron(3).
pmmgr is largely self-configuring and perseveres despite most run-time errors. pmmgr runs in the
foreground until interrupted. When signaled, it will stop its running daemons before exiting.
ALGORITHM
Each poll interval, pmmgr computes a list of possible targets for a pmcd search. This list is assembled
from several configuration files, and may include explicitly listed specifications, hosts discovered
through several different mechanisms, and/or individual containers running within them. Once the list is
assembled, pmmgr attempts to make a brief pmNewContext connection to each target, in order to check for
the existence of an actual running pmcd instance, and to extract a hostid. The hostid is treated as a
unique identifier for the instance, so that redundant connection paths to the same server can be filtered
out. Once the final list of live pmcd instances is identified, along with their unique hostids, pmmgr
ensures that any requested pcp client daemons are started (or restarted) for them. If any pmcd instances
disappear from the list, its pcp client daemons are stopped. This entire cycle repeats every poll
interval.
OPTIONS
A description of the command line options specific to pmmgr follows:
-c directory adds a given configuration directory to pmmgr. pmmgr can supervise multiple different
configurations at the same time, so this option may be repeated. Errors in the configuration may be
noted to standard error, but pmmgr will fill in missing information with built-in defaults. The
default directory is $PCP_SYSCONF_DIR/pmmgr
-p polling-interval sets the host-discovery polling interval to the given number of seconds. The
default is 60. Daemons for a particular target host will be restarted no more frequently than this
interval. There may be a short-lived thread inside pmmgr for startup and shutdown of each daemon
for each target host.
-l log-file redirects standard output and error to the given log file, which is created anew
-v adds more verbose tracing to standard output.
CONFIGURATION
FILES
A pmmgr configuration identifies which hosts should be monitored, which daemons should be maintained for
them, and what options those daemons should be run with. pmmgr uses a small number of files in a
configuration directory, instead of lines in a single text file. The individual files carry zero or more
lines of 100% pure configuration text, and no comments. (If desired, a configuration may be commented
upon within other files, such as a free-form README.)
Some of the configuration files are forked into pairs: per-hostid and common. This permits numerous
almost-identical-configuration targets to be managed from the same configuration directory. For these
files, marked with * below, pmmgr will concatenate a per-hostid file (if it exists) and a common file (if
it exists) in order to form the complete configuration item.
For example, for pmie configuration for target hostid foo, pmmgr will search files named pmie.foo then
pmie. For single-line configuration items, the first file & line found will "win"; for multi-line
configuration items, they all "win".
TARGET SELECTION
This set of configuration files identifies where pmmgr should search for pmcd instances, how to uniquely
identify them, and where state such as log files should be kept for each. Ideally, a persistent and
unique hostid string is computed for each potential target pmcd from specified metric values. This
hostid is also used as a subdirectory name for locating daemon data. The rare empty hostid is mapped to
"-".
hostid-static
This file contains one or more lines specifying the static string that should be used as multiple
distinct hostids for the same target pmcd. Treatment of the distinct hostids may be customized
using per-hostid configuration files. Specifying values in this file overrides the hostid-metrics
file specified below. It should be noted that using this option will cause all target pmcds to be
assigned the same set of hostids. Thus, this is useful in monitoring single hosts or if each
monitored host has its own configuration directory.
hostid-metrics
This file contains one or more lines of metric specifications in the format accepted by
pmParseMetricSpec. Metrics without instance specifiers mean all instances of that metric. These
are used to generate the unique hostid string for each pmcd server that pmmgr discovers. Upon
discovery, all the metrics/instances named are queried, string values fetched, and
normalized/concatenated into a single hyphenated printable string. The default is the single
metric pmcd.hostname, which is sufficient if all the hosts discovered have unique hostname(2). If
they don't, you should add other pcp metric specifications to set them apart at your site. The
more you add, the longer the hostid string, but the more likely that accidental duplication is
prevented.
However, it may be desirable for a hostid to also be persistent, so that if the target host goes offline
and later returns, the new hostid matches the previous one, because then old and new histories can be
joined. This argues against using metrics whose values vary from boot to boot.
Some candidate metrics to consider: network.interface.hw_addr, network.interface.inet_addr["eth0"],
network.interface.ipv6_addr, kernel.uname.nodename
log-directory
This file contains the path of a directory beneath which the per-hostid subdirectories are to be
created by pmmgr. If it is not a full path, it is implicitly relative to the configuration
directory itself. The default is $PCP_LOG_DIR/pmmgr/.
target-host
This file contains one or more lines containing pmcd host specifications, as described on the
PCPintro(1) manual page. The default is to target pmcd at local:.
target-discovery
This file contains one or more lines containing specifications for the pmDiscoverServices PMAPI
call, each of which may map onto a fluctuating set of local or remote pmcd servers. Each poll
interval, pmmgr will attempt to rerun discovery with all of the given specifications. Again, it
is not a problem if more than one specification matches the same actual pmcd: one confirmed access
path is arbitrarily selected. The default is to do no discovery. Consider including
avahi,timeout=5 to rely in pmcd self-announcements on the local network (searching for up to five
seconds each time). Consider including probe=192.168.1.0/24 to quickly scan the given IP address
range.
subtarget-containers
If this file exists, pmmgr will scan each host that is found for running containers. For each
running container, it will create independent subtargets for running requested daemons. The
hostid string for these subtargets is the host's hostid string, followed by a double-hyphen, then
the full unique container instance-name string.
target-threads
This file contains a limit on the number of concurrent threads that analyze potential target pmcds
for their hostids and/or containers. The default is a few dozen threads per CPU core, if known.
Set this to zero if remote pmcds should be analyzed sequentially. A small number of threads is
not a good idea if any potential target pmcds are unreachable, since $PMCD_CONNECT_TIMEOUT may be
several seconds long each.
log-subdirectory-gc
This file may contain a time interval specification as per the PCPintro(1) manual page. All
subdirectories of the log-directory are presumed to contain data for pmmgr-monitored servers.
Those that have not been modified in at least that long, and not associated with a currently
monitored target, are deleted entirely. This value should be longer than the longest interval
that pmmgr normally recreates archives (such as due to pmmgr restarts, and pmlogmerge intervals).
The default value is 90days.
PMLOGGER CONFIGURATION
This group of configuration options controls a pmlogger daemon for each host. This may include
generating its configuration, and managing its archives.
pmlogger*
If and only if this file exists, pmmgr will maintain a pmlogger daemon for each targeted host.
This file contains one line of additional space-separated options for the pmlogger daemon. (pmmgr
already adds -h, -H, -f, -r, -l, and perhaps -c.) The default is to maintain no pmlogger (and no
other configuration in this section is processed).
pmlogger-timefmt*
Specify a time format to use in the archive-* name for pmlogger generated archives. The default is
"%Y%m%d.%H%M%S". Expected to be in strftime(3) format.
pmlogconf*
If and only if this file exists, pmmgr will run pmlogconf to generate a configuration file for
each target pmcd. The file contains one line of space-separated additional options for the
pmlogconf program. pmlogconf's generated output file will be stored under the
log-directory/hostid subdirectory. (pmmgr already adds -c, -r, and -h.) The default is no
pmlogconf, so instead, the pmlogger file above should probably contain a -c option, to specify a
fixed pmlogger configuration.
ARCHIVE LOG MANAGEMENT
Default pmlogger configurations can collect tens of megabytes of data per day (possibly split into
multiple archives), per target host. If your disk space is less than infinite, or archive-splitting
unwieldy, this should be managed. In the default, unmanaged case, the system administrator is
responsible for managing the individual archive-* files from the per-host logging subdirectories. pmmgr
offers several other options, each representing different performance and usability tradeoffs.
ARCHIVE LOG MANAGEMENT - pmlogmerge
This style of archive log management regularly creates a single merged archive from prior archives for
each target host, in effect lopping off old data and appending the new. A single merged archive can be
relatively large (defaults to approximately 100-400 MB per host), and puts a corresponding I/O load on
storage, but is most convenient for a detailed long-timeframe analysis. Once pmlogger is restarted, it
always creates a new archive, so in the steady state, there will be one merged archive of recent history,
and one current archive being written-to by pmlogger.
pmlogmerge*
If this file exists, pmmgr will run pmlogextract to periodically merge together preexisting log
archives for each target pmcd into a single large one. Then, the preexisting log archives are
deleted (including any prior merged ones). This configuration file may contain a time interval
specification as per the PCPintro(1) manual page, representing the period after which pmlogger
should be temporarily stopped, and archives merged. It represents the maximum amount of time that
the merged archive lags the present time. The default is 24hours.
pmlogmerge-granular*
If this file also exists, pmmgr will merge only a subset of preexisting log archives into the new
one, instead of all of them, so as to approximate a granular, aligned set of merged archives. The
subset chosen corresponds to the previous time interval specified by the pmlogmerge control file.
The default is no granularity.
pmlogcheck-corrupt-gc*
Before archives are considered for merging, they are processed through pmlogcheck to check for
corruption. In the unlikely case of a problem, such archives are renamed out of the way (named
"corrupt-*"), and retained up to a limited time. This file specifies how long. If this file
exists, it the time interval it contains is the maximum age. The default is 90days. To store
corrupt archives indefinitely, set this to a large quantity like "99999weeks".
pmlogmerge-rewrite*
If this file exists, pmmgr will run pmlogrewrite -i (plus any other options listed in this file)
on each input archive before merging it. This will naturally require more disk I/O. The default
is no rewriting.
pmlogmerge-retain*
pmmgr reduces/deletes any original-resolution archives after a time period specified by this file,
as measured by the file mtime. The period will also be passed to pmlogextract as a negative
parameter to -S. The default is 14days. To store archives indefinitely, set this to a large
quantity like "99999weeks".
pmlogreduce*
If this file exists, then prior to removing archives that expire past the pmlogmerge-retain
period, they are processed with pmlogreduce to create reduced archives (named reduced-*). If the
file contains space-separated options, they are passed onto pmlogreduce. (By default, pmlogreduce
down-samples to a 600-second interval.)
pmlogreduce-retain*
If this file exists, then reduced archives (identified by the reduced-* pattern) are deleted after
a time period specified by this file, as measured from the file mtime. Since this time is likely
that of the pmlogreduce run, the total retention time will be approximately the pmlogmerge-retain
time plus the pmlogreduce-retain time. The default is 90days. To store reduced archives
indefinitely, set this to a large quantity like "99999weeks".
disk-full-threshold
If this file exists, then pmmgr will track the disk space available where pmlogger archives are
kept. If that partition fills up past the configured percentage, pmmgr will linearly reduce the
duration logs are kept via the disk-full-retention variable.
Values must be greater than zero, and expressed either a value between 0 and 1, or decimal value
between 1 and 100.
disk-full-retention
If expressed, this variable scales the rate at which logs are culled when disk-full-threshold has
been surpassed. A lower percentage will cull logs more quickly (in favour of preserving disk
space), while a higher percentage will opt to retain more pcp archives.
┌───────────────────────────┬────────────────┬────────────────────────┐
│ Normalized Full Threshold │ Full Retention │ Final Retention Factor │
├───────────────────────────┼────────────────┼────────────────────────┤
│ 1 │ 0.0 │ 0.0 │
├───────────────────────────┼────────────────┼────────────────────────┤
│ 0.75 │ 0.0 │ 0.25 │
├───────────────────────────┼────────────────┼────────────────────────┤
│ 0.5 │ 0.0 │ 0.5 │
├───────────────────────────┼────────────────┼────────────────────────┤
│ 0.0 │ 0.0 │ 1.0 │
├───────────────────────────┼────────────────┼────────────────────────┤
│ 1 │ 0.5 │ 0.5 │
├───────────────────────────┼────────────────┼────────────────────────┤
│ 0.75 │ 0.5 │ 0.625 │
├───────────────────────────┼────────────────┼────────────────────────┤
│ 0.5 │ 0.5 │ 0.75 │
├───────────────────────────┼────────────────┼────────────────────────┤
│ 0.0 │ 0.5 │ 1.0 │
├───────────────────────────┼────────────────┼────────────────────────┤
│ 1 │ 1.0 │ 1.0 │
├───────────────────────────┼────────────────┼────────────────────────┤
│ 0.75 │ 1.0 │ 1.0 │
├───────────────────────────┼────────────────┼────────────────────────┤
│ 0.5 │ 1.0 │ 1.0 │
├───────────────────────────┼────────────────┼────────────────────────┤
│ 0.0 │ 1.0 │ 1.0 │
└───────────────────────────┴────────────────┴────────────────────────┘
PMIE CONFIGURATION
This group of configuration options controls a pmie daemon for each host. This may include generating a
custom configuration.
pmie* If and only if this file exists, pmmgr will maintain a pmie daemon for each targeted pmcd. This
file contains one line of additional space-separated options for the pmie daemon. (pmmgr already
adds -h, -f, -l, and perhaps -c.) The default is to maintain no pmie (and no other configuration
in this section is processed).
pmieconf*
If and only if this file exists, pmmgr will run pmieconf to generate a configuration file for each
target pmcd. The file contains one line of space-separated additional options for the pmieconf
program. pmieconf- generated output file will be stored under the log-directory/hostid
subdirectory. (pmmgr already adds -F, -c, and -f.) The default is no pmieconf, so instead, the
pmie file above should probably contain a -c option, to specify a fixed pmie configuration.
MONITOR DAEMON MANAGEMENT
pmmgr may be used to invoke arbitrary PCP client programs for each target pmcd. This can enable
automated invocation of reporting or relaying tools, such as pmrep, pcp2graphite or pcp2influxdb without
needing a specialized system service.
monitor*
If this file exists, then for each line in this file, a new background process will be invoked.
(It is restarted if it exits.) The line specifies the beginning of the command line (including
the program name); pmmgr appends a -h HOSTSPEC, and arranges to collect the standard output and
standard error into separate monitor-NN.out and monitor-NN.err files under the log directory.
Errors messages in the latter are transcribed to pmmgr's own logs.
FILES
$PCP_SYSCONF_DIR/pmmgr/
default configuration directory
$PCP_LOG_DIR/pmmgr/
default logging directory
PCP ENVIRONMENT
Environment variables with the prefix PCP_ are used to parametrize the file and directory names used by
PCP. On each installation, the file /etc/pcp.conf contains the local values for these variables. The
$PCP_CONF variable may be used to specify an alternative configuration file, as described in pcp.conf(5).
SEE ALSO
PCPIntro(1), cron(1), pmcd(1), pmlogconf(1), pmlogger(1), pmlogger_daily(1), pmieconf(1), pmie(1),
pmie_daily(1), pmrep(1), pcp2graphite(1), pcp2influxdb(1), pmlogreduce(1), pcp.conf(5) and pcp.env(5).
Performance Co-Pilot PCP PMMGR(1)