Ubuntu Manpage: pmlogger_check, pmlogger_daily - administration of Performance Co-Pilot archive log files

NAME

       pmlogger_check, pmlogger_daily - administration of Performance Co-Pilot archive log files

SYNOPSIS

       $PCP_BINADM_DIR/pmlogger_check [-CNsTV?]  [-c control] [-l logfile]
       $PCP_BINADM_DIR/pmlogger_daily  [-KMNoprRV?]   [-c  control] [-k discard] [-l logfile] [-m addresses] [-s
       size] [-t want] [-x compress] [-X program] [-Y regex]

DESCRIPTION

These shell scripts and associated control files may be used to create a customized regime of
administration and management for Performance Co-Pilot (see PCPIntro(1)) archive log files.

pmlogger_check may be run at any time of the day and is intended to check that a desired set of
pmlogger(1) processes are running. If not, it (re-)starts any missing logger processes.

pmlogger_daily is intended to be run once per day, preferably in the early morning, as soon after
midnight as practicable. Its task is to aggregate, rotate and perform general housekeeping one or more
sets of PCP archives.

To accommodate the evolution of PMDAs and changes in production logging environments, pmlogger_daily is
integrated with pmlogrewrite(1) to allow optional and automatic rewriting of archives before merging. If
there are global rewriting rules to be applied across all archives mentioned in the control file(s), then
create the directory $PCP_SYSCONF_DIR/pmlogrewrite and place any pmlogrewrite(1) rewriting rules in this
directory. For rewriting rules that are specific to only one family of archives, use the directory name
from the control file(s) - i.e. the fourth field - and create a file, or a directory, or a symbolic link
named pmlogrewrite within this directory and place the required rewriting rule(s) in the pmlogrewrite
file or in files within the pmlogrewrite subdirectory. pmlogger_daily will choose rewriting rules from
the archive directory if they exist, else rewriting rules from $PCP_SYSCONF_DIR/pmlogrewrite if that
directory exists, else no rewriting is attempted.

As an alternate mechanism, if the file $PCP_LOG_DIR/pmlogger/.NeedRewrite exists when pmlogger_daily
starts then this is treated the same as specifying -R on the command line and
$PCP_LOG_DIR/pmlogger/.NeedRewrite will be removed once all the rewriting has been done.

OPTIONS

-c control, --control=control
Both pmlogger_check and pmlogger_daily are controlled by PCP logger control file(s) that specifies
the pmlogger instances to be managed. The default control file is $PCP_PMLOGGERCONTROL_PATH, but an
alternate may be specified using the -c option. If the directory $PCP_PMLOGGERCONTROL_PATH.d (or
control.d from the -c option) exists, then the contents of any additional control files therein will
be appended to the main control file (which must exist).

-C This option causes pmlogger_check to query the system service runlevel information for pmie, and use
that to determine whether to start processes or not.

-k period, --discard=period
After some period, old PCP archives are discarded. This period is 14 days by default, but may be
changed using this option. Some special values are recognized for the period, namely 0 to keep no
archives beyond the current one, and forever or never to prevent any archives being discarded. Note
that the semantics of discard are that it is measured from the time of last modification of each
archive, and not from the current day. This has subtle implications for compression (see below) -
the compression process results in the creation of new archive files which have new modification
times. In this case, the discard period (re)starts from the time of compression.

-K When this option is specified for pmlogger_daily then only the compression tasks are attempted, so
no pmlogger rotation, no culling, no rewriting, etc. When -K is used and a compress value of 0 is
in effect (from -x on the command line or PCP_COMPRESSAFTER in the environment or via the control
file) this is intended for environments where compression of archives is desired before the
scheduled daily processing happens. To achieve this, once pmlogger_check has completed regular
processing, it calls pmlogger_daily with just the -K option. Provided PCP_COMPRESSAFTER is set to 0
along with any other required compression options to match the scheduled invocation of
pmlogger_daily, then this will compress all volumes except the ones being currently written by
pmlogger(1).

-l file, --logfile=file
In order to ensure that mail is not unintentionally sent when these scripts are run from cron(8)
diagnostics are always sent to log files. By default, this file is
$PCP_LOG_DIR/pmlogger/pmlogger_check.log or $PCP_LOG_DIR/pmlogger/pmlogger_daily.log but this can be
changed using the -l option. If this log file already exists when the script starts, it will be
renamed with a .prev suffix (overwriting any log file saved earlier) before diagnostics are
generated to the log file. The -l and -t options cannot be used together.

-m addresses, --mail=addresses
Use of this option causes pmlogger_daily to construct a summary of the ``notices'' file entries
which were generated in the last 24 hours, and e-mail that summary to the set of space-separated
addresses. This daily summary is stored in the file $PCP_LOG_DIR/NOTICES.daily, which will be empty
when no new ``notices'' entries were made in the previous 24 hour period.

-M This option may be used to disable archive merging (or renaming) and rewriting (-M implies -r).
This is most useful in cases where the archives are being incrementally copied to a remote
repository, e.g. using rsync(1). Merging, renaming and rewriting all risk an increase in the
synchronization load, especially immediately after pmlogger_daily has run, so -M may be useful in
these cases.

-N, --showme
This option enables a ``show me'' mode, where the programs actions are echoed, but not executed, in
the style of ``make -n''. Using -N in conjunction with -V maximizes the diagnostic capabilities for
debugging.

-o By default all possible archives will be merged. This option reinstates the old behaviour in which
only yesterday's archives will be considered as merge candidates. In the special case where only a
single input archive needs to be merged, pmlogmv(1) is used to rename the archive, otherwise
pmlogger_merge(1) is used to merge all of the archives for a single host and a single day into a new
PCP archive and the individual archives are removed.

-p If this option is specified for pmlogger_daily then the status of the daily processing is polled and
if the daily pmlogger(1) rotation, culling, rewriting, compressing, etc. has not been done in the
last 24 hours then it is done now. The intent is to have pmlogger_daily called regularly with the
-p option (at 30 mins past the hour, every hour in the default cron(8) set up) to ensure daily
processing happens as soon as possible if it was missed at the regularly scheduled time (which is
00:10 by default), e.g. if the system was down or suspended at that time. With this option
pmlogger_daily simply exits if the previous day's processing has already been done. The -K and -p
options to pmlogger_daily are mutually exclusive.

-r, --norewrite
This command line option acts as an override and prevents all archive rewriting with pmlogrewrite(1)
independent of the presence of any rewriting rule files or directories.

-R, --rewriteall
Sometimes PMDA changes require all archives to be rewritten, not just the ones involved in any
current merging. This is required for example after a PCP upgrade where a new version of an
existing PMDA has revised metadata. The -R command line forces this universal-style of rewriting.
The -R option to pmlogger_daily is mutually exclusive with both the -r and -M options.

-s size, --rotate=size
If the PCP ``notices'' file ($PCP_LOG_DIR/NOTICES) is larger than 20480 bytes, pmlogger_daily will
rename the file with a ``.old'' suffix, and start a new ``notices'' file. The rotate threshold may
be changed from 20480 to size bytes using the -s option.

-s, --stop
Use of this option provides the reverse pmlogger_check functionality, allowing the set of pmlogger
processes to be cleanly shutdown.

-t period
To assist with debugging or diagnosing intermittent failures the -t option may be used. This will
turn on very verbose tracing (-VV) and capture the trace output in a file named
$PCP_LOG_DIR/pmlogger/daily.datestamp.trace, where datestamp is the time pmlogger_daily was run in
the format YYYYMMDD.HH.MM. In addition, the period argument will ensure that trace files created
with -t will be kept for period days and then discarded.

-T, --terse
This option to pmlogger_check produces less verbose output than the default. This is most suitable
for a pmlogger ``farm'' where many instances of pmlogger are expected to be running.

-V, --verbose
The output from the cron execution of the scripts may be extended using the -V option to the scripts
which will enable verbose tracing of their activity. By default the scripts generate no output
unless some error or warning condition is encountered. Using -N in conjunction with -V maximizes
the diagnostic capabilities for debugging.

-x period, --compress-after=period
Archive data files can optionally be compressed after some period to conserve disk space. This is
particularly useful for large numbers of pmlogger processes under the control of pmlogger_check. If
transparent_decompress is enabled when libpcp was built (can be checked with the pmconfig(1)
-Loption), then the default behaviour is compression ``as soon as possible''. Otherwise the default
behaviour is to not compress files (which matches the historical default behaviour in earlier PCP
releases). The -x option specifies the number of days after which to compress archive data and
metadata files. If compress is 0 then compression will be applied as soon as possible. If compress
is never or forever then no compression will be done. The environment variable PCP_COMPRESSAFTER
may be used as an alternative mechanism to define compress. If both PCP_COMPRESSAFTER and -x
specify different values for compress then the environment variable value is used and a warning is
issued.

-X program, --compressor=program
This option specifies the program to use for compression - by default this is xz(1). The
environment variable PCP_COMPRESS may be used as an alternative mechanism to define program. If
both PCP_COMPRESS and -X specify different compression programs then the environment variable value
is used and a warning is issued.

-Y regex, --regex=regex
This option allows a regular expression to be specified causing files in the set of files matched
for compression to be omitted - this allows only the data file to be compressed, and also prevents
the program from attempting to compress it more than once. The default regex is
".(index|Z|gz|bz2|zip|xz|lzma|lzo|lz4)$" - such files are filtered using the -v option to egrep(1).
The environment variable PCP_COMPRESSREGEX may be used as an alternative mechanism to define regex.
If both PCP_COMPRESSREGEX and -Y specify different values for regex then the environment variable
value is used and a warning is issued.

-?, --help
Display usage message and exit.

CONFIGURATION

Warning: The $PCP_PMLOGGERCONTROL_PATH and $PCP_PMLOGGERCONTROL_PATH.d files must not be writable by any
user other than root.

The control file(s) should be customized according to the following rules that define for the current
version (1.1) of the control file format.

1. Lines beginning with a ``#'' are comments.
2. Lines beginning with a ``$'' are assumed to be assignments to environment variables in the style of
sh(1), and all text following the ``$'' will be eval'ed by the script reading the control file, and
the corresponding variable exported into the environment. This is particularly useful to set and
export variables into the environment of the administrative scripts, e.g.
$ PMCD_CONNECT_TIMEOUT=20
3. There must be a version line in the initial control file of the form:
$ version=1.1
4. There should be one line in the control file(s) for each pmlogger instance of the form:

host y|n y|n directory args

5. Fields within a line of the control file(s) are usually separated by one or more spaces or tabs
(although refer to the description of the directory field for some important exceptions).
6. The first field is the name of the host that is the source of the performance metrics for this
pmlogger instance.
7. The second field indicates if this is a primary pmlogger instance (y) or not (n). Since the primary
logger must run on the local host, and there may be at most one primary logger for a particular host,
this field can be y for at most one pmlogger instance, in which case the host name must be the name
of the local host.
8. The third field indicates if this pmlogger instance needs to be started under the control of
pmsocks(1) to connect to a pmcd through a firewall (y or n).
9. The fourth field is a directory name. All files associated with this pmlogger instance will be
created in this directory, and this will be the current directory for the execution of any programs
required in the maintenance of those archives. A useful convention is that primary logger archives
for the local host with hostname myhost are maintained in the directory $PCP_ARCHIVE_DIR/myhost (this
is where the default pmlogger start-up script in $PCP_RC_DIR/pcp will create the archives), while
archives for the remote host mumble are maintained in $PCP_ARCHIVE_DIR/mumble.
10. The directory field may contain embedded shell syntax that will be evaluated by sh(1) to produce the
real directory name to be used. The allowed constructs are:
• Any text (including white space) enclosed with $( and ).
• Any text (including white space) enclosed with ` and ` (back quotes).
• Any text (including white space) enclosed with " and " (double quotes).
• Any word containing a $ (assumed to introduce an environment variable name).
11. All other fields are interpreted as arguments to be passed to pmlogger(1) and/or pmnewlog(1). Most
typically this would be the -c option.

The following sample control lines specify a primary logger on the local host (bozo), and non-primary
loggers to collect and log performance metrics from the hosts wobbly and boing.

$version=1.1
bozo y n $PCP_ARCHIVE_DIR/bozo -c config.default
wobbly n n "/store/wobbly/$(date +%Y)" -c ./wobbly.config
boing n n $PCP_ARCHIVE_DIR/boing -c ./pmlogger.config

Typical crontab(5) entries for periodic execution of pmlogger_daily and pmlogger_check are given in
$PCP_SYSCONF_DIR/pmlogger/crontab (unless installed by default in /etc/cron.d already) and shown below.

# daily processing of archive logs
14 0 * * * $PCP_BINADM_DIR/pmlogger_daily
# every 30 minutes, check pmlogger instances are running
25,55 * * * * $PCP_BINADM_DIR/pmlogger_check

When using systemd(1) on Linux, no crontab entries are needed as the timer mechanism provided by systemd
is used instead.

FILES

       $PCP_PMLOGGERCONTROL_PATH
            the PCP logger control file
            Warning: this file must not be writable by any user other than root.

       $PCP_PMLOGGERCONTROL_PATH.d
            optional directory containing additional PCP logger control files, typically one per host
            Warning: the files herein must not be writable by any user other than root.

       $PCP_SYSCONF_DIR/pmlogger/crontab
            sample crontab for automated script execution by $PCP_USER (or root).  Exists only if  the  platform
            does not support the /etc/cron.d mechanism.

       $PCP_VAR_DIR/config/pmlogger/config.default
            default  pmlogger  configuration  file  location  for  the local primary logger, typically generated
            automatically by pmlogconf(1).

       $PCP_ARCHIVE_DIR/<hostname>
            default location for archives of performance information collected from the host hostname

       $PCP_ARCHIVE_DIR/<hostname>/lock
            transient lock file to guarantee mutual  exclusion  during  pmlogger  administration  for  the  host
            hostname  -  if  present,  can  be  safely  removed if neither pmlogger_daily nor pmlogger_check are
            running

       $PCP_ARCHIVE_DIR/<hostname>/Latest
            PCP archive folio created by mkaf(1) for the most recently launched archive  containing  performance
            metrics from the host hostname

       $PCP_LOG_DIR/NOTICES
            PCP ``notices'' file used by pmie(1) and friends

       $PCP_LOG_DIR/pmlogger/pmlogger_check.log
            if  the  previous execution of pmlogger_check produced any output it is saved here.  The normal case
            is no output in which case the file does not exist.

       $PCP_LOG_DIR/pmlogger/pmlogger_daily.log
            if the previous execution of pmlogger_daily produced any output it is saved here.  The  normal  case
            is no output in which case the file does not exist.

       $PCP_ARCHIVE_DIR/<hostname>/SaveLogs
            if this directory exists, then the log file from the -l argument of a newly launched pmlogger(1) for
            hostname will be linked into this directory with the name archive.log where archive is the  basename
            of  the  associated  pmlogger(1)  PCP  archive files.  This allows the log file to be inspected at a
            later time, even if several pmlogger(1) instances for hostname have been launched  in  the  interim.
            Because  the  cron-driven  PCP  archive  management  scripts  run under the uid of the user ``pcp'',
            $PCP_ARCHIVE_DIR/hostname/SaveLogs typically needs to be owned by the user ``pcp''.

       $PCP_LOG_DIR/pmlogger/.NeedRewrite
            if this file exists, then this is treated as equivalent to using -R on the command line and the file
            will be removed once all rewriting has been done.

PCP ENVIRONMENT

       Environment  variables with the prefix PCP_ are used to parameterize the file and directory names used by
       PCP.  On each installation, the file /etc/pcp.conf contains the local values for  these  variables.   The
       $PCP_CONF variable may be used to specify an alternative configuration file, as described in pcp.conf(5).

       The  default  behaviour,  when  pmlogger(1)  configuration  comes from pmlogconf(1), is to regenerate the
       configuration file and check for changes whenever pmlogger(1) is started  from  pmlogger_check.   If  the
       PMDA  configuration  is  stable,  this  is not necessary, and setting $PMLOGGER_CHECK_SKIP_LOGCONF to yes
       disables the regeneration and checking.