xenial (1) pmlogextract.1.gz

Provided by: pcp_3.10.8build1_amd64 bug

NAME

       pmlogextract - reduce, extract, concatenate and merge Performance Co-Pilot archives

SYNOPSIS

       pmlogextract  [-dfwz]  [-c  configfile]  [-S  starttime]  [-s  samples]  [-T endtime] [-v volsamples] [-Z
       timezone] input [...] output

DESCRIPTION

       pmlogextract reads one or more Performance Co-Pilot (PCP) archive logs identified by input and creates  a
       temporally  merged  and/or reduced PCP archive log in output.  The nature of merging is controlled by the
       number of input archive logs, while the nature of data  reduction  is  controlled  by  the  command  line
       arguments.   The input(s) must be PCP archive logs created by pmlogger(1) with performance data collected
       from the same host, but usually over different time periods and  possibly  (although  not  usually)  with
       different performance metrics being logged.

       If  only  one input is specified, then the default behavior simply copies the input PCP archive log, into
       the output PCP archive log.  When two or more PCP archive logs are  specified  as  input,  the  logs  are
       merged (or concatenated) and written to output.

       In  the  output archive log a ``mark'' record will be inserted at a time just past the end of each of the
       input archive logs to indicate a possible temporal discontinuity between the end of one input archive log
       and  the  start  of the next input archive log.  See the MARK RECORDS section below for more information.
       There is no ``mark'' record after the end of the last (in temporal order) of the input archive logs.

COMMAND LINE OPTIONS

       The command line options for pmlogextract are as follows:

       -c configfile
              Extract only the metrics  specified  in  configfile  from  the  input  PCP  archive  log(s).   The
              configfile  syntax  accepted by pmlogextract is explained in more detail in the Configuration File
              Syntax section.

       -d     Desperate mode.  Normally if a fatal error occurs, all trace of the partially written PCP  archive
              output is removed.  With the -d option, the output archive log is not removed.

       -f     For most common uses, all of the input archive logs will have been collected in the same timezone.
              But if this is not the case, then pmlogextract must choose one of the  timezones  from  the  input
              archive  logs  to  be  used as the timezone for the output archive log.  The default is to use the
              timezone from the last input archive log.  The -f option forces the timezone from the first  input
              archive log to be used.

       -S starttime
              Define  the  start  of  a  time  window to restrict the samples retrieved or specify a ``natural''
              alignment of the output sample times; refer to PCPIntro(1).  See also the -w option.

       -s samples
              The argument samples defines the number of samples to be written to output.  If samples is 0 or -s
              is not specified, pmlogextract will sample until the end of the PCP archive log, or the end of the
              time window as specified by -T, whichever comes first.  The -s option will override the -T  option
              if it occurs sooner.

       -T endtime
              Define the termination of a time window to restrict the samples retrieved or specify a ``natural''
              alignment of the output sample times; refer to PCPIntro(1).  See also the -w option.

       -v volsamples
              The output archive log  is  potentially  a  multi-volume  data  set,  and  the  -v  option  causes
              pmlogextract  to  start a new volume after volsamples log records have been written to the archive
              log.

              Independent of any -v option, each volume of an archive is limited to no more than 2^31 bytes,  so
              pmlogextract will automatically create a new volume for the archive before this limit is reached.

       -w     Where  -S and -T specify a time window within the same day, the -w flag will cause the data within
              the time window to be extracted, for every day in the archive log.  For example, the options -w -S
              @11:00 -T @15:00 specify that pmlogextract should include archive log records only for the periods
              from 11am to 3pm on each day.  When -w is used, the  output  archive  log  will  contain  ``mark''
              records to indicate the temporal discontinuity between the end of one time window and the start of
              the next.

       -Z timezone
              Use timezone when displaying the date and time.  Timezone is in  the  format  of  the  environment
              variable TZ as described in environ(7).

       -z     Use  the  local timezone of the host from the input archive logs.  The default is to initially use
              the timezone of the local host.

CONFIGURATION FILE SYNTAX

       The configfile contains metrics of interest - only those metrics (or instances) mentioned  explicitly  or
       implicitly  in  the  configuration file will be included in the output archive.  Each specifications must
       begin on a new line, and may span multiple lines in  the  configuration  file.   Instances  may  also  be
       specified, but they are optional.  The format for each specification is

               metric [[instance[,instance...]]]

       where  metric may be a leaf or a non-leaf name in the Performance Metrics Name Space (PMNS, see pmns(5)).
       If a metric refers to a non-leaf node in the PMNS, pmlogextract will recursively  descend  the  PMNS  and
       include all metrics corresponding to descendent leaf nodes.

       Instances  are  optional, and may be specified as a list of one or more space (or comma) separated names,
       numbers or strings (enclosed in single or double quotes).  Elements in the  list  that  are  numbers  are
       assumed  to  be  internal instance identifiers - see pmGetInDom(3) for more information.  If no instances
       are given, then all instances of the associated metric(s) will be extracted.

       Any additional white space is ignored and comments may be added with a `#' prefix.

CONFIGURATION FILE EXAMPLE

       This is an example of a valid configfile:

               #
               # config file for pmlogextract
               #

               kernel.all.cpu
               kernel.percpu.cpu.sys ["cpu0","cpu1"]
               disk.dev ["dks0d1"]

MARK RECORDS

       When more than one input archive log contributes  performance  data  to  the  output  archive  log,  then
       ``mark'' records are inserted to indicate a possible discontinuity in the performance data.

       A  ``mark''  record  contains a timestamp and no performance data and is used to indicate that there is a
       time period in the PCP archive log where we do not know the values of any  performance  metrics,  because
       there  was  no pmlogger(1) collecting performance data during this period.  Since these periods are often
       associated with the restart of a service or pmcd(1) or a system, there may be considerable  doubt  as  to
       the continuity of performance data across this time period.

       The  rationale  behind  ``mark'' records may be demonstrated with an example.  Consider one input archive
       log that starts at 00:10 and ends at 09:15 on the same day, and another input archive log that starts  at
       09:20  on  the  same  day  and  ends at 00:10 the following morning.  The would be a very common case for
       archives managed and rotated by pmlogger_check(1) and pmlogger_daily(1).

       The output archive log would contain:
       00:10.000   first record from first input archive log
       ...
       09:15.000   last record from first input archive log
       09:15.001   <mark record>
       09:20.000   first record from second input archive log
       ...
       01:10.000   last record from second input archive log

       The time period where the performance data is missing starts just after 09:15 and ends just before 09:20.
       When the output archive log is processed with any of the PCP reporting tools, the ``mark'' record is used
       to indicate a period of missing data.  For example in the archive above, if one was reporting the average
       I/O  rate  at 30 minute intervals, aligned on the hour, then there would be data for the intervals ending
       at 09:00 and 10:00 but no data reported for the interval ending at 09:30 as this spans a ``mark'' record.

       The presence of ``mark'' records in a PCP archive log can  be  established  using  pmdumplog(1)  where  a
       timestamp and the annotation <mark> is used to indicate a ``mark'' record.

METADATA CHECKS

       When  more  than  one  input archive is specified, pmlogextract performs a number of checks to ensure the
       metadata is consistent for metrics appearing in more than  one  of  the  input  archives.   These  checks
       include:

       * metric data type is the same
       * metric semantics are the same
       * metric units are the same
       * metric is either always singular or always has the same instance domain
       * metrics with the same name have the same PMID
       * metrics with the same PMID have the same name

       If  any  of  these  checks  fail, pmlogextract reports the details and aborts without creating the output
       archive.

       To address these semantic issues, use pmlogrewrite(1) to translate the  input  archives  into  equivalent
       archives with consistent metdadata before using pmlogextract.

FILES

       For each of the input and output archive logs, several physical files are used.
       archive.meta
                 metadata (metric descriptions, instance domains, etc.) for the archive log
       archive.0 initial volume of metrics values (subsequent volumes have suffixes 1, 2, ...) - for input these
                 files may have been previously compressed with  bzip2(1)  or  gzip(1)  and  thus  may  have  an
                 additional .bz2 or .gz suffix.
       archive.index
                 temporal index to support rapid random access to the other files in the archive log.

PCP ENVIRONMENT

       Environment  variables with the prefix PCP_ are used to parameterize the file and directory names used by
       PCP.  On each installation, the file /etc/pcp.conf contains the local values for  these  variables.   The
       $PCP_CONF variable may be used to specify an alternative configuration file, as described in pcp.conf(5).

SEE ALSO

       PCPIntro(1),   pmdumplog(1),  pmlc(1),  pmlogger(1),  pmlogreduce(1),  pmlogrewrite(1),  pcp.conf(5)  and
       pcp.env(5).

DIAGNOSTICS

       All error conditions detected by pmlogextract are reported on stderr with textual  (if  sometimes  terse)
       explanation.

       Should  one  of the input archive logs be corrupted (this can happen if the pmlogger instance writing the
       log suddenly dies), then pmlogextract will detect and report the position of the corruption in the  file,
       and any subsequent information from that archive log will not be processed.

       If any error is detected, pmlogextract will exit with a non-zero status.

CAVEATS

       The  preamble  metrics  (pmcd.pmlogger.archive,  pmcd.pmlogger.host,  and  pmcd.pmlogger.port), which are
       automatically recorded by pmlogger at the start of the archive, may not be present in the archive  output
       by  pmlogextract.   These  metrics  are  only  relevant  while  the archive is being created, and have no
       significance once recording has finished.