Provided by: pcp_6.0.3-1_amd64 bug

NAME

       pmdaopenmetrics - OpenMetrics PMDA

SYNOPSIS

       $PCP_PMDAS_DIR/openmetrics/pmdaopenmetrics  [-D] [-n] [-c config] [-d domain] [-l logfile]
       [-r root] [-t timeout] [-u user]

DESCRIPTION

       pmdaopenmetrics is a Performance Metrics Domain Agent (PMDA) which dynamically creates PCP
       metrics  from  configured  OpenMetrics  endpoints,  which  provide  HTTP  based  access to
       application metrics.  The PMDA essentially implements a bridge between Prometheus and PCP,
       allowing  PCP  to  easily ingest performance data from more than 650 registered end-points
       and many other application specific end-points.

       The default config directory is $PCP_PMDAS_DIR/openmetrics/config.d/, see  ``CONFIGURATION
       SOURCES''  below.   The  default URL fetch timeout is 2 seconds.  The default user, if not
       specified with the -u option, is the current user.  If the -n option is given, the list of
       configuration  files  will  not  be  sorted  prior  to processing.  This list is sorted by
       default but that can be expensive if there are a large number of configuration files (URLs
       and/or scripts).

       If  the -D option is given, additional diagnostic messages will be written to the PMDA log
       file, which is  $PCP_LOG_DIR/pmcd/openmetrics.log  by  default  (see  also  -lbelow).   In
       addition, the metric openmetrics.control.debug controls the same debug flag and can be set
       with the following command:
            pmstore openmetrics.control.debug value
       where value is either 1 (to enable verbose log messages) or  0  (to  disable  verbose  log
       messages).   This  is  particularly  useful  for examining the http headers passed to each
       fetch request, filter settings and other processing  details  that  are  logged  when  the
       debugging flag is enabled.

       The -d option may be used to override the default performance metrics domain number, which
       defaults to 144.  It is strongly recommended not to change this.  The domain number should
       be different for every PMDA on the one host, and the same domain number should be used for
       pmdaopenmetrics PMDA on all hosts.  See also the -r option, which allows the root  of  the
       dynamic namespace to be changed from the default openmetrics.

       The  -l option may be used to specify logfile as the destination for PMDA messages instead
       of the default, $PCP_LOG_DIR/pmcd/openmetrics.log.  As a special case, logfile may be  "-"
       to  send  messages  to  the  stderr stream instead, e.g.  -l-.  This would normally be the
       stderr stream for the parent process, pmcd(1), which may itself  have  redirected  stderr.
       This  redirection  is  normally  most useful in a containerized environment, or when using
       dbpmda(1).

       The -r option allows the root of the dynamic namespace to be  changed  to  root  from  the
       default,  openmetrics.   In  conjunction  with  other  command  line  options, this allows
       pmdaopenmetrics to be deployed as a different PMDA with  distinct  metrics  namespace  and
       metrics  domain  on  the  same  host  system.  Note that all PMDAs require a unique domain
       number so the -d option must also be specified.  Use of the -r option may also change  the
       defaults  for  some  other  command  line  options, e.g. the default log file name and the
       default configuration directory.

CONFIGURATION SOURCES

       As     it     runs,     pmdaopenmetrics     periodically     recursively     scans     the
       $PCP_PMDAS_DIR/openmetrics/config.d  directory  (or  the  directory  specified with the -c
       option), looking for source URL files (*.url) and executable  scripts  or  binaries.   Any
       files  that  do  not have the .url suffix or are not executable, are ignored - this allows
       documentation  files  such  as  "README"  and  non-executable  "common"  script   function
       definitions to be present without being considered as config files.

       A  remote  server  does not have to be up or stay running - the PMDA tolerates remote URLs
       that may come and go over time.  The PMDA will relay data and metadata  when/if  they  are
       available,  and  will  return  errors when/if they are down.  PCP metric IDs, internal and
       external instance domain identifiers are persisted and will be  restored  when  individual
       metric  sources become available and/or when the PMDA is restarted.  In addition, the PMDA
       checks directory modification times and will rescan for new or changed configuration files
       dynamically.   It  is  not necessary to restart the PMDA when adding, removing or changing
       configuration files.

URL SOURCES

       Each file with the .url suffix found in the config directory or sub-directory contains one
       complete  HTTP  or  HTTPS  URL  at which pmdaopenmetrics can reach a OpenMetrics endpoint.
       Local file access is also supported with a conventional file:///somepath/somefile URL,  in
       which case somepath/somefile should contain openmetrics formatted metric data.

       The  first  line  of a .url config file should be the URL, as described above.  Subsequent
       lines, if any, are prefixed with a keyword that can be used to alter the http GET request.
       A  keyword must end with ':' (colon) and the text extends to the end of the line.  Comment
       lines that start with # and  blank  lines  are  ignored.   The  only  currently  supported
       keywords are HEADER: and FILTER:.

       HEADER: headername: value ... to end of line
       Adds  headername  and  its  value  to  the  headers passed in the http GET request for the
       configured URL.  An example configuration file that provides 3 commonly used  headers  and
       an authentication token might be :

          http://somehost/path/endpoint.html
          # this is a comment
          HEADER: Accept: text/html
          HEADER: Keep-Alive: 300
          HEADER: Connection: keep-alive
          HEADER: Authorization: token ABCDEF1234567890

       As  mentioned  above,  header  values extend to the end of the line.  They may contain any
       valid characters, including colons.  Multiple spaces will be collapsed to a single  space,
       and  leading  and trailing spaces are trimmed.  A common use for headers is to configure a
       proxy agent and the assorted parameters it may require.

METRIC FILTERING

       Metric filtering is a configuration file  feature  that  allows  ingested  metrics  to  be
       included  or  excluded,  i.e.  filtered.   This  is  useful because most end-points return
       multiple metrics, and usually only some are  interesting  for  monitoring  purposes.   The
       syntax is:
       FILTER: INCLUDE METRIC regex
       or
       FILTER: EXCLUDE METRIC regex
       Dynamically  created  metric names that match regex will be either included or excluded in
       the name space, as specified.  Note that only the PMNS leaf component of the  metric  name
       (as  ingested from the URL source) is compared with the regex pattern.  The simple rule is
       that the first matching filter regex for a particular metric leaf name is  the  rule  that
       prevails.   If  no  filter  regex  matches  (or  there are no filters), then the metric is
       included by default, i.e. the default filter if none  are  specified  is  FILTER:  INCLUDE
       METRIC  .*  This is backward compatible with older versions of the configuration file that
       did not support filters.  Multiple FILTER: lines would normally be used, e.g.  to  include
       some metrics but exclude all others, use FILTER: EXCLUDE METRIC .*  as the last of several
       filters that include the desired metrics.  Conversely, to exclude some metrics but include
       all  others,  use  FILTER:  EXCLUDE METRIC regex.  In this case it's not necessary (though
       doesn't hurt) to specify the final FILTER: INCLUDE METRIC .*  because,  as  stated  above,
       any metric that does not match any filter regex will be included by default.

LABEL FILTERING

       Label  filtering  uses  similar FILTER: syntax and semantics as metric filtering.  FILTER:
       EXCLUDE LABEL regex will delete all labels with label name matching regex from all metrics
       defined  by  the configuration file.  The same rules as for metric filters apply for label
       filters too - an implicit rule: FILTER: INCLUDE LABEL .*  applies to all  labels  that  do
       not  match  any  earlier  label  filter rule.  FILTER: OPTIONAL LABEL regex specifies that
       matching label names are to be included in the returned metric labelsets (i.e.  included),
       but  are  not  to be used as part of the the external instance names.  All included labels
       that are not optional (i.e. the intrinsic labels) will be concatenated together  and  used
       for  external  instance  naming.  In addition, non-intrinsic labels (i.e. labels tagged as
       OPTIONAL) will have the PM_LABEL_OPTIONAL flag set in  the  labelsets  returned  by  notes
       callbacks.   This  flag  affects  how the labels are used in certain clients.  For further
       details, see pmLookupLabels(3) and related man  pages  for  further  details.   Note  that
       external  instance  names  begin  with  the  unique  numeric  internal instance identifier
       followed by a space, so external instance names are always unique.

       Caution is needed with label  filtering  because  by  default,  all  labels  are  used  to
       construct the PCP instance name.  By excluding some labels (or changing them to optional),
       the instance names will change.  In addition, excluding all labels for a particular metric
       changes  that  metric  to  be  singular,  i.e. have no instance domain.  By excluding some
       labels, different instances returned by the URL or scripted configuration  entry  for  the
       same  metric  may  become  duplicates.   When  such  duplicates  occur, the last duplicate
       instance returned by the end-point URL or script prevails over any earlier instances.  For
       these  reasons,  it  is  recommended  that  label  filtering  rules be configured when the
       configuration file is first defined, and not changed thereafter.   If  a  label  filtering
       change  is required, the configuration file should be renamed, which effectively defines a
       new metric (or set of peer metrics as returned by the URL or script),  with  the  new  (or
       changed) instance naming.

       Unrecognized  keywords  in  configuration  files  are  reported  in  the PMDA log file but
       otherwise ignored.

SCRIPTED SOURCES

       Executable scripts present in the $PCP_PMDAS_DIR/openmetrics/config.d  directory  or  sub-
       directories will be executed and the stdout stream containing openmetrics formatted metric
       data will be parsed as though it had come from a URL or file.  The stderr  stream  from  a
       script   will  be  sent  to  the  PMDA  log  file,  which  by  default  can  be  found  in
       $(PCP_LOG_DIR)/pmcd/openmetrics.log.

       Note that scripted sources do not support label or metric filtering  (as  described  above
       for  URL  sources)  -  they  can  simply  do their own filtering in the script itself with
       sed(1), awk(1), or whatever tool is desired.

       A simple example of a scripted config entry follows:

          #! /bin/sh
          awk '{
              print("# HELP loadavg local load average")
              print("# TYPE loadavg gauge")
              printf("loadavg {interval=\"1-minute\"} %.2f\n", $1)
              printf("loadavg {interval=\"5-minute\"} %.2f\n", $2)
              printf("loadavg {interval=\"15-minute\"} %.2f\n", $3)
          }' /proc/loadavg

       This script produces the following OpenMetrics-formatted metric data when run:

          # HELP loadavg local load average
          # TYPE loadavg gauge
          loadavg {interval="1-minute"} 0.12
          loadavg {interval="5-minute"} 0.27
          loadavg {interval="15-minute"} 0.54

       If   the   above   script   was   saved   and   made   executable   in   a   file    named
       $PCP_PMDAS_DIR/openmetrics/config.d/local/system.sh  then  this  would result in a new PCP
       metric named openmetrics.local.system.loadavg which would have  three  instances  for  the
       current load average values: 1-minute, 5-minute and 15-minute.

       Scripted  config entries may produce more than one PCP leaf metric name.  For example, the
       above "system.sh" script could also export  other  metrics  such  as  CPU  statistics,  by
       reading  /proc/stat  on  the  local  system.  Such additional metrics would appear as peer
       metrics in the same PCP metric subtree.  In the case of  CPU  counters,  the  metric  type
       definition  should  be counter, not gauge.  For full details of the openmetrics exposition
       formats,                                                                               see
       https://github.com/OpenObservability/OpenMetrics/blob/master/specification/OpenMetrics.md.

SELinux CONSIDERATIONS

       Scripted  config  files  are  executed  by  the pmdaopenmetrics PMDA with the same SELinux
       context and policy as the local pmcd(1).  For simple scripts, such  as  the  load  average
       example described above, this is normally fine.  However AVC errors may result for scripts
       that make library or system calls that are restricted by the  prevailing  SELinux  context
       and  policies.  In these cases it is not feasible to unilaterally grant pmcd or it's PMDAs
       an unconfined execution policy.  In these site specific cases  it  will  be  necessary  to
       create  a  local  SELinux  policy module.  This can be done by capturing the AVC record(s)
       from the local audit log, generate a local policy module using audit2allow, and then  load
       the new module using semodule, e.g. as follows :

              $ sudo grep '^type=AVC.*pcp' /var/log/audit/audit.log \
              | audit2allow -M mypolicy
              $ sudo semodule -i mypolicy.pp

       If  these local policies need to be persistent across reboots, then a scriptlet similar to
       the  above   example   may   be   added   to   the   local   pmcd   RC   file   (typically
       /etc/pcp/pmcd/rc.local).  For further details, see audit2allow(1) and semodule(1).

METRIC NAMING

       All  metrics  from  a  file  named  JOB.*   will  be  exported  as  PCP  metrics  with the
       openmetrics.JOB metric name prefix.  Therefore, the JOB name must be a valid non-leaf name
       for  PCP  PMNS  metric  names.  If the JOB name has multiple dot-separated components, the
       resulting PMNS names will include those components and care is needed to ensure there  are
       no  overlapping definitions, e.g. metrics returned by JOB.response may overlap or conflict
       with metrics returned by JOB.response.time.

       Config file entries (URLs or scripts) found in subdirectories of the config directory will
       also   result   in   hierarchical   metric  names.   For  example,  a  config  file  named
       $PCP_PMDAS_DIR/openmetrics/config.d/mysource/latency/get.url will result in metrics  being
       created  (by  fetching  that source URL) below openmetrics.mysource.latency.get in the PCP
       namespace.  Scripts found in subdirectories of the config directory  similarly  result  in
       hierarchical PCP metric names.

DYNAMIC METRIC NAMES

       As  described  above,  changes and new additions can be made to files in the configuration
       directory without having to restart the PMDA.  These changes  are  detected  automatically
       and  the  PCP metric names below openmetrics in the PMNS will be updated accordingly, i.e.
       new metrics will be dynamically added  and/or  existing  metrics  removed.   In  addition,
       pmdaopenmetrics  honors  the  PMCD_NAMES_CHANGE pmFetch(3) protocol that was introduced in
       PCP version 4.0.  In particular, if openmetrics metrics are being logged by a PCP  version
       4.0  or  later  pmlogger(1),  new  metrics  that appear as a result of changes in the PMDA
       configuration directory will automatically start to be logged, provided the  root  of  the
       openmetrics  PMDA  namespace is configured for logging in the pmlogger configuration file.
       See pmlogger(1) for details.  An example of such a pmlogger configuration file is :

          log mandatory on 2 second {
               # log all metrics below the root of the openmetrics namespace
               openmetrics
          }

METADATA

       Metric data returned by URL or scripted configuration files may contain metadata that  can
       be  used by the openmetrics PMDA to specify the semantics, data type, scaling and units of
       dynamically created metrics.  This metadata is prefixed with  #  PCP5  or  #  PCP  in  the
       ingested  metric data.  For additional information about PCP metadata, see pmLookupDesc(3)
       and pmParseUnitsStr(3) and examples in shipped configuration files.

       In-line "PCP5" metadata must be supplied by the metrics source end-pont (URL  or  script).
       An  alternative  is  to specify this in the URL configuration file directly, which has the
       advantage of not having to modify the source/end-point if the  metadata  is  incorrect  or
       missing.   Metadata  specified  in  the  URL  configuration  file  over-rides  any in-line
       metadata.

       The configuration file syntax for specifying metadata is:
       METADATA: regex type indom semantics units ... to EOL
       Where:
       METADATA: is literal
       regex is an extended regular expression to match one or more metric names returned by  the
       URL,
       type is one of the PCP numeric types (32, u32, 64, u64, float or double),
       indom is an arbitrary instance domain name, or PM_INDOM_NULL,
       semantics is either counter, instant or discrete and
       units  is  either  none  or  a  string  extending  to  end  of  line  that is parseable by
       pmParseUnitsStr(3), i.e. the units,  dimensions  and  scaling  to  be  used  for  matching
       metrics.

       An  example configuration file that ingests metrics from the Grafana /metrics end-point on
       localhost,   filters   out   all   metrics   returned   by    that    URL    except    for
       grafana_api_response_status_total and then specifies the metric type is an unsigned 32 bit
       integer with a non-singular instance domain named  response  and  counter  semantics  with
       units of count.

       http://localhost:3000/metrics
       FILTER: INCLUDE METRIC grafana_api_response_status_total
       FILTER: EXCLUDE METRIC .*
       METADATA: grafana_api_response_status_total u32 response counter count

       Note  that the name in the indom field is presently ignored unless it is PM_INDOM_NULL, in
       which case the metric has no instance domain (i.e. singular), even if it has labels  which
       would otherwise be used for instance naming.

CONTROL METRICS

       The   PMDA   maintains   special   control   metrics,  as  described  below.   Apart  from
       openmetrics.control.debug, each of these metrics has  one  instance  for  each  configured
       metric  source.   All  of these metrics have integer values with counter semantics, except
       openmetrics.control.status, which has a string  value.   It  is  important  to  note  that
       fetching  any  of the openmetrics.control metrics will only update the counters and status
       values if the corresponding URL is actually fetched.  If the source URL  is  not  fetched,
       the  control  metric  values  do  not  trigger  a  refresh and the control values reported
       represent the most recent fetch of each corresponding source.

       The instance domain for the openmetrics.control metrics is  adjusted  dynamically  as  new
       sources  are  discovered.   If there are no sources configured, the metric names are still
       defined but the instance domain will be empty and a fetch will return no values.

       openmetrics.control.status
              A string representing the status of the last fetch  of  the  corresponding  source.
              This  will  generally be success for an http response code of 200.  This metric can
              be used for service availability  monitoring  -  provided,  as  stated  above,  the
              corresponding source URL is fetched too.

       openmetrics.control.status_code
              This  metric is similar to openmetrics.control.status except that it is the integer
              response code of the last fetch.  A value of 200 usually signifies success and  any
              other  value  failure.   This  metric  can  also  be  used for service availability
              monitoring, with the same caveats as openmetrics.control.status.

       openmetrics.control.calls
              total number of times each configured metric source has been  fetched  (if  it's  a
              URL)  or  executed  (if  it's  a  script), since the PMDA started.  This metric has
              counter semantics and would normally be converted to a rate/second by client tools.

       openmetrics.control.fetch_time
              Total time in milliseconds that each configured metric source has taken to return a
              document,  excluding  the  time  to  parse  the  document.  This metric has counter
              semantics and would normally be rate converted by client tools but is  also  useful
              in raw form as the accumulated parse time since the PMDA was started.

       openmetrics.control.parse_time
              Total  time  in  milliseconds that each configured metric source has taken to parse
              each document, excluding the time to fetch the document.  This metric  has  counter
              semantics  and  would normally be rate converted by client tools but is also useful
              in raw form as the accumulated parse time since the PMDA was started.

       When converted to a rate, the calls metric represents  the  average  fetch  rate  of  each
       source  over  the  sampling  interval  (time  delta  between samples).  The fetch_time and
       parse_time counters, when converted to a rate, represent the  average  fetch  and  parsing
       latency (respectfully), during the sampling interval.

       The openmetrics.control.debug metric has a singular value, defaulting to 0.  If a non-zero
       value is stored into this metric using  pmstore(1),  additional  debug  messages  will  be
       written to the PMDA log file.

LIMITATIONS

       pmdaopenmetrics and libpcp internals impose some numerical constraints about the number of
       sources (4095),  metrics  (1024)  within  each  source,  and  instances  for  each  metric
       (4194304).

INSTALLATION

       Install the OpenMetrics PMDA by using the Install script as root:

           # cd $PCP_PMDAS_DIR/openmetrics
           # ./Install

       To uninstall, the following must be done as root:

           # cd $PCP_PMDAS_DIR/openmetrics
           # ./Remove

       pmdaopenmetrics is launched by pmcd(1) and should never be executed directly.  The Install
       and Remove scripts notify pmcd when the agent is installed or removed.

       When scripts and .url files are added, removed or changed in the configuration  directory,
       it is usually not necessary to restart the PMDA - the changes will be detected and managed
       on subsequent requests to the PMDA.

FILES

       $PCP_PMDAS_DIR/openmetrics/Install
           installation script for the pmdaopenmetrics agent

       $PCP_PMDAS_DIR/openmetrics/Remove
           undo installation script for the pmdaopenmetrics agent

       $PCP_PMDAS_DIR/openmetrics/config.d/
           contains URLs and scripts used by the pmdaopenmetrics agent as sources of  openmetrics
           metric data.

       $PCP_LOG_DIR/pmcd/openmetrics.log
           default log file for error messages from pmdaopenmetrics

       $PCP_VAR_DIR/config/144.*
           files containing internal tables for metric and instance ID number persistence (domain
           144).

PCP ENVIRONMENT

       Environment variables with the prefix PCP_ are used to parameterize the file and directory
       names used by PCP.  On each installation, the file /etc/pcp.conf contains the local values
       for these variables.  The $PCP_CONF  variable  may  be  used  to  specify  an  alternative
       configuration file, as described in pcp.conf(5).

SEE ALSO

       PCPIntro(1),  audit2allow(1),  pmcd(1),  pminfo(1),  pmlogger(1), pmstore(1), PMWEBAPI(3),
       pmFetch(3),               pmLookupLabels(3),               semodule(1),                and
       https://prometheus.io/docs/instrumenting/exposition_formats.