NAME

       pmdaopenmetrics - OpenMetrics PMDA

SYNOPSIS

       $PCP_PMDAS_DIR/openmetrics/pmdaopenmetrics  [-D]  [-n] [-c config] [-d domain] [-l logfile] [-r root] [-t
       timeout] [-u user]

DESCRIPTION

       pmdaopenmetrics is a Performance Metrics Domain Agent (PMDA) which creates PCP metrics  from  OpenMetrics
       endpoints,  which  provide  HTTP  based  access  to application metrics.  The default config directory is
       $PCP_PMDAS_DIR/openmetrics/config.d/, see ``CONFIGURATION SOURCES'' below.  The default URL fetch timeout
       is  2  seconds.   The  default user, if not specified with the -u option, is the current user.  If the -n
       option is given, the list of configuration files will not be sorted prior to processing.   This  list  is
       sorted  by  default  but  that  can be expensive if there are a large number of configuration files (URLs
       and/or scripts).

       If the -D option is given, additional diagnostic messages will be written to the PMDA log file, which  is
       $PCP_LOG_DIR/pmcd/openmetrics.log   by   default   (see   also   -lbelow).    In   addition,  the  metric
       openmetrics.control.debug controls the same debug flag and can be set with the following command:
            pmstore openmetrics.control.debug value
       where value is either 1 (to enable verbose log messages) or 0 (to disable verbose log messages).  This is
       particularly  useful  for  examining  the  http headers passed to each fetch request, filter settings and
       other processing details that are logged when the debugging flag is enabled.

       The -d option may be used to override the default performance metrics domain number,  which  defaults  to
       144.   It  is  strongly  recommended not to change this.  The domain number should be different for every
       PMDA on the one host, and the same domain number should be used for pmdaopenmetrics PMDA  on  all  hosts.
       See  also  the  -r  option, which allows the root of the dynamic namespace to be changed from the default
       openmetrics.

       The -l option may be used to specify logfile as the destination for PMDA messages instead of the default,
       $PCP_LOG_DIR/pmcd/openmetrics.log.   As a special case, logfile may be "-" to send messages to the stderr
       stream instead, e.g.  -l-.  This would normally be the stderr stream for  the  parent  process,  pmcd(1),
       which  may  itself  have  redirected stderr.  This redirection is normally most useful in a containerized
       environment, or when using dbpmda(1).

       The -r option allows the root of  the  dynamic  namespace  to  be  changed  to  root  from  the  default,
       openmetrics.   In conjunction with other command line options, this allows pmdaopenmetrics to be deployed
       as a different PMDA with distinct metrics namespace and metrics domain on the  same  host  system.   Note
       that  all  PMDAs  require  a unique domain number so the -d option must also be specified.  Use of the -r
       option may also change the defaults for some other command line options, e.g. the default log  file  name
       and the default configuration directory.

CONFIGURATION SOURCES

       As  it  runs,  pmdaopenmetrics  periodically  recursively  scans  the $PCP_PMDAS_DIR/openmetrics/config.d
       directory (or the directory specified with the -c option), looking  for  source  URL  files  (*.url)  and
       executable  scripts  or  binaries.  Any files that do not have the .url suffix or are not executable, are
       ignored - this allows documentation files such as "README" and non-executable  "common"  script  function
       definitions to be present without being considered as config files.

       A remote server does not have to be up or stay running - the PMDA tolerates remote URLs that may come and
       go over time.  The PMDA will relay data and metadata when/if they are available, and will  return  errors
       when/if  they  are down.  PCP metric IDs, internal and external instance domain identifiers are persisted
       and will be restored when individual metric sources become available and/or when the PMDA  is  restarted.
       In  addition,  the  PMDA  checks  directory  modification  times  and  will  rescan  for  new  or changed
       configuration files dynamically.  It is not necessary to  restart  the  PMDA  when  adding,  removing  or
       changing configuration files.

URL SOURCES

       Each  file with the .url suffix found in the config directory or sub-directory contains one complete HTTP
       or HTTPS URL at which pmdaopenmetrics can reach a  OpenMetrics  endpoint.   Local  file  access  is  also
       supported  with  a  conventional  file:///somepath/somefile  URL,  in which case somepath/somefile should
       contain openmetrics formatted metric data.

       The first line of a .url config file should be the URL, as described above.  Subsequent  lines,  if  any,
       are  prefixed with a keyword that can be used to alter the http GET request.  A keyword must end with ':'
       (colon) and the text extends to the end of the line.  Comment lines that start with # and blank lines are
       ignored.  The only currently supported keywords are HEADER: and FILTER:.

       HEADER: headername: value ... to end of line
       Adds  headername  and its value to the headers passed in the http GET request for the configured URL.  An
       example configuration file that provides 3 commonly used headers and an authentication token might be :

          http://somehost/path/endpoint.html
          # this is a comment
          HEADER: Accept: text/html
          HEADER: Keep-Alive: 300
          HEADER: Connection: keep-alive
          HEADER: Authorization: token ABCDEF1234567890

       As mentioned above, header values extend to the end of the line.  They may contain any valid  characters,
       including  colons.   Multiple spaces will be collapsed to a single space, and leading and trailing spaces
       are trimmed.  A common use for headers is to configure a proxy agent and the assorted parameters  it  may
       require.

       FILTER: INCLUDE METRIC regex
       or
       FILTER: EXCLUDE METRIC regex
       Dynamically  created metric names that match regex will be either included or excluded in the name space,
       as specified.  The simple rule is that the first matching filter regex for a particular  metric  name  is
       the  rule  that  prevails.   If  no  filter  regex  matches (or there are no filters), then the metric is
       included by default, i.e. the default filter if none are specified is FILTER: INCLUDE METRIC .*  This  is
       backward compatible with older versions of the configuration file that did not support filters.  Multiple
       FILTER: lines would normally be used, e.g. to include some metrics but exclude all  others,  use  FILTER:
       EXCLUDE  METRIC  .*   as  the  last  of several filters that include the desired metrics.  Conversely, to
       exclude some metrics but include all others, use FILTER: EXCLUDE METRIC regex.  In  this  case  it's  not
       necessary (though doesn't hurt) to specify the final FILTER: INCLUDE METRIC .*  because, as stated above,
       any metric that does not match any filter regex will be included by default.

       Label filtering uses similar FILTER: syntax and semantics.  FILTER: EXCLUDE LABEL regex will  delete  all
       labels  matching regex from all metrics defined in the configuration file.  The same rules as for metrics
       apply for labels too - an implicit rule: FILTER: INCLUDE LABEL .*  applies to  all  labels  that  do  not
       match any earlier filter rule.

       Caution  is  needed  with  label  filtering  because by default, all labels are used to construct the PCP
       instance name.  By excluding some labels, the instance names will change.  Excluding  all  labels  for  a
       particular  metric  changes  that  metric  to be singular, i.e. have no instance domain.  In addition, by
       excluding some labels, different  instances  of  the  same  metric  may  become  duplicates.   When  such
       duplicates  occur,  the  last  duplicate instance returned by the end-point URL prevails over any earlier
       instances.  For these reasons, it is recommended that  label  filtering  rules  be  configured  when  the
       configuration  file  is  first  defined,  and  not  changed  thereafter.   If a label filtering change is
       required, the configuration file should be renamed, which effectively defines a new metric, with the  new
       (or changed) instance naming.

       Unrecognized keywords in configuration files are reported in the PMDA log file but otherwise ignored.

SCRIPTED SOURCES

       Executable  scripts  present in the $PCP_PMDAS_DIR/openmetrics/config.d directory or sub-directories will
       be executed and the stdout stream containing openmetrics formatted metric data will be parsed  as  though
       it had come from a URL or file.  The stderr stream from a script will be sent to the PMDA log file, which
       by default can be found in $(PCP_LOG_DIR)/pmcd/openmetrics.log.

       Note that scripted sources do not support label or metric filtering (as described above for URL  sources)
       -  they  can  simply do their own filtering in the script itself with sed(1), awk(1), or whatever tool is
       desired.

       A simple example of a scripted config entry follows:

          #! /bin/sh
          awk '{
              print("# HELP loadavg local load average")
              print("# Type loadavg gauge")
              printf("loadavg {interval=\"1-minute\"} %.2f\n", $1)
              printf("loadavg {interval=\"5-minute\"} %.2f\n", $2)
              printf("loadavg {interval=\"15-minute\"} %.2f\n", $3)
          }' /proc/loadavg

       This script produces the following OpenMetrics-formatted metric data when run:

          # HELP loadavg local load average
          # Type loadavg gauge
          loadavg {interval="1-minute"} 0.12
          loadavg {interval="5-minute"} 0.27
          loadavg {interval="15-minute"} 0.54

       If    the    above    script    was    saved    and    made    executable     in     a     file     named
       $PCP_PMDAS_DIR/openmetrics/config.d/local/system.sh  then  this  would  result  in a new PCP metric named
       openmetrics.local.system.loadavg which would have three instances for the current  load  average  values:
       1-minute, 5-minute and 15-minute.

       Scripted  config  entries  may  produce  more  than  one  PCP  leaf  metric name.  For example, the above
       "system.sh" script could also export other metrics such as CPU statistics, by reading /proc/stat  on  the
       local  system.   Such additional metrics would appear as peer metrics in the same PCP metric subtree.  In
       the case of CPU counters, the metric type definition should be counter, not gauge.  For full  details  of
       the openmetrics exposition formats, see https://openmetrics.io/docs/instrumenting/exposition_formats.

METRIC NAMING

       All metrics from a file named JOB.*  will be exported as PCP metrics with the openmetrics.JOB metric name
       prefix.  Therefore, the JOB name must be a valid non-leaf name for PCP PMNS metric  names.   If  the  JOB
       name  has  multiple  dot-separated components, the resulting PMNS names will include those components and
       care is needed to ensure there are no overlapping definitions, e.g. metrics returned by JOB.response  may
       overlap or conflict with metrics returned by JOB.response.time.

       Config file entries (URLs or scripts) found in subdirectories of the config directory will also result in
       hierarchical      metric      names.       For       example,       a       config       file       named
       $PCP_PMDAS_DIR/openmetrics/config.d/mysource/latency/get.url  will  result  in  metrics being created (by
       fetching that source URL) below openmetrics.mysource.latency.get in the PCP namespace.  Scripts found  in
       subdirectories of the config directory similarly result in hierarchical PCP metric names.

DYNAMIC METRIC NAMES

       As described above, changes and new additions can be made to files in the configuration directory without
       having to restart the PMDA.  These changes are detected automatically and  the  PCP  metric  names  below
       openmetrics  in  the  PMNS will be updated accordingly, i.e. new metrics will be dynamically added and/or
       existing metrics removed.  In addition, pmdaopenmetrics honors the PMCD_NAMES_CHANGE pmFetch(3)  protocol
       that  was introduced in PCP version 4.0.  In particular, if openmetrics metrics are being logged by a PCP
       version 4.0 or later  pmlogger(1),  new  metrics  that  appear  as  a  result  of  changes  in  the  PMDA
       configuration  directory will automatically start to be logged, provided the root of the openmetrics PMDA
       namespace is configured for logging in the pmlogger configuration file.  See pmlogger(1) for details.  An
       example of such a pmlogger configuration file is :

          log mandatory on 2 second {
               # log all metrics below the root of the openmetrics namespace
               openmetrics
          }

CONTROL METRICS

       The  PMDA  maintains  special control metrics, as described below.  Apart from openmetrics.control.debug,
       each of these metrics is a counter and has one instance for each configured metric source.  The  instance
       domain  is  adjusted  dynamically as new sources are discovered.  If there are no sources configured, the
       metric names are still defined but the instance domain will be empty and a fetch will return no values.

       openmetrics.control.calls
              total number of times each configured metric source has been fetched (if it's a URL)  or  executed
              (if it's a script), since the PMDA started.

       openmetrics.control.fetch_time
              Total  time  in  milliseconds  that  each configured metric source has taken to return a document,
              excluding the time to parse the document.

       openmetrics.control.parse_time
              Total time in milliseconds that each configured metric source has taken to  parse  each  document,
              excluding the time to fetch the document.

       When  converted  to  a  rate,  the calls metric represents the average fetch rate of each source over the
       sampling interval (time delta between samples).  The fetch_time and parse_time counters,  when  converted
       to a rate, represent the average fetch and parsing latency (respectfully), during the sampling interval.

       The  openmetrics.control.debug  metric  has  a  singular  value, defaulting to 0.  If a non-zero value is
       stored into this metric using pmstore(1), additional debug messages will be written to the PMDA log file.

LIMITATIONS

       pmdaopenmetrics and libpcp internals impose some  numerical  constraints  about  the  number  of  sources
       (4095), metrics (1024) within each source, and instances for each metric (4194304).

INSTALLATION

       Install the OpenMetrics PMDA by using the Install script as root:

             # cd $PCP_PMDAS_DIR/openmetrics
             # ./Install

       To uninstall, do the following as root:

             # cd $PCP_PMDAS_DIR/openmetrics
             # ./Remove

       pmdaopenmetrics  is  launched  by  pmcd(1) and should never be executed directly.  The Install and Remove
       scripts notify pmcd when the agent is installed or removed.

       When scripts and .url files are added, removed or changed in the configuration directory, it  is  usually
       not  necessary  to  restart the PMDA - the changes will be detected and managed on subsequent requests to
       the PMDA.

FILES

       $PCP_PMDAS_DIR/openmetrics/Install
           installation script for the pmdaopenmetrics agent

       $PCP_PMDAS_DIR/openmetrics/Remove
           undo installation script for the pmdaopenmetrics agent

       $PCP_PMDAS_DIR/openmetrics/config.d/
           contains URLs and scripts used by the pmdaopenmetrics agent as sources of openmetrics metric data.

       $PCP_LOG_DIR/pmcd/openmetrics.log
           default log file for error messages from pmdaopenmetrics

       $PCP_VAR_DIR/config/144.*
           files containing internal tables for metric and instance ID number persistence (domain 144).

PCP ENVIRONMENT

       Environment variables with the prefix PCP_ are used to parameterize the file and directory names used  by
       PCP.   On  each  installation, the file /etc/pcp.conf contains the local values for these variables.  The
       $PCP_CONF variable may be used to specify an alternative configuration file, as described in pcp.conf(5).

SEE ALSO

       PCPIntro(1),   pmcd(1),    pminfo(1),    pmlogger(1),    pmstore(1),    PMWEBAPI(3),    pmFetch(3)    and
       https://openmetrics.io/docs/instrumenting/exposition_formats.