noble (1) pmdaopenmetrics.1.gz
NAME
pmdaopenmetrics - OpenMetrics PMDA
SYNOPSIS
$PCP_PMDAS_DIR/openmetrics/pmdaopenmetrics [-D] [-n] [-c config] [-d domain] [-l logfile] [-r root] [-t
timeout] [-u user]
DESCRIPTION
pmdaopenmetrics is a Performance Metrics Domain Agent (PMDA) which dynamically creates PCP metrics from
configured OpenMetrics endpoints, which provide HTTP based access to application metrics. The PMDA
essentially implements a bridge between Prometheus and PCP, allowing PCP to easily ingest performance
data from more than 650 registered end-points and many other application specific end-points.
The default config directory is $PCP_PMDAS_DIR/openmetrics/config.d/, see ``CONFIGURATION SOURCES''
below. The default URL fetch timeout is 2 seconds. The default user, if not specified with the -u
option, is the current user. If the -n option is given, the list of configuration files will not be
sorted prior to processing. This list is sorted by default but that can be expensive if there are a
large number of configuration files (URLs and/or scripts).
If the -D option is given, additional diagnostic messages will be written to the PMDA log file, which is
$PCP_LOG_DIR/pmcd/openmetrics.log by default (see also -lbelow). In addition, the metric
openmetrics.control.debug controls the same debug flag and can be set with the following command:
pmstore openmetrics.control.debug value
where value is either 1 (to enable verbose log messages) or 0 (to disable verbose log messages). This is
particularly useful for examining the http headers passed to each fetch request, filter settings and
other processing details that are logged when the debugging flag is enabled.
The -d option may be used to override the default performance metrics domain number, which defaults to
144. It is strongly recommended not to change this. The domain number should be different for every
PMDA on the one host, and the same domain number should be used for pmdaopenmetrics PMDA on all hosts.
See also the -r option, which allows the root of the dynamic namespace to be changed from the default
openmetrics.
The -l option may be used to specify logfile as the destination for PMDA messages instead of the default,
$PCP_LOG_DIR/pmcd/openmetrics.log. As a special case, logfile may be "-" to send messages to the stderr
stream instead, e.g. -l-. This would normally be the stderr stream for the parent process, pmcd(1),
which may itself have redirected stderr. This redirection is normally most useful in a containerized
environment, or when using dbpmda(1).
The -r option allows the root of the dynamic namespace to be changed to root from the default,
openmetrics. In conjunction with other command line options, this allows pmdaopenmetrics to be deployed
as a different PMDA with distinct metrics namespace and metrics domain on the same host system. Note
that all PMDAs require a unique domain number so the -d option must also be specified. Use of the -r
option may also change the defaults for some other command line options, e.g. the default log file name
and the default configuration directory.
CONFIGURATION SOURCES
As it runs, pmdaopenmetrics periodically recursively scans the $PCP_PMDAS_DIR/openmetrics/config.d
directory (or the directory specified with the -c option), looking for source URL files (*.url) and
executable scripts or binaries. Any files that do not have the .url suffix or are not executable, are
ignored - this allows documentation files such as "README" and non-executable "common" script function
definitions to be present without being considered as config files.
A remote server does not have to be up or stay running - the PMDA tolerates remote URLs that may come and
go over time. The PMDA will relay data and metadata when/if they are available, and will return errors
when/if they are down. PCP metric IDs, internal and external instance domain identifiers are persisted
and will be restored when individual metric sources become available and/or when the PMDA is restarted.
In addition, the PMDA checks directory modification times and will rescan for new or changed
configuration files dynamically. It is not necessary to restart the PMDA when adding, removing or
changing configuration files.
URL SOURCES
Each file with the .url suffix found in the config directory or sub-directory contains one complete HTTP
or HTTPS URL at which pmdaopenmetrics can reach a OpenMetrics endpoint. Local file access is also
supported with a conventional file:///somepath/somefile URL, in which case somepath/somefile should
contain openmetrics formatted metric data.
The first line of a .url config file should be the URL, as described above. Subsequent lines, if any,
are prefixed with a keyword that can be used to alter the http GET request. A keyword must end with ':'
(colon) and the text extends to the end of the line. Comment lines that start with # and blank lines are
ignored. The only currently supported keywords are HEADER: and FILTER:.
HEADER: headername: value ... to end of line
Adds headername and its value to the headers passed in the http GET request for the configured URL. An
example configuration file that provides 3 commonly used headers and an authentication token might be :
http://somehost/path/endpoint.html
# this is a comment
HEADER: Accept: text/html
HEADER: Keep-Alive: 300
HEADER: Connection: keep-alive
HEADER: Authorization: token ABCDEF1234567890
As mentioned above, header values extend to the end of the line. They may contain any valid characters,
including colons. Multiple spaces will be collapsed to a single space, and leading and trailing spaces
are trimmed. A common use for headers is to configure a proxy agent and the assorted parameters it may
require.
METRIC FILTERING
Metric filtering is a configuration file feature that allows ingested metrics to be included or excluded,
i.e. filtered. This is useful because most end-points return multiple metrics, and usually only some are
interesting for monitoring purposes. The syntax is:
FILTER: INCLUDE METRIC regex
or
FILTER: EXCLUDE METRIC regex
Dynamically created metric names that match regex will be either included or excluded in the name space,
as specified. Note that only the PMNS leaf component of the metric name (as ingested from the URL
source) is compared with the regex pattern. The simple rule is that the first matching filter regex for
a particular metric leaf name is the rule that prevails. If no filter regex matches (or there are no
filters), then the metric is included by default, i.e. the default filter if none are specified is
FILTER: INCLUDE METRIC .* This is backward compatible with older versions of the configuration file that
did not support filters. Multiple FILTER: lines would normally be used, e.g. to include some metrics but
exclude all others, use FILTER: EXCLUDE METRIC .* as the last of several filters that include the
desired metrics. Conversely, to exclude some metrics but include all others, use FILTER: EXCLUDE METRIC
regex. In this case it's not necessary (though doesn't hurt) to specify the final FILTER: INCLUDE METRIC
.* because, as stated above, any metric that does not match any filter regex will be included by
default.
LABEL FILTERING
Label filtering uses similar FILTER: syntax and semantics as metric filtering. FILTER: EXCLUDE LABEL
regex will delete all labels with label name matching regex from all metrics defined by the configuration
file. The same rules as for metric filters apply for label filters too - an implicit rule: FILTER:
INCLUDE LABEL .* applies to all labels that do not match any earlier label filter rule. FILTER:
OPTIONAL LABEL regex specifies that matching label names are to be included in the returned metric
labelsets (i.e. included), but are not to be used as part of the the external instance names. All
included labels that are not optional (i.e. the intrinsic labels) will be concatenated together and used
for external instance naming. In addition, non-intrinsic labels (i.e. labels tagged as OPTIONAL) will
have the PM_LABEL_OPTIONAL flag set in the labelsets returned by notes callbacks. This flag affects how
the labels are used in certain clients. For further details, see pmLookupLabels(3) and related man pages
for further details. Note that external instance names begin with the unique numeric internal instance
identifier followed by a space, so external instance names are always unique.
Caution is needed with label filtering because by default, all labels are used to construct the PCP
instance name. By excluding some labels (or changing them to optional), the instance names will change.
In addition, excluding all labels for a particular metric changes that metric to be singular, i.e. have
no instance domain. By excluding some labels, different instances returned by the URL or scripted
configuration entry for the same metric may become duplicates. When such duplicates occur, the last
duplicate instance returned by the end-point URL or script prevails over any earlier instances. For
these reasons, it is recommended that label filtering rules be configured when the configuration file is
first defined, and not changed thereafter. If a label filtering change is required, the configuration
file should be renamed, which effectively defines a new metric (or set of peer metrics as returned by the
URL or script), with the new (or changed) instance naming.
Unrecognized keywords in configuration files are reported in the PMDA log file but otherwise ignored.
SCRIPTED SOURCES
Executable scripts present in the $PCP_PMDAS_DIR/openmetrics/config.d directory or sub-directories will
be executed and the stdout stream containing openmetrics formatted metric data will be parsed as though
it had come from a URL or file. The stderr stream from a script will be sent to the PMDA log file, which
by default can be found in $(PCP_LOG_DIR)/pmcd/openmetrics.log.
Note that scripted sources do not support label or metric filtering (as described above for URL sources)
- they can simply do their own filtering in the script itself with sed(1), awk(1), or whatever tool is
desired.
A simple example of a scripted config entry follows:
#! /bin/sh
awk '{
print("# HELP loadavg local load average")
print("# TYPE loadavg gauge")
printf("loadavg {interval=\"1-minute\"} %.2f\n", $1)
printf("loadavg {interval=\"5-minute\"} %.2f\n", $2)
printf("loadavg {interval=\"15-minute\"} %.2f\n", $3)
}' /proc/loadavg
This script produces the following OpenMetrics-formatted metric data when run:
# HELP loadavg local load average
# TYPE loadavg gauge
loadavg {interval="1-minute"} 0.12
loadavg {interval="5-minute"} 0.27
loadavg {interval="15-minute"} 0.54
If the above script was saved and made executable in a file named
$PCP_PMDAS_DIR/openmetrics/config.d/local/system.sh then this would result in a new PCP metric named
openmetrics.local.system.loadavg which would have three instances for the current load average values:
1-minute, 5-minute and 15-minute.
Scripted config entries may produce more than one PCP leaf metric name. For example, the above
"system.sh" script could also export other metrics such as CPU statistics, by reading /proc/stat on the
local system. Such additional metrics would appear as peer metrics in the same PCP metric subtree. In
the case of CPU counters, the metric type definition should be counter, not gauge. For full details of
the openmetrics exposition formats, see
https://github.com/OpenObservability/OpenMetrics/blob/master/specification/OpenMetrics.md.
SELinux CONSIDERATIONS
Scripted config files are executed by the pmdaopenmetrics PMDA with the same SELinux context and policy
as the local pmcd(1). For simple scripts, such as the load average example described above, this is
normally fine. However AVC errors may result for scripts that make library or system calls that are
restricted by the prevailing SELinux context and policies. In these cases it is not feasible to
unilaterally grant pmcd or it's PMDAs an unconfined execution policy. In these site specific cases it
will be necessary to create a local SELinux policy module. This can be done by capturing the AVC
record(s) from the local audit log, generate a local policy module using audit2allow, and then load the
new module using semodule, e.g. as follows :
$ sudo grep '^type=AVC.*pcp' /var/log/audit/audit.log \
| audit2allow -M mypolicy
$ sudo semodule -i mypolicy.pp
If these local policies need to be persistent across reboots, then a scriptlet similar to the above
example may be added to the local pmcd RC file (typically /etc/pcp/pmcd/rc.local). For further details,
see audit2allow(1) and semodule(1).
METRIC NAMING
All metrics from a file named JOB.* will be exported as PCP metrics with the openmetrics.JOB metric name
prefix. Therefore, the JOB name must be a valid non-leaf name for PCP PMNS metric names. If the JOB
name has multiple dot-separated components, the resulting PMNS names will include those components and
care is needed to ensure there are no overlapping definitions, e.g. metrics returned by JOB.response may
overlap or conflict with metrics returned by JOB.response.time.
Config file entries (URLs or scripts) found in subdirectories of the config directory will also result in
hierarchical metric names. For example, a config file named
$PCP_PMDAS_DIR/openmetrics/config.d/mysource/latency/get.url will result in metrics being created (by
fetching that source URL) below openmetrics.mysource.latency.get in the PCP namespace. Scripts found in
subdirectories of the config directory similarly result in hierarchical PCP metric names.
DYNAMIC METRIC NAMES
As described above, changes and new additions can be made to files in the configuration directory without
having to restart the PMDA. These changes are detected automatically and the PCP metric names below
openmetrics in the PMNS will be updated accordingly, i.e. new metrics will be dynamically added and/or
existing metrics removed. In addition, pmdaopenmetrics honors the PMCD_NAMES_CHANGE pmFetch(3) protocol
that was introduced in PCP version 4.0. In particular, if openmetrics metrics are being logged by a PCP
version 4.0 or later pmlogger(1), new metrics that appear as a result of changes in the PMDA
configuration directory will automatically start to be logged, provided the root of the openmetrics PMDA
namespace is configured for logging in the pmlogger configuration file. See pmlogger(1) for details. An
example of such a pmlogger configuration file is :
log mandatory on 2 second {
# log all metrics below the root of the openmetrics namespace
openmetrics
}
METADATA
Metric data returned by URL or scripted configuration files may contain metadata that can be used by the
openmetrics PMDA to specify the semantics, data type, scaling and units of dynamically created metrics.
This metadata is prefixed with # PCP5 or # PCP in the ingested metric data. For additional information
about PCP metadata, see pmLookupDesc(3) and pmParseUnitsStr(3) and examples in shipped configuration
files.
In-line "PCP5" metadata must be supplied by the metrics source end-pont (URL or script). An alternative
is to specify this in the URL configuration file directly, which has the advantage of not having to
modify the source/end-point if the metadata is incorrect or missing. Metadata specified in the URL
configuration file over-rides any in-line metadata.
The configuration file syntax for specifying metadata is:
METADATA: regex type indom semantics units ... to EOL
Where:
METADATA: is literal
regex is an extended regular expression to match one or more metric names returned by the URL,
type is one of the PCP numeric types (32, u32, 64, u64, float or double),
indom is an arbitrary instance domain name, or PM_INDOM_NULL,
semantics is either counter, instant or discrete and
units is either none or a string extending to end of line that is parseable by pmParseUnitsStr(3), i.e.
the units, dimensions and scaling to be used for matching metrics.
An example configuration file that ingests metrics from the Grafana /metrics end-point on localhost,
filters out all metrics returned by that URL except for grafana_api_response_status_total and then
specifies the metric type is an unsigned 32 bit integer with a non-singular instance domain named
response and counter semantics with units of count.
http://localhost:3000/metrics
FILTER: INCLUDE METRIC grafana_api_response_status_total
FILTER: EXCLUDE METRIC .*
METADATA: grafana_api_response_status_total u32 response counter count
Note that the name in the indom field is presently ignored unless it is PM_INDOM_NULL, in which case the
metric has no instance domain (i.e. singular), even if it has labels which would otherwise be used for
instance naming.
CONTROL METRICS
The PMDA maintains special control metrics, as described below. Apart from openmetrics.control.debug,
each of these metrics has one instance for each configured metric source. All of these metrics have
integer values with counter semantics, except openmetrics.control.status, which has a string value. It
is important to note that fetching any of the openmetrics.control metrics will only update the counters
and status values if the corresponding URL is actually fetched. If the source URL is not fetched, the
control metric values do not trigger a refresh and the control values reported represent the most recent
fetch of each corresponding source.
The instance domain for the openmetrics.control metrics is adjusted dynamically as new sources are
discovered. If there are no sources configured, the metric names are still defined but the instance
domain will be empty and a fetch will return no values.
openmetrics.control.status
A string representing the status of the last fetch of the corresponding source. This will
generally be success for an http response code of 200. This metric can be used for service
availability monitoring - provided, as stated above, the corresponding source URL is fetched too.
openmetrics.control.status_code
This metric is similar to openmetrics.control.status except that it is the integer response code
of the last fetch. A value of 200 usually signifies success and any other value failure. This
metric can also be used for service availability monitoring, with the same caveats as
openmetrics.control.status.
openmetrics.control.calls
total number of times each configured metric source has been fetched (if it's a URL) or executed
(if it's a script), since the PMDA started. This metric has counter semantics and would normally
be converted to a rate/second by client tools.
openmetrics.control.fetch_time
Total time in milliseconds that each configured metric source has taken to return a document,
excluding the time to parse the document. This metric has counter semantics and would normally be
rate converted by client tools but is also useful in raw form as the accumulated parse time since
the PMDA was started.
openmetrics.control.parse_time
Total time in milliseconds that each configured metric source has taken to parse each document,
excluding the time to fetch the document. This metric has counter semantics and would normally be
rate converted by client tools but is also useful in raw form as the accumulated parse time since
the PMDA was started.
When converted to a rate, the calls metric represents the average fetch rate of each source over the
sampling interval (time delta between samples). The fetch_time and parse_time counters, when converted
to a rate, represent the average fetch and parsing latency (respectfully), during the sampling interval.
The openmetrics.control.debug metric has a singular value, defaulting to 0. If a non-zero value is
stored into this metric using pmstore(1), additional debug messages will be written to the PMDA log file.
LIMITATIONS
pmdaopenmetrics and libpcp internals impose some numerical constraints about the number of sources
(4095), metrics (1024) within each source, and instances for each metric (4194304).
INSTALLATION
Install the OpenMetrics PMDA by using the Install script as root:
# cd $PCP_PMDAS_DIR/openmetrics
# ./Install
To uninstall, the following must be done as root:
# cd $PCP_PMDAS_DIR/openmetrics
# ./Remove
pmdaopenmetrics is launched by pmcd(1) and should never be executed directly. The Install and Remove
scripts notify pmcd when the agent is installed or removed.
When scripts and .url files are added, removed or changed in the configuration directory, it is usually
not necessary to restart the PMDA - the changes will be detected and managed on subsequent requests to
the PMDA.
FILES
$PCP_PMDAS_DIR/openmetrics/Install
installation script for the pmdaopenmetrics agent
$PCP_PMDAS_DIR/openmetrics/Remove
undo installation script for the pmdaopenmetrics agent
$PCP_PMDAS_DIR/openmetrics/config.d/
contains URLs and scripts used by the pmdaopenmetrics agent as sources of openmetrics metric data.
$PCP_LOG_DIR/pmcd/openmetrics.log
default log file for error messages from pmdaopenmetrics
$PCP_VAR_DIR/config/144.*
files containing internal tables for metric and instance ID number persistence (domain 144).
PCP ENVIRONMENT
Environment variables with the prefix PCP_ are used to parameterize the file and directory names used by
PCP. On each installation, the file /etc/pcp.conf contains the local values for these variables. The
$PCP_CONF variable may be used to specify an alternative configuration file, as described in pcp.conf(5).
SEE ALSO
PCPIntro(1), audit2allow(1), pmcd(1), pminfo(1), pmlogger(1), pmstore(1), PMWEBAPI(3), pmFetch(3), pmLookupLabels(3), semodule(1), and https://prometheus.io/docs/instrumenting/exposition_formats.