Provided by: pcp_4.3.4-1build2_amd64 
NAME
pmdaprometheus - Prometheus PMDA
SYNOPSIS
$PCP_PMDAS_DIR/prometheus/pmdaprometheus [-D] [-n] [-c config] [-d domain] [-l logfile]
[-r root] [-t timeout] [-u user]
DESCRIPTION
pmdaprometheus is a Performance Metrics Domain Agent (PMDA) which creates PCP metrics from
Prometheus endpoints, which provide HTTP based access to application metrics. The default
config directory is $PCP_PMDAS_DIR/prometheus/config.d/, see ``CONFIGURATION SOURCES''
below. The default URL fetch timeout is 2 seconds. The default user, if not specified
with the -u option, is the current user. If the -n option is given, the list of
configuration files will not be sorted prior to processing. This list is sorted by
default but that can be expensive if there are a large number of configuration files (URLs
and/or scripts).
If the -D option is given, additional diagnostic messages will be written to the PMDA log
file, which is $PCP_LOG_DIR/pmcd/prometheus.log by default (see also -lbelow). In
addition, the metric prometheus.control.debug controls the same debug flag and can be set
with the following command:
pmstore prometheus.control.debug value
where value is either 1 (to enable verbose log messages) or 0 (to disable verbose log
messages). This is particularly useful for examining the http headers passed to each
fetch request, filter settings and other processing details that are logged when the
debugging flag is enabled.
The -d option may be used to override the default performance metrics domain number, which
defaults to 144. It is strongly recommended not to change this. The domain number should
be different for every PMDA on the one host, and the same domain number should be used for
pmdaprometheus PMDA on all hosts. See also the -r option, which allows the root of the
dynamic namespace to be changed from the default prometheus.
The -l option may be used to specify logfile as the destination for PMDA messages instead
of the default, $PCP_LOG_DIR/pmcd/prometheus.log. As a special case, logfile may be "-"
to send messages to the stderr stream instead, e.g. -l-. This would normally be the
stderr stream for the parent process, pmcd(1), which may itself have redirected stderr.
This redirection is normally most useful in a containerized environment, or when using
dbpmda(1).
The -r option allows the root of the dynamic namespace to be changed to root from the
default, prometheus. In conjunction with other command line options, this allows
pmdaprometheus to be deployed as a different PMDA with distinct metrics namespace and
metrics domain on the same host system. Note that all PMDAs require a unique domain
number so the -d option must also be specified. Use of the -r option may also change the
defaults for some other command line options, e.g. the default log file name and the
default configuration directory.
CONFIGURATION SOURCES
As it runs, pmdaprometheus periodically recursively scans the
$PCP_PMDAS_DIR/prometheus/config.d directory (or the directory specified with the -c
option), looking for source URL files (*.url) and executable scripts or binaries. Any
files that do not have the .url suffix or are not executable, are ignored - this allows
documentation files such as "README" and non-executable "common" script function
definitions to be present without being considered as config files.
A remote server does not have to be up or stay running - the PMDA tolerates remote URLs
that may come and go over time. The PMDA will relay data and metadata when/if they are
available, and will return errors when/if they are down. PCP metric IDs, internal and
external instance domain identifiers are persisted and will be restored when individual
metric sources become available and/or when the PMDA is restarted. In addition, the PMDA
checks directory modification times and will rescan for new or changed configuration files
dynamically. It is not necessary to restart the PMDA when adding, removing or changing
configuration files.
URL SOURCES
Each file with the .url suffix found in the config directory or sub-directory contains one
complete HTTP or HTTPS URL at which pmdaprometheus can reach a Prometheus endpoint. Local
file access is also supported with a conventional file://somepath/somefile URL, in which
case somepath/somefile should contain prometheus formatted metric data.
The first line of a .url config file should be the URL, as described above. Subsequent
lines, if any, are prefixed with a keyword that can be used to alter the http GET request.
A keyword must end with ':' (colon) and the text extends to the end of the line. Comment
lines that start with # and blank lines are ignored. The only currently supported
keywords are HEADER: and FILTER:.
HEADER: headername: value ... to end of line
Adds headername and its value to the headers passed in the http GET request for the
configured URL. An example configuration file that provides 3 commonly used headers and
an authentication token might be :
http://somehost/path/endpoint.html
# this is a comment
HEADER: Accept: text/html
HEADER: Keep-Alive: 300
HEADER: Connection: keep-alive
HEADER: Authorization: token ABCDEF1234567890
As mentioned above, header values extend to the end of the line. They may contain any
valid characters, including colons. Multiple spaces will be collapsed to a single space,
and leading and trailing spaces are trimmed. A common use for headers is to configure a
proxy agent and the assorted parameters it may require.
FILTER: INCLUDE METRIC regex
or
FILTER: EXCLUDE METRIC regex
Dynamically created metric names that match regex will be either included or excluded in
the name space, as specified. The simple rule is that the first matching filter regex for
a particular metric name is the rule that prevails. If no filter regex matches (or there
are no filters), then the metric is included by default, i.e. the default filter if none
are specified is FILTER: INCLUDE METRIC .* This is backward compatible with older
versions of the configuration file that did not support filters. Multiple FILTER: lines
would normally be used, e.g. to include some metrics but exclude all others, use FILTER:
EXCLUDE METRIC .* as the last of several filters that include the desired metrics.
Conversely, to exclude some metrics but include all others, use FILTER: EXCLUDE METRIC
regex. In this case it's not necessary (though doesn't hurt) to specify the final FILTER:
INCLUDE METRIC .* because, as stated above, any metric that does not match any filter
regex will be included by default.
Label filtering uses similar FILTER: syntax and semantics. FILTER: EXCLUDE LABEL regex
will delete all labels matching regex from all metrics defined in the configuration file.
The same rules as for metrics apply for labels too - an implicit rule: FILTER: INCLUDE
LABEL .* applies to all labels that do not match any earlier filter rule.
Caution is needed with label filtering because by default, all labels are used to
construct the PCP instance name. By excluding some labels, the instance names will
change. Excluding all labels for a particular metric changes that metric to be singular,
i.e. have no instance domain. In addition, by excluding some labels, different instances
of the same metric may become duplicates. When such duplicates occur, the last duplicate
instance returned by the end-point URL prevails over any earlier instances. For these
reasons, it is recommended that label filtering rules be configured when the configuration
file is first defined, and not changed thereafter. If a label filtering change is
required, the configuration file should be renamed, which effectively defines a new
metric, with the new (or changed) instance naming.
Unrecognized keywords in configuration files are reported in the PMDA log file but
otherwise ignored.
SCRIPTED SOURCES
Executable scripts present in the $PCP_PMDAS_DIR/prometheus/config.d directory or sub-
directories will be executed and the stdout stream containing prometheus formatted metric
data will be parsed as though it had come from a URL or file. The stderr stream from a
script will be sent to the PMDA log file, which by default can be found in
$(PCP_LOG_DIR)/pmcd/prometheus.log.
Note that scripted sources do not support label or metric filtering (as described above
for URL sources) - they can simply do their own filtering in the script itself with
sed(1), awk(1), or whatever tool is desired.
A simple example of a scripted config entry follows:
#! /bin/sh
awk '{
print("# HELP loadavg local load average")
print("# Type loadavg gauge")
printf("loadavg {interval=\"1-minute\"} %.2f\n", $1)
printf("loadavg {interval=\"5-minute\"} %.2f\n", $2)
printf("loadavg {interval=\"15-minute\"} %.2f\n", $3)
}' /proc/loadavg
This script produces the following Prometheus-formatted metric data when run:
# HELP loadavg local load average
# Type loadavg gauge
loadavg {interval="1-minute"} 0.12
loadavg {interval="5-minute"} 0.27
loadavg {interval="15-minute"} 0.54
If the above script was saved and made executable in a file named
$PCP_PMDAS_DIR/prometheus/config.d/local/system.sh then this would result in a new PCP
metric named prometheus.local.system.loadavg which would have three instances for the
current load average values: 1-minute, 5-minute and 15-minute.
Scripted config entries may produce more than one PCP leaf metric name. For example, the
above "system.sh" script could also export other metrics such as CPU statistics, by
reading /proc/stat on the local system. Such additional metrics would appear as peer
metrics in the same PCP metric subtree. In the case of CPU counters, the metric type
definition should be counter, not gauge. For full details of the prometheus exposition
formats, see https://prometheus.io/docs/instrumenting/exposition_formats.
METRIC NAMING
All metrics from a file named JOB.* will be exported as PCP metrics with the
prometheus.JOB metric name prefix. Therefore, the JOB name must be a valid non-leaf name
for PCP PMNS metric names. If the JOB name has multiple dot-separated components, the
resulting PMNS names will include those components and care is needed to ensure there are
no overlapping definitions, e.g. metrics returned by JOB.response may overlap or conflict
with metrics returned by JOB.response.time.
Config file entries (URLs or scripts) found in subdirectories of the config directory will
also result in hierarchical metric names. For example, a config file named
$PCP_PMDAS_DIR/prometheus/config.d/mysource/latency/get.url will result in metrics being
created (by fetching that source URL) below prometheus.mysource.latency.get in the PCP
namespace. Scripts found in subdirectories of the config directory similarly result in
hierarchical PCP metric names.
DYNAMIC METRIC NAMES
As described above, changes and new additions can be made to files in the configuration
directory without having to restart the PMDA. These changes are detected automatically
and the PCP metric names below prometheus in the PMNS will be updated accordingly, i.e.
new metrics will be dynamically added and/or existing metrics removed. In addition,
pmdaprometheus honors the PMCD_NAMES_CHANGE pmFetch(3) protocol that was introduced in PCP
version 4.0. In particular, if prometheus metrics are being logged by a PCP version 4.0
or later pmlogger(1), new metrics that appear as a result of changes in the PMDA
configuration directory will automatically start to be logged, provided the root of the
prometheus PMDA namespace is configured for logging in the pmlogger configuration file.
See pmlogger(1) for details. An example of such a pmlogger configuration file is :
log mandatory on 2 second {
# log all metrics below the root of the prometheus namespace
prometheus
}
CONTROL METRICS
The PMDA maintains special control metrics, as described below. Apart from
prometheus.control.debug, each of these metrics is a counter and has one instance for each
configured metric source. The instance domain is adjusted dynamically as new sources are
discovered. If there are no sources configured, the metric names are still defined but
the instance domain will be empty and a fetch will return no values.
prometheus.control.calls
total number of times each configured metric source has been fetched (if it's a
URL) or executed (if it's a script), since the PMDA started.
prometheus.control.fetch_time
Total time in milliseconds that each configured metric source has taken to return a
document, excluding the time to parse the document.
prometheus.control.parse_time
Total time in milliseconds that each configured metric source has taken to parse
each document, excluding the time to fetch the document.
When converted to a rate, the calls metric represents the average fetch rate of each
source over the sampling interval (time delta between samples). The fetch_time and
parse_time counters, when converted to a rate, represent the average fetch and parsing
latency (respectfully), during the sampling interval.
The prometheus.control.debug metric has a singular value, defaulting to 0. If a non-zero
value is stored into this metric using pmstore(1), additional debug messages will be
written to the PMDA log file.
LIMITATIONS
pmdaprometheus and libpcp internals impose some numerical constraints about the number of
sources (4095), metrics (1024) within each source, and instances for each metric
(4194304).
INSTALLATION
Install the Prometheus PMDA by using the Install script as root:
# cd $PCP_PMDAS_DIR/prometheus
# ./Install
To uninstall, do the following as root:
# cd $PCP_PMDAS_DIR/prometheus
# ./Remove
pmdaprometheus is launched by pmcd(1) and should never be executed directly. The Install
and Remove scripts notify pmcd when the agent is installed or removed.
When scripts and .url files are added, removed or changed in the configuration directory,
it is usually not necessary to restart the PMDA - the changes will be detected and managed
on subsequent requests to the PMDA.
FILES
$PCP_PMDAS_DIR/prometheus/Install
installation script for the pmdaprometheus agent
$PCP_PMDAS_DIR/prometheus/Remove
undo installation script for the pmdaprometheus agent
$PCP_PMDAS_DIR/prometheus/config.d/
contains URLs and scripts used by the pmdaprometheus agent as sources of prometheus
metric data.
$PCP_LOG_DIR/pmcd/prometheus.log
default log file for error messages from pmdaprometheus
$PCP_VAR_DIR/config/144.*
files containing internal tables for metric and instance ID number persistence (domain
144).
PCP ENVIRONMENT
Environment variables with the prefix PCP_ are used to parameterize the file and directory
names used by PCP. On each installation, the file /etc/pcp.conf contains the local values
for these variables. The $PCP_CONF variable may be used to specify an alternative
configuration file, as described in pcp.conf(5).
SEE ALSO
pmcd(1), pminfo(1), pmlogger(1), pmstore(1), PMWEBAPI(3), pmFetch(3) and
https://prometheus.io/docs/instrumenting/exposition_formats.