Provided by: pcp_5.0.3-1_amd64 
NAME
pmseries - display information about performance metric timeseries
SYNOPSIS
pmseries [-adFiIlLmMnqsStvV?] [-c config] [-g pattern] [-h host] [-p port] [-Z timezone] [query | labels
... | series ... | source ... ]
DESCRIPTION
pmseries displays various types of information about performance metrics available through the scalable
timeseries facilities of the Performance Co-Pilot (PCP) and the Redis distributed data store.
By default pmseries communicates with a local redis-server(1), however the -h and -p options can be used
to specify an alternate Redis instance. If this instance is a node of a Redis cluster, all other
instances in the cluster will be discovered and used automatically.
pmseries runs in several different modes - either querying timeseries identifiers, metadata or values
(already stored in Redis), or manually loading timeseries into Redis. The latter mode is seldom used,
however, since pmproxy(1) will automatically perform this function for local pmlogger(1) instances, when
running in its default time series mode.
Without command line options specifying otherwise, pmseries will issue a timeseries query to find
matching timeseries and values. All timeseries are identified using a unique SHA-1 hash which is always
displayed in a 40-hexdigit human readable form. These hashes are formed using the metadata associated
with every metric.
Importantly, this includes all metric metadata (labels, names, descriptors). Metric labels in particular
are (as far as possible) unique for every machine - on Linux for example the labels associated with every
metric include the unique /etc/machine-id, the hostname, domainname, and other automatically generated
machine labels, as well as any administrator-defined labels from /etc/pcp/labels. These labels can be
reported with pminfo(1) and the pmcd.labels metric.
See pmLookupLabels(3), pmLookupInDom(3), pmLookupName(3) and pmLookupDesc(3) for detailed information
about metric labels and other metric metadata used in each timeseries identifier hash calculation.
The timeseries identifiers provide a higher level (and machine independent) identifier than the
traditional PCP performance metric identifiers (pmID), instance domain identifiers (pmInDom) and metric
names. See PCPIntro(1) for more details about these traditional identifiers. However, pmseries uses
timeseries identifiers in much the same way that pminfo(1) uses the lower level indom, metric identifiers
and metric names.
The default mode of pmseries operation (i.e. with no command line options) depends on the arguments it is
presented. If all non-option arguments appear to be timeseries identifiers (in 40 hex digit form)
pmseries will report metadata for these timeseries - refer to the -a option for details. Otherwise, the
parameters will be treated as a timeseries query.
TIMESERIES QUERIES
Query expressions are formed using the pmseries query language described below, but can be as simple as a
metric name.
The following is an example of querying timeseries from all hosts that match a metric name pattern
(globbed):
$ pmseries kernel.all.cpu*
1d7b0bb3f6aec0f49c54f5210885464a53629b60
379db729afd63fb9eff436625bd6c55a7adc5cfd
3dd3b45bb05f96636043e5d58b52b441ce542285
[...]
ed2bf325ff6dc7589ec966698e5404b67252306a
dcb2a032a308b5717bf605ba8f8737e9c6e1ed19
To identify timeseries, the query language uses the general syntax:
[metric.name] '{metadata qualifiers}' '[time specification]'
The metric.name component restricts the timeseries query to any matching PCP metric name (the list of
metric names for a PCP archive or live host is reported by pminfo(1) with no arguments beyond --host or
--archive). The pmseries syntax extends on that of pminfo and allows for glob(7) based pattern matching
within the metric name.
METADATA QUALIFIERS AND OPERATORS
Metadata qualifiers are enclosed by ``curly'' braces ({}), and further restrict the query results to
timeseries with various metadata properties. These qualifiers are based on metric or instance names, and
metric label values, and take the general form metadata.name OPERATOR value, such as:
instance.name == "cpu0"
metric.name != "kernel.all.pswitch"
When using label names, the metadata qualifier is optional and can be dropped, such as:
label.hostname == "www.acme.com"
hostname == "www.acme.com"
For metric and instance names only the string operators apply, but for metric label values all operators
are available. The set of available operators is:
Boolean operators
All string (label, metrics and instances) and numeric (label) values can be tested for equality ("==") or
inequality ("!=").
String operators
Strings can be subject to pattern matching in the form of glob matching ("~~"), regular expression match‐
ing ("=~"), and regular expression non-matching ("!~"). The ":" operator is equivalent to "~~" - i.e.,
regular expression matching.
Relational operators (numeric label values only)
Numeric label values can be subject to the less than ("<"), greater than (">"), less than or equal
("<="), greater than or equal (">="), equal ("==") and not equal ("!=") operators.
Logical operators
Multiple metadata qualifiers can be combined with the logical operators for AND ("&&") and OR ("||") as
in many programming languages. The comma (",") character is equivalent to logical AND ("&&").
TIME SPECIFICATION
The final (optional) component of a query allows the user to specify a specific time window of interest.
Any time specification will result in values being returned for all matching timeseries only for the time
window specified.
The specification is ``square'' bracket ([]) enclosed, and consists of one or more comma-separated compo‐
nents. Each component specifies some aspect related to time, taking the general form: keyword: value,
such as:
samples: 10
Sample count
The number of samples to return, specified via either the samples or (equivalent) count keyword. The
value provided must be a positive integer. If no end time is explicitly set (see ``Time window'' later)
then the most recent samples will be returned.
Sample interval
An interval between successive samples can be requested using the interval or (equivalent) delta keyword.
The value provided should be either a numeric or string value that will be parsed by pmParseInterval(3),
such as 5 (seconds) or 2min (minutes).
Time window
Start and end times, and alignments, affecting the returned values. The keywords match the parameters to
the pmParseTimeWindow(3) function which will be used to parse them, and are: start or (equivalent) begin,
finish or (equivalent) end, align and offset.
Time zones
The resulting timestamps can be returned having been evaluated for a specific timezone, using the time‐
zone or hostzone keywords. The value associated with timezone will be interpreted by pmNewZone(3). A
true or false value should be associated with hostzone, and when set to true this has the same effect as
described by pmNewContextZone(3).
TIMESERIES METADATA
Using command line options, pmseries can be requested to provide metadata (metric names, instance names,
labels, descriptors) associated with either individual timeseries or a group of timeseries, for example:
$ pmseries -a dcb2a032a308b5717bf605ba8f8737e9c6e1ed19
dcb2a032a308b5717bf605ba8f8737e9c6e1ed19
PMID: 60.0.21
Data Type: 64-bit unsigned int InDom: PM_INDOM_NULL 0xffffffff
Semantics: counter Units: millisec
Source: f5ca7481da8c038325d15612bb1c6473ce1ef16f
Metric: kernel.all.cpu.nice
labels {"agent":"linux","domainname":"localdomain",\
"groupid":1000,"hostname":"shard",\
"latitude":-25.28496,"longitude":152.87886,\
"machineid":"295b16e3b6074cc8bdbda8bf96f6930a",\
"userid":1000}
The complete set of pmseries metadata reporting options are:
-a, --all
Convenience option to report all metadata for the given timeseries, equivalent to -dilms.
-d, --desc
Metric descriptions detailing the PMID, data type, data semantics, units, scale and associated in‐
stance domain. This option has a direct pminfo(1) equivalent.
-g pattern, --glob=pattern
Provide a glob(7) pattern to restrict the report provided by the -i, -l, -m, and -S.
-i, --instances
Metric descriptions detailing the PMID, data type, data semantics, units, scale and associated in‐
stance domain.
-I, --fullindom
Print the InDom in verbose mode. This option has a direct pminfo(1) equivalent.
-l, --labels
Print label sets associated with metrics and instances. Labels are optional metric metadata de‐
scribed in detail in pmLookupLabels(3). This option has a direct pminfo(1) equivalent.
-m, --metrics
Print metric names.
-M, --fullpmid
Print the PMID in verbose mode. This option has a direct pminfo(1) equivalent.
-n, --names
Print comma-separated label names only (not values) for the labels associated with metrics and in‐
stances.
-s, --series
Print timeseries identifiers associated with metrics, instances and sources. These unique identi‐
fiers are calculated from intrinsic (non-optional) labels and other metric metadata associated with
each PMAPI context (sources), metrics and instances. Archive, local context or pmcd(1) connections
for the same host all produce the same source identifier. This option has a direct pminfo(1) equiv‐
alent. See also pmLookupLabels(3) and the -l/--labels option.
TIMESERIES SOURCES
A source is a unique identifier (represented externally as a 40-byte hexadecimal SHA-1 hash) that repre‐
sents both the live host and/or archives from which each timeseries originated. The context for a source
identifier (obtained with -s) can be reported with:
-S, --sources
Print names for timeseries sources. These names are either hostnames or fully qualified archive
paths.
It is important to note that live and archived sources can and will generate the same SHA-1 source iden‐
tifier hash, provided that the context labels remain the same for that host (labels are stored in PCP
archives and can also be fetched live from pmcd(1)).
TIMESERIES LOADING
Timeseries metadata and data are loaded either automatically by a local pmproxy(1), or manually using a
specially crafted pmseries query and the -L/--load option:
$ pmseries --load "{source.path: \"$PCP_LOG_DIR/pmlogger/acme\"}"
pmseries: [Info] processed 2275 archive records from [...]
This query must specify a source archive path, but can also restrict the import to specific timeseries
(using metric names, labels, etc) and to a specific time window using the time specification component of
the query language.
As a convenience, if the argument to load is a valid file path as determined by access(2), then a short-
hand form can be used:
$ pmseries --load $PCP_LOG_DIR/pmlogger/acme.0
OPTIONS
The available command line options, in addition to timeseries metadata and sources options described
above, are:
-c config, --config=config
Specify the config file to use.
-h host, --host=host
Connect Redis server at host, rather than the one the localhost.
-L, --load
Load timeseries metadata and data into the Redis cluster.
-p port, --port=port
Connect Redis server at port, rather than the default 6379.
-q, --query
Perform a timeseries query. This is the default action.
-t, --times
Report time stamps numerically (in milliseconds) instead of the default human readable form.
-v, --values
Report all of the known values for given label name(s).
-V, --version
Display version number and exit.
-Z timezone, --timezone=timezone
Use timezone for the date and time. Timezone is in the format of the environment variable TZ as de‐
scribed in environ(7).
-?, --help
Display usage message and exit.
EXAMPLES
The following sample query shows several fundamental aspects of the pmseries query language:
$ pmseries 'kernel.all.load{hostname:"toium"}[count:2]'
eb713a9cf472f775aa59ae90c43cd7f960f7870f
[Thu Nov 14 05:57:06.082861000 2019] 1.0e-01 b84040ffccd54f839b65140cf139bab51cbbcf62
[Thu Nov 14 05:57:06.082861000 2019] 6.8e-01 a60b5b3bf25e71071c41934fa4d7d251f765f30c
[Thu Nov 14 05:57:06.082861000 2019] 6.4e-01 e1974a062375e6e62370ffadf5b0650dad739480
[Thu Nov 14 05:57:16.091546000 2019] 1.6e-01 b84040ffccd54f839b65140cf139bab51cbbcf62
[Thu Nov 14 05:57:16.091546000 2019] 6.7e-01 a60b5b3bf25e71071c41934fa4d7d251f765f30c
[Thu Nov 14 05:57:16.091546000 2019] 6.4e-01 e1974a062375e6e62370ffadf5b0650dad739480
This query returns the two most recent values for all instances of the kernel.all.load metric with a la‐
bel.hostname matching the regular expression "toium". This is a set-valued metric (i.e., a metric with
an ``instance domain'' which in this case consists of three instances: 1, 5 and 15 minute averages). The
first column returned is a timestamp, then a floating point value, and finally an instance identifier
timeseries hash (two values returned for three instances, so six rows are returned). The metadata for
these timeseries can then be further examined:
$ pmseries -a eb713a9cf472f775aa59ae90c43cd7f960f7870f
eb713a9cf472f775aa59ae90c43cd7f960f7870f
PMID: 60.2.0
Data Type: float InDom: 60.2 0xf000002
Semantics: instant Units: none
Source: 0e89c1192db79326900d82131c31399524f0b3ee
Metric: kernel.all.load
inst [1 or "1 minute"] series b84040ffccd54f839b65140cf139bab51cbbcf62
inst [5 or "5 minute"] series a60b5b3bf25e71071c41934fa4d7d251f765f30c
inst [15 or "15 minute"] series e1974a062375e6e62370ffadf5b0650dad739480
inst [1 or "1 minute"] labels {"agent":"linux","hostname":"toium"}
inst [5 or "5 minute"] labels {"agent":"linux","hostname":"toium"}
inst [15 or "15 minute"] labels {"agent":"linux","hostname":"toium"}
PCP ENVIRONMENT
Environment variables with the prefix PCP_ are used to parameterize the file and directory names used by
PCP. On each installation, the file /etc/pcp.conf contains the local values for these variables. The
$PCP_CONF variable may be used to specify an alternative configuration file, as described in pcp.conf(5).
For environment variables affecting PCP tools, see pmGetOptions(3).
SEE ALSO
PCPIntro(1), pmcd(1), pminfo(1), pmproxy(1), redis-server(1), access(2), PMAPI(3), PMWEBAPI(3), pm‐
LookupDesc(3), pmLookupInDom(3), pmLookupLabels(3), pmLookupName(3), pmNewContextZone(3), pmNewZone(3),
pmParseInterval(3), pmParseTimeWindow(3), pcp.conf(5), environ(7), glob(7) and regex(7).
Performance Co-Pilot PCP PMSERIES(1)