Provided by: pcp_4.0.1-1_amd64 
NAME
pcp-atopsar - Advanced System Activity Report (pcp-atop related)
SYNOPSIS
pcp [pcp options] atopsar [atop options] [-r file|date] [-h host] [-R cnt] [-b hh:mm] [-e hh:mm]
pcp [pcp options] atopsar [atop options] interval [samples]
DESCRIPTION
The pcp-atopsar program can be used to report statistics at the system level.
In the first synopsis line (no sampling interval specified), pcp-atopsar extracts data from a raw logfile
that has been recorded previously by pmlogger(1) (or via the -w option of the pcp-atop program).
You can specify the name of the logfile with the -r option of the pcp-atopsar program. When a pmlogger
daily logfile is used, named $PCP_LOG_DIR/pmlogger/[host]/YYYYMMDD (where YYYYMMDD reflects the date),
the required date of the form YYYYMMDD can be specified with the -r option instead of the filename, or
the symbolic name 'y' can be used for yesterday's daily logfile (this can be repeated so 'yyyy' indicates
the logfile of four days ago). If the -r option is not specified at all, today's daily logfile is used
by default.
By default, the hostname of the localhost will be used when resolving pmlogger archives, however an
alternative host can be specified using the -h option.
The starting and ending times of the report can be defined using the options -b and -e followed by a time
argument of the form hh:mm.
In the second synopsis line, pcp-atopsar reads actual activity counters from the kernel with the
specified interval (in seconds) and the specified number of samples (optionally). When pcp-atopsar is
activated in this way it immediately sends the output for every requested report to standard output. If
only one type of report is requested, the header is printed once and after every interval seconds the
statistical counters are shown for that period. If several reports are requested, a header is printed
per sample followed by the statistical counters for that period.
When invoked via the pcp(1) command, the PCPIntro(1) options -h/--host, -a/--archive, -O/--origin,
-s/--samples, -t/--interval, -Z/--timezone and several other pcp options become indirectly available.
Some generic flags can be specified to influence the behaviour of the pcp-atopsar program:
-S By default the timestamp at the beginning of a line is suppressed if more lines are shown for one
interval. With this flag a timestamp is given for every output-line (easier for post-processing).
-a By default certain resources as disks and network interfaces are only shown when they were active
during the interval. With this flag all resources of a given type are shown, even if they were
inactive during the interval.
-x By default pcp-atopsar only uses colors if output is directed to a terminal (window). These colors
might indicate that a critical occupation percentage has been reached (red) or has been almost
reached (cyan) for a particular resource. See the man-page of atop for a detailed description of
this feature (section COLORS).
With the flag -x the use of colors is suppressed unconditionally.
-C By default pcp-atopsar only uses colors if output is directed to a terminal (window). These colors
might indicate that a critical occupation percentage has been reached (red) or has been almost
reached (cyan) for a particular resource. See the man-page of atop for a detailed description of
this feature (section COLORS).
With the flag -C colors will always be used, even if output is not directed to a terminal.
-M Use markers at the end of a line to indicate that a critical occupation percentage has been reached
('*') or has been almost reached ('+') for particular resources. The marker '*' is similar to the
color red and the marker '+' to the color cyan. See the man-page of atop for a detailed description
of these colors (section COLORS).
-H Repeat the header line within a report for every N detail lines. The value of N is determined
dynamically in case of output to a tty/window (depending on the number of lines); for output to a
file or pipe this value is 23.
-R Summarize cnt samples into one sample. When the logfile contains e.g. samples of 10 minutes, the use
of the flag '-R 6' shows a report with one sample for every hour.
Other flags are used to define which reports are required:
-A Show all possible reports.
-c Report about CPU utilization (in total and per cpu).
-p Report about processor-related matters, like load-averages and hardware interrupts.
-P Report about processes.
-m Current memory- and swap-occupation.
-s Report about paging- and swapping-activity, and overcommitment.
-l Report about utilization of logical volumes.
-f Report about utilization of multiple devices.
-d Report about utilization of disks.
-n Report about NFS mounted filesystems on NFS client.
-j Report about NFS client activity.
-J Report about NFS server activity.
-i Report about the network interfaces.
-I Report about errors for network-interfaces.
-w Report about IP version 4 network traffic.
-W Report about errors for IP version 4 traffic.
-y General report about ICMP version 4 layer activity.
-Y Per-type report about ICMP version 4 layer activity.
-u Report about UDP version 4 network traffic.
-z Report about IP version 6 network traffic.
-Z Report about errors for IP version 6 traffic.
-k General report about ICMP version 6 layer activity.
-K Per-type report about ICMP version 6 layer activity.
-U Report about UDP version 6 network traffic.
-t Report about TCP network traffic.
-T Report about errors for TCP-traffic.
-O Report about top-3 processes consuming most processor capacity. This report is only available when
using a log file (not when specifying an interval).
-G Report about top-3 processes consuming most resident memory. This report is only available when
using a log file (not when specifying an interval).
-D Report about top-3 processes issueing most disk transfers. This report is only available when using
a log file (not when specifying an interval).
-N Report about top-3 processes issueing most IPv4/IPv6 socket transfers. This report is only
available when using a log file (not when specifying an interval).
OUTPUT DESCRIPTION
Depending on the requested report, a number of columns with output values are produced. The values are
mostly presented as a number of events per second.
The output for the flag -c contains the following columns per cpu:
usr% Percentage of cpu-time consumed in user mode (program text) for all active processes running
with a nice value of zero (default) or a negative nice value (which means a higher priority
than usual). The cpu consumption in user mode of processes with a nice value larger than
zero (lower priority) is indicated in the nice%-column.
nice% Percentage of cpu time consumed in user mode (i.e. program text) for all processes running
witn a nice value larger than zero (which means with a lower priority than average).
sys% Percentage of cpu time consumed in system mode (kernel text) for all active processes. A high
percentage usually indicates a lot of system calls being issued.
irq% Percentage of cpu time consumed for handling of device interrupts.
softirq% Percentage of cpu time consumed for soft interrupt handling.
steal% Percentage of cpu time stolen by other virtual machines running on the same hardware.
guest% Percentage of cpu time used by other virtual machines running on the same hardware (overlaps
with usr%/nice%).
wait% Percentage of unused cpu time while at least one of the processes in wait-state awaits
completion of disk I/O.
idle% Percentage of unused cpu time because all processes are in a wait-state but not waiting for
disk-I/O.
The output for the flag -p contains the following values:
pswch/s Number of process switches (also called context switches) per second on this cpu. A process
switch occurs at the moment that an active thread (i.e. the thread using a cpu) enters a
wait state or has used its time slice completely; another thread will then be chosen to use
the cpu.
devintr/s Number of hardware interrupts handled per second on this cpu.
clones/s The number of new threads started per second.
loadavg1 Load average reflecting the average number of threads in the runqueue or in non-interruptible
wait state (usually waiting for disk or tape I/O) during the last minute.
loadavg5 Load average reflecting the average number of threads in the runqueue or in non-interruptible
wait state (usually waiting for disk or tape I/O) during the last 5 minutes.
loadavg15 Load average reflecting the average number of threads in the runqueue or in non-interruptible
wait state (usually waiting for disk or tape I/O) during the last 15 minutes.
The output for the flag -P contains information about the processes and threads:
clones/s The number of new threads started per second.
pexit/s
curproc Total number of processes present in the system.
curzomb Number of zombie processes present in the system.
thrrun Total number of threads present in the system in state 'running'.
thrslpi Total number of threads present in the system in state 'interruptible sleeping'.
thrslpu Total number of threads present in the system in state 'uninterruptible sleeping'.
The output for the flag -m contains information about the memory- and swap-utilization:
memtotal Total usable main memory size.
memfree Available main memory size at this moment (snapshot).
buffers Main memory used at this moment to cache metadata-blocks (snapshot).
cached Main memory used at this moment to cache data-blocks (snapshot).
dirty Amount of memory in the page cache that still has to be flushed to disk at this moment
(snapshot).
slabmem Main memory used at this moment for dynamically allocated memory by the kernel (snapshot).
swptotal Total swap space size at this moment (snapshot).
swpfree Available swap space at this moment (snapshot).
The output for the flag -s contains information about the frequency of swapping:
pagescan/s Number of scanned pages per second due to the fact that free memory drops below a particular
threshold.
swapin/s The number of memory-pages the system read from the swap-device per second.
swapout/s The number of memory-pages the system wrote to the swap-device per second.
commitspc The committed virtual memory space i.e. the reserved virtual space for all allocations of
private memory space for processes.
commitlim The maximum limit for the committed space, which is by default swap size plus 50% of memory
size. The kernel only verifies whether the committed space exceeds the limit if strict
overcommit handling is configured (vm.overcommit_memory is 2).
The output for the flags -l (LVM), -f (MD), and -d (hard disk) contains the following columns per active
unit:
disk Name.
busy Busy-percentage of the unit (i.e. the portion of time that the device was busy handling
requests).
read/s Number of read-requests issued per second on this unit.
KB/read Average number of Kbytes transferred per read-request for this unit.
writ/s Number of write-requests issued per second on this unit.
KB/writ Average number of Kbytes transferred per write-request for this unit.
avque Average number of requests outstanding in the queue during the time that the unit is busy.
avserv Average number of milliseconds needed by a request on this unit (seek, latency and data-
transfer).
The output for the flag -n contains information about activity on NFS mounted filesystems (client):
mounted_device
Mounted device containing server name and server directory being mounted.
physread/s Kilobytes data physically read from the NFS server by processes running on the NFS client.
KBwrite/s Kilobytes data physically written to the NFS server by processes running on the NFS client.
When the NFS filesystem was mounted during the interval, the state 'M' is shown.
The output for the flag -j contains information about NFS client activity:
rpc/s Number of RPC calls per second issued to NFS server(s).
rpcread/s Number of read RPC calls per second issued to NFS server(s).
rpcwrite/s Number of write RPC calls per second issued to NFS server(s).
retrans/s Number of retransmitted RPC calls per second.
autrefresh/s
Number of authorization refreshes per second.
The output for the flag -J contains information about NFS server activity:
rpc/s Number of RPC calls per second received from NFS client(s).
rpcread/s Number of read RPC calls per second received from NFS client(s).
rpcwrite/s Number of write RPC calls per second received from NFS client(s).
MBcr/s Number of Megabytes per second returned to read requests by clients.
MBcw/s Number of Megabytes per second passed in write requests by clients.
nettcp/s Number of requests per second handled via TCP.
netudp/s Number of requests per second handled via UDP.
The output for the flag -i provides information about utilization of network interfaces:
interf Name of interface.
busy Busy percentage for this interface. If the linespeed of this interface could not be
determined (e.g. for virtual interfaces), a question mark is shown.
ipack/s Number of packets received from this interface per second.
opack/s Number of packets transmitted to this interface per second.
iKbyte/s Number of Kbytes received from this interface per second.
oKbyte/s Number of Kbytes transmitted via this interface per second.
imbps/s Effective number of megabits received per second.
ombps/s Effective number of megabits transmitted per second.
maxmbps/s Linespeed as number of megabits per second. If the linespeed could not be determined (e.g.
virtual interfaces), value 0 is shown.
The linespeed is followed by the indication 'f' (full duplex) or 'h' (half duplex).
The output for the flag -I provides information about the failures that were detected for network
interfaces:
interf Name of interface.
ierr/s Number of bad packets received from this interface per second.
oerr/s Number of times that packet transmission to this interface failed per second.
coll/s Number of collisions encountered per second while transmitting packets.
idrop/s Number of received packets dropped per second due to lack of buffer-space in the local
system.
odrop/s Number of transmitted packets dropped per second due to lack of buffer-space in the local
system.
iframe/s Number of frame alignment-errors encountered per second on received packets.
ocarrier/s Number of carrier-errors encountered per second on transmitted packets.
The output for the flag -w provides information about the utilization of the IPv4-layer (formal SNMP-
names between brackets):
inrecv/s Number of IP datagrams received from interfaces per second, including those received in error
(ipInReceives).
outreq/s Number of IP datagrams that local higher-layer protocols supplied to IP in requests for
transmission per second (ipOutRequests).
indeliver/s Number of received IP datagrams that have been successfully delivered to higher protocol-
layers per second (ipInDelivers).
forward/s Number of received IP datagrams per second for which this entity was not their final IP
destination, as a result of which an attempt was made to forward (ipForwDatagrams).
reasmok/s Number of IP datagrams successfully reassembled per second (ipReasmOKs).
fragcreat/s Number of IP datagram fragments generated per second at this entity (ipFragCreates).
The output for the flag -W provides information about the failures that were detected in the IPv4-layer
(formal SNMP-names between brackets):
in: dsc/s Number of input IP datagrams per second for which no problems were encountered to prevent
their continued processing but that were discarded, e.g. for lack of buffer space
(ipInDiscards).
in: hder/s Number of input IP datagrams per second discarded due to errors in the IP header
(ipInHdrErrors).
in: ader/s Number of input IP datagrams per second discarded because the IP address in the destination
field was not valid to be received by this entity (ipInAddrErrors).
in: unkp/s Number of inbound packets per second that were discarded because of an unknown or unsupported
protocol (ipInUnknownProtos).
in: ratim/s Number of timeout-situations per second while other fragments were expected for successful
reassembly (ipReasmTimeout).
in: rfail/s Number of failures detected per second by the IP reassembly algorithm (ipReasmFails).
out: dsc/s Number of output IP datagrams per second for which no problems were encountered to prevent
their continued processing but that were discarded, e.g. for lack of buffer space
(ipOutDiscards).
out: nrt/s Number of IP datagrams per second discarded because no route could be found (ipOutNoRoutes).
The output for the flag -y provides information about the general utilization of the ICMPv4-layer and
some information per type of ICMP-message (formal SNMP-names between brackets):
intot/s Number of ICMP messages (any type) received per second at this entity (icmpInMsgs).
outtot/s Number of ICMP messages (any type) transmitted per second from this entity (icmpOutMsgs).
inecho/s Number of ICMP Echo (request) messages received per second (icmpInEchos).
inerep/s Number of ICMP Echo-Reply messages received per second (icmpInEchoReps).
otecho/s Number of ICMP Echo (request) messages transmitted per second (icmpOutEchos).
oterep/s Number of ICMP Echo-Reply messages transmitted per second (icmpOutEchoReps).
The output for the flag -Y provides information about other types of ICMPv4-messages (formal SNMP-names
between brackets):
ierr/s Number of ICMP messages received per second but determined to have ICMP-specific errors
(icmpInErrors).
isq/s Number of ICMP Source Quench messages received per second (icmpInSrcQuenchs).
ird/s Number of ICMP Redirect messages received per second (icmpInRedirects).
idu/s Number of ICMP Destination Unreachable messages received per second (icmpInDestUnreachs).
ite/s Number of ICMP Time Exceeded messages received per second (icmpOutTimeExcds).
oerr/s Number of ICMP messages transmitted per second but determined to have ICMP-specific errors
(icmpOutErrors).
osq/s Number of ICMP Source Quench messages transmitted per second (icmpOutSrcQuenchs).
ord/s Number of ICMP Redirect messages transmitted per second (icmpOutRedirects).
odu/s Number of ICMP Destination Unreachable messages transmitted per second (icmpOutDestUnreachs).
ote/s Number of ICMP Time Exceeded messages transmitted per second (icmpOutTimeExcds).
The output for the flag -u provides information about the utilization of the UDPv4-layer (formal SNMP-
names between brackets):
indgram/s Number of UDP datagrams per second delivered to UDP users (udpInDatagrams).
outdgram/s Number of UDP datagrams transmitted per second from this entity (udpOutDatagrams).
inerr/s Number of received UDP datagrams per second that could not be delivered for reasons other
than the lack of an application at the destination port (udpInErrors).
noport/s Number of received UDP datagrams per second for which there was no application at the
destination port (udpNoPorts).
The output for the flag -z provides information about the utilization of the IPv6-layer (formal SNMP-
names between brackets):
inrecv/s Number of input IPv6-datagrams received from interfaces per second, including those received
in error (ipv6IfStatsInReceives).
outreq/s Number of IPv6-datagrams per second that local higher-layer protocols supplied to IP in
requests for transmission (ipv6IfStatsOutRequests). This counter does not include any
forwarded datagrams.
inmc/s Number of multicast packets per second that have been received by the interface
(ipv6IfStatsInMcastPkts).
outmc/s Number of multicast packets per second that have been transmitted to the interface
(ipv6IfStatsOutMcastPkts).
indeliv/s Number of IP datagrams successfully delivered per second to IPv6 user-protocols, including
ICMP (ipv6IfStatsInDelivers).
reasmok/s Number of IPv6 datagrams successfully reassembled per second (ipv6IfStatsReasmOKs).
fragcre/s Number of IPv6 datagram fragments generated per second at this entity
(ipv6IfStatsOutFragCreates).
The output for the flag -Z provides information about the failures that were detected in the IPv6-layer
(formal SNMP-names between brackets):
in: dsc/s Number of input IPv6 datagrams per second for which no problems were encountered to prevent
their continued processing but that were discarded, e.g. for lack of buffer space
(ipv6IfStatsInDiscards).
in: hder/s Number of input datagrams per second discarded due to errors in the IPv6 header
(ipv6IfStatsInHdrErrors).
in: ader/s Number of input datagrams per second discarded because the IPv6 address in the destination
field was not valid to be received by this entity (ipv6IfStatsInAddrErrors).
in: unkp/s Number of locally-addressed datagrams per second that were discarded because of an unknown or
unsupported protocol (ipv6IfStatsInUnknownProtos).
in: ratim/s Number of timeout-situations per second while other IPv6 fragments were expected for
successful reassembly (ipv6ReasmTimeout).
in: rfail/s Number of failures detected per second by the IPv6 reassembly-algorithm
(ipv6IfStatsReasmFails).
out: dsc/s Number of output IPv6 datagrams per second for which no problems were encountered to prevent
their continued processing but that were discarded, e.g. for lack of buffer space
(ipv6IfStatsOutDiscards).
out: nrt/s Number of IPv6 datagrams per second discarded because no route could be found
(ipv6IfStatsInNoRoutes).
The output for the flag -k provides information about the general utilization of the ICMPv6-layer and
some information per type of ICMP-message (formal SNMP-names between brackets):
intot/s Number of ICMPv6 messages (any type) received per second at the interface (ipv6IfIcmpInMsgs).
outtot/s Number of ICMPv6 messages (any type) transmitted per second from this entity
(ipv6IfIcmpOutMsgs).
inerr/s Number of ICMPv6 messages received per second that had ICMP-specific errors, such as bad ICMP
checksums, bad length, etc (ipv6IfIcmpInErrors).
innsol/s Number of ICMP Neighbor Solicit messages received per second (ipv6IfIcmpInNeighborSolicits).
innadv/s Number of ICMP Neighbor Advertisement messages received per second
(ipv6IfIcmpInNeighborAdvertisements).
otnsol/s Number of ICMP Neighbor Solicit messages transmitted per second
(ipv6IfIcmpOutNeighborSolicits).
otnadv/s Number of ICMP Neighbor Advertisement messages transmitted per second
(ipv6IfIcmpOutNeighborAdvertisements).
The output for the flag -K provides information about other types of ICMPv6-messages (formal SNMP-names
between brackets):
iecho/s Number of ICMP Echo (request) messages received per second (ipv6IfIcmpInEchos).
ierep/s Number of ICMP Echo-Reply messages received per second (ipv6IfIcmpInEchoReplies).
oerep/s Number of ICMP Echo-Reply messages transmitted per second (ipv6IfIcmpOutEchoReplies).
idu/s Number of ICMP Destination Unreachable messages received per second
(ipv6IfIcmpInDestUnreachs).
odu/s Number of ICMP Destination Unreachable messages transmitted per second
(ipv6IfIcmpOutDestUnreachs).
ird/s Number of ICMP Redirect messages received per second (ipv6IfIcmpInRedirects).
ord/s Number of ICMP Redirect messages transmitted per second (ipv6IfIcmpOutRedirect).
ite/s Number of ICMP Time Exceeded messages received per second (ipv6IfIcmpInTimeExcds).
ote/s Number of ICMP Time Exceeded messages transmitted per second (ipv6IfIcmpOutTimeExcds).
The output for the flag -U provides information about the utilization of the UDPv6-layer (formal SNMP-
names between brackets):
indgram/s Number of UDPv6 datagrams per second delivered to UDP users (udpInDatagrams),
outdgram/s Number of UDPv6 datagrams transmitted per second from this entity (udpOutDatagrams),
inerr/s Number of received UDPv6 datagrams per second that could not be delivered for reasons other
than the lack of an application at the destination port (udpInErrors).
noport/s Number of received UDPv6 datagrams per second for which there was no application at the
destination port (udpNoPorts).
The output for the flag -t provides information about the utilization of the TCP-layer (formal SNMP-names
between brackets):
insegs/s Number of received segments per second, including those received in error (tcpInSegs).
outsegs/s Number of transmitted segments per second, excluding those containing only retransmitted
octets (tcpOutSegs).
actopen/s Number of active opens per second that have been supported by this entity (tcpActiveOpens).
pasopen/s Number of passive opens per second that have been supported by this entity (tcpPassiveOpens).
nowopen Number of connections currently open (snapshot), for which the state is either ESTABLISHED or
CLOSE-WAIT (tcpCurrEstab).
The output for the flag -T provides information about the failures that were detected in the TCP-layer
(formal SNMP-names between brackets):
inerr/s Number of received segments per second received in error (tcpInErrs).
retrans/s Number of retransmitted segments per second (tcpRetransSegs).
attfail/s Number of failed connection attempts per second that have occurred at this entity
(tcpAttemptFails).
estabreset/s
Number of resets per second that have occurred at this entity (tcpEstabResets).
outreset/s Number of transmitted segments per second containing the RST flag (tcpOutRsts).
The output for the flag -O provides information about the top-3 of processes with the highest processor
consumption:
pid Process-id (if zero, the process has exited while the pid could not be determined).
command The name of the process.
cpu% The percentage of cpu-capacity being consumed. This value can exceed 100% for a
multithreaded process running on a multiprocessor machine.
The output for the flag -G provides information about the top-3 of processes with the highest memory
consumption:
pid Process-id (if zero, the process has exited while the pid could not be determined).
command The name of the process.
mem% The percentage of resident memory-utilization by this process.
The output for the flag -D provides information about the top-3 of processes that issue the most read and
write accesses to disk:
pid Process-id (if zero, the process has exited while the pid could not be determined).
command The name of the process.
dsk% The percentage of read and write accesses related to the total number of read and write
accesses issued on disk by all processes, so a high percentage does not imply a high disk
load on system level.
The output for the flag -N provides information about the top-3 of processes that issue the most socket
transfers for IPv4/IPv6:
pid Process-id (if zero, the process has exited while the pid could not be determined).
command The name of the process.
net% The percentage of socket transfers related to the total number of transfers issued by all
processes, so a high percentage does not imply a high network load on system level.
EXAMPLES
To see today's cpu-activity so far (supposed that atop is logging in the background):
pcp-atopsar
To see the memory occupation for June 5, 2012 between 10:00 and 12:30 (supposed that pmlogger has been
logging daily in the background on host acme.com):
pcp-atopsar -m -r $PCP_LOG_DIR/pmlogger/acme.com/20120605 -b 10:00 -e 12:30
or
pcp-atopsar -m -r 20120605 -b 10:00 -e 12:30
or, suppose it is June 8, 2012 at this moment
pcp-atopsar -m -r yyy -b 10:00 -e 12:30
Write a logfile with atop to record the system behaviour for 30 minutes (30 samples of one minute) and
produce all available reports afterwards:
pcp-atop -w /tmp/atoplog 60 30
pcp-atopsar -A -r /tmp/atoplog
To watch TCP activity evolve for ten minutes (10 samples with sixty seconds interval):
pcp-atopsar -t 60 10
To watch the header-lines ('_' as last character) of all reports with only the detail-lines showing
critical resource consumption (marker '*' or '+' as last character):
pcp-atopsar -AM | grep '[_*+]$'
FILES
/etc/atoprc
Configuration file containing system-wide default values (mainly flags). See related man-page.
~/.atoprc
Configuration file containing personal default values (mainly flags). See related man-page.
$PCP_LOG_DIR/pmlogger/HOST/YYYYMMDD
Daily data file, where YYYYMMDD are digits representing the date, and HOST is the hostname of the
machine being logged.
SEE ALSO
pcp(1), pcp-atop(1), mkaf(1), pmlogger(1), pmlogger_daily(1), PCPIntro(1) and pcp-atoprc(5).