Provided by: libpcp3-dev_6.3.3-1_amd64 
NAME
__pmFaultInject, __pmFaultSummary, PM_FAULT_POINT, PM_FAULT_RETURN, PM_FAULT_CHECK, PM_FAULT_CLEAR -
Fault Injection Infrastructure for QA
C SYNOPSIS
#include <pcp/pmapi.h>
#include <pcp/fault.h>
void __pmFaultInject(const char *ident, int class);
void __pmFaultSummary(FILE *f);
PM_FAULT_POINT(ident, class);
PM_FAULT_RETURN(retvalue);
PM_FAULT_CHECK;
PM_FAULT_CLEAR;
cc -DPM_FAULT_INJECTION=1 ... -lpcp_fault
DESCRIPTION
As part of the coverage-driven changes to QA in PCP 3.6, it became apparent that we needed someway to
exercise the ``uncommon'' code paths associated with error detection and recovery.
The facilities described below provide a basic fault injection infrastructure (for libpcp only at this
stage, although the mechanism is far more general and could easily be extended).
A special build is required to create libpcp_fault and the associated <pcp/fault.h> header file. Once
this has been done, new QA applications may be built with -DPM_FAULT_INJECTION=1 and/or existing
applications can be exercised in presence of fault injection by forcing libpcp_fault to be used in
preference to libpcp as described below.
In the code to be tested, __pmFaultInject defines a fault point at which a fault of type class may be
injected. ident is a string to uniquely identify the fault point across all of the PCP source code, so
something like "libpcp/" __FILE__ ":<number>" works just fine. The ident string also determines if a
fault will be injected at run-time or not - refer to the RUN-TIME CONTROL section below. class selects a
failure type, using one of the following defined values (this list may well grow over time):
PM_FAULT_ALLOC
Will cause the next call to malloc(3), realloc(3) or strdup(3) to fail, returning NULL and setting
errno to ENOMEM. We could extend the coverage to all of the malloc-related routines, but these
three are sufficient to cover the vast majority of the uses within libpcp.
PM_FAULT_CALL
Will cause the next call to an instrumented routine to fail by returning an error code (possibly
the new PM_ERR_FAULT code). The actual error code is defined in the PM_FAULT_RETURN macro at the
head of an instrumented routine. Initially, only __pmRegisterAnon(3) (returns PM_ERR_FAULT),
__pmGetPDU(3) (returns PM_ERR_TIMEOUT) and __pmAllocResult(3) (returns NULL) were instrumented as
a proof of concept for this part of the facility, however other routines may have this fault
injection capability added over time.
PM_FAULT_MISC
The ``other'' class, currently used with PM_FAULT_CHECK as described below.
To allow fault injection to co-exist within the production source code, PM_FAULT_POINT is a macro that
emits no code by default, but when PM_FAULT_INJECTION is defined this becomes a call to __pmFaultInject.
Throughout libpcp we use PM_FAULT_POINT and not __pmFaultInject so that both libpcp and libpcp_fault can
be built from the same source code.
Similarly, the macro PM_FAULT_RETURN emits no code unless PM_FAULT_INJECTION is defined, in which case if
a fault of type PM_FAULT_CALL has been armed with __pmFaultInject then, the enclosing routine return with
the function value retvalue.
The PM_FAULT_CHECK macro returns a value that may be 0 or 1. If PM_FAULT_INJECTION is defined then if a
fault of type PM_FAULT_MISC has been armed with __pmFaultInject then the value is 1 else it is 0.
PM_FAULT_CHECK is most often used in concert with the PM_FAULT_POINT macro with the PM_FAULT_MISC class
to potentially arm a trigger, then test PM_FAULT_CHECK and if this has the value 1, then the
PM_FAULT_CLEAR macro is used to clear any armed faults, and the fault injection code is executed.
This is illustrated in the example below from src/libpcp/src/exec.c:
pid = fork();
/* begin fault-injection block */
PM_FAULT_POINT("libpcp/" __FILE__ ":4", PM_FAULT_MISC);
if (PM_FAULT_CHECK) {
PM_FAULT_CLEAR;
if (pid > (pid_t)0)
kill(pid, SIGKILL);
setoserror(EAGAIN);
pid = -1;
}
/* end fault-injection block */
A summary of fault points seen and faults injected is produced on stdio stream f by __pmFaultSummary.
Additional tracing (via -Dfault or pmDebugOptions.fault) and a new PMAPI error code (PM_ERR_FAULT) are
also defined, although these will only ever be seen or used in libpcp_fault. If pmDebugOptions.fault is
set the first time __pmFaultInject is called, then __pmFaultSummary will be called automatically to
report on stderr when the application exits (via atexit(3)).
Fault injection cannot be nested. Each call to __pmFaultInject clears any previous fault injection that
has been armed, but not yet executed.
The fault injection infrastructure is not thread-safe and should only be used with applications that are
known to be single-threaded.
RUN-TIME CONTROL
By default, no fault injection is enabled at run-time, even when __pmFaultInject is called.
Faults are selectively enabled using a control file, identified by the environment variable
$PM_FAULT_CONTROL; if this is not set, no faults are enabled.
The control file (if it exists) is read the first time __pmFaultInject is called, and contains lines of
the form:
ident op number
that define fault injection guards.
ident is a fault point string (as defined by a call to __pmFaultInject, or more usually the
PM_FAULT_POINT macro). So one needs access to the libpcp source code to determine the available ident
strings and their semantics.
op is one of the C-style operators >=, >, ==, <, <=, != or % and number is an unsigned integer. op
number is optional and the default is >0
The semantics of the fault injection guards are that each time __pmFaultInject is called for a particular
ident, a trip count is incremented (the first trip is 1); if the C-style expression tripcount op number
has the value 1 (so true for most ops, or the remainder equals 1 for the % op), then a fault of the class
defined for the fault point associated with ident will be armed, and executed as soon as possible.
Within the control file, blank lines are ignored and lines beginning with # are treated as comments.
For an existing application linked with libpcp fault injection may still be used by forcing libpcp_fault
to be used in the place of libpcp. The following example shows how this might be done.
$ export PM_FAULT_CONTROL=/tmp/control
$ cat $PM_FAULT_CONTROL
# ok for 2 trips, then inject errors
libpcp/events.c:1 >2
$ export LD_PRELOAD=/usr/lib/libpcp_fault.so
$ pmevent -Dfault -s 3 sample.event.records
host: localhost
samples: 3
interval: 1.00 sec
sample.event.records[fungus]: 0 event records
__pmFaultInject(libpcp/events.c:1) ntrip=1 SKIP
sample.event.records[bogus]: 2 event records
10:46:12.413 --- event record [0] flags 0x1 (point) ---
sample.event.param_string "fetch #0"
10:46:12.413 --- event record [1] flags 0x1 (point) ---
sample.event.param_string "bingo!"
__pmFaultInject(libpcp/events.c:1) ntrip=2 SKIP
sample.event.records[fungus]: 1 event records
10:46:03.416 --- event record [0] flags 0x1 (point) ---
__pmFaultInject(libpcp/events.c:1) ntrip=3 INJECT
sample.event.records[bogus]: pmUnpackEventRecords: Cannot allocate memory
__pmFaultInject(libpcp/events.c:1) ntrip=4 INJECT
sample.event.records[fungus]: pmUnpackEventRecords: Cannot allocate memory
__pmFaultInject(libpcp/events.c:1) ntrip=5 INJECT
sample.event.records[bogus]: pmUnpackEventRecords: Cannot allocate memory
=== Fault Injection Summary Report ===
libpcp/events.c:1: guard trip>2, 5 trips, 3 faults
EXAMPLES
Refer to the PCP and PCP QA source code.
The macro definitions are in src/include/pcp/fault.h.
src/libpcp/src/fault.c contains all of the the underlying implementation.
src/libpcp_fault and src/libpcp_fault/src contains the recipe and Makefiles for creating and installing
libpcp_fault.so and <pcp/fault.h>.
PM_FAULT_RETURN was initiallly used in the following libpcp source files: derive_parser.y.in, pdu.c and
result.c.
PM_FAULT_POINT. was initiallly used in the following libpcp source files: derive_parser.y.in, desc.c,
e_indom.c, e_labels.c, err.c, events.c, exec.c, fetch.c, help.c, instance.c, interp.c, labels.c,
logmeta.c, pmns.c, p_profile.c and store.c.
The ``fault'' group of QA tests show examples of control file use. To see which tests are involved
$ cd qa
$ check -n -g fault
DIAGNOSTICS
Some non-recoverable errors are reported on stderr.
ENVIRONMENT
PM_FAULT_CONTROL
Full path to the fault injection control file.
LD_PRELOAD
Force libpcp_fault to be used in preference to libpcp.
SEE ALSO
PMAPI(3)