Provided by: erlang-manpages_16.b.3-dfsg-1ubuntu2.2_all 
NAME
test_server - This module provides support for test suite authors.
DESCRIPTION
The test_server module aids the test suite author by providing various support functions.
The supported functionality includes:
* Logging and timestamping
* Capturing output to stdout
* Retrieving and flushing the message queue of a process
* Watchdog timers, process sleep, time measurement and unit conversion
* Private scratch directory for all test suites
* Start and stop of slave- or peer nodes
For more information on how to write test cases and for examples, please see the Test
Server User's Guide.
TEST SUITE SUPPORT FUNCTIONS
The following functions are supposed to be used inside a test suite.
EXPORTS
os_type() -> OSType
Types:
OSType = term()
This is the same as returned from os:type/0
This function is equivalent to os:type/0. It is kept for backwards compatibility.
fail()
fail(Reason)
Types:
Reason = term()
The reason why the test case failed.
This will make the test suite fail with a given reason, or with suite_failed if no
reason was given. Use this function if you want to terminate a test case, as this
will make it easier to read the log- and HTML files. Reason will appear in the
comment field in the HTML log.
timetrap(Timout) -> Handle
Types:
Timeout = integer() | {hours,H} | {minutes,M} | {seconds,S}
H = M = S = integer()
Pid = pid()
The process that is to be timetrapped (self()by default)
Sets up a time trap for the current process. An expired timetrap kills the process
with reason timetrap_timeout. The returned handle is to be given as argument to
timetrap_cancel before the timetrap expires. If Timeout is an integer, it is
expected to be milliseconds.
Note:
If the current process is trapping exits, it will not be killed by the exit signal
with reason timetrap_timeout. If this happens, the process will be sent an exit
signal with reason kill 10 seconds later which will kill the process. Information
about the timetrap timeout will in this case not be found in the test logs.
However, the error_logger will be sent a warning.
timetrap_cancel(Handle) -> ok
Types:
Handle = term()
Handle returned from timetrap
This function cancels a timetrap. This must be done before the timetrap expires.
timetrap_scale_factor() -> ScaleFactor
Types:
ScaleFactor = integer()
This function returns the scale factor by which all timetraps are scaled. It is
normally 1, but can be greater than 1 if the test_server is running cover, using a
larger amount of scheduler threads than the amount of logical processors on the
system, running under purify, valgrind or in a debug-compiled emulator. The scale
factor can be used if you need to scale you own timeouts in test cases with same
factor as the test_server uses.
sleep(MSecs) -> ok
Types:
MSecs = integer() | float() | infinity
The number of milliseconds to sleep
This function suspends the calling process for at least the supplied number of
milliseconds. There are two major reasons why you should use this function instead
of timer:sleep, the first being that the module timer may be unavailable at the
time the test suite is run, and the second that it also accepts floating point
numbers.
adjusted_sleep(MSecs) -> ok
Types:
MSecs = integer() | float() | infinity
The default number of milliseconds to sleep
This function suspends the calling process for at least the supplied number of
milliseconds. The function behaves the same way as test_server:sleep/1, only MSecs
will be multiplied by the 'multiply_timetraps' value, if set, and also
automatically scaled up if 'scale_timetraps' is set to true (which it is by
default).
hours(N) -> MSecs
minutes(N) -> MSecs
seconds(N) -> MSecs
Types:
N = integer()
Value to convert to milliseconds.
Theese functions convert N number of hours, minutes or seconds into milliseconds.
Use this function when you want to test_server:sleep/1 for a number of seconds,
minutes or hours(!).
format(Format) -> ok
format(Format, Args)
format(Pri, Format)
format(Pri, Format, Args)
Types:
Format = string()
Format as described for io_:format.
Args = list()
List of arguments to format.
Formats output just like io:format but sends the formatted string to a logfile. If
the urgency value, Pri, is lower than some threshold value, it will also be written
to the test person's console. Default urgency is 50, default threshold for display
on the console is 1.
Typically, the test person don't want to see everything a test suite outputs, but
is merely interested in if the test cases succeeded or not, which the test server
tells him. If he would like to see more, he could manually change the threshold
values by using the test_server_ctrl:set_levels/3 function.
capture_start() -> ok
capture_stop() -> ok
capture_get() -> list()
These functions makes it possible to capture all output to stdout from a process
started by the test suite. The list of characters captured can be purged by using
capture_get.
messages_get() -> list()
This function will empty and return all the messages currently in the calling
process' message queue.
timecall(M, F, A) -> {Time, Value}
Types:
M = atom()
The name of the module where the function resides.
F = atom()
The name of the function to call in the module.
A = list()
The arguments to supply the called function.
Time = integer()
The number of seconds it took to call the function.
Value = term()
Value returned from the called function.
This function measures the time (in seconds) it takes to call a certain function.
The function call is not caught within a catch.
do_times(N, M, F, A) -> ok
do_times(N, Fun)
Types:
N = integer()
Number of times to call MFA.
M = atom()
Module name where the function resides.
F = atom()
Function name to call.
A = list()
Arguments to M:F.
Calls MFA or Fun N times. Useful for extensive testing of a sensitive function.
m_out_of_n(M, N, Fun) -> ok | exit({m_out_of_n_failed, {R,left_to_do}}
Types:
N = integer()
Number of times to call the Fun.
M = integer()
Number of times to require a successful return.
Repeatedly evaluates the given function until it succeeds (doesn't crash) M times.
If, after N times, M successful attempts have not been accomplished, the process
crashes with reason {m_out_of_n_failed, {R,left_to_do}}, where R indicates how many
cases that was still to be successfully completed.
For example:
m_out_of_n(1,4,fun() -> tricky_test_case() end)
Tries to run tricky_test_case() up to 4 times, and is happy if it succeeds once.
m_out_of_n(7,8,fun() -> clock_sanity_check() end)
Tries running clock_sanity_check() up to 8 times,and allows the function to fail
once. This might be useful if clock_sanity_check/0 is known to fail if the clock
crosses an hour boundary during the test (and the up to 8 test runs could never
cross 2 boundaries)
call_crash(M, F, A) -> Result
call_crash(Time, M, F, A) -> Result
call_crash(Time, Crash, M, F, A) -> Result
Types:
Result = ok | exit(call_crash_timeout) | exit({wrong_crash_reason, Reason})
Crash = term()
Crash return from the function.
Time = integer()
Timeout in milliseconds.
M = atom()
Module name where the function resides.
F = atom()
Function name to call.
A = list()
Arguments to M:F.
Spawns a new process that calls MFA. The call is considered successful if the call
crashes with the gives reason (Crash) or any reason if not specified. The call must
terminate within the given time (default infinity), or it is considered a failure.
temp_name(Stem) -> Name
Types:
Stem = string()
Returns a unique filename starting with Stem with enough extra characters appended
to make up a unique filename. The filename returned is guaranteed not to exist in
the filesystem at the time of the call.
break(Comment) -> ok
Types:
Comment = string()
Comment is a string which will be written in the shell, e.g. explaining what to do.
This function will cancel all timetraps and pause the execution of the test case
until the user executes the continue/0 function. It gives the user the opportunity
to interact with the erlang node running the tests, e.g. for debugging purposes or
for manually executing a part of the test case.
When the break/1 function is called, the shell will look something like this:
--- SEMIAUTOMATIC TESTING ---
The test case executes on process <0.51.0>
"Here is a comment, it could e.g. instruct to pull out a card"
-----------------------------
Continue with --> test_server:continue().
The user can now interact with the erlang node, and when ready call
test_server:continue().
Note that this function can not be used if the test is executed with
ts:run/0/1/2/3/4 in batch mode.
continue() -> ok
This function must be called in order to continue after a test case has called
break/1.
run_on_shielded_node(Fun, CArgs) -> term()
Types:
Fun = function() (arity 0)
Function to execute on the shielded node.
CArg = string()
Extra command line arguments to use when starting the shielded node.
Fun is executed in a process on a temporarily created hidden node with a proxy for
communication with the test server node. The node is called a shielded node (should
have been called a shield node). If Fun is successfully executed, the result is
returned. A peer node (see start_node/3) started from the shielded node will be
shielded from test server node, i.e. they will not be aware of each other. This is
useful when you want to start nodes from earlier OTP releases than the OTP release
of the test server node.
Nodes from an earlier OTP release can normally not be started if the test server
hasn't been started in compatibility mode (see the +R flag in the erl(1)
documentation) of an earlier release. If a shielded node is started in
compatibility mode of an earlier OTP release than the OTP release of the test
server node, the shielded node can start nodes of an earlier OTP release.
Note:
You must make sure that nodes started by the shielded node never communicate
directly with the test server node.
Note:
Slave nodes always communicate with the test server node; therefore, never start
slave nodes from the shielded node, always start peer nodes.
start_node(Name, Type, Options) -> {ok, Node} | {error, Reason}
Types:
Name = atom() | string()
Name of the slavenode to start (as given to -sname or -name)
Type = slave | peer
The type of node to start.
Options = [{atom(), term()]
Tuplelist of options
This functions starts a node, possibly on a remote machine, and guarantees cross
architecture transparency. Type is set to either slave or peer.
slave means that the new node will have a master, i.e. the slave node will
terminate if the master terminates, TTY output produced on the slave will be sent
back to the master node and file I/O is done via the master. The master is normally
the target node unless the target is itself a slave.
peer means that the new node is an independent node with no master.
Options is a tuplelist which can contain one or more of
{remote, true}:
Start the node on a remote host. If not specified, the node will be started on
the local host. Test cases that require a remote host will fail with a
reasonable comment if no remote hosts are available at the time they are run.
{args, Arguments}:
Arguments passed directly to the node. This is typically a string appended to
the command line.
{wait, false}:
Don't wait until the node is up. By default, this function does not return
until the node is up and running, but this option makes it return as soon as
the node start command is given..
Only valid for peer nodes
{fail_on_error, false}:
Returns {error, Reason} rather than failing the test case.
Only valid for peer nodes. Note that slave nodes always act as if they had
fail_on_error=false
{erl, ReleaseList}:
Use an Erlang emulator determined by ReleaseList when starting nodes, instead
of the same emulator as the test server is running. ReleaseList is a list of
specifiers, where a specifier is either {release, Rel}, {prog, Prog}, or
'this'. Rel is either the name of a release, e.g., "r12b_patched" or 'latest'.
'this' means using the same emulator as the test server. Prog is the name of an
emulator executable. If the list has more than one element, one of them is
picked randomly. (Only works on Solaris and Linux, and the test server gives
warnings when it notices that nodes are not of the same version as itself.)
When specifying this option to run a previous release, use
is_release_available/1 function to test if the given release is available and
skip the test case if not.
In order to avoid compatibility problems (may not appear right away), use a
shielded node (see run_on_shielded_node/2) when starting nodes from different
OTP releases than the test server.
{cleanup, false}:
Tells the test server not to kill this node if it is still alive after the test
case is completed. This is useful if the same node is to be used by a group of
test cases.
{env, Env}:
Env should be a list of tuples {Name, Val}, where Name is the name of an
environment variable, and Val is the value it is to have in the started node.
Both Name and Val must be strings. The one exception is Val being the atom
false (in analogy with os:getenv/1), which removes the environment variable.
Only valid for peer nodes. Not available on VxWorks.
{start_cover, false}:
By default the test server will start cover on all nodes when the test is run
with code coverage analysis. To make sure cover is not started on a new node,
set this option to false. This can be necessary if the connection to the node
at some point will be broken but the node is expected to stay alive. The reason
is that a remote cover node can not continue to run without its main node.
Another solution would be to explicitly stop cover on the node before breaking
the connection, but in some situations (if old code resides in one or more
processes) this is not possible.
stop_node(NodeName) -> bool()
Types:
NodeName = term()
Name of the node to stop
This functions stops a node previously started with start_node/3. Use this function
to stop any node you start, or the test server will produce a warning message in
the test logs, and kill the nodes automatically unless it was started with the
{cleanup, false} option.
is_commercial() -> bool()
This function test whether the emulator is commercially supported emulator. The
tests for a commercially supported emulator could be more stringent (for instance,
a commercial release should always contain documentation for all applications).
is_release_available(Release) -> bool()
Types:
Release = string() | atom()
Release to test for
This function test whether the release given by Release (for instance,
"r12b_patched") is available on the computer that the test_server controller is
running on. Typically, you should skip the test case if not.
Caution: This function may not be called from the suite clause of a test case, as
the test_server will deadlock.
is_native(Mod) -> bool()
Types:
Mod = atom()
A module name
Checks whether the module is natively compiled or not
app_test(App) -> ok | test_server:fail()
app_test(App,Mode)
Types:
App = term()
The name of the application to test
Mode = pedantic | tolerant
Default is pedantic
Checks an applications .app file for obvious errors. The following is checked:
* required fields
* that all modules specified actually exists
* that all requires applications exists
* that no module included in the application has export_all
* that all modules in the ebin/ dir is included (If Mode==tolerant this only
produces a warning, as all modules does not have to be included)
comment(Comment) -> ok
Types:
Comment = string()
The given String will occur in the comment field of the table on the HTML result
page. If called several times, only the last comment is printed. comment/1 is also
overwritten by the return value {comment,Comment} from a test case or by fail/1
(which prints Reason as a comment).
TEST SUITE EXPORTS
The following functions must be exported from a test suite module.
EXPORTS
all(suite) -> TestSpec | {skip, Comment}
Types:
TestSpec = list()
Comment = string()
This comment will be printed on the HTML result page
This function must return the test specification for the test suite module. The
syntax of a test specification is described in the Test Server User's Guide.
init_per_suite(Config0) -> Config1 | {skip, Comment}
Types:
Config0 = Config1 = [tuple()]
Comment = string()
Describes why the suite is skipped
This function is called before all other test cases in the suite. Config is the
configuration which can be modified here. Whatever is returned from this function
is given as Config to the test cases.
If this function fails, all test cases in the suite will be skipped.
end_per_suite(Config) -> void()
Types:
Config = [tuple()]
This function is called after the last test case in the suite, and can be used to
clean up whatever the test cases have done. The return value is ignored.
init_per_testcase(Case, Config0) -> Config1 | {skip, Comment}
Types:
Case = atom()
Config0 = Config1 = [tuple()]
Comment = string()
Describes why the test case is skipped
This function is called before each test case. The Case argument is the name of the
test case, and Config is the configuration which can be modified here. Whatever is
returned from this function is given as Config to the test case.
end_per_testcase(Case, Config) -> void()
Types:
Case = atom()
Config = [tuple()]
This function is called after each test case, and can be used to clean up whatever
the test case has done. The return value is ignored.
Case(doc) -> [Decription]
Case(suite) -> [] | TestSpec | {skip, Comment}
Case(Config) -> {skip, Comment} | {comment, Comment} | Ok
Types:
Description = string()
Short description of the test case
TestSpec = list()
Comment = string()
This comment will be printed on the HTML result page
Ok = term()
Config = [tuple()]
Elements from the Config parameter can be read with the ?config macro, see
section about test suite support macros
The documentation clause (argument doc) can be used for automatic generation of
test documentation or test descriptions.
The specification clause (argument spec) shall return an empty list, the test
specification for the test case or {skip,Comment}. The syntax of a test
specification is described in the Test Server User's Guide.
The execution clause (argument Config) is only called if the specification clause
returns an empty list. The execution clause is the real test case. Here you must
call the functions you want to test, and do whatever you need to check the result.
If something fails, make sure the process crashes or call test_server:fail/0/1
(which also will cause the process to crash).
You can return {skip,Comment} if you decide not to run the test case after all,
e.g. if it is not applicable on this platform.
You can return {comment,Comment} if you wish to print some information in the
'Comment' field on the HTML result page.
If the execution clause returns anything else, it is considered a success, unless
it is {'EXIT',Reason} or {'EXIT',Pid,Reason} which can't be distinguished from a
crash, and thus will be considered a failure.
A conf test case is a group of test cases with an init and a cleanup function. The
init and cleanup functions are also test cases, but they have special rules:
* They do not need a specification clause.
* They must always have the execution clause.
* They must return the Config parameter, a modified version of it or
{skip,Comment} from the execution clause.
* The cleanup function may also return a tuple {return_group_result,Status},
which is used to return the status of the conf case to Test Server and/or to a
conf case on a higher level. (Status = ok | skipped | failed).
* init_per_testcase and end_per_testcase are not called before and after these
functions.
TEST SUITE LINE NUMBERS
If a test case fails, the test server can report the exact line number at which it failed.
There are two ways of doing this, either by using the line macro or by using the
test_server_line parse transform.
The line macro is described under TEST SUITE SUPPORT MACROS below. The line macro will
only report the last line executed when a test case failed.
The test_server_line parse transform is activated by including the headerfile
test_server_line.hrl in the test suite. When doing this, it is important that the
test_server_line module is in the code path of the erlang node compiling the test suite.
The parse transform will report a history of a maximum of 10 lines when a test case fails.
Consecutive lines in the same function are not shown.
The attribute -no_lines(FuncList). can be used in the test suite to exclude specific
functions from the parse transform. This is necessary e.g. for functions that are executed
on old (i.e. <R10B) OTP releases. FuncList = [{Func,Arity}].
If both the line macro and the parse transform is used in the same module, the parse
transform will overrule the macro.
TEST SUITE SUPPORT MACROS
There are some macros defined in the test_server.hrl that are quite useful for test suite
programmers:
The line macro, is quite essential when writing test cases. It tells the test server
exactly what line of code that is being executed, so that it can report this line back if
the test case fails. Use this macro at the beginning of every test case line of code.
The config macro, is used to retrieve information from the Config variable sent to all
test cases. It is used with two arguments, where the first is the name of the
configuration variable you wish to retrieve, and the second is the Config variable
supplied to the test case from the test server.
Possible configuration variables include:
* data_dir - Data file directory.
* priv_dir - Scratch file directory.
* nodes - Nodes specified in the spec file
* nodenames - Generated nodenames.
* Whatever added by conf test cases or init_per_testcase/2
Examples of the line and config macros can be seen in the Examples chapter in the user's
guide.
If the line_trace macro is defined, you will get a timestamp (erlang:now()) in your minor
log for each line macro in your suite. This way you can at any time see which line is
currently being executed, and when the line was called.
The line_trace macro can also be used together with the test_server_line parse transform
described above. A timestamp will then be written for each line in the suite, except for
functions stated in the -no_lines attribute.
The line_trace macro can e.g. be defined as a compile option, like this:
erlc -W -Dline_trace my_SUITE.erl