Ubuntu Manpage: mailcross - a cross-validation simulator for use with dbacl.

NAME

       mailcross - a cross-validation simulator for use with dbacl.

SYNOPSIS


       mailcross command [ command_arguments ]

DESCRIPTION

mailcross automates the task of cross-validating email filtering and classification programs such as
dbacl(1). Given a set of categorized documents, mailcross initiates simulation runs to estimate the
classification errors and thereby permits fine tuning of the parameters of the classifier.

Cross-validation is a method which is widely used to compare the quality of classification and learning
algorithms, and as such permits rudimentary comparisons between those classifiers which make use of
dbacl(1) and bayesol(1), and other competing classifiers.

The mechanics of cross-validation are as follows: A set of pre-classified email messages is first split
into a number of roughly equal-sized subsets. For each subset, the filter (by default, dbacl(1)) is used
to classify each message within this subset, based upon having learned the categories from the remaining
subsets. The resulting classification errors are then averaged over all subsets.

The results obtained by cross validation essentially do not depend upon the ordering of the sample
emails. Other methods (see mailtoe(1),mailfoot(1)) attempt to capture the behaviour of classification
errors over time.

mailcross uses the environment variables MAILCROSS_LEARNER and MAILCROSS_FILTER when executing, which
permits the cross-validation of arbitrary filters, provided these satisfy the compatibility conditions
stated in the ENVIRONMENT section below.

For convenience, mailcross implements a testsuite framework with predefined wrappers for several open
source classifiers. This permits the direct comparison of dbacl(1) with competing classifiers on the same
set of email samples. See the USAGE section below.

During preparation, mailcross builds a subdirectory named mailcross.d in the current working directory.
All needed calculations are performed inside this subdirectory.

EXIT STATUS

       mailcross returns 0 on success, 1 if a problem occurred.

COMMANDS

prepare size
Prepares a subdirectory named mailcross.d in the current working directory, and populates it with
empty subdirectories for exactly size subsets.

add category [FILE]...
Takes a set of emails from either FILE if specified, or STDIN, and associates them with category.
All emails are distributed randomly into the subdirectories of mailcross.d for later use. For each
category, this command can be repeated several times, but should be executed at least once.

clean Deletes the directory mailcross.d and all its contents.

learn For every previously built subset of email messages, pre-learns all the categories based on the
contents of all the subsets except this one. The command_arguments are passed to
MAILCROSS_LEARNER.

run For every previously built subset of email messages, performs the classification based upon the
pre-learned categories associated with all but this subset. The command_arguments are passed to
MAILCROSS_FILTER.

summarize
Prints statistics for the latest cross-validation run.

review truecat predcat
Scans the last run statistics and extracts all the messages which belong to category truecat but
have been classified into category predcat. The extracted messages are copied to the directory
mailcross.d/review for perusal.

testsuite list
Shows a list of available filters/wrapper scripts which can be selected.

testsuite select [FILTER]...
Prepares the filter(s) named FILTER to be used for simulation. The filter name is the name of a
wrapper script located in the directory /usr/share/dbacl/testsuite. Each filter has a rigid
interface documented below, and the act of selecting it copies it to the mailcross.d/filters
directory. Only filters located there are used in the simulations.

testsuite deselect [FILTER]...
Removes the named filter(s) from the directory mailcross.d/filters so that they are not used in
the simulation.

testsuite run
Invokes every selected filter on the datasets added previously, and calculates misclassification
rates.

testsuite status
Describes the scheduled simulations.

testsuite summarize
Shows the cross validation results for all filters. Only makes sense after the run command.

USAGE

The normal usage pattern is the following: first, you should separate your email collection into several
categories (manually or otherwise). Each category should be associated with one or more folders, but each
folder should not contain more than one category. Next, you should decide how many subsets to use, say
10. Note that too many subsets will slow down the calculations rapidly. Now you can type

% mailcross prepare 10

Next, for every category, you must add every folder associated with this category. Suppose you have three
categories named spam, work, and play, which are associated with the mbox files spam.mbox, work.mbox, and
play.mbox respectively. You would type

% mailcross add spam spam.mbox
% mailcross add work work.mbox
% mailcross add play play.mbox

You can now perform as many simulations as desired. Every cross validation consists of a learning, a
running and a summarizing stage. These operations are performed on the classifier specified in the
MAILCROSS_FILTER and MAILCROSS_LEARNER variables. By setting these variables appropriately, you can
compare classification performance as you vary the command line options of your classifier(s).

% mailcross learn
% mailcross run
% mailcross summarize

The testsuite commands are designed to simplify the above steps and allow comparison of a wide range of
email classifiers, including but not limited to dbacl. Classifiers are supported through wrapper
scripts, which are located in the /usr/share/dbacl/testsuite directory.

The first stage when using the testsuite is deciding which classifiers to compare. You can view a list
of available wrappers by typing:

% mailcross testsuite list

Note that the wrapper scripts are NOT the actual email classifiers, which must be installed separately by
your system administrator or otherwise. Once this is done, you can select one or more wrappers for the
simulation by typing, for example:

% mailcross testsuite select dbaclA ifile

If some of the selected classifiers cannot be found on the system, they are not selected. Note also that
some wrappers can have hard-coded category names, e.g. if the classifier only supports binary
classification. Heed the warning messages.

It remains only to run the simulation. Beware, this can take a long time (several hours depending on the
classifier).

% mailcross testsuite run
% mailcross testsuite summarize

Once you are all done with simulations, you can delete the working files, log files etc. by typing

% mailcross clean

The progress of the cross validation is written silently in various log files which are located in the
mailcross.d/log directory. Check these in case of problems.

SCRIPT INTERFACE

mailcross testsuite takes care of learning and classifying your prepared email corpora for each selected
classifier. Since classifiers have widely varying interfaces, this is only possible by wrapping those
interfaces individually into a standard form which can be used by mailcross testsuite.

Each wrapper script is a command line tool which accepts a single command followed by zero or more
optional arguments, in the standard form:

wrapper command [argument]...

Each wrapper script also makes use of STDIN and STDOUT in a well defined way. If no behaviour is
described, then no output or input should be used. The possible commands are described below:

filter In this case, a single email is expected on STDIN, and a list of category filenames is expected in
$2, $3, etc. The script writes the category name corresponding to the input email on STDOUT. No
trailing newline is required or expected.

learn In this case, a standard mbox stream is expected on STDIN, while a suitable category file name is
expected in $2. No output is written to STDOUT.

clean In this case, a directory is expected in $2, which is examined for old database information. If
any old databases are found, they are purged or reset. No output is written to STDOUT.

describe
IN this case, a single line of text is written to STDOUT, describing the filter's functionality.
The line should be kept short to prevent line wrapping on a terminal.

bootstrap
In this case, a directory is expected in $2. The wrapper script first checks for the existence of
its associated classifier, and other prerequisites. If the check is successful, then the wrapper
is cloned into the supplied directory. A courtesy notification should be given on STDOUT to
express success or failure. It is also permissible to give longer descriptions caveats.

toe Used by mailtoe(1).

foot Used by mailfoot(1).

ENVIRONMENT

       Right after loading, mailcross reads the hidden file .mailcrossrc in the $HOME directory, if  it  exists,
       so this would be a good place to define custom values for environment variables.

       MAILCROSS_FILTER
              This  variable  contains  a shell command to be executed repeatedly during the running stage.  The
              command should accept an email message on STDIN and output a resulting category  name.  It  should
              also  accept  a list of category file names on the command line.  If undefined, mailcross uses the
              default value MAILCROSS_FILTER="dbacl -T email -T xml -v" (and also magically adds the  -c  option
              before each category).

       MAILCROSS_LEARNER
              This  variable  contains  a shell command to be executed repeatedly during the learning stage. The
              command should accept a mbox type stream of emails on STDIN for learning, and the file name of the
              category  on   the   command   line.    If   undefined,   mailcross   uses   the   default   value
              MAILCROSS_LEARNER="dbacl -H 19 -T email -T xml -l".

       TEMPDIR
              This  directory  is  exported  for  the  benefit  of wrapper scripts. Scripts which need to create
              temporary files should place them a the location given in TEMPDIR.

NOTES

       The subdirectory mailcross.d can grow quite large. It contains a full copy of the  training  corpora,  as
       well as learning files for size times all the added categories, and various log files.

WARNING

       Cross-validation  is  a  widely  used, but ad-hoc statistical procedure, completely unrelated to Bayesian
       theory, and subject to controversy.  Use this at your own risk.

SOURCE

       The source code for the latest version of this program is available at the following locations:

       http://www.lbreyer.com/gpl.html
       http://dbacl.sourceforge.net

AUTHOR

       Laird A. Breyer <laird@lbreyer.com>