Provided by: dbacl_1.12-3_amd64 bug

NAME

       mailtoe - a train-on-error simulator for use with dbacl.

SYNOPSIS

       mailtoe command [ command_arguments ]

DESCRIPTION

       mailtoe  automates the task of testing email filtering and classification programs such as
       dbacl(1).  Given a set of categorized documents, mailtoe initiates test runs  to  estimate
       the  classification  errors  and  thereby  permit  fine  tuning  of  the parameters of the
       classifier.

       Train-on-error (TOE)  is  a  learning  method  which  is  sometimes  advocated  for  email
       classifiers. Given an incoming email stream, the method consists in reusing a fixed set of
       category databases until the first misclassification occurs. At that point, the  offending
       email  is used to relearn the relevant category, until the next misclassification. In this
       way, categories are only updated when errors occur. This directly models the way that some
       email classifiers are used in practice.

       TOE's  error  rates depend directly on the order in which emails are seen.  A small change
       in ordering, as might happen due to networking delays, can have  a  large  impact  on  the
       number  of  misclassifications.   Consequently,  mailtoe does not give meaningful results,
       unless the sample emails are chosen carefully.  However, as this method is  commonly  used
       by  spam  filters,  it is still worth computing to foster comparisons. Other methods  (see
       mailcross(1),mailfoot(1)) attempt to capture the behaviour  of  classification  errors  in
       other ways.

       To  improve and stabilize the error rate calculation, mailtoe performs the TOE simulations
       several times  on  slightly  reordered  email  streams,  and  averages  the  results.  The
       reorderings  occur  by multiplexing the emails from each category mailbox in random order.
       Thus if there are three categories, the first email classified is chosen randomly from the
       front  of the sample email streams of each type.  The second email is also chosen randomly
       among the three types, from the front of the
        streams after the first email was removed. Simulation stops when all sample  streams  are
       exhausted.

       mailtoe  uses  the  environment  variable MAILTOE_FILTER when executing, which permits the
       simulation of arbitrary filters,  provided  these  satisfy  the  compatibility  conditions
       stated in the ENVIRONMENT section below.

       For  convenience,  mailtoe  implements  a testsuite framework with predefined wrappers for
       several open source classifiers. This permits  the  direct  comparison  of  dbacl(1)  with
       competing classifiers on the same set of email samples. See the USAGE section below.

       During  preparation,  mailtoe builds a subdirectory named mailtoe.d in the current working
       directory.  All needed calculations are performed inside this subdirectory.

EXIT STATUS

       mailtoe returns 0 on success, 1 if a problem occurred.

COMMANDS

       prepare size
              Prepares a subdirectory named mailtoe.d  in  the  current  working  directory,  and
              populates it with empty subdirectories for exactly size subsets.

       add category [ FILE ]...
              Takes  a set of emails from either FILE if specified, or STDIN, and associates them
              with category.  The ordering of emails within FILE  is  preserved,  and  subsequent
              FILEs  are  appended  to  the first in each category.  This command can be repeated
              several times, but should be executed at least once.

       clean  Deletes the directory mailtoe.d and all its contents.

       run    Multiplexes randomly from the email streams added earlier, and relearns  categories
              only when a misclassification occurs. The simulation is repeated size times.

       summarize
              Prints average error rates for the simulations.

       plot [ ps | logscale ]...
              Plots  the  number  of  errors  over  simulation time. The "ps" option, if present,
              writes the plot to a postscript file in the  directory  mailtoe/plots,  instead  of
              being  shown on-screen. The "logscale" option, if present, causes the plot to be on
              the log scale for both ordinates.

       review truecat predcat
              Scans the last run statistics  and  extracts  all  the  messages  which  belong  to
              category  truecat  but  have  been classified into category predcat.  The extracted
              messages are copied to the directory mailtoe.d/review for perusal.

       testsuite list
              Shows a list of available filters/wrapper scripts which can be selected.

       testsuite select [ FILTER ]...
              Prepares the filter(s) named FILTER to be used for simulation. The filter  name  is
              the  name  of a wrapper script located in the directory /usr/share/dbacl/testsuite.
              Each filter has a rigid interface documented below, and the  act  of  selecting  it
              copies  it  to the mailtoe.d/filters directory. Only filters located there are used
              in the simulations.

       testsuite deselect [ FILTER ]...
              Removes the named filter(s) from the directory mailtoe.d/filters so that  they  are
              not used in the simulation.

       testsuite run [ plots ]
              Invokes  every  selected  filter  on  the datasets added previously, and calculates
              misclassification rates. If the "plots" option is present, each  filter  simulation
              is plotted as a postscript file in the directory mailtoe.d/plots.

       testsuite status
              Describes the scheduled simulations.

       testsuite summarize
              Shows  the cross validation results for all filters. Only makes sense after the run
              command.

USAGE

       The normal usage  pattern  is  the  following:  first,  you  should  separate  your  email
       collection  into  several  categories  (manually  or  otherwise).  Each category should be
       associated with one or more folders, but each folder should  not  contain  more  than  one
       category.  Next,  you  should decide how many runs to use, say 10.  The more runs you use,
       the better the predicted error rates. However, more runs take more time.  Now you can type

       % mailtoe prepare 10

       Next, for every category, you must add every folder associated with this category. Suppose
       you  have  three categories named spam, work, and play, which are associated with the mbox
       files spam.mbox, work.mbox, and play.mbox respectively. You would type

       % mailtoe add spam spam.mbox
       % mailtoe add work work.mbox
       % mailtoe add play play.mbox

       You should aim for a similar number of emails in each category, as the random multiplexing
       will  be  unbalanced  otherwise. The ordering of the email messages in each *.mbox file is
       important, and is preserved during each simulation. If you  repeatedly  add  to  the  same
       category,  the  later  mailboxes  will  be  appended  to the first, preserving the implied
       ordering.

       You can now perform as many  TOE  simulations  as  desired.  The  multiplexed  emails  are
       classified  and  learned  one at a time, by executing the command given in the environment
       variable MAILTOE_FILTER. If not set, a default value is used.

       % mailtoe run
       % mailtoe summarize

       The testsuite commands are designed to simplify the above steps and allow comparison of  a
       wide  range  of  email  classifiers,  including but not limited to dbacl.  Classifiers are
       supported through wrapper scripts, which are  located  in  the  /usr/share/dbacl/testsuite
       directory.

       The  first  stage  when using the testsuite is deciding which classifiers to compare.  You
       can view a list of available wrappers by typing:

       % mailtoe testsuite list

       Note that the wrapper scripts  are  NOT  the  actual  email  classifiers,  which  must  be
       installed  separately  by  your system administrator or otherwise.  Once this is done, you
       can select one or more wrappers for the simulation by typing, for example:

       % mailtoe testsuite select dbaclA ifile

       If some of the selected classifiers cannot be found on the system, they are not  selected.
       Note  also  that  some wrappers can have hard-coded category names, e.g. if the classifier
       only supports binary classification. Heed the warning messages.

       It remains only to run the simulation. Beware, this can take a long  time  (several  hours
       depending on the classifier).

       % mailtoe testsuite run
       % mailtoe testsuite summarize

       Once you are all done, you can delete the working files, log files etc. by typing

       % mailtoe clean

SCRIPT INTERFACE

       mailtoe  testsuite  takes care of learning and classifying your prepared email corpora for
       each selected classifier. Since classifiers have widely varying interfaces, this  is  only
       possible  by wrapping those interfaces individually into a standard form which can be used
       by mailtoe testsuite.

       Each wrapper script is a command line tool which accepts a single command followed by zero
       or more optional arguments, in the standard form:

       wrapper command [argument]...

       Each  wrapper  script  also  makes  use  of  STDIN and STDOUT in a well defined way. If no
       behaviour is described, then no output or input should be used.  The possible commands are
       described below:

       filter In this case, a single email is expected on STDIN, and a list of category filenames
              is expected in $2, $3, etc. The script writes the category  name  corresponding  to
              the input email on STDOUT. No trailing newline is required or expected.

       learn  In  this  case,  a  standard  mbox  stream  is  expected on STDIN, while a suitable
              category file name is expected in $2. No output is written to STDOUT.

       clean  In this case, a directory is expected in $2, which is  examined  for  old  database
              information. If any old databases are found, they are purged or reset. No output is
              written to STDOUT.

       describe
              IN this case, a single line of text is written to STDOUT, describing  the  filter's
              functionality.  The  line  should  be  kept  short  to  prevent  line wrapping on a
              terminal.

       bootstrap
              In this case, a directory is expected in $2. The wrapper script  first  checks  for
              the  existence  of its associated classifier, and other prerequisites. If the check
              is successful, then the wrapper is cloned into the supplied directory.  A  courtesy
              notification  should  be given on STDOUT to express success or failure.  It is also
              permissible to give longer descriptions caveats.

       toe    In this case, a list of categories is expected  in  $3,  $4,  etc.  Every  possible
              category must be listed. Preceding this list, the true category is given in $2.

       foot   Used by mailfoot(1).

ENVIRONMENT

       Right  after  loading, mailtoe reads the hidden file .mailtoerc in the $HOME directory, if
       it exists, so this would  be  a  good  place  to  define  custom  values  for  environment
       variables.

       MAILTOE_FILTER
              This variable contains a shell command to be executed repeatedly during the running
              stage.  The command should accept an email message on STDIN and output a  resulting
              category  name.  On the command line, it should also accept first the true category
              name, then a list of all possible category file names.  If the output category does
              not  match the true category, then the relevant categories are assumed to have been
              silently updated/relearned.  If MAILTOE_FILTER is undefined, mailtoe uses a default
              value.

       TEMPDIR
              This  directory  is exported for the benefit of wrapper scripts. Scripts which need
              to create temporary files should place them a the location given in TEMPDIR.

NOTES

       The subdirectory mailtoe.d can grow quite large. It contains a full copy of  the  training
       corpora,  as  well  as learning files for size times all the added categories, and various
       log files.

       While TOE simulations for dbacl(1) can be used to  compare  with  other  classifiers,  TOE
       should  not  be  used  for  real world classifications. This is because, unlike many other
       filters, dbacl(1) learns evidence weights in  a  nonlinear  way,  and  does  not  preserve
       relative weights between tokens, even if those tokens aren't seen in new emails.

WARNING

       Because  the  ordering  of  emails within the added mailboxes matters, the estimated error
       rates are not well defined or even meaningful in an  objective  sense.   However,  if  the
       sample  emails  represent  an  actual  snapshot of a user's incoming email, then the error
       rates are somewhat meaningful. The  simulations  can  then  be  interpreted  as  alternate
       realities where a given classifier would have intercepted the incoming mail.

SOURCE

       The  source  code  for  the  latest  version of this program is available at the following
       locations:

       http://www.lbreyer.com/gpl.html
       http://dbacl.sourceforge.net

AUTHOR

       Laird A. Breyer <laird@lbreyer.com>

SEE ALSO

       bayesol(1) dbacl(1), mailinspect(1), mailcross(1), mailfoot(1), regex(7)