Provided by: spamoracle_1.4-14build3_amd64 bug

NAME

       spamoracle - a spam classification tool

SYNOPSIS

       spamoracle [-config conf] [-f database] mark [ mailbox ...  ]

       spamoracle [-config conf] [-f database] add [-v] -spam spambox ...  -good goodbox ...

       spamoracle [-config conf] [-f database] test [-min prob] [-max prob] [ mailbox ...  ]

       spamoracle [-config conf] [-f database] stat [ mailbox ...  ]

       spamoracle [-config conf] [-f database] list regexp ...

       spamoracle [-config conf] [-f database] backup > backupfile

       spamoracle [-config conf] [-f database] restore < backupfile

       spamoracle [-config conf] [-f database] words [ mailbox ...  ]

DESCRIPTION

       SpamOracle  is  a  tool  to  help detect and filter away "spam" (unsolicited commercial e-
       mail).  It proceeds by statistical analysis of  the  words  that  appear  in  the  e-mail,
       comparing  the  frequencies  of  words with those found in a user-provided corpus of known
       spam and known legitimate  e-mail.   The  classification  algorithm  is  based  on  Bayes'
       formula,    and    is   described   in   Paul   Graham's   paper,   A   plan   for   spam,
       http://www.paulgraham.com/spam.html.

       This program is designed to work in conjunction  with  procmail(1).   The  result  of  the
       analysis is output as an additional message header X-Spam: followed by yes, no or unknown,
       plus additional details.  A procmail rule can then test this X-Spam:  header  and  deliver
       the e-mail to the appropriate mailbox.

       In  addition,  SpamOracle  also analyses MIME attachments, extracting relevant information
       such as MIME type, character encoding and attached file name, and summarizing them  in  an
       additional   X-Attachments:  header.   This  allows  procmail  to  easily  reject  e-mails
       containing suspicious attachments, e.g. Windows executables which often indicate a virus.

REQUIREMENTS AND LIMITATIONS

       To use SpamOracle, your mail must be delivered to a Unix machine on which you have a shell
       account.   This  machine  must  have procmail(1) (see http://www.procmail.org/) installed.
       Your ~/.forward file must be set up to run all incoming e-mail  through  procmail(1).   If
       your  mail  server  supports  the  POP or IMAP protocols, you can also use fetchmail(1) to
       fetch your mail from the server and have it delivered to your local machine.

       To provide the corpus of messages from which SpamOracle "learns", an archive of about 1000
       of  your e-mails is needed.  The archive must be manually or semi-automatically split into
       known spams and known good messages.  Mis-classified messages in the  corpus  (e.g.  spams
       mistakenly   stored  among  the  good  messages)  will  decrease  the  efficiency  of  the
       classification.  The archive must be in Unix mailbox format, or in "one message per  file"
       format (a la MH).  Other formats, such as Emacs' Babyl, are not supported.

       The  notion  of  "word"  used by SpamOracle is slanted towards Western European languages,
       i.e. the ISO Latin-1 and Latin-9 character  sets.   Preliminary  support  for  JIS-encoded
       Japanese  can  be  selected at compile-time.  SpamOracle will not work well if you receive
       many legitimate e-mails written in other character sets, such as Chinese or Korean sets.

INITIALIZATION

       To build the database of word frequencies from the corpus, do:

              rm ~/.spamoracle.db
              spamoracle add -v -good goodmails -spam spammails

       By default, the database is stored in the file  .spamoracle.db  in  your  home  directory.
       This  can be overriden with the -f option: spamoracle -f mydatabase add ...  The -v option
       prints progress information during the processing of the corpus.

       This assumes that the good, non-spam messages from the  corpus  are  stored  in  the  file
       goodmails,  and  the known spam messages in the file spammails.  You can also fetch corpus
       messages from several files, and/or process them via several invocations of SpamOracle:

              spamoracle add -good goodmails1 ... goodmailsN
              spamoracle add -spam spammails1 ... spammailsP

TESTING THE DATABASE

       To check that the  database  was  built  correctly,  and  familiarize  yourself  with  the
       statistical analysis performed by SpamOracle, invoke the "test" mode on the mailboxes that
       you just used for building the corpus:

              spamoracle test goodmails | more
              spamoracle test spammails | more

       For each message in the given mailboxes, you'll see a summary like this:

               From: bbo <midhack@ureach.com>
               Subject: Check This Out
               Score: 1.00 -- 15
               Details: refid:98 $$$$:98 surfing:98 asp:95 click:93 cable:92
                 instantly:90 https:88 internet:87 www:86 U4:85 isn't:14 month:81
                 com:75 surf:75
              Attachments: cset="GB2312" type="application/octet-stream"
                 name="Guangwen4.zip"
               File: inbox/314

       The first two lines are just the From: and Subject: fields of the original message.

       The Score: line summarizes the result of the analysis.  The first number (between 0.0  and
       1.0) is the probability that the message is actually spam --- or, equivalently, the degree
       of similarity of the message with the spam messages in the corpus.  The second number  (an
       integer  between  0  and  15)  is  the number of "interesting" words found in the message.
       "Interesting" words are those that occur at least 5 times in the corpus.  In the  example,
       we  have  15  interesting  words (the maximum) and a score of 1.00, indicating a spam with
       high certainty.

       The Details: line provides an explanation of the score.  It lists the 15 most  interesting
       words  found  in  the  message,  that  is,  the  15 interesting words whose probability of
       denoting a spam is farthest away from the neutral  0.5.   Each  word  is  given  with  its
       individual score, written as a percentage (between 01 and 99) rather than as a probability
       so as to save space.  Here, we see a number of very  "spammish"  words  such  as  $$$$  or
       click,  with  probability  0.98  and 0.93 respectively, and a few "innocent" words such as
       isn't (probability 0.14).  The U4 word with probability 0.85  is  actually  a  pseudo-word
       representing a 4-letter word all in uppercase -- something spammers are fond of.

       The Attachments: line summarizes some information about MIME attachments for this message.
       Here, we have one attachment of type application/octect-stream, file  name  Guangwen4.zip,
       and character set GB2312 (an encoding for Chinese).

       The File: line shows the file that is being tested.

       Normally,  when  running spamoracle test goodmails, most messages should come out with low
       score (0.2 or less), and when running spamoracle test spammails, most messages should come
       out  with  a  high  score (0.8 or more).  If not, your corpus isn't very good, or not well
       classified into spam and non-spam.  To quickly  see  the  outliers,  you  can  reduce  the
       interval of scores for which message summaries are displayed, as follows:

              spamoracle test -min 0.2 goodmails | more
                     # Shows only good mails with score >= 0.2
              spamoracle test -max 0.8 spammails | more
                     # Shows only spam mails with score <= 0.8

       Now,  for a more challenging test, take a mailbox that contains unfiltered e-mails, i.e. a
       mixture of spam and legitimate e-mails, and run it through SpamOracle:

              spamoracle test mymailbox | less

       Marvel at how well the oracle recognizes spam from the rest!  If  the  result  isn't  that
       marvelous to you, keep in mind that certain spams are just too short to be recognized (not
       enough significant words).   Also,  perhaps  your  corpus  was  too  small,  or  not  well
       categorized...

MARKING AND FILTERING INCOMING E-MAIL

       Once  the database is built, you're ready to run incoming e-mails through SpamOracle.  The
       command spamoracle mark reads one e-mail from standard input, and copies  it  to  standard
       output, with two headers inserted: X-Spam: and X-Attachments:.  The X-Spam: header has one
       the following formats:

          X-Spam: yes; score; details

       or

          X-Spam: no; score; details

       or

          X-Spam: unknown; score; details

       The score and details are as described for spamoracle test.

       The yes/no/unknown tag synthesizes the results of the analysis: yes means that  the  score
       is  >=  0.8 and at least 5 interesting words were found; no means that the score is <= 0.2
       and at least 5 interesting words were found; unknown is returned otherwise.   The  unknown
       case  generally  occurs  for  very short messages, where not enough interesting words were
       found.

       The X-Attachments: header contains the same information  as  the  Attachments:  output  of
       spamoracle test, that is, a summary of the message attachments.

       To  process automatically your incoming e-mail through SpamOracle and act upon the results
       of the analysis, just insert the following "recipes" in the file ~/.procmailrc:

              :0fw
              | /usr/local/bin/spamoracle mark

              :0
              * ^X-Spam: yes;
              spambox

       What these cryptic commands mean is:

       - Run every mail through the spamoracle mark command.  (If spamoracle wasn't installed  in
       /usr/local/bin,  adjust  the path as necessary.)  This adds two headers to the message: X-
       Spam: and X-Attachments:, describing the results of the spam analysis and  the  attachment
       analysis.

       - If we have an X-Spam: yes header, deliver the message to the file spambox rather than to
       your regular mailbox.  Presumably, you'll read spambox once in a  while,  but  less  often
       than  your  regular  mailbox.   Daring  users can put /dev/null instead of spambox to just
       throw away the message, but please don't do that until you've used SpamOracle for a  while
       and  are  happy with the results.  SpamOracle's false positive rate (i.e. legitimate mails
       classified as spam) is low (0.1%) but not  null.   So,  better  save  the  presumed  spams
       somewhere, and scan them quickly from time to time.

       If  you'd  like to enjoy a bit of attachment-based filtering, here are some procmail rules
       for that:

              :0
              * ^X-Attachments:.*name=".*\.(pif|scr|exe|bat|com)"
              spambox

              :0
              * ^X-Attachments:.*type="audio/(x-wav|x-midi)
              spambox

              :0
              * ^(Content-type:.*|X-Attachments:.*cset="|^Subject:.*=\?)(ks_c|gb2312|iso-2|euc-|big5|windows-1251)
              spambox

       The first rule treats as spam every mail that has  a  Windows  executable  as  attachment.
       These mails are typically sent by viruses.  The second rule does the same with attachments
       of type x-wav or x-midi.  I never normally receive music by e-mail, however  some  popular
       e-mail  viruses  seem fond of these attachment types.  The third rule treats as spam every
       mail that uses  character  encodings  corresponding  to  Korean,  Chinese,  Japanese,  and
       Cyrillic.

UPDATING THE DATABASE

       At  any time, you can add more known spams or known legitimate messages to the database by
       using the spamoracle add command.

       For instance, if you find a spam message that was not classified as such, run  it  through
       spamoracle  add -spam, so that SpamOracle can learn from its mistake.  (Without additional
       arguments, this command will read a single message from standard input and  record  it  as
       spam.)  Under mutt(1) for instance, just highlight the spam message and type

              |spamoracle add -spam

       Similarly,  if  you find a legitimate message while checking your spam box, run it through
       spamoracle add -good.

       Another option is to collect more known spams or more known  good  messages  into  mailbox
       files,  and once in a while do spamoracle add -good new_good_mails or spamoracle add -spam
       new_spam_mails.

QUERYING THE DATABASE

       For your edification and entertainment, the contents of the database  can  be  queried  by
       regular  expressions.   The spamoracle list regexp command lists all words in the database
       that match regexp  (an  Emacs-style  regular  expression),  along  with  their  number  of
       occurrences in spam mail and in good mail.  For instance:

              spamoracle list '.*' # show all words -- big list!
              spamoracle list 'sex.*'
              spamoracle list 'linux.*'

DATABASE BACKUPS

       The  database used by SpamOracle is stored in a compact, binary format that is not humanly
       readable.  Moreover, this format is subject to change in later versions of SpamOracle.  To
       facilitate  backups  and  upgrades,  the  database  contents  can also be manipulated in a
       portable, text format.

       The spamoracle backup command dumps the contents of the database to standard output, in  a
       textual, portable format.

       The  spamoracle  restore  command  reads  such a dump from standard input and rebuilds the
       database with this data.

       The recommended procedure for upgrading to a newer version of SpamOracle is:

              # Before the upgrade:
              spamoracle backup > backupfile
              # Upgrade SpamOracle
              # Restore the database
              spamoracle restore < backupfile

CONFIGURING FILTERING PARAMETERS

       Many of the parameters  that  govern  message  classification  can  be  configured  via  a
       configuration  file.  By default, the configuration is read from the file .spamoracle.conf
       in the user's home directory.  A different configuration file  can  be  specified  on  the
       command line using the -config option: spamoracle -config myconfigfile ...

       The list of configurable parameters and the format of the configuration file are described
       in spamoracle.conf(5).

       All parameters have reasonable defaults, but  you  can  try  to  improve  the  quality  of
       classification  further  by  tweaking  them.  To determine the impact of your changes, use
       either the test or stat commands to spamoracle.  The spamoracle stat command prints a one-
       line  summary of how many spam, non-spam, and unknown messages were found in the mailboxes
       given as arguments.

TECHNICAL DETAILS

       SpamOracle's notion of "word" is any run of 3 to 12 of the following characters:  letters,
       single quotes, and dashes (-).  If support for non-English european languages was compiled
       in, word characters also include the  relevant  accented  letters  for  the  languages  in
       question.   All  words  are  mapped  to  lowercase, and accented letters are mapped to the
       corresponding non-accented letters.

       A run of 3 to 12 of the following  characters  also  constitutes  a  word:  digits,  dots,
       commas, and dollar, Euro and percent signs.

       In  addition,  a run of three or more uppercase letters generates a pseudo-word Un where n
       is the length of the run.  Similarly, a run of three or more non-ASCII characters (code >=
       128) generates a pseudo-word Wn where n is the length of the run.

       For instance, the following text:

              SUMMER in English is written "ete" in French

       is  processed  into  the following words, assuming French support was selected at compile-
       time:

              U5 summer english written ete french W3

       and if French support was not selected:

              U5 summer english written french W3

       To see the words that are extracted from a message, issue the  spamoracle  words  command.
       It  reads  either  a  single message from standard input, or all messages from the mailbox
       files given as arguments, decomposes the messages into words and prints the words.

RANDOM NOTES

       The database file can be compressed with gzip(1) to save disk space,  at  the  expense  of
       slower  spamoracle  operations.  If the database file specified with the -f option has the
       extension .gz, spamoracle will automatically uncompress it on start-up, and re-compress it
       after updates.

       If your mail is stored in MH format, you may run into "command line too long" errors while
       trying to process a lot of small files with the spamoracle add command, e.g. when doing
       spamoracle add -good archives/*/* -spam spam/*
       Instead, do something like:
       find archives -type f -print | xargs spamoracle add -good
       find spam -type f -print | xargs spamoracle add -spam

AUTHOR

       Xavier Leroy <Xavier.Leroy@inria.fr>

SEE ALSO

       spamoracle.conf(5); procmail(1); fetchmail(1)

       http://cristal.inria.fr/~xleroy/software/ (SpamOracle distribution site)

       http://www.paulgraham.com/spam.html (Paul Graham's seminal paper)

                                                                                    SPAMORACLE(1)