Provided by: libdbix-fulltextsearch-perl_0.73-10_all bug

NAME

       DBIx::FullTextSearch - Indexing documents with MySQL as storage

SYNOPSIS

       DBIx::FullTextSearch uses a MySQL database backend to index files, web documents and
       database fields.  Supports must include, can include, and cannot include words and
       phrases.  Support for boolean (AND/OR) queries, stop words and stemming.

           use DBIx::FullTextSearch;
           use DBI;
           # connect to database (regular DBI)
           my $dbh = DBI->connect('dbi:mysql:database', 'user', 'passwd');

           # create a new stoplist
           my $sl = DBIx::FullTextSearch::StopList->create_default($dbh, 'sl_en', 'English');

           # create a new index with default english stoplist and english stemmer
           my $fts = DBIx::FullTextSearch->create($dbh, 'fts_web_1',
                       frontend => 'string', backend => 'blob',
                       stoplist => 'sl_en', stemmer => 'en-us');
           # or open existing one
           # my $fts = DBIx::FullTextSearch->open($dbh, 'fts_web_1');

           # index documents
           $fts->index_document('krtek', 'krtek leze pod zemi');
           $fts->index_document('jezek', 'Jezek ma ostre bodliny.');

           # search for matches
           my @docs = $fts->contains('foo');
           my @docs = $fts->econtains('+foo', '-Bar');
           my @docs = $fts->search('+foo -Bar');
           my @docs = $fts->search('foo AND (bar OR baz)');

DESCRIPTION

       DBIx::FullTextSearch is a flexible solution for indexing contents of documents.  It uses
       the MySQL database to store the information about words and documents and provides Perl
       interface for indexing new documents, making changes and searching for matches.  For
       DBIx::FullTextSearch, a document is nearly anything -- Perl scalar, file, Web document,
       database field.

       The basic style of interface is shown above. What you need is a MySQL database and a DBI
       with DBD::mysql. Then you create a DBIx::FullTextSearch index -- a set of tables that
       maintain all necessary information. Once created it can be accessed many times, either for
       updating the index (adding documents) or searching.

       DBIx::FullTextSearch uses one basic table to store parameters of the index. Second table
       is used to store the actual information about documents and words, and depending on the
       type of the index (specified during index creation) there may be more tables to store
       additional information (like conversion from external string names (eg. URL's) to internal
       numeric form). For a user, these internal thingies and internal behaviour of the index are
       not important. The important part is the API, the methods to index document and ask
       questions about words in documents. However, certain understanding of how it all works may
       be usefull when you are deciding if this module is for you and what type of index will
       best suit your needs.

       Frontends

       From the user, application point of view, the DBIx::FullTextSearch index stores documents
       that are named in a certain way, allows adding new documents, and provides methods to ask:
       "give me list of names of documents that contain this list of words". The
       DBIx::FullTextSearch index doesn't store the documents itself. Instead, it stores
       information about words in the documents in such a structured way that it makes easy and
       fast to look up what documents contain certain words and return names of the documents.

       DBIx::FullTextSearch provides a couple of predefined frontend classes that specify various
       types of documents (and the way they relate to their names).

       default
           By default, user specifies the integer number of the document and the content (body)
           of the document. The code would for example read

                   $fts->index_document(53, 'zastavujeme vyplaty vkladu');

           and DBIx::FullTextSearch will remember that the document 53 contains three words.
           When looking for all documents containing word (string) vklad, a call

                   my @docs = $fts->contains('vklad*');

           would return numbers of all documents containing words starting with 'vklad', 53 among
           them.

           So here it's user's responsibility to maintain a relation between the document numbers
           and their content, to know that a document 53 is about vklady. Perhaps the documents
           are already stored somewhere and have unique numeric id.

           Note that the numeric id must be no larger than 2^"doc_id_bits".

       string
           Frontend string allows the user to specify the names of the documents as strings,
           instead of numbers. Still the user has to specify both the name of the document and
           the content:

                   $fts->index_document('foobar',
                                   'the quick brown fox jumped over lazy dog!');

           After that,

                   $fts->contains('dog')

           will return 'foobar' as one of the names of documents with word 'dog' in it.

       file
           To index files, use the frontend file. Here the content of the document is clearly the
           content of the file specified by the filename, so in a call to index_document, only
           the name is needed -- the content of the file is read by the DBIx::FullTextSearch
           transparently:

                   $fts->index_document('/usr/doc/FAQ/Linux-FAQ');
                   my @files = $fts->contains('penguin');

       url Web document can be indexed by the frontend url. DBIx::FullTextSearch uses LWP to get
           the document and then parses it normally:

                   $fts->index_document('http://www.perl.com/');

           Note that the HTML tags themselves are indexed along with the text.

       table
           You can have a DBIx::FullTextSearch index that indexes char or blob fields in MySQL
           table. Since MySQL doesn't support triggers, you have to call the "index_document"
           method of DBIx::FullTextSearch any time something changes in the table. So the
           sequence probably will be

                   $dbh->do('insert into the_table (id, data, other_fields)
                           values (?, ?, ?)', {}, $name, $data, $date_or_something);
                   $fts->index_document($name);

           When calling "contains", the id (name) of the record will be returned. If the id in
           the_table is numeric, it's directly used as the internal numeric id, otherwise a
           string's way of converting the id to numeric form is used.

           When creating this index, you'll have to pass it three additionial options,
           "table_name", "column_name", and "column_id_name".  You may use the optional
           column_process option to pre-process data in the specified columns.

       The structure of DBIx::FullTextSearch is very flexible and adding new frontend (what will
       be indexed) is very easy.

       Backends

       While frontend specifies what is indexed and how the user sees the collection of
       documents, backend is about low level database way of actually storing the information in
       the tables. Three types are available:

       blob
           For each word, a blob holding list of all documents containing that word is stored in
           the table, with the count (number of occurencies) associated with each document
           number. That makes it for very compact storage. Since the document names (for example
           URL) are internally converted to numbers, storing and fetching the data is fast.
           However, updating the information is very slow, since information concerning one
           document is spread across all table, without any direct database access.  Updating a
           document (or merely reindexing it) requires update of all blobs, which is slow.

           The list of documents is stored sorted by document name so that fetching an
           information about a document for one word is relatively easy, still a need to update
           (or at least scan) all records in the table makes this storage unsuitable for
           collections of documents that often change.

       column
           The column backend stores a word/document pair in database fields, indexing both, thus
           allowing both fast retrieval and updates -- it's easy to delete all records describing
           one document and insert new ones.  However, the database indexes that have to be
           maintained are large.

           Both blob and column backends only store a count -- number of occurencies of the word
           in the document (and even this can be switched off, yielding just a yes/no information
           about the word's presence).  This allows questions like

                   all documents containing words 'voda' or 'Mattoni'
                           but not a word 'kyselka'

           but you cannot ask whether a document contains a phrase 'kyselka Mattoni' because such
           information is not maintained by these types of backends.

       phrase
           To allow phrase matching, a phrase backend is available. For each word and document
           number it stores a blob of lists of positions of the word in the document. A query

                   $fts->contains('kyselk* Mattoni');

           then only returns those documents (document names/numbers) where word kyselka (or
           kyselky, or so) is just before word Mattoni.

       Mixing frontends and backends

       Any frontend can be used with any backend in one DBIx::FullTextSearch index. You can index
       Web documents with "url" frontend and "phrase" backend to be able to find phrases in the
       documents. And you can use the default, number based document scheme with "blob" backend
       to use the disk space as efficiently as possible -- this is usefull for example for
       mailing-list archives, where we need to index huge number of documents that do not change
       at all.

       Finding optimal combination is very important and may require some analysis of the
       document collection and manipulation, as well as the speed and storage requirements.
       Benchmarking on actual target platform is very useful during the design phase.

METHODS

       The following methods are available on the user side as DBIx::FullTextSearch API.

       create
                   my $fts = DBIx::FullTextSearch->create($dbh, $index_name, %opts);

           The class method "create" creates index of given name (the name of the index is the
           name of its basic parameter table) and all necessary tables, returns an object --
           newly created index. The options that may be specified after the index name define the
           frontend and backend types, storage parameters (how many bits for what values), etc.
           See below for list of create options and discussion of their use.

       open
                   my $fts = DBIx::FullTextSearch->open($dbh, $index_name);

           Opens and returns object, accessing specifies DBIx::FullTextSearch index. Since all
           the index parameters and information are stored in the $index_name table (including
           names of all other needed tables), the database handler and the name of the parameter
           table are the only needed arguments.

       index_document
                   $fts->index_document(45, 'Sleva pri nakupu stribra.');
                   $fts->index_document('http://www.mozilla.org/');
                   $fts->index_document('http://www.mozilla.org/','This is the mozilla web site');

           For the "default" and "string" frontends, two arguments are expected -- the name
           (number or string) of the document and its content. For "file", "url", and "table"
           frontends the content is optional.  Any content that you pass will be appended to the
           content from the file, URL, or database table.

       delete_document
                   $fts->delete_document('http://www.mozilla.org/');

           Removes information about document from the index. Note that for "blob" backend this
           is very time consuming process.

       contains
                   my @docs = $fts->contains('sleva', 'strib*');

           Returns list of names (numbers or strings, depending on the frontend) of documents
           that contain some of specified words.

       econtains
                   my @docs = $fts->contains('foo', '+bar*', '-koo');

           Econtains stands for extended contains and allows words to be prefixed by plus or
           minus signs to specify that the word must or mustn't be present in the document for it
           to match.

       search
            my @docs = $fts->search(qq{+"this is a phrase" -koo +bar foo});
            my @docs = $fts->search("(foo OR baz) AND (bar OR caz)");

           This is a wrapper to econtains which takes a user input string and parses it into can-
           include, must-include, and must-not-include words and phrases.  It also can handle
           boolean (AND/OR) queries.

       contains_hashref, econtains_hashref, search_hashref
           Similar to "contains", "econtains" and "search", only instead of list of document
           names, these methods return a hash reference to a hash where keys are the document
           names and values are the number of occurencies of the words.

       drop
           Removes all tables associated with the index, including the base parameter table.
           Effectivelly destroying the index form the database.

            $fts->drop;

       empty
           Emptys the index so you can reindex the data.

            $fts->empty;

INDEX OPTIONS

       Here we list the options that may be passed to "create" method.  These allow to specify
       the style and storage parameters in great detail.

       backend
           The backend type, default "blob", possible values "blob", "column" and "phrase" (see
           above for explanation).

       frontend
           The frontend type. The "default" frontend requires the user to specify numeric id of
           the document together with the content of the document, other possible values are
           "string", "file" and "url" (see above for more info).

       word_length
           Maximum length of words that may be indexed, default 30.

       data_table
           Name of the table where the actual data about word/document relation is stored. By
           default, the name of the index (of the base table) with _data suffix is used.

       name_length
           Any frontend that uses strings as names of documents needs to maintain a conversion
           table from these names to internal integer ids. This value specifies maximum length of
           these string names (URLs, file names, ...).

       blob_direct_fetch
           Only for "blob" backend. When looking for information about specific document in the
           list stored in the blob, the blob backend uses division of interval to find the
           correct place in the blob. When the interval gets equal or shorter that this value,
           all values are fetched from the database and the final search is done in Perl code
           sequentially.

       word_id_bits
           With "column" or "phase" backends, DBIx::FullTextSearch maintains a numeric id for
           each word to optimize the space requirements. The word_id_bits parameter specifies the
           number of bits to reserve for this conversion and thus effectively limits number of
           distinct words that may be indexed. The default is 16 bits and possible values are 8,
           16, 24 or 32 bits.

       word_id_table
           Name of the table that holds conversion from words to their numeric id (for "column"
           and "phrase" backends). By default is the name of the index with _words suffix.

       doc_id_bits
           A number of bits to hold a numeric id of the document (that is either provided by the
           user (with "default" frontend) or generated by the module to accomplish the conversion
           from the string name of the document). This value limits the maximum number of
           documents to hold. The default is 16 bits and possible values are 8, 16 and 32 bits
           for "blob" backend and 8, 16, 24 and 32 bits for "column" and "phrase" backends.

       doc_id_table
           Name of the table that holds conversion from string names of documents to their
           numeric id, by default the name of the index with _docid suffix.

       count_bits
           Number of bits reserved for storing number of occurencies of each word in the
           document. The default is 8 and possible values are the same as with doc_id_bits.

       position_bits
           With "phrase backend", DBIx::FullTextSearch stores positions of each word of the
           documents. This value specifies how much space should be reserved for this purpose.
           The default is 32 bits and possible values are 8, 16 or 32 bits. This value limits the
           maximum number of words of each document that can be stored.

       index_splitter
           DBIx::FullTextSearch allows the user to provide any Perl code that will be used to
           split the content of the document to words when indexing documents.  The code will be
           evalled inside of the DBIx::FullTextSearch code. The default is

                   /(\w{2,$word_length})/g

           and shows that the input is stored in the variable $data and the code may access any
           other variable available in the perl_and_index_data_* methods (see source), especially
           $word_length to get the maximum length of words and $backend to get the backend
           object.

           The default value also shows that by default, the minimum length of words indexed is
           2.

       search_splitter
           This is similar to the "index_splitter" method, except that it is used in the
           "contains_hashref" method when searching for documents instead of when indexing
           documents.  The default is

                  /(\w{2,$word_length}\*?)/g

           Which, unlike the default "index_splitter", allows for the wild card character (*).

       filter
           The output words of splitter (and also any parameter of (e)contains* methods) are send
           to filter that may do further processing. Filter is again a Perl code, the default is

                   map { lc $_ }

           showing that the filter operates on input list and by default does conversion to
           lowercase (yielding case insensitive index).

       init_env
           Because user defined splitter or filter may depend on other things that it is
           reasonable to set before the actual procession of words, you can use yet another Perl
           hook to set things up. The default is no initialization hook.

       stoplist
           This is the name of a DBIx::FullTextSearch::StopList object that is used for stop
           words.

       stemmer
           If this option is set, then word stemming will be enabled in the indexing and
           searching.

           The value is the name of a Lingua::Stem recognized locale.  Currently, 'en', 'en-us'
           and 'en-uk' are the only recognized locales.  All locale identifiers are converted to
           lowercase.

       table_name
           For "table" frontend; this is the name of the table that will be indexed.

       column_name
           For "table" frontend; this is a reference to an array of columns in the "table_name"
           that contains the documents -- data to be indexed. It can also have a form
           table.column that will be used if the "table_name" option is not specified.

       column_id_name
           For "table" frontend; this is the name of the field in "table_name" that holds names
           (ids) of the records. If not specified, a field that has primary key on it is used. If
           this field is numeric, it's values are directly used as identifiers, otherwise a
           conversion to numeric values is made.

NOTES

       To handle internationalization, it may help to use the following in your code (for example
       Spanish in Chile):

         use POSIX;
         my $loc = POSIX::setlocale( &POSIX::LC_ALL, "es_CL" );

       I haven't tested this, so I would be interested in hearing whether this works.

ERROR HANDLING

       The create and open methods return the DBIx::FullTextSearch object on success, upon
       failure they return undef and set error message in $DBIx::FullTextSearch::errstr variable.

       All other methods return reasonable (documented above) value on success, failure is
       signalized by unreasonable (typically undef or null) return value; the error message may
       then be retrieved by "$fts->errstr" method call.

VERSION

       This documentation describes DBIx::FullTextSearch module version 0.73.

BUGS

       Error handling needs more polishing.

       We do not check if the stored values are larger that specified by the *_bits parameters.

       No CGI administration tool at the moment.

       No scoring algorithm implemented.

DEVELOPMENT

       These modules are under active development.  If you would like to contribute, please
       e-mail tjmather@maxmind.com

       There are two mailing lists for this module, one for users, and another for developers.
       To subscribe, visit http://sourceforge.net/mail/?group_id=8645

AUTHOR

       (Original) Jan Pazdziora, adelton@fi.muni.cz, http://www.fi.muni.cz/~adelton/ at Faculty
       of Informatics, Masaryk University in Brno, Czech Republic

       (Current Maintainer) T.J. Mather, tjmather@maxmind.com,
       http://www.maxmind.com/app/opensourceservices Princeton, NJ USA

       Paid support is available from directly from the maintainers of this package.  Please see
       <http://www.maxmind.com/app/opensourceservices> for more details.

CREDITS

       Fixes, Bug Reports, Docs have been generously provided by:

         Vladimir Bogdanov
         Ade Olonoh
         Kate Pugh
         Sven Paulus
         Andrew Turner
         Tom Bille
         Joern Reder
         Tarik Alkasab
         Dan Collis Puro
         Tony Bowden
         Mario Minati
         Miroslav Suchy
         Stephen Patterson
         Joern Reder
         Hans Poo

       Of course, big thanks to Jan Pazdziora, the original author of this module.  Especially
       for providing a clean, modular code base!

COPYRIGHT

       All rights reserved. This package is free software; you can redistribute it and/or modify
       it under the same terms as Perl itself.

SEE ALSO

       DBIx::FullTextSearch::StopWord, Class::DBI::mysql::FullTextSearch

OTHER PRODUCTS and why I've written this module

       I'm aware of DBIx::TextIndex and DBIx::KwIndex modules and about UdmSearch utility, and
       about htdig and glimpse on the non-database side of the world.

       To me, using a database gives reasonable maintenance benefits. With products that use
       their own files to store the information (even if the storage algorithms are efficient and
       well thought of), you always struggle with permissions on files and directories for
       various users, with files that somebody accidently deleted or mungled, and making the
       index available remotely is not trivial.

       That's why I've wanted a module that will use a database as a storage backend. With MySQL,
       you get remote access and access control for free, and on many web servers MySQL is part
       of the standard equipment. So using it for text indexes seemed natural.

       However, existing DBIx::TextIndex and UdmSearch are too narrow-aimed to me. The first only
       supports indexing of data that is stored in the database, but you may not always want or
       need to store the documents in the database as well. The UdmSearch on the other hand is
       only for web documents, making it unsuitable for indexing mailing-list archives or local
       data.

       I believe that DBIx::FullTextSearch is reasonably flexible and still very efficient. It
       doesn't enforce its own idea of what is good for you -- the number of options is big and
       you can always extend the module with your own backend of frontend if you feel that those
       provided are not sufficient. Or you can extend existing by adding one or two parameters
       that will add new features. Of course, patches are always welcome.  DBIx::FullTextSearch
       is a tool that can be deployed in many projects. It's not a complete environment since
       different people have different needs. On the other hand, the methods that it provides
       make it easy to build a complete solution on top of this in very short course of time.