Ubuntu Manpage: SWISH-CONFIG - Configuration File Directives

Provided by: swish-e_2.4.7-4build1_amd64

NAME

       SWISH-CONFIG - Configuration File Directives

OVERVIEW

       This document lists the available configuration directives available in Swish-e.

CONFIGURATION FILE

What files Swish-e indexes and how they are indexed, and where the index is written can be
controlled by a configuration file.

The configuration file is a text file composed of comments, blank lines, and configuration
directives. The order of the directives is not important. Some directives may be used
more than once in the configuration file, while others can only be used once (e.g.
additional directives will overwrite preceding directives). Case of the directive is not
important -- you may use upper, lower, or mixed case.

Comments are any line that begin with a "#".

# This is a comment

As of 2.4.3 lines may be continued by placing a backslas as the last character on the
line:

IgnoreWords \
am \
the \
foo

Directives may take more than one parameter. Enclose single parameters that include
whitespace in quotes (single or double). Inside of quotes the backslash escapes the next
character.

ReplaceRules append "foo bar" <- define "foo bar" as a single parameter

If you need to include a quote character in the value either use a backslash to escape it,
or enclose it in quotes of the other type.

Backslashes also have special meaning in regular expressions.

FileFilterMatch pdftotext "'%p' -" /\.pdf$/

This says that the dot is a real dot (instead of matching any character). If you place
the regular expression in quotes then you must use double-backslashes.

FileFilterMatch pdftotext "'%p' -" "/\\.pdf$/"

Swish-e will convert the double backslash into a single backslash before passing the
parameter to the regular expression compiler.

Commented example configuration files are included in the conf directory of the Swish-e
distribution.

Some command line arguments can override directives specified in the configuration file.
Please see also the SWISH-RUN for instructions on running Swish-e, and the SWISH-SEARCH
page for information and examples on how to search your index.

The configuration file is specified to Swish-e by the "-c" switch. For example,

swish-e -c myconfig.conf

You may also split your directives up into different configuration files. This allows you
to have a master configuration file used for many different indexes, and smaller
configuration files for each separate index. You can specify the different configuration
files when running from the command line with the "-c" switch (see SWISH-RUN), or you may
include other Configuration file with the IncludeConfigFile directive below.

Typically, in a configuration file the directives are grouped together in some logical
order -- that is, directives that control the source of the documents would be grouped
together first, and directives that control how each document is filtered or its words
index in another group of directives. (The directives listed below are grouped in this
order).

The configuration file directives are listed below in these groups:

• "Administrative Headers Directives" -- You may add administrative information to the
header of the index file.

• "Document Source Directives" -- Directives for selecting the source documents and the
location of the index file.

• "Document Contents Directives" -- Directives that control how a document content is
indexed.

• "Directives for the File Access method only" -- These directives are only applicable
to the File Access indexing method.

• "Directives for the HTTP Access Method Only" -- Likewise, these only apply to the HTTP
Access method.

• "Directives for the prog Access Method Only" -- These only apply to the prog Access
method.

• "Document Filter Directives" -- This is a special section that describes using
document filters with Swish-e.

Alphabetical Listing of Directives

• AbsoluteLinks [yes⎪NO]

• BeginCharacters *string of characters*

• BumpPositionCounterCharacters *string*

• Buzzwords [*list of buzzwords*⎪File: path]

• CompressPositions [yes⎪NO]

• ConvertHTMLEntities [YES⎪no]

• DefaultContents [TXT⎪HTML⎪XML⎪TXT2⎪HTML2⎪XML2⎪TXT*⎪HTML*⎪XML*]

• Delay *seconds*

• DontBumpPositionOnEndTags *list of names*

• DontBumpPositionOnStartTags *list of names*

• EnableAltSearchSyntax [yes⎪NO]

• EndCharacters *string of characters*

• EquivalentServer *server alias*

• ExtractPath *metaname* [replace⎪remove⎪prepend⎪append⎪regex]

• FileFilter *suffix* *program* [options]

• FileFilterMatch *program* *options* *regex* [*regex* ...]

• FileInfoCompression [yes⎪NO]

• FileMatch [contains⎪is⎪regex] *regular expression*

• FileRules [contains⎪is⎪regex] *regular expression*

• FuzzyIndexingMode [NONE⎪Stemming⎪Soundex⎪Metaphone⎪DoubleMetaphone]

• FollowSymLinks [yes⎪NO]

• HTMLLinksMetaName *metaname*

• IgnoreFirstChar *string of characters*

• IgnoreLastChar *string of characters*

• IgnoreLimit *integer integer*

• IgnoreMetaTags *list of names*

• IgnoreNumberChars *list of characters*

• IgnoreTotalWordCountWhenRanking [YES⎪no]

• IgnoreWords [*list of stop words*⎪File: path]

• ImageLinksMetaName *metaname*

• IncludeConfigFile

• IndexAdmin *text*

• IndexAltTagMetaName *tagname*⎪as-text

• IndexComments [yes⎪NO]

• IndexContents [TXT⎪HTML⎪XML⎪TXT2⎪HTML2⎪XML2⎪TXT*⎪HTML*⎪XML*] *file extensions*

• IndexDescription *text*

• IndexDir [URL⎪directories or files]

• IndexFile *path*

• IndexName *text*

• IndexOnly *list of file suffixes*

• IndexPointer *text*

• IndexReport [0⎪1⎪2⎪3]

• MaxDepth *integer*

• MaxWordLimit *integer*

• MetaNameAlias *meta name* *list of aliases*

• MetaNames *list of names*

• MinWordLimit *integer*

• NoContents *list of file suffixes*

• obeyRobotsNoIndex [yes⎪NO]

• ParserWarnLevel [0⎪1⎪2⎪3]

• PreSortedIndex *list of property names*

• PropCompressionLevel [0-9]

• PropertyNameAlias *property name* *list of aliases*

• PropertyNames *list of meta names*

• PropertyNamesCompareCase *list of meta names*

• PropertyNamesIgnoreCase *list of meta names*

• PropertyNamesNoStripChars *list of meta names*

• PropertyNamesDate *list of meta names*

• PropertyNamesNumeric *list of meta names*

• PropertyNamesMaxLength integer *list of meta names*

• PropertyNamesSortKeyLength integer *list of meta names*

• ReplaceRules [replace⎪remove⎪prepend⎪append⎪regex]

• ResultExtFormatName name -x format string

• SpiderDirectory *path*

• StoreDescription [XML <tag>⎪HTML <meta>⎪TXT size]

• "SwishProgParameters *list of parameters*

• SwishSearchDefaultRule [<AND-WORD>⎪<or-word>]

• TmpDir *path*

• TranslateCharacters [*string1 string2*⎪:ascii7:]

• TruncateDocSize *number of characters*

• UndefinedMetaTags [error⎪ignore⎪INDEX⎪auto]

• UndefinedXMLAttributes [DISABLE⎪error⎪ignore⎪index⎪auto]

• UseStemming [yes⎪NO]

• UseSoundex [yes⎪NO]

• UseWords [*list of words*⎪File: path]

• WordCharacters *string of characters*

• XMLClassAttributes *list of XML attribute names*

Directives that Control Swish

These configuration directives control the general behavior of Swish-e.

IncludeConfigFile *path to config file*
This directive can be used to include configuration directives located in another
file.

IncludeConfigFile /usr/local/swish/conf/site_config.config

IndexReport [0⎪1⎪2⎪3]
This is how detailed you want reporting while indexing. You can specify numbers 0 to
3. 0 is totally silent, 3 is the most verbose. The default is 1.

This may be overridden from the command line via the "-v" switch (see SWISH-RUN).

ParserWarnLevel [0⎪1⎪2⎪3]
Sets the error level when using the libxml2 parser for XML and HTML. libxml2 will
point out structural errors in your documents.

0 = no report
1 = fatal errors
2 = errors
3 = warnings

Currently (as of 2.4.4 - early 2005) libxml2 only reports errors at level 2. The
default as of 2.4.4 is "2" which should report any errors that might indicate a
problem parsing a document.

The exception to this is UTF-8 to Latin-1 conversion errors are reported at level 3
(changed from 1 in 2.4.4). Although these errors indicate a problem indexing text,
they are only reported at level 3 because they can be very common.

It is recommended that you index at ParserWarnLevel 3 when first starting out to see
what errors and warnings are reported. Then reduce the level when you understand what
documents are causing parsing problems and why.

IndexFile *path*
Index file specifies the location of the generated index file. If not specified,
Swish-e will create the file index.swish-e in the current directory.

IndexFile /usr/local/swish/site.index

obeyRobotsNoIndex [yes⎪NO]
When enabled, Swish-e will not index any HTML file that contains:

The default is to ignore these meta tags and index the document. This tag is
described at http://www.robotstxt.org/wc/exclusion.html.

Note: This feature is only available with the libxml2 HTML parser.

Also, if you are using the libxml2 parser (HTML2 and XML2) then you can use the
following comments in your documents to prevent indexing:

and/or these may be used also:

For example, these are very helpful to prevent indexing of common headers, footers,
and menus.

NOTE: This following items are currently not available. These items require Swish-e to
parse the configuration file while searching.

EnableAltSearchSyntax [yes⎪NO]
NOTE: This following item is currently not available.

Enable alternate search syntax. Allows the usage of a basic "Altavista(c)",
"Lycos(c)", etc. like search syntax. This means a search query can contain "+" and
"-" as syntax parameter.

Example:

swish-e -w "+word1 +word2 -word3 word4 word5"
"+" = following word has to be in all found documents
"-" = following word may not be in any document found
" " = following word will be searched in documents

SwishSearhOperators <and-word> <or-word> <not-word>
NOTE: This following item is currently not available.

Using this config directive you can change the boolean search operators of Swish-e,
e.g. to adapt these to your language. The default is: AND OR NOT

Example (german):

SwishSearchOperators UND ODER NICHT

SwishSearchDefaultRule [<AND-WORD>⎪<or-word>]
NOTE: This following item is currently not available.

"SwishSearchDefaultRule" defines the default Boolean operator to use if none is
specified between words or phrases. The default is "AND".

The word you specify must match one of the available "SwishSearchOperators".

Example:

SwishSearchOperators UND ODER NICHT
# Make it act like a web search engine
SwishSearchDefaultRule ODER

ResultExtFormatName name -x format string
NOTE: This following item is currently not available.

The output of Swish-e can be defined by specifying a format string with the "-x"
command line argument. Using "ResultExtFormatName" you can assign a predefined format
string to a name.

Examples:

ResultExtFormatName moreinfo "%c⎪%r⎪%t⎪%p⎪<author>⎪<publishyear>\n"

Then when searching you can specify the format string's name

swish-e ... -x moreinfo ...

See the "-x" switch in SWISH-RUN for more information about output formats.

Administrative Headers Directives

Swish-e stores configuration information in the header of the index file. This
information can be retrieved while searching or by functions in the Swish-e C library.
There are a number of fields available for your own use. None of these fields are
required:

IndexName *text*
IndexDescription *text*
IndexPointer *text*
IndexAdmin *text*
These variables specify information that goes into index files to help users and
administrators. IndexName should be the name of your index, like a book title.
IndexDescription is a short description of the index or a URL pointing to a more full
description. IndexPointer should be a pointer to the original information, most
likely a URL. IndexAdmin should be the name of the index maintainer and can include
name and email information. These values should not be more than 70 or so characters
and should be contained in quotes. Note that the automatically generated date in
index files is in D/M/Y and 24-hour format.

Examples:

IndexName "Linux Documentation"
IndexDescription "This is an index of /usr/doc on our Linux machine."
IndexPointer http://localhost/swish/linux/index.html
IndexAdmin webmaster

Document Source Directives

These directives control what documents are indexed and how they are accessed. See also
Directives for the File Access method only and Directives for the HTTP Access Method Only
for directives that are specific to those access methods.

IndexDir [directories or files⎪URL⎪external program]
IndexDir defines the source of the documents for Swish-e. Swish-e currently supports
three file access methods: File system, HTTP (also called spidering), and prog for
reading files from an external program.

The "-S" command line argument is used to select the file access method.

swish-e -c swish.config -S fs - file system
swish-e -c swish.config -S http - internal http spider
swish-e -c swish.config -S prog - external program of any type

For the fs method of access IndexDir is a space-separated list of files and
directories to index. Use a forward slash as the path separator in MS Windows.

For the http method the IndexDir setting is a list of space-separated URLs.

For the prog method the IndexDir setting is a list of space-separated programs to run
(which generate documents for swish to index).

You may specify more than one IndexDir directive.

Any sub-directories of any listed directory will also be indexed.

Note: While processing directories, Swish-e will ignore any files or directories that
begin with a dot ("."). You may index files or directories that begin with a dot by
specifying their name with "IndexDir" or "-i".

Examples:

# Index this directory an any subdirectories
IndexDir /usr/local/home/http

# Index the docs directory in current directory
IndexDir ./docs

# Index these files in the current directory
IndexDir ./index.html ./page1.html ./page2.html
# and index this directory, too
IndexDir ../public_html

For the HTTP method of access specify the URL's from which you want the spidering to
begin.

Example:

IndexDir http://www.my-site.com/index.html
IndexDir http://localhost/index.html

Obviously, using the HTTP method to index is much slower than indexing local files.
Be well aware that some sites do not appreciate spidering and may block your IP
address. You may wish to contact the remote site before spidering their web site.
More information about spidering can be found in Directives for the HTTP Access Method
Only below.

For the prog method of access IndexDir specifies the path to the program(s) to
execute. The external program must correctly format the documents being passed back
to Swish-e. Examples of external programs are provided in the prog-bin directory.

IndexDir ./myprogram.pl

See prog for details.

Note: Not all directives work with all methods.

NoContents *list of file suffixes*
Files with these suffixes will not have their contents indexed, but will have their
path name (file name) indexed instead.

If the file's type is HTML or HTML2 (as set by "IndexContents" or "DefaultContents")
then the file will be parsed for a HTML title and that title will be indexed. Note
that you must set the file's type with "IndexContents" or "DefaultContents": ".html"
and ".htm" are NOT type HTML by default. For example:

IndexContents HTML* .htm .html

If a title is found, it will still be checked for "FileRules title", and the file will
be skipped if a match is found. See "FileRules".

If the file's type is not HTML, or it is HTML and no title is found, then the file's
path will be indexed.

For example, this will allow searching by image file name.

NoContents .gif .xbm .au .mov .mpg .pdf .ps

Note: Using this directive will not cause files with those suffixes to be indexed.
That is, if you use "IndexOnly" to limit the types of files that are indexed, then you
must specify in "IndexOnly" the same suffixes listed in "NoContents".

This does not work:

# Wrong!
IndexOnly .htm .html
NoContents .gif .xbm .au .mov .mpg .pdf .ps

A "-S prog" program may set the "No-Contents:" header to enable this feature for a
specific document (although it would be smarter for the "-S prog" program to simply
only send the pathname or title to be indexed.

ReplaceRules [replace⎪remove⎪prepend⎪append⎪regex]
ReplaceRules allows you to make changes to file pathnames before they're indexed.
These changed file names or URLs will be returned in search results.

For example, you may index your files locally (with the File system indexing method),
yet return a URL in search results. This directive can be used to map the file names
to their respective URLs on your web server.

There are five operations you can specify: replace, append, remove, prepend, and regex
They will parse the pathname in the order you've typed these commands.

This directive uses C library regex.h regular expressions.

replace "the string you want replaced" "what to change it to"
remove "a string to remove"
prepend "a string to add before the result"
append "a string to add after the result"
regex "/search string/replace string/options"

Remember, quotes are needed if an expression contains white space, and backslashes
have special meaning.

Regex is an Extended Regular Expression. The first character found is the delimiter
(but it's not smart enough to use matched chars such as [], (), and {}).

The replace string may use substitution variables:

$0 the entire matched (sub)string
$1-$9 returns patterns captured in "(" ")" pairs
$` the string before the matched pattern
$' the string after the matched pattern

The options change the behavior of expression:

i ignore the case when matching
g repeat the substitution for the entire pattern

Examples:

ReplaceRules replace testdir/ anotherdir/
ReplaceRules replace [a-z_0-9]*_m.*\.html index.html

ReplaceRules remove testdir/

ReplaceRules prepend http://localhost/
ReplaceRules append .html

ReplaceRules regex !^/web/(.+)/!http://$1.domain.com/!
replaces a file path:
/web/search/foo/index.html
with
http://search.domain.com/foo/index.html

ReplaceRules regex #^#http://localhost/www#
ReplaceRules prepend http://localhost/www (same thing)

# Remove all extensions from C source files
ReplaceRules remove .c # ERROR! That "." is *any char*
ReplaceRules remove \.c # much better...

ReplaceRules remove "\\.c" # if in quotes you need double-backslash!
ReplaceRules remove "\.c" # ERROR! "\." -> "." and is *any char*

IndexContents [TXT⎪HTML⎪XML⎪TXT2⎪HTML2⎪XML2⎪TXT*⎪HTML*⎪XML*] *file extensions*
The "IndexContents" directive assigns one of Swish-e's document parsers to a document,
based on the its extension. Swish-e currently knows how to parse TXT, HTML, and XML
documents.

The XML2, HTML2, and TXT2 parsers are currently only available when Swish-e is
configured to use libxml2.

You may use XML*, HTML*, and TXT* to select the parser automatically. If libxml2 is
installed then it will be used to parse the content. Otherwise, Swish-e's internal
parsers will be used.

Documents that are not assigned a parser with "IndexContents" will, by default, use
the HTML2 parser if libxml2 is installed, otherwise will use Swish-e's internal HTML
parser. The "DefaultContents" directive may be used to assign a parser to documents
that do not match a file extension defined with the "IndexContents" directive.

Example:

IndexContents HTML* .htm .html .shtml
IndexContents TXT* .txt .log .text
IndexContents XML* .xml

HTML* is the default type for all files, unless otherwise specified (and this default
can be changed by the DefaultContents directive. Swish-e parses titles from HTML
files, if available, and keeps track of the context of the text for context searching
(see "-t" in SWISH-RUN).

If using filters (with the "FileFilter" directive) to convert documents you should
include those extensions, too. For example, if using a filter to convert .pdf to
.html, you need to tell Swish-e that .pdf should be indexed by the internal HTML
parser:

FileFilter .pdf pdf2html
IndexContent HTML .pdf

See also "ExtractPath".

MetaNameAlias *meta name* *list of aliases*
MetaNameAlias assigns aliases for a meta name. For example, if your documents contain
meta tags "description", "summary", and "overview" that all give a summary of your
documents you could do this:

MetaNames summary
MetaNameAlias summary description overview

Then all three tags will get indexed as meta tag "summary". You can then search all
the fields as:

-w summary=foo

The Alias work at search time, too. So these will also limit the search to the
"summary" meta name.

-w description=foo
-w overview=foo

MetaNamesRank integer *list of meta names*
You can assign a bias to metanames that will affect how ranking is calculated. The
range of values is from -10 to +10, with zero being no bias.

MetaNamesRank 4 subject
MetaNamesRank 3 swishdefault
MetaNamesRank 2 author publisher
MetaNamesRank -5 wrongwords

This feature is still considered experimental. If you use it, please send feedback to
the discussion list.

HTMLLinksMetaName *metaname*
Allows indexing of HTML links. Normally, HTML links (href tags) are not indexed by
Swish-e. This directive defines a metaname, and links will be indexed under this meta
name.

Example:

HTMLLinksMetaName links

Now, to limit searches to files with a link to "home.html" do this:

-w links='"home.html"'

The double quotes force a phrase search.

To make Swish-e index links as normal text, you may use:

HTMLLinksMetaName swishdefault

This feature is only available with the libxml2 HTML parser.

ImageLinksMetaName *metaname*
Allows indexing of image links under a metaname. Normally, image URLs are not
indexed.

Example:

ImagesLinksMetaName images

Now, if you would like to find pages that include a nice image of a beach:

-w images='beach'

To make Swish-e index links as normal text, you may use:

ImageLinksMetaName swishdefault

This feature is only available with the libxml2 HTML parser.

IndexAltTagMetaName *tagname*⎪as-text
Allows indexing of images <IMG> ALT tag text. Specify either a tag name which will be
used as a metaname, or the special text "as-text" which says to index the ALT text as
if it were plain text at the current location.

For example, by specifying a tag name:

IndexAltTagMetaName bar

would make this markup:

appear like

Then the normal rules ("MetaNames" and "PropertyNames") apply to how that text is
indexed.

If you use the special tag "as-text" then

simply becomes

This feature is only available when using the libxml2 parser (HTML2 and XML2).

AbsoluteLinks [yes⎪NO]
If this is set true then Swish-e will attempt to convert relative URIs extracted from
HTML documents for use with "HTMLLinksMetaName" and "ImageLinksMetaName" into absolute
URIs. Swish-e will use any <BASE> tag found in the document, otherwise it will use
the file's pathname. The pathname used will be the pathname *after* "ReplaceRules"
has been applied to the document's pathname.

For example, say you wish to index image links under the metaname "images".

ImageLinksMetaName images

If an image is located in http://localhost/vacations/france/index.html and
"AbsoluteLinks" is set to no, then a image within that document:

will only index "beach.jpeg".

But, if you want more detail when searching, you can enable "AbsoluteLinks" and Swish-
e will index "http://localhost/vacations/france/beach.jpeg". You can then look for
images of beaches, but only in France:

-w images=(beach and france)

This also means you can search for any images within France:

-w images=(france)

This feature is only available with the libxml2 HTML parser.

UndefinedMetaTags [error⎪ignore⎪INDEX⎪auto]
This directive defines the behavior of Swish-e during indexing when a meta name is
found but is not listed in MetaNames. There are four choices:

error
If a meta name is found that is not listed in MetaNames then indexing will be halted
and an error reported.

ignore
The contents of the meta tag are ignored and not indexed unless a metaname has been
defined with the "MetaNames" directive.

index
The contents of the meta tag are indexed, but placed in the main index unless
there's an enclosing metatag already in force. This is the default.

auto
This method create meta tags automatically for HTML meta names and XML elements.
Using this is the same as specifying all the meta names explicitly in a MetaNames
directive.

UndefinedXMLAttributes [DISABLE⎪error⎪ignore⎪index⎪auto]
This is similar to "UndefinedMetaTags", but only applies to XML documents (parsed with
libxml2). This allows indexing of attribute content, and provides a way to index the
content under a metaname. For example, "UndefinedXMLAttributes" can make

look like the following to swish:

What happens to the text "23" will depend on the setting of "UndefinedXMLAttributes":

disable
XML attributes are not parsed and not indexed. This is the default.

error
If the concatenated meta name (e.g. person.age) is not listed in MetaNames then
indexing will be halted and an error reported.

ignore
The contents of the meta tag are ignored and not indexed unless a metaname has been
defined with the "MetaNames" directive.

index
The contents of the meta tag are indexed, but placed in the main index unless
there's an enclosing metatag already in force.

auto
This method will create meta tags from the combined element and attributes (and XML
Class name) This options should be used with caution as it can generate a lot of
metaname entries.

See also the example below "XMLClassAttribues".

XMLClassAttributes *list of XML attribute names*
Combines an XML class name with the element name to make up a metaname. For example:

XMLClassAttributes class

Will appear to Swish-e as:

How the data is indexed depends on "MetaNames" and "UndefinedMetaTags".

Here's an example using the following configuration which combines the two directives
"XMLClassAttributes" and "UndefinedXMLAttributes".

XMLClassAttributes class
UndefinedMetaTags auto
UndefinedXMLAttributes auto
IndexContents XML2 .xml

The source XML file looks like:

Swish-e parses as:

./swish-e -c 2 -i 1.xml -T parsed_tags parsed_text -v 0
Indexing Data Source: "File-System"

<xml> (MetaName)

<person> (MetaName)
<person.student> (MetaName)
<person.student.phone> (MetaName)
555-1212
</person.student.phone>
<person.student.age> (MetaName)
102
</person.student.age>
John
</person>

<person> (MetaName)
<person.greeting> (MetaName)
howdy
</person.greeting>
Bill
</person>

</xml>
Indexing done!

One thing to note is that the first <person> block finds a class name "student" so all
metanames that are created from attributes use the combined name "person.student".
The second <person> block doesn't contain a "class" so, the attribute name is combined
directly with the element name (e.g. "person.greeting").

ExtractPath *metaname* [replace⎪remove⎪prepend⎪append⎪regex]
This directive can be used to index extracted parts of a document's path. A common
use would be to limit searches to specific areas of your file tree.

The extracted string will be indexed under the specified meta name.

See "ReplaceRules" for a description of the various pattern replacement methods, but
you will use the regex method.

For example, say your file system (or web tree) was organized into departments:

/web/sales/foo...
/web/parts/foo...
/web/accounting/foo...

And you wanted a way to limit searches to just documents under "sales".

ExtractPath department regex !^/web/([^/]+)/.*$!$1!

Which says, extract out the department name (as substring $1) and index it as meta
name "department". Then to limit a search to the sales department:

swish-e -w foo AND department=sales

Note that the "regex" method uses a substitution pattern, so to index only a sub-
string match the entire document path in the regular expression, as shown above.
Otherwise any part that is not matched will end up in the substitution pattern.

See the "ExtractPathDefault" option for a way to set a value if not patterns match.

Although unlikely, you may use more than one "ExtractPath" directive. More than one
directive of the same meta name will operate successively (in order listed in the
configuration file) on the path. This allows you to use regular expressions on the
results of the previous pattern substitution (as if piping the output from one
expression to the patter of the next).

ExtractPath foo regex !^(...).+$!$1!
ExtractPath foo regex !^.+(.)$!$1!

So, the third letter is indexed as meta name "foo" if both patterns match.

ExtractPath foo regex !^X(...).+$!$1!
ExtractPath foo regex !^.+(.)$!$1!

Now (not the "X"), if the first pattern doesn't match, the last character of the path
name is indexed. You must be clear on this behavior if you are using more than one
"ExtractPath" directive with the same metaname.

The document path operated on is the real path swish used to access the document.
That is, the "ReplaceRules" directive has no effect on the path used with
"ExtractPath".

The full path is used for each meta name if more than one "ExtractPath" directive is
used. That is, changes to the path used in "ExtractPath foo" do not affect the path
used by "ExtractPath bar".

ExtractPathDefault *metaname* default_value
This can be used with "ExtractPath" to set a default string to index under the given
metaname if none of the "ExtractPath" patterns match.

For example, say your want to index each document with a metaname "department" based
on the following path examples:

/web/sales/foo...
/web/parts/foo...
/web/accounting/foo...

But you are also indexing documents that do not follow that pattern and you want to
search those separately, too.

ExtractPath department regex !^/web/([^/]+)/.*$!$1!
ExtractPathDefault department other

Now, you may search like this:

-w foo department=(sales) - limit searches to the sales documents
-w foo department=(parts) - limit searches to the parts documents
-w foo department=(accounting) - limit searches to the accounting documents
-w foo department=(other) - everything but sales, parts, and accounting.

This basically is a shortcut for:

-w foo not department=(sales or parts or accounting)

but you don't need to keep track of what was extracted.

PropertyNames *list of meta names*
PropertyNamesCompareCase *list of meta names*
PropertyNamesIgnoreCase *list of meta names*
Swish-e allows you to specify certain META tags that can be used as document
properties. The contents of any META tag that has been identified as a document
property can be returned as part of the search results along with the rank, file name,
title, and document size (see the "-p" and "-x" switches in SWISH-RUN).

Properties are useful for returning additional data from documents in search results
-- this saves the effort of reading and parsing the source files while reading Swish-e
search results, and is especially useful when the source documents are no longer
available or slow to access (e.g. over http).

Another feature of properties is that Swish-e can use the PropertyNames for sorting
the search results (see the "-s" switch).

PropertyNames author subjects

Two variations are available. "PropertyNamesCompareCase" and
"PropertyNamesIgnoreCase". These tell Swish-e to either ignore or compare case when
sorting results. The default for "PropertyNames" is to ignore the case.

PropertyNamesIgnoreCase subject
PropertyNamesCompareCase keyword

The defaults for "internal" properties are:

swishtitle -- ignore the case
swishdocpath -- compare case
swishdescription -- compare case

These can be overridden with "PropertyNamesCompareCase" and "PropertyNamesIgnoreCase".

PropertyNamesCompareCase swishtitle

Use of PropertyNames will increase the size of your index files, sometimes
significantly. Properties will be compressed if Swish-e is compiled with zlib as
described in the INSTALL manual page.

If Swish-e finds more than one property of the same name in a document the property's
contents will be concatinated for strings, and a warning issues for numeric (or date)
properties.

PropertyNamesNoStripChars
PropertyNamesNoStripChars specifies that the listed properties should not have strings
of low ASCII characters replaced with a space character. Properties will be stored as
found in the document.

When printing properties with the swish-e binary newlines are replaced with a space
character. Use the swish-e library (or SWISH::API perl module) to fetch properties
without newlines replaced.

PropertyNamesNumeric
This directive is similar to "PropertyNames", but it flags the property as being a
string of digits (integer value) that will be stored as binary data instead of a
string. This allows sorting with "-s" and limiting with "-L" to sort and limit the
property correctly.

Swish-e uses strtoul(3) to convert the string into an unsigned long integer.
Therefore, only positive integers can be stored.

Future versions of Swish-e may be able to store different property types (such as
negative integers and real numbers). This directive may change in future releases of
Swish.

PropertyNamesDate
This directive is exactly like "PropertyNamesNumeric", but it also flags the number as
a machine timestamp (seconds since Epoch), and will print a formatted date when
returning this property. See "-x" in SWISH-RUN.

Swish-e will not parse dates when indexing; you must use a timestamp.

PropertyNameAlias *property name* *list of aliases*
This allows aliases for a property name. For example, if you are indexing HTML files,
plus XML files that are written in English, German, and Spanish and thus use the tags
"title", "titel", and "título" you can use:

PropertyNameAlias swishtitle title titel título titulo

Note that "swishtitle" is the built-in property used to store the title of a document,
and therefore you do not need to specify it as a PropertyName before use.

PropertyNamesMaxLength integer *list of meta names*
This option will set the max length of the text stored in a property. You must
specify a number between 0 and the max integer size on your platform, and a list of
properties. The properties specified must not be aliases.

If any of the property names do not exist they will be created (e.g. you do not need
to define the property with PropertyNames first).

In general, this feature will only be useful when parsing HTML or XML with the libxml2
parser.

For example:

PropertyNamesMaxLength 1000 swishdescription
PropertyNameAlias swishdescription body

Is somewhat like

StoreDescription HTML <body> 1000
StoreDescription XML <body> 1000
StoreDescription HTML2 <body> 1000
StoreDescription XML2 <body> 1000

but StoreDescription allows setting the tag for each parser type.

PropertyNamesMaxLength 1000 headings
PropertyNameAlias headings h1 h2 h3 h4

collects all the heading text into a single property called "headings", not to exceed
1000 characters.

PropertyNamesSortKeyLength integer *list of meta names*
Sets the length of the string used when sorting. The default is 100 characters. The
-T metanames debugging option will list the current values for an index.

This setting is used when sorting during indexing, and perhaps when sorting while
searching. It also effects the order when limiting to a range of values with the -L
option.

PreSortedIndex *list of property names*
By default Swish-e generates presorted tables while indexing for each property name.
This allows faster sorting when generating results. On large document collections
this presorting may add to the indexing time, and also adds to the total size of the
index. This directive can be used to customize exactly which properties will be
presorted.

If "PreSortedIndex" it is not present in the config file (default action), all the
properties will be presorted at indexing time. If it is present without any
parameter, no properties will be presorted. Otherwise, only the property names
specified will be presorted.

For example, if you only wish to sort results by a property called "title":

PropertyNames title age time
PreSortedIndex title

StoreDescription [XML <tag> size⎪HTML <meta> size⎪TXT size]
StoreDescription allows you to store a document description in the index file. This
description can be returned in your search results when the "-x" switch is used to
include the swishdescription for extended results, or by using "-p swishdescription".

The document type (XML, HTML and TXT) must match the document type currently being
indexed as set by "IndexContents" or "DefaultContents". See those directives for
possible values. A common problem is using "StoreDescription" yet not setting the
document's type with "IndexContents" or "DefaultContents". Another problem is
different types:

IndexContents HTML2 .html
StoreDescription HTML <body>

Then .html documents are assigned a type of HTML2 (and parsed by the libxml2 parser),
but the description will not be stored since it is type HTML instead of HTML2.

For text documents you specify the type TXT (or TXT2 or TXT*) and the number of
characters to capture.

StoreDescription TXT 20

The above stores only the first twenty characters from the text file in the Swish-e
index file.

For HTML, and XML file types, specify the tag to use for the description, and
optionally the number of characters to capture. If not specified will capture the
entire contents of the tag.

StoreDescription HTML <body> 20000
StoreDescription XML <desc> 40

Again, note that documents must be assigned a document type with "IndexContents" or
"DefaultContents" to use this feature.

Swish-e will compress the descriptions (or any other large property) if compiled to
use zlib (see INSTALL). This is recommended when using StoreDescription and a large
number of documents. Compression of 30% to 50% is not uncommon with HTML files.

PropCompressionLevel [0-9]
This directive sets the compression level used when storing properties to disk. A
setting of zero is no compression, and a setting of nine is the most compression.

The default depends on the default setting compiled with zlib, but is typically six.

This option is useful when using "StoreDescription" to store a large amount text in
properties (or if using "PropertyNames" with large property sizes).

Properties must be over a value defined in config.h (100 is the default) before
compression will be attempted. Swish-e will never store the results of the
compression if the compressed data is larger than the original data.

This option is only available when Swish-e is compiled with zlib support.

TruncateDocSize *number of characters*
TruncateDocSize limits the size of a document while indexing documents and/or using
filters. This config directive truncates the numbers of read bytes of a document to
the specified size. This means: if a document is larger, read only the specified
numbers of bytes of the document.

Example:

TruncateDocSize 10000000

The default is zero, which means read all data.

Warning: If you use TruncateDocSize, use it with care! TruncateDocSize is a safety
belt only, to limit e.g. filteroutput, when accessing databases, or to limit
"runnaway" filters. Truncating doc input may destroy document structures for Swish-e
(e.g. swish may miss closing tags for XML or HTML documents).

TruncateDocSize does not currently work with the "prog" input source method.

FuzzyIndexingMode NONE⎪Stemming⎪Soundex⎪Metaphone⎪DoubleMetaphone
Selects the type of index to create. Only one type of index may be created.

It's a good idea to create both a normal index and a fuzzy index and allow your search
interface select which index to use. Many people find the fuzzy searches to be too
fuzzy.

The available fuzzy indexing options can be displayed by running

swish-e -T LIST_FUZZY_MODES

Available options include:

None
Words are stored in the index without any conversion. This is the default.

Stemming_*
This options uses one of the installed Snowball stemmers
(http://snowball.tartarus.org/).

The installed stemmers can be viewed by running

swish-e -T LIST_FUZZY_MODES

For example, to use the Spanish stemming module:

FuzzyIndexingMode Stemming_es

Stem or Stemming_en
**This option is no longer supported.**

Selects the legacy Swish-e English stemmer.

This is deprecated in favor of the Snowball English stemmer Stemming_en1.

Words are converted using the Porter stemming algorithm.

From: http://www.tartarus.org/~martin/PorterStemmer/

The Porter stemming algorithm (or Porter stemmer) is a
process for removing the commoner morphological and inflexional
endings from words in English. Its main use is as part of a
term normalisation process that is usually done when setting up
Information Retrieval systems.

This will help a search for "running" to also find "run" and "runs", for example.

The stemming function does not convert words to their root, rather
programmatically removes endings on words in an attempt to make similar words with
different endings stem to the same string of characters. It's not a perfect
system, and searches on stemmed indexes often return curious results. For
example, two entirely different words may stem to the same word.

Stemming also can be confusing when used with a wildcard (truncation). For
example, you might expect to find the word "running" by searching for "runn*".
But this fails when using a stemmed index, as "running" stems to "run", yet
searching for "runn*" looks for words that start with "runn".

Soundex
Soundex was developed in the 1880s so records for people with similar sounding
names could be found more readily. Soundex is a coded surname based on the way a
surname sounds rather than spelling. Surnames that sound similar, like Smith and
Smyth, are filed together under the same Soundex code. This is mostly useful for
US English.

Soundex should not be used to search for sound-alike words. Metaphone would be
more appropriate for generic sound matching of words. Soundex should only be used
where you need to search multiple documents for proper names which sound similar.
This is primarily used for indexing genealogical records. This may be useful for
indexing other collections of data consisting mostly of names. Many common name
variations are matched by Soundex. The only notable exception is the first letter
of the name. The first letter is not matched for sound.

Metaphone and DoubleMetaphone
Words are transformed into a short series of letters representing the sound of the
word (in English). Metaphone algorithms are often used for looking up mis-spelled
words in dictionary programs.

From: http://aspell.sourceforge.net/metaphone/

Lawrence Philips' Metaphone Algorithm is an algorithm which returns
the rough approximation of how an English word sounds.

The "DoubleMetaphone" mode will sometimes generate two different metaphones for
the same word. This is supposed to be useful when a word may be pronounced more
than one way.

A metaphone index should give results somewhere in between Soundex and Stemming.

UseStemming [yes⎪NO]
Put yes to apply word stemming algorithm during indexing, else no.

UseStemming no
UseStemming yes

When UseStemming is set to "yes" every word is stemmed before placing it in to the
index.

This option is deprecated. It has been superceded by "FuzzyIndexingMode".

UseSoundex [yes⎪NO]
When UseSoundex is set to "yes" every word is converted to a Soundex code before
placing it in to the index.

This option is deprecated. It has been superceded by "FuzzyIndexingMode".

IgnoreTotalWordCountWhenRanking [YES⎪no]
Put yes to ignore the total number of words in the file when calculating ranking.
Often better with merges and small files. Default is yes.

IgnoreTotalWordCountWhenRanking no

The default was changed from no to yes in version 2.2.

NOTE: must be set to no if you intend to use the -R 1 option when searching.

MinWordLimit *integer*
Set the minimum length of an word. Shorter words will not be indexed. The default is
1 (as defined in src/config.h).

MinWordLimit 5

MaxWordLimit *integer*
Set the maximum length of an indexable word. Every longer word will not be indexed.
The Default is 40 (as defined in src/config.h).

WordCharacters *string of characters*
IgnoreFirstChar *string of characters*
IgnoreLastChar *string of characters*
BeginCharacters *string of characters*
EndCharacters *string of characters*
These settings define what a word consists of to the Swish-e indexing engine.
Compiled in defaults are in src/config.h.

When indexing Swish-e uses WordCharacters to split up the document into words. Words
are defined by any string of non-blank characters that contain only the characters
listed in WordCharacters. If a string of characters includes a character that is not
in WordCharacters then the word will be spit into two or more separate words.

For example:

WordCharacters abde

Would turn "abcde" into two words "ab" and "de".

Next, of these words, any characters defined in IgnoreFirstChar are stripped off the
start of the word, and IgnoreLastChar characters are stripped off the end of the word.
This allows, for example, periods within a word (www.slashdot.com), but not at the end
of a word. Characters in IgnoreFirstChar and IgnoreLastChar must be in
WordCharacters.

Finally, the resulting words MUST begin with one of the characters listed in
BeginCharacters and end with one of the characters listed in EndCharacters.
BeginCharacters and EndCharacters must be a subset of the characters in
WordCharacters. Often, WordCharacters, BeginCharacters and EndCharacters will all be
the same.

Note that the same process applies to the query while searching.

Getting these settings correct will take careful consideration and practice. It's
helpful to create an index of a single test file, and then look at the words that are
placed in the index (see the "-v 4", "-D" and "-k" searching switches).

Currently there is only support for eight-bit characters.

Example:

WordCharacters .abcdefghijklmnopqrstuvwxyz
BeginCharacters abcdefghijklmnopqrstuvwxyz
EndCharacters abcdefghijklmnopqrstuvwxyz
IgnoreFirstChar .
IgnoreLastChar .

So the string

Please visit http://www.example.com/path/to/file.html.

will be indexed as the following words:

please
visit
http
www.example.com
path
to
file.html

Which means that you can search for "www.example.com" as a single word, but searching
for just "example" will not find the document.

Note: when indexing HTML documents HTML entities are converted to their character
equivalents before being processed with these directives. This is a change from
previous versions of Swish-e where you were required to include the characters
"0123456789&#;" to index entities. See also "ConvertHTMLEntities"

Buzzwords [*list of buzzwords*⎪File: path]
The Buzzwords option allows you to specify words that will be indexed regardless of
WordCharacters, BeginCharacters, EndCharacters, stemming, soundex and many of the
other checks done on words while indexing.

Buzzwords are case insensitive.

Buzzwords should be separated by spaces and may span multiple directives. If the
special format "File:filename" is used then the Buzzwords will be read from an
external file during indexing.

Examples:

Buzzwords C++ TCP/IP

Buzzwords File: ./buzzwords.lst

If a Buzzword contains search operator characters they must be backslashed when
searching. For example:

Buzzwords C++ TCP/IP web=http

./swish-e -w 'web\=http'

Buzzwords are found by splitting the text on whitespace, removing "IgnoreFirstChar"
and "IgnoreLastChar" characters from the word, and then comparing with the list of
"Buzzwords". Therefore, if adding "Buzzwords" to an index you will probably want to
define "IgnoreFirstChar" and "IgnoreLastChar" settings.

Note: Buzzwords specific settings for "IgnoreFirstChar" and "IgnoreLastChar" may be
used in the future.

CompressPositions [yes⎪NO]
This option enables zlib compression for individual word data in the index file. The
default is NO, that is the index word data is not compressed by default.

Enabling this option can reduced the size of the index file, but at the expense of
slower wildcard search times.

The default changed from YES to NO starting with version 2.4.3.

IgnoreWords [*list of stop words*⎪File: path]
The IgnoreWords option allows you to specify words to ignore, called stopwords. The
default is to not use any stopwords.

Words should be separated by spaces and may span multiple directives. If the special
format "File:filename" is used then the stop words will be read from an external file
during indexing.

In previous versions of Swish-e you could use the directive

IgnoreWords swishdefault - obsolete!

to include a default list of compiled in stopwords. This keyword is no longer
supported.

Examples:

IgnoreWords www http a an the of and or

IgnoreWords File: ./stopwords.de

UseWords [*list of words*⎪File: path]
UseWords defines the words that Swish-e will index. Only the words listed will be
indexed.

You can specify a list of words following the directive (you may specify more than one
"UseWords" directive in a config file), and/or use the "File:" form to specify a path
to a file containing the words:

UseWords perl python pascal fortran basic cobal php
UseWords File: /path/to/my/wordlist

Please drop the Swish-e list a note if you actually use this feature. It may be
removed from future versions.

IgnoreLimit *integer integer*
This automatically omits words that appear too often in the files (these words are
called stopwords). Specify a whole percentage and a number, such as "80 256". This
omits words that occur in over 80% of the files and appear in over 256 files. Comment
out to turn off auto-stopwording.

IgnoreLimit 50 1000

Swish-e must do extra processing to adjust the entire index when this feature is used.
It is recommended that instead of using this feature that you decided what words are
stopwords and add them to IngoreWords in your configuration file. To do this, use
IgnoreLimit one time and note the stop words that are found while indexing. Add this
list to IgnoreWords, and then remove IgnoreLimit from the configuration file.

IgnoreMetaTags *list of names*
"IgnoreMetaTags" defines a list of metatags to ignore while indexing XML files (and
HTML files if using libxml2 for parsing HTML). All text within the tags will be
ignored -- both for indexing ("MetaNames") and properties ("PropertyNames"). To still
parse properties, yet do not index the text, see "UndefinedMetaTags".

This option is useful to avoid indexing specific data from a file. For example:

<person>
<first_name>
William
</first_name> <last_name>
Shakespeare
</last_name> <updated_date>
April 25, 1999
</updated_date>
</person>

In the above example you might not want to index the updated date, and therefore
prevent finding this record by searching

-w 'person=(April)'

This is solved by:

IgnoreMetaTags updated_date

See also the "-l" switch in SWISH-RUN.

FileRules [type] [contains⎪is⎪regex] *regular expression*
FileMatch [type] [contains⎪is⎪regex] *regular expression*
FileRules and FileMatch are used to, respectively, exclude and include files and
directories to index. Since, by default, Swish-e indexes all files and recurses all
directories (but see also "FollowSymLinks") you will typically only use "FileRules" to
exclude files or directories. "FileMatch" is useful in a few cases, for example, to
override the behavior of "IndexOnly". Some examples are included below.

Except for "FileRules title ...", this feature is only available for file access
method (-S fs), which is the default indexing mode. Also, any pathname modification
with "ReplaceRules" happens after the check for "FileRules". (It's unlikely that you
would exclude files with "FileRules" based on text you added with "ReplaceRules"!)

The regular expression is a C regex.h extended regular expression. You may supply
more than one regular expression per line, or use separate directives. Preceding the
regular expression with the word "not" negates the match.

The regular expression is compared against [type] as described below.

For historical reasons, you can specify "contains" or "is". "is" simply forces the
regular expression to match at the start and end of the string (by internally
prepending "^" and appending "$" to the regular expression).

The "regex" option requires delimiter characters:

FileRules title regex /^private/i

The only advantage of "regex" is if you want to do case insensitive matches, or simply
like your regular expressions to look like perl regular expressions. You must use
matching delimiters; (), {}, and [], are not currently supported for no good reason
other than laziness.

Use quotes (" or ') around a pattern if it contains any white space. Note that the
backslash character becomes the escape character within quotes.

For example, these sets generate the same regular expressions.

FileRules title is hello
FileRules title contains ^hello$
FileRules title regex /^hello$/

These all need quotes due to the included space character

FileRules title is "hello there"
FileRules title contains "^hello there$"
FileRules title regex "!^hello there$!"

These show how the backslash must be doubled inside of quotes. Swish-e converts a
double-backslash into a single backslash, and then passes that single onto the regular
expression compiler.

FileRules filename regex /\.pdf/
FileRules filename regex "/\\.pdf/"

FileRules filename regex !hello\\there! # need double for real backslash
FileRules filename regex "!hello\\\\there!" # need double-double inside of quotes

Matching Types

The following types of match strings my be supplied:

FileRules pathname
FileRules dirname
FileRules filename
FileRules directory
FileRules title

FileMatch pathname
FileMatch filename
FileMatch dirname
FileMatch directory

pathname matches the regular expression against the current pathname. The pathname
may or may not be absolute depending on what you supplied to "IndexDir".

Example:

# Don't index paths that contain private or hidden
FileRules pathname contains (private⎪hidden)

# Same thing
FileRules pathname regex /(private⎪hidden)/

# Don't index exe files
FileRules pathname contains \.exe$

dirname and filename split the path name by the last delimiter character into a
directory name, and a file name. Then these are compared against the patterns
supplied. Directory names do not have a trailing slash. All path names use the
forward slash as a delimiter within Swish-e.

Example:

# Same as last example - don't index *.exe files.
FileRules filename contains \.exe$

# Don't index any file called test.html files
FileRules filename contains ^test\.html$

# Same thing
FileRules filename is test\.html

# Don't index any directories that contain "old" (/usr/local/myold/docs)
FileRules dirname contains old

# Don't index any directories that contain the path segment "old" (/usr/local/old/foo)
FileRules dirname contains /old/

# Index only .htm, .html, plus any all-digit file names
IndexOnly .htm .html
FileMatch filename contains ^\d+$

# Same as previous, but maybe a little slower
FileRules filename regex not !\.(htm⎪html)$!
FileMatch filename contains ^\d+$

Swish-e checks these settings in the order of "pathname", "dirname", and "filename",
and "FileMatch" patterns are checked before "FileRules", in general. This allows you
to exclude most files with "FileRules", yet allow in a few special cases with
"FileMatch". For example:

# Exclude all files of .exe, .bin, and .bat
FileRules filename contains \.(exe⎪bin⎪bat)$
# But, let these two in
FileMatch filename is baseball\.bat incoming_mail\.bin

# Same, but as a single pattern
FileMatch filename is (baseball\.bat⎪incoming_mail\.bin)

The "directory" type is somewhat unique. When Swish-e recurses into a directory it
will compare all the files in the directory with the pattern and then decide if that
entire directory should or should not be indexed (or recursed). Note that you are
matching against file names in a directory -- and some of those names may be directory
names.

A "FileRules directory" match will cause Swish-e to ignore all files and sub-
directories in the current directory.

Warning: A match with "FileMatch directory" says to index everything in the *current*
directory and ignore any FileRules for this directory.

Example:

# Don't index any directories (and sub directories) that contain
# a file (or sub-directory) called "index.skip"
FileRules directory contains ^index\.skip$

# Don't index directories that contain a .htaccess file.
FileRules directory contains ^\.htaccess

"title" checks for a pattern match in an HTML title.

Example:

FileRules title contains construction example pointers

# This example says to ignore case
FileRules title regex "/^Internal document/i"

Note: "FileRules title" works for any input method (fs, prog, or http) that is parsed
as HTML, and where a title was found in the document.

In case all this seems a bit confusing, processing a directory happens in the
following order.

First the directory name is checked:

FileRules dirname - reject entire directory if matches

Next the directory is scanned and each file name (which might be the name of a
sub-directory) is checked:

FileRules directory - reject entire dir if *any* files match
FileMatch directory - accept entire dir if *any* files match

Then, unless "FileMatch directory" matched, each file is tested with FileMatch. A
match says to index the file without further testing (i.e. overrides FileRules and
IndexOnly):

FileMatch pathname \
FileMatch dirname - file is accepted if any match
FileMatch filename /

otherwise

IndexOnly - file is checked for the correct file extension

FileRules pathname \
FileRules dirname - file is rejected if any match
FileRules filename /

finally, the file is indexed.

Files (not directories) listed with "IndexDir" or "-i" are processed in a similar way:

FileMatch pathname \
FileMatch dirname - file is accepted if any match
FileMatch filename /

otherwise, the file is rejected if it doesn't have the correct extension or a
FileRules matches.

IndexOnly - file is checked for the correct file extension

FileRules pathname \
FileRules dirname - file is rejected if any match
FileRules filename /

Note: If things are not indexing as you expect, create a directory with some test
files and use the "-T regex" trace option to see how file names are checked. Start
with very simple tests!

Directives for the HTTP Access Method Only

The HTTP Access method is enabled by the "-S http" switch when indexing. It works by
running a Perl program called SwishSpider which fetches documents from a web server.

Only text files (content-type of "text/*") are indexed with the HTTP Access Method. Other
document types (e.g. PDF or MSWord) may be indexed as well. The SwishSpider will attempt
to make use of the SWISH::Filter module (included with the Swish-e distribution) to
convert documents into a format that Swish-e can index.

Note: The -S prog method of spidering (using spider.pl) can be a replacement for the -S
http method. It offers more configuration options and better spidering speed.

These directives below are available when using the HTTP Access Method of indexing.

MaxDepth *integer*
MaxDepth defines how many links the spider should follow before stopping. A value of
0 configures the spider to traverse all links. The default is MaxDepth 0.

MaxDepth 5

Note: The default was changed from 5 to 0 in release 2.4.0

Delay *seconds*
The number of seconds to wait between issuing requests to a server. This setting
allows for more friendly spidering of remote sites. The default is 5 seconds.

Delay 1

Note: The default was changed from 60 to 5 seconds in release 2.4.0

TmpDir *path*
The location of a writable temp directory on your system. The HTTP access method
tells the Perl helper to place its files in this location, and the "-e" switch causes
Swish-e to use this directory while indexing. There is no default.

TmpDir /tmp/swish

If this directory does not exist or is not writable Swish-e will fail with an error
during indexing.

Note, the environment variables of "TMPDIR", "TMP", and "TEMP" (in that order) will
override this setting.

SpiderDirectory *path*
The location of the Perl helper script called swishspider. If you use a relative
directory, it is relative to your directory when you run Swish-e, not to the directory
that Swish-e is in. The default is the location swishspider was installed. Normally
this does not need to be set.

SpiderDirectory /usr/local/swish

EquivalentServer *server alias*
Often times the same site may be referred to by different names. A common example is
that often http://www.some-server.com and http://some-server.com are the same. Each
line should have a list of all the method/names that should be considered equivalent.
Multiple EquivalentServer directives may be used. Each directive defines its own set
of equivalent servers.

EquivalentServer http://library.berkeley.edu http://www.lib.berkeley.edu
EquivalentServer http://sunsite.berkeley.edu:2000 http://sunsite.berkeley.edu

Directives for the prog Access Method Only

This section details the directives that are only available for the "prog" document source
feature of Swish-e. The "prog" access method runs an external program that "feeds"
documents to Swish-e. This allows indexing and filtering of documents from any source.

See prog - general purpose access method in the SWISH-RUN man page for more information.

A number of example programs for use with the "prog" access method are provided in the
prog-bin directory. Please see those example if you have questions about implementing a
"prog" input program.

SwishProgParameters *list of parameters*
This is a list of parameters that will be sent to the external program when running
with the "prog" document source method.

SwishProgParameters /path/to/config hello there
IndexDir /path/to/program.pl

Then running:

swish-e -c config -S prog

Swish-e will execute "/path/to/program.pl" and pass "/path/to/config hello there" as
three command line arguments to the program. This directive makes it easy to pass
settings from the Swish-e configuration file to the external program.

For example, the "spider.pl" program (included in the "prog-bin" directory) uses the
"SwishProgParameters" to specify what file to read for configuration information.

SwishProgParameters spider.config
IndexDir ./spider.pl

The "spider.pl" program also has a default action so you can avoid using a
configuration file:

SwishProgParameters default http://www.swishe.org/ http://some.other.site/
IndexDir ./spider.pl

And the spider program will use default settings for spidering those sites.

Swish-e can read documents from standard input, so another way to run an external
program with parameters is:

./spider.pl spider.conf ⎪ ./swish-e -S prog -i stdin

Notes when using MS Windows

You should use unix style path separators to specify your external program. Swish will
convert forward slashes to backslashes before calling the external program. This is only
true for the program name specified with "IndexDir" or the "-i" command line option.

In addition, Swish-e will make sure the program specified actually exists, which means you
need to use the full name of the program.

For example, to run the perl spider program spider.pl you would need a Swish-e
configuration file such as:

IndexDir e:/perl/bin/perl.exe
SwishProgParameters prog-bin/spider.pl default http://swish-e.org

and run indexing with the command:

swish-e -c swish.cfg -S prog -v 9

The "IndexDir" command tells Swish-e the name of the program to run. Under unix you can
just specify the name of the script, since unix will figure out the program from the first
line of the script.

The "SwishProgParameters" are the parameters passed to the program specified by "IndexDir"
(perl.exe in this case). The first parameter is the perl script to run
(prog-bin/spider.pl). Perl passes the rest of the parameters directly to the perl script.
The second parameter default tells the spider.pl program to use default settings for
spidering (or you could specify a spider config file -- see "perldoc spider.pl" for
details), and lastly, the URL is passed into the spider program.

Document Filter Directives

Internally, Swish-e knows how to parse only text, HTML, and XML documents. With "filters"
you can index other types of documents. For example, if all your web pages are in gzip
format a filter can uncompress these on the fly for indexing.

You may wish to read the Swish-e FAQ question on filtering before continuing here. How Do
I filter documents?

There are two suggested methods for filtering.

Filtering with SWISH::Filter

The Swish-e distribution includes a Perl module called SWISH::Filter and individual
filters located in the filters directory. This system uses plug-in filters to extend the
types of documents that Swish-e can index. The plug-in filters do not actually do the
filtering, but rather provide a standard interface for accessing programs that can filter
or convert documents. The programs that do the filtering are not part of the Swish-e
distribution; they must be downloaded and installed separately.

The advantage of this method is that new filtering methods can be installed easily.

This system is designed to work with the -S http and -prog methods, but may also be used
with the "FileFilter" feature and -S fs indexing method. See
$prefix/share/doc/swish-e/examples/filter-bin/swish_filter.pl for an example.

See the filters/README file for more information.

Filtering with the FileFilter feature

A filter is an external program that Swish-e executes while processing a document of a
given type. Swish-e will execute the filter program for each file that matches the file
suffix (extension) set in the FileFilter or FileFilterMatch directives. FileFilterMatch
matches using regular expressions and is described below.

Filters may be used with any type of input method (i.e. -S fs, -S http, or -S prog). But
because

Swish-e calls the external program passing as default arguments:

$0 the name of the filter program

$1 the physical path name of the file to read. This may be a temporary file location if
indexing by the http method.

$2 When indexing under the file system this will be the same as $1 (the path to the
source file), but when indexing under the http method this will be the URL of the
source document.

Swish-e can also pass other parameters to the filter program. These parameters can be
defined using the FileFilter or FileFilterMatch directives. See Filter Options below.

The filter program must open the file, process its contents, and return it to Swish-e by
printing to STDOUT.

Note that this can add a significant amount of time to the indexing process if your
external program is a perl or shell script. If you have many files to filter you should
consider writing your filter in C instead of a shell or perl script, or using the "prog"
Access Method along with SWISH::Filter.

FilterDir *path-to-directory*
Deprecated.

This is the path to a directory where the filter programs are stored. Swish-e looks
in this directory to find the filter specified in the FileFilter directive.

This directive is not needed if the filter program can be found in your system's path.
Even if your filter is not in your system's path you can specify the full path to the
filter in the FileFilter or FileFilterMatch directives.

Example:

FilterDir /usr/local/swish/filters

FileFilter *suffix* "filter-prog" ["filter-options"]
This maps file suffix (extension) to a filter program. If filter-prog starts with a
directory delimiter (absolute path), Swish-e doesn't use the FilterDir settings, but
uses the given filter-prog path directly.

On systems that have a working fork(2) system call the filter program is run by
forking swish then executing the filter. This mean the shell is not used for running
the filter and no arguments are passed through the shell.

On other systems (e.g. Windows) the arguments are double-quoted and popen(3) is used
to run the program. This does pass argument though the shell and may be a security
concern depending on the abilities of the shell.

Filter options:

Filter options are a string passed as arguments to the filter-prog. Filter options
can contain variables, replaced by Swish-e. If you omit filter-options Swish-e will
use default parameters for the options listed above.

Default: %p %P
Which means: pass "workfile path" and "documentfile path" to filter.

Variables in filter options:

%% = %
%P = Full document pathname (e.g. URL, or path on filesystem)
%p = Full pathname to work file (maybe a tmpfile or the real document path on filesystem)
%F = Filename stripped from full document pathname
%f = Filename stripped from "work" pathname
%D = Directoryname stripped from full document pathname
%d = Directoryname stripped from full "work" pathname

Examples of strings passed:

%P = document pathname: http://myserver/path1/mydoc.txt
%p = work pathname: /tmp/tmp.1234.mydoc.txt
%F = mydoc.txt
%f = tmp.1234.mydoc.txt
%D = http://myserver/path1
%d = /tmp

Notes when using MS Windows

Windows uses double quotes to escape shell metacharacters, so if you need to use
quotes then use single quotes around the entire option string.

FileFiler .mydoc mydocfilter.exe '--title "text with spaces"'

You can specify the filter program using forward slashes (unix style). Swish will
convert the slashes to backslashes before running your program.

FileFilter .mydoc c:/some/path/mydocfilter.exe '-d "%d" -example -url "%P" "%f"'

Examples of filters:

FileFilter .doc /usr/local/bin/catdoc "-s8859-1 -d8859-1 %p"
FileFilter .pdf pdftotext "%p -"
FileFilter .html.gz gzip "-c %p"
FileFilter .mydoc "/some/path/mydocfilter" "-d %d -example -url %P %f"

The above examples are running a binary filter program. For more complicated
filtering needs you may use a scripting language such as Perl or a shell script.
Here's some examples of calling a shell and perl script:

FileFilter .pdf pdf2html.sh
FileFilter .ps ghostscript-filter.pl

Using a scripting language (or any language that has a large startup cost) can greatly
increase the indexing time. For small indexing jobs, this may not be an issue, but
for large collections of files that require processing by a scripting language, you
may be better off using the "-S prog" access method where the script will only be
compiled once, instead of for each document.

Filters are probably easier to write than a "-S prog" program. Which you decide to
use depends on your requirements. Examples of filter scripts can be found in the
filter-bin directory, and examples of "-S prog" programs can be found in the prog-bin
directory.

FileFilterMatch *filter-prog* *filter-options* *regex* [*regex* ...]
This is similar to "FileMatch" except uses regular expressions to match against the
file name. *filter-prog* is the path to the program. Unlike "FileFilter" this does
not use the "FilterDir" option. Also unlike "FileFilter" you must specify the
*filter-options*.

Examples:

FileFilterMatch ./pdftotext "%p -" /\.pdf$/

Note that will also match a file called ".pdf", so you may want to use something that
requires a filename that has more than just an extension. For example:

FileFilterMatch ./pdftotext "%p -" /.\.pdf$/

To specify more than one extension:

FileFilterMatch ./check_title.pl "%p" /\.html$/ /\.htm$/

Or a few ways to do the same thing:

FileFilterMatch ./check_title.pl %p /\.(html⎪html)$/
FileFilterMatch ./check_title.pl %p /\.html?$/

And to ignore case:

FileFilterMatch ./check_title.pl %p /\.html?$/i

You may also precede an expression with "not" to negate regular expression that
follow. For example, to match files that do not have an extension:

FileFilterMatch ./convert "%p %P" not /\..+$/

Document Info

       $Id: SWISH-CONFIG.pod 1846 2006-10-20 20:18:30Z whmoseley $

       .