Plucene::Analysis::Standard::StandardTokenizer
standard tokenizer
- Provided by: libplucene-perl (Version: 1.25-3)
- Report a bug
SYNOPSIS
# isa Plucene::Analysis::CharTokenizer
DESCRIPTION
This is the standard tokenizer.
This should be a good tokenizer for most European-language documents.
METHODS
token_re
The regular expression for tokenising.
normalize
Remove 's and .