Provided by: libplucene-perl_1.25-4_all
Plucene::Analysis::CharTokenizer - base class for character tokenisers
# isa Plucene::Analysis::Tokenizer my $next = $chartokenizer->next;
This is an abstract base class for simple, character-oriented tokenizers.
token_re This should be defined in subclasses. normalize This will normalise the character before it is added to the token. next my $next = $chartokenizer->next; This will return the next token in the string, or undef at the end of the string.