NAME

       Plucene::Analysis::CharTokenizer - base class for character tokenisers

SYNOPSIS

               # isa Plucene::Analysis::Tokenizer

               my $next = $chartokenizer->next;

DESCRIPTION

       This is an abstract base class for simple, character-oriented tokenizers.

METHODS

   token_re
       This should be defined in subclasses.

   normalize
       This will normalise the character before it is added to the token.

   next
               my $next = $chartokenizer->next;

       This will return the next token in the string, or undef at the end of the string.