NAME

       Plucene::Analysis::Tokenizer - base class for tokenizers

SYNOPSIS

               my $tokenizer = Plucene::Analysis::Tokenizer::Subclass
                       ->new({ reader => $reader });

DESCRIPTION

       This is an abstract base class for tokenizers.

       A Tokenizer is a TokenStream whose input is a Reader.

METHODS

   new
               my $tokenizer = Plucene::Analysis::Tokenizer::Subclass
                       ->new({ reader => $reader });

       This will create a new tokenizer.

   next
       This must be defined in a subclass

   close
       Close the input reader.