Plucene::Analysis::Tokenizer
base class for tokenizers
- Provided by: libplucene-perl (Version: 1.25-3)
- Report a bug
SYNOPSIS
my $tokenizer = Plucene::Analysis::Tokenizer::Subclass
->new({ reader => $reader });
DESCRIPTION
This is an abstract base class for tokenizers.
A Tokenizer is a TokenStream whose input is a Reader.
METHODS
new
my $tokenizer = Plucene::Analysis::Tokenizer::Subclass
->new({ reader => $reader });
This will create a new tokenizer.
next
This must be defined in a subclass
close
Close the input reader.