Plucene::Analysis::WhitespaceTokenizer

white space tokenizer

Provided by: libplucene-perl (Version: 1.25-3)
Report a bug

SYNOPSIS

        # isa Plucene::Analysis::CharTokenizer

DESCRIPTION

A WhitespaceTokenizer is a tokenizer that divides text at whitespace. Adjacent sequences of non-Whitespace characters form tokens.