Provided by: libstring-tokenizer-perl_0.06-3_all 
NAME
String::Tokenizer - A simple string tokenizer.
SYNOPSIS
use String::Tokenizer;
# create the tokenizer and tokenize input
my $tokenizer = String::Tokenizer->new("((5+5) * 10)", '+*()');
# create tokenizer
my $tokenizer = String::Tokenizer->new();
# ... then tokenize the string
$tokenizer->tokenize("((5 + 5) - 10)", '()');
# will print '(, (, 5, +, 5, ), -, 10, )'
print join ", " => $tokenizer->getTokens();
# create tokenizer which retains whitespace
my $st = String::Tokenizer->new(
'this is a test with, (significant) whitespace',
',()',
String::Tokenizer->RETAIN_WHITESPACE
);
# this will print:
# 'this', ' ', 'is', ' ', 'a', ' ', 'test', ' ', 'with', ' ', '(', 'significant', ')', ' ', 'whitespace'
print "'" . (join "', '" => $tokenizer->getTokens()) . "'";
# get a token iterator
my $i = $tokenizer->iterator();
while ($i->hasNextToken()) {
my $next = $i->nextToken();
# peek ahead at the next token
my $look_ahead = $i->lookAheadToken();
# ...
# skip the next 2 tokens
$i->skipTokens(2);
# ...
# then backtrack 1 token
my $previous = $i->prevToken();
# ...
# get the current token
my $current = $i->currentToken();
# ...
}
DESCRIPTION
A simple string tokenizer which takes a string and splits it on whitespace. It also
optionally takes a string of characters to use as delimiters, and returns them with the
token set as well. This allows for splitting the string in many different ways.
This is a very basic tokenizer, so more complex needs should be either addressed with a
custom written tokenizer or post-processing of the output generated by this module.
Basically, this will not fill everyone's needs, but it spans a gap between simple "split /
/, $string" and the other options that involve much larger and complex modules.
Also note that this is not a lexical analyser. Many people confuse tokenization with
lexical analysis. A tokenizer merely splits its input into specific chunks, a lexical
analyzer classifies those chunks. Sometimes these two steps are combined, but not here.
METHODS
new ($string, $delimiters, $handle_whitespace)
If you do not supply any parameters, nothing happens, the instance is just created.
But if you do supply parameters, they are passed on to the "tokenize" method and that
method is run. For information about those arguments, see "tokenize" below.
setDelimiter ($delimiter)
This can be used to set the delimiter string, this is used by "tokenize".
handleWhitespace ($value)
This can be used to set the whitespace handling. It accepts one of the two constant
values "RETAIN_WHITESPACE" or "IGNORE_WHITESPACE".
tokenize ($string, $delimiters, $handle_whitespace)
Takes a $string to tokenize, and optionally a set of $delimiter characters to
facilitate the tokenization and the type of whitespace handling with
$handle_whitespace. The $string parameter and the $handle_whitespace parameter are
pretty obvious, the $delimiter parameter is not as transparent. $delimiter is a string
of characters, these characters are then separated into individual characters and are
used to split the $string with. So given this string:
(5 + (100 * (20 - 35)) + 4)
The "tokenize" method without a $delimiter parameter would return the following comma
separated list of tokens:
'(5', '+', '(100', '*', '(20', '-', '35))', '+', '4)'
However, if you were to pass the following set of delimiters "(, )" to "tokenize", you
would get the following comma separated list of tokens:
'(', '5', '+', '(', '100', '*', '(', '20', '-', '35', ')', ')', '+', '4', ')'
We now can differentiate the parens from the numbers, and no globbing occurs. If you
wanted to allow for optionally leaving out the whitespace in the expression, like
this:
(5+(100*(20-35))+4)
as some languages do. Then you would give this delimiter "+*-()" to arrive at the same
result.
If you decide that whitespace is significant in your string, then you need to specify
that like this:
my $st = String::Tokenizer->new(
'this is a test with, (significant) whitespace',
',()',
String::Tokenizer->RETAIN_WHITESPACE
);
A call to "getTokens" on this instance would result in the following token set.
'this', ' ', 'is', ' ', 'a', ' ', 'test', ' ', 'with', ' ', '(', 'significant', ')', ' ', 'whitespace'
All running whitespace is grouped together into a single token, we make no attempt to
split it into its individual parts.
getTokens
Simply returns the array of tokens. It returns an array-ref in scalar context.
iterator
Returns a String::Tokenizer::Iterator instance, see below for more details.
INNER CLASS
A String::Tokenizer::Iterator instance is returned from the String::Tokenizer's "iterator"
method and serves as yet another means of iterating through an array of tokens. The
simplest way would be to call "getTokens" and just manipulate the array yourself, or push
the array into another object. However, iterating through a set of tokens tends to get
messy when done manually. So here I have provided the String::Tokenizer::Iterator to
address those common token processing idioms. It is basically a bi-directional iterator
which can look ahead, skip and be reset to the beginning.
NOTE: String::Tokenizer::Iterator is an inner class, which means that only
String::Tokenizer objects can create an instance of it. That said, if
String::Tokenizer::Iterator's "new" method is called from outside of the String::Tokenizer
package, an exception is thrown.
new ($tokens_array_ref)
This accepts an array reference of tokens and sets up the iterator. This method can
only be called from within the String::Tokenizer package, otherwise an exception will
be thrown.
reset
This will reset the internal counter, bringing it back to the beginning of the token
list.
hasNextToken
This will return true (1) if there are more tokens to be iterated over, and false (0)
otherwise.
hasPrevToken
This will return true (1) if the beginning of the token list has been reached, and
false (0) otherwise.
nextToken
This dispenses the next available token, and move the internal counter ahead by one.
prevToken
This dispenses the previous token, and moves the internal counter back by one.
currentToken
This returns the current token, which will match the last token retrieved by
"nextToken".
lookAheadToken
This peeks ahead one token to the next one in the list. This item will match the next
item dispensed with "nextToken". This is a non-destructive look ahead, meaning it does
not alter the position of the internal counter.
skipToken
This will jump the internal counter ahead by 1.
skipTokens ($number_to_skip)
This will jump the internal counter ahead by $number_to_skip.
skipTokenIfWhitespace
This will skip the next token if it is whitespace.
skipTokensUntil ($token_to_match)
Given a string as a $token_to_match, this will skip all tokens until it matches that
string. If the $token_to_match is never matched, then the iterator will return the
internal pointer to its initial state.
collectTokensUntil ($token_to_match)
Given a string as a $token_to_match, this will collect all tokens until it matches
that string, at which point the collected tokens will be returned. If the
$token_to_match is never matched, then the iterator will return the internal pointer
to its initial state and no tokens will be returned.
TO DO
Inline token expansion
The Java StringTokenizer class allows for a token to be tokenized further, therefore
breaking it up more and including the results into the current token stream. I have
never used this feature in this class, but I can see where it might be a useful one.
This may be in the next release if it works out.
Possibly compliment this expansion with compression as well, so for instance double
quoted strings could be compressed into a single token.
Token Bookmarks
Allow for the creation of "token bookmarks". Meaning we could tag a specific token
with a label, that index could be returned to from any point in the token stream. We
could mix this with a memory stack as well, so that we would have an ordering to the
bookmarks as well.
BUGS
None that I am aware of. Of course, if you find a bug, let me know, and I will be sure to
fix it.
CODE COVERAGE
I use Devel::Cover to test the code coverage of my tests, below is the Devel::Cover report
on this module's test suite.
------------------------ ------ ------ ------ ------ ------ ------ ------
File stmt branch cond sub pod time total
------------------------ ------ ------ ------ ------ ------ ------ ------
String/Tokenizer.pm 100.0 100.0 64.3 100.0 100.0 100.0 97.6
------------------------ ------ ------ ------ ------ ------ ------ ------
Total 100.0 100.0 64.3 100.0 100.0 100.0 97.6
------------------------ ------ ------ ------ ------ ------ ------ ------
SEE ALSO
The interface and workings of this module are based largely on the StringTokenizer class
from the Java standard library.
Below is a short list of other modules that might be considered similar to this one. If
this module does not suit your needs, you might look at one of these.
String::Tokeniser
Along with being a tokenizer, it also provides a means of moving through the resulting
tokens, allowing for skipping of tokens and such. It was last updated in 2011.
Parse::Tokens
This one hasn't been touched since 2001, although it did get up to version 0.27. It
looks to lean over more towards the parser side than a basic tokenizer.
Text::Tokenizer
This is both a lexical analyzer and a tokenizer. It also uses XS, where
String::Tokenizer is pure perl. This is something maybe to look into if you were to
need a more beefy solution than String::Tokenizer provides.
THANKS
Thanks to Stephan Tobias for finding bugs and suggestions on whitespace handling.
AUTHOR
stevan little, <stevan@cpan.org>
COPYRIGHT AND LICENSE
Copyright 2004-2016 by Infinity Interactive, Inc.
<http://www.iinteractive.com>
This library is free software; you can redistribute it and/or modify it under the same
terms as Perl itself.