Provided by: libppix-regexp-perl_0.047-1_all 
NAME
PPIx::Regexp::Tokenizer - Tokenize a regular expression
SYNOPSIS
use PPIx::Regexp::Dumper;
PPIx::Regexp::Dumper->new( 'qr{foo}smx' )
->print();
INHERITANCE
"PPIx::Regexp::Tokenizer" is a PPIx::Regexp::Support.
"PPIx::Regexp::Tokenizer" has no descendants.
DESCRIPTION
This class provides tokenization of the regular expression.
METHODS
This class provides the following public methods. Methods not documented here (or
documented below under "EXTERNAL TOKENIZERS") are private, and unsupported in the sense
that the author reserves the right to change or remove them without notice.
new
my $tokenizer = PPIx::Regexp::Tokenizer->new( 'xyzzy' );
This static method instantiates the tokenizer. You must pass it the regular expression to
be parsed, either as a string or as a PPI::Element of some sort. You can also pass
optional name/value pairs of arguments. The option names are specified without a leading
dash. Supported options are:
default_modifiers array_reference
This argument specifies default statement modifiers. It is optional, but if specified
must be an array reference. See the PPIx::Regexp new() documentation for the details.
encoding name
This option specifies the encoding of the string to be tokenized. If specified, an
"Encode::decode" is done on the string (or the "content" of the PPI class) before it
is tokenized.
postderef boolean
This option specifies whether the tokenizer recognizes postfix dereferencing. See the
PPIx::Regexp new() documentation for the details.
$PPIx::Regexp::Tokenizer::DEFAULT_POSTDEREF is not exported.
trace number
Specifying a positive value for this option causes a trace of the tokenization. This
option is unsupported in the sense that the author reserves the right to alter it
without notice.
If this option is unspecified, the value comes from environment variable
"PPIX_REGEXP_TOKENIZER_TRACE" (see "ENVIRONMENT VARIABLES"). If this environment
variable does not exist, the default is 0.
Undocumented options are unsupported.
The returned value is the instantiated tokenizer, or "undef" if instantiation failed. In
the latter case a call to "errstr" will return the reason.
content
print $tokenizer->content();
This method returns the string being tokenized. This will be the result of the
PPI::Element->content() method if the object was instantiated with a PPI::Element.
default_modifiers
print join ', ', @{ $tokenizer->default_modifiers() };
This method returns a reference to a copy of the array passed to the "default_modifiers"
argument to new(). If this argument was not used to instantiate the object, the return is
a reference to an empty array.
encoding
This method returns the encoding of the data being parsed, if one was set when the class
was instantiated; otherwise it simply returns undef.
errstr
my $tokenizer = PPIx::Regexp::Tokenizer->new( 'xyzzy' )
or die PPIx::Regexp::Tokenizer->errstr();
This static method returns an error description if tokenizer instantiation failed.
failures
print $tokenizer->failures(), " tokenization failures\n";
This method returns the number of tokenization failures encountered. A tokenization
failure is represented in the output token stream by a PPIx::Regexp::Token::Unknown.
modifier
$tokenizer->modifier( 'x' )
and print "Tokenizing an extended regular expression\n";
This method returns true if the given modifier character was found on the end of the
regular expression, and false otherwise.
Starting with version 0.036_01, if the argument is a single-character modifier followed by
an asterisk (intended as a wild card character), the return is the number of times that
modifier appears. In this case an exception will be thrown if you specify a multi-
character modifier (e.g. 'ee*'), or if you specify one of the match semantics modifiers
(e.g. 'a*').
next_token
my $token = $tokenizer->next_token();
This method returns the next token in the token stream, or nothing if there are no more
tokens.
significant
This method exists simply for the convenience of PPIx::Regexp::Dumper. It always returns
true.
tokens
my @tokens = $tokenizer->tokens();
This method returns all remaining tokens in the token stream.
EXTERNAL TOKENIZERS
This class does very little of its own tokenization. Instead the token classes contain
external tokenization routines, whose name is '__PPIX_TOKENIZER__' concatenated with the
current mode of the tokenizer ('regexp' for regular expressions, 'repl' for the
replacement string).
These external tokenizers are called as static methods, and passed the
"PPIx::Regexp::Tokenizer" object and the current character in the character stream.
If the external tokenizer wants to make one or more tokens, it returns an array containing
either length in characters for tokens of the tokenizer's own class, or the results of one
or more "make_token" calls for tokens of an arbitrary class.
If the external tokenizer is not interested in the characters starting at the current
position it simply returns.
The following methods are for the use of external tokenizers, and are not part of the
public interface to this class.
capture
if ( $tokenizer->find_regexp( qr{ \A ( foo ) }smx ) ) {
foreach ( $tokenizer->capture() ) {
print "$_\n";
}
}
This method returns all the contents of any capture buffers from the previous call to
"find_regexp". The first element of the array (i.e. element 0) corresponds to $1, and so
on.
The captures are cleared by "make_token", as well as by another call to "find_regexp".
cookie
$tokenizer->cookie( foo => sub { 1 } );
my $cookie = $tokenizer->cookie( 'foo' );
my $old_hint = $tokenizer->cookie( foo => undef );
This method either creates, deletes, or accesses a cookie.
A cookie is a code reference which is called whenever the tokenizer makes a token. If it
returns a false value, it is deleted. Explicitly setting the cookie to "undef" also
deletes it.
When you call "$tokenizer->cookie( 'foo' )", the current cookie is returned. If you pass a
new value of "undef" to delete the token, the deleted cookie (if any) is returned.
When the "make_token" method calls a cookie, it passes it the tokenizer and the token just
made. If a token calls a cookie, it is recommended that it merely pass the tokenizer,
though of course the token can do whatever it wants.
The cookie mechanism seems to be a bit of a crock, but it appeared to be more work to fix
things up in the lexer after the tokenizer got something wrong.
The recommended way to write a cookie is to use a closure to store any necessary data, and
have a call to the cookie return the data; otherwise the ultimate consumer of the cookie
has no way to access the data. Of course, it may be that the presence of the cookie at a
certain point in the parse is all that is required.
expect
$tokenizer->expect( 'PPIx::Regexp::Token::Code' );
This method inserts a given class at the head of the token scan, for the next iteration
only. More than one class can be specified. Class names can be abbreviated by removing the
leading 'PPIx::Regexp::'.
If no class is specified, this method does nothing.
The expectation lasts from the next time "get_token" is called until the next time
make_token makes a significant token, or until the next "expect" call if that is done
sooner.
find_regexp
my $end = $tokenizer->find_regexp( qr{ \A \w+ }smx );
my ( $begin, $end ) = $tokenizer->find_regexp(
qr{ \A \w+ }smx );
This method finds the given regular expression in the content, starting at the current
position. If called in scalar context, the offset from the current position to the end of
the matched string is returned. If called in list context, the offsets to both the
beginning and the end of the matched string are returned.
find_matching_delimiter
my $offset = $tokenizer->find_matching_delimiter();
This method is used by tokenizers to find the delimiter matching the character at the
current position in the content string. If the delimiter is an opening bracket of some
sort, bracket nesting will be taken into account.
When searching for the matching delimiter, the back slash character is considered to
escape the following character, so back-slashed delimiters will be ignored. No other
quoting mechanisms are recognized, though, so delimiters inside quotes still count. This
is actually the way Perl works, as
$ perl -e 'qr<(?{ print "}" })>'
demonstrates.
This method returns the offset from the current position in the content string to the
matching delimiter (which will always be positive), or undef if no match can be found.
get_mode
This method returns the name of the current mode of the tokenizer.
get_start_delimiter
my $start_delimiter = $tokenizer->get_start_delimiter();
This method is used by tokenizers to access the start delimiter for the regular
expression.
get_token
my $token = $tokenizer->make_token( 3 );
my @tokens = $tokenizer->get_token();
This method returns the next token that can be made from the input stream. It is not part
of the external interface, but is intended for the use of an external tokenizer which
calls it after making and retaining its own token to look at the next token ( if any ) in
the input stream.
If any external tokenizer calls get_token without first calling make_token, a fatal error
occurs; this is better than the infinite recursion which would occur if the condition were
not trapped.
An external tokenizer must return anything returned by get_token; otherwise tokens get
lost.
interpolates
This method returns true if the top-level structure being tokenized interpolates; that is,
if the delimiter is not a single quote.
make_token
return $tokenizer->make_token( 3, 'PPIx::Regexp::Token::Unknown' );
This method is used by this class (and possibly by individual tokenizers) to manufacture a
token. Its arguments are the number of characters to include in the token, and optionally
the class of the token. If no class name is given, the caller's class is used. Class names
may be shortened by removing the initial 'PPIx::Regexp::', which will be restored by this
method.
The token will be manufactured from the given number of characters starting at the current
cursor position, which will be adjusted.
If the given length would include characters past the end of the string being tokenized,
the length is reduced appropriately. If this means a token with no characters, nothing is
returned.
match
if ( $tokenizer->find_regexp( qr{ \A \w+ }smx ) ) {
print $tokenizer->match(), "\n";
}
This method returns the string matched by the previous call to "find_regexp".
The match is set to "undef" by "make_token", as well as by another call to "find_regexp".
modifier_duplicate
$tokenizer->modifier_duplicate();
This method duplicates the modifiers on the top of the modifier stack, with the intent of
creating a locally-scoped copy of the modifiers. This should only be called by an external
tokenizer that is actually creating a modifier scope. In other words, only when creating a
PPIx::Regexp::Token::Structure token whose content is '('.
modifier_modify
$tokenizer->modifier_modify( name => $value ... );
This method sets new values for the modifiers in the local scope. Only the modifiers whose
names are actually passed have their values changed.
This method is intended to be called after manufacturing a PPIx::Regexp::Token::Modifier
token, and passed the results of its "modifiers" method.
modifier_pop
$tokenizer->modifier_pop();
This method removes the modifiers on the top of the modifier stack. This should only be
called by an external tokenizer that is ending a modifier scope. In other words, only when
creating a PPIx::Regexp::Token::Structure token whose content is ')'.
Note that this method will never pop the last modifier item off the stack, to guard
against unmatched right parentheses.
peek
my $character = $tokenizer->peek();
my $next_char = $tokenizer->peek( 1 );
This method returns the character at the given non-negative offset from the current
position. If no offset is given, an offset of 0 is used.
If you ask for a negative offset or an offset off the end of the sting, "undef" is
returned.
ppi_document
This method makes a PPI document out of the remainder of the string, and returns it.
prior
$tokenizer->prior( 'can_be_quantified' )
and print "The prior token can be quantified.\n";
This method is deprecated in favor of prior_significant_tokrn(). Six months after the
release of version 0.044_01 it will start warning on the first use. Six months after that
it will warn on every use. Six months after that a fatal exception will be thrown when it
is called. I am not sure I need to put this though a deprecation cycle, given that this
method is documented as not being part of the public interface, but I choose to err on the
side of caution.
This method calls the named method on the most-recently-instantiated significant token,
and returns the result. Any arguments subsequent to the method name will be passed to the
method.
Because this method is designed to be used within the tokenizing system, it will die
horribly if the named method does not exist.
If called with no arguments at all the most-recently-instantiated significant token is
returned.
prior_significant_token
$tokenizer->prior_significant_token( 'can_be_quantified' )
and print "The prior token can be quantified.\n";
This method calls the named method on the most-recently-instantiated significant token,
and returns the result. Any arguments subsequent to the method name will be passed to the
method.
Because this method is designed to be used within the tokenizing system, it will die
horribly if the named method does not exist.
If called with no arguments at all the most-recently-instantiated significant token is
returned.
ENVIRONMENT VARIABLES
A tokenizer trace can be requested by setting environment variable
PPIX_REGEXP_TOKENIZER_TRACE to a numeric value other than 0. Use of this environment
variable is unsupported in the same sense that the "trace" option of "new" is unsupported.
Explicitly specifying the "trace" option to "new" overrides the environment variable.
The real reason this is documented is to give the user a way to troubleshoot funny output
from the tokenizer.
SUPPORT
Support is by the author. Please file bug reports at <http://rt.cpan.org>, or in
electronic mail to the author.
AUTHOR
Thomas R. Wyant, III wyant at cpan dot org
COPYRIGHT AND LICENSE
Copyright (C) 2009-2016 by Thomas R. Wyant, III
This program is free software; you can redistribute it and/or modify it under the same
terms as Perl 5.10.0. For more details, see the full text of the licenses in the directory
LICENSES.
This program is distributed in the hope that it will be useful, but without any warranty;
without even the implied warranty of merchantability or fitness for a particular purpose.