Provided by: libsyntax-highlight-perl-improved-perl_1.01-5_all bug


       Syntax::Highlight::Perl::Improved - Highlighting of Perl Syntactical Structures


       This file documents Syntax::Highlight::Perl::Improved version 1.0.


           # simple procedural
           use Syntax::Highlight::Perl::Improved ':BASIC';  # or ':FULL'

           print format_string($my_string);

           # OO
           use Syntax::Highlight::Perl::Improved;

           my $formatter = new Syntax::Highlight::Perl::Improved;
           print $formatter->format_string($my_string);


       This module provides syntax highlighting for Perl code.  The design bias is roughly line-
       oriented and streamed (ie, processing a file line-by-line in a single pass).  Provisions
       may be made in the future for tasks related to "back-tracking" (ie, re-doing a single line
       in the middle of a stream) such as speeding up state copying.

       The only constructor provided is "new()".  When called on an existing object, "new()" will
       create a new copy of that object.  Otherwise, "new()" creates a new copy of the (internal)
       Default Object.  Note that the use of the procedural syntax modifies the Default Object
       and that those changes will be reflected in any subsequent "new()" calls.

       Formatting is done using the "format_string()" method.  Call "format_string()" with one or
       more strings to format, or it will default to using $_.

   Setting and Getting Formats
       You can set the text used for formatting a syntax element using "set_format()" (or set the
       start and end format individually using "set_start_format()" and "set_end_format()",

       You can also retrieve the text used for formatting for an element via "get_start_format()"
       or "get_end_format".  Bulk retrieval of the names or values of defined formats is possible
       via "get_format_names_list()" (names), "get_start_format_values_list()" and

       See "FORMAT TYPES" later in this document for information on what format elements can be

   Checking and Setting the State
       You can check certain aspects of the state of the formatter via the methods:
       "in_heredoc()", "in_string()", "in_pod()", "was_pod()", "in_data()", and "line_count()".

       You can reset all of the above states (and a few other internal ones) using "reset()".

   Stable and Unstable Formatting Modes
       You can set or check the stability of formatting via "unstable()".

       In unstable (TRUE) mode, formatting is not considered to be persistent with nested
       formats.  Or, put another way, when unstable, the formatter can only "remember" one format
       at a time and must reinstate formatting for each token.  An example of unstable formatting
       is using ANSI color escape sequences in a terminal.

       In stable (FALSE) mode (the default), formatting is considered persistent within
       arbitrarily nested formats.  Even in stable mode, however, formatting is never allowed to
       span multiple lines; it is always fully closed at the end of the line and reinstated at
       the beginning of a new line, if necessary.  This is to ensure properly balanced tags when
       only formatting a partial code snippet.  An example of stable formatting is HTML.

       Using "define_substitution()", you can have the formatter substitute certain strings with
       others, after the original string has been parsed (but before formatting is applied).
       This is useful for escaping characters special to the output mode (eg, > and < in HTML)
       without them affecting the way the code is parsed.

       You can retrieve the current substitutions (as a hash-ref) via "substitutions()".


       The Syntax::Highlight::Perl::Improved formatter recognizes and differentiates between many
       Perl syntactical elements.  Each type of syntactical element has a Format Type associated
       with it.  There is also a 'DEFAULT' type that is applied to any element who's Format Type
       does not have a value.

       Several of the Format Types have underscores in their name.  This underscore is special,
       and indicates that the Format Type can be "generalized."  This means that you can assign a
       value to just the first part of the Format Type name (the part before the underscore) and
       that value will be applied to all Format Types with the same first part.  For example, the
       Format Types for all types of variables begin with "Variable_".  Thus, if you assign a
       value to the Format Type "Variable", it will be applied to any type of variable.
       Generalized Format Types take precedence over non-generalized Format Types.  So the value
       assigned to "Variable" would be applied to "Variable_Scalar", even if "Variable_Scalar"
       had a value explicitly assigned to it.

       You can also define a "short-cut" name for each Format Type that can be generalized.  The
       short-cut name would be the part of the Format Type name after the underscore.  For
       example, the short-cut for "Variable_Scalar" would be "Scalar".  Short-cut names have the
       least precedence and are only assigned if neither the generalized Type name, nor the full
       Type name have values.

       Following is a list of all the syntactical elements that Syntax::Highlight::Perl::Improved
       currently recognizes, along with a short description of what each would be applied to.

           A normal Perl comment.  Starts with '#' and goes until the end of the line.

           Inline documentation.  Starts with a line beginning with an equal sign ('=') followed
           by a word (eg: '=pod') and continuing until a line beginning with '=cut'.

           Either the "she-bang" line at the beginning of the file, or a line directive altering
           what the compiler thinks the current line and file is.

           A loop or statement label (to be the target of a goto, next, last or redo).

           Any string or character that begins or ends a String.  Including, but not necessarily
           limited to: quote-like regular expression operators ("m//", "s///", "tr///", etc), a
           Here-Document terminating line, the lone period terminating a format, and, of course,
           normal quotes ("'", """, "`", "q{}", "qq{}", "qr{}", "qx{}").

           Any text within quotes, "format"s, Here-Documents, Regular Expressions, and the like.

           The identifier used to define, identify, or call a subroutine (or method).  Note that
           Syntax::Highlight::Perl::Improved cannot recognize a subroutine if it is called
           without using parentheses or an ampersand, or methods called using the indirect object
           syntax.  It formats those as barewords.

           A scalar variable.

           Note that (theoretically) this format is not applied to non-scalar variables that are
           being used as scalars (ie: array or hash lookups, nor references to anything other
           than scalars).  Syntax::Highlight::Perl::Improved figures out (or at least tries to)
           the actual type of the variable being used (by looking at how you're subscripting it)
           and formats it accordingly.  The first character of the variable (ie, the "$", "@",
           "%", or "*") tells you the type of value being used, and the color (hopefully) tells
           you the type of variable being used to get that value.

           (See "KNOWN ISSUES" for information about when this doesn't work quite right.)

           An array variable (but not usually a slice; see above).

           A hash variable.

           A typeglob.  Note that typeglobs not beginning with an asterisk (*) (eg: filehandles)
           are formatted as barewords.  This is because, well, they are.

           Whitespace.  Not usually formatted but it can be.

           A special, or backslash-escaped, character.  For example: "\n" (newline), or "\d"

           Only occurs within strings or regular expressions.

           A Perl keyword.  Some examples include: my, local, sub, next.

           Note that Perl does not make any distinction between keywords and built-in functions
           (at least not in the documentation).  Thus I had to make a subjective call as to what
           would be considered keywords and what would be built-in functions.

           The list of keywords can be found (and overloaded) in the variable
           $Syntax::Highlight::Perl::Improved::keyword_list_re as a pre-compiled regular

           A Perl built-in function, called as a function (ie, using parentheses).

           The list of built-in functions can be found (and overloaded) in the variable
           $Syntax::Highlight::Perl::Improved::builtin_list_re as a pre-compiled regular

           A Perl built-in function, called as a list or unary operator (ie, without using

           The list of built-in functions can be found (and overloaded) in the variable
           $Syntax::Highlight::Perl::Improved::builtin_list_re as a pre-compiled regular

           A Perl operator.

           The list of operators can be found (and overloaded) in the variable
           $Syntax::Highlight::Perl::Improved::operator_list_re as a pre-compiled regular

           A bareword.  This can be user-defined subroutine called without parentheses, a
           typeglob used without an asterisk (*), or just a plain old bareword.

           The name of a package or pragmatic module.

           Note that this does not apply to the package portion of a fully qualified variable

           A numeric literal.

           A symbol (ie, non-operator punctuation).

           The special tokens that signal the end of executable code and the beginning of the
           DATA section.  Specifically, '"__END__"' and '"__DATA__"'.

           Anything in the DATA section (see "CodeTerm").


       Syntax::Highlight::Perl::Improved uses OO method-calls internally (and actually defines a
       Default Object that is used when the functions are invoked procedurally) so you will not
       gain anything (efficiency-wise) by using the procedural interface.  It is just a matter of

       It is actually recommended that you use the OO interface, as this allows you to
       instantiate multiple, concurrent-yet-separate formatters.  Though I cannot think of why
       you would need multiple formatters instantiated. :-)

       One point to note: the "new()" method uses the Default Object to initialize new objects.
       This means that any changes to the state of the Default Object (including Format
       definitions) made by using the procedural interface will be reflected in any subsequently
       created objects.  This can be useful in some cases (eg, call "set_format()" procedurally
       just before creating a batch of new objects to define default Formats for them all) but
       will most likely lead to trouble.


       new PACKAGE
       new OBJECT
           Creates a new object.  If called on an existing object, creates a new copy of that
           object (which is thenceforth totally separate from the original).

           Resets the object's internal state.  This breaks out of strings and here-docs, ends
           PODs, resets the line-count, and otherwise gets the object back into a "normal" state
           to begin processing a new stream.

           Note that this does not reset any user options (including formats and format

       unstable EXPR
           Returns true if the formatter is in unstable mode.

           If called with a non-zero number, puts the formatter into unstable formatting mode.

           In unstable mode, it is assumed that formatting is not persistent one token to the
           next and that each token must be explicitly formatted.

           Returns true if the next string to be formatted will be inside a Here-Document.

           Returns true if the next string to be formatted will be inside a multi-line string.

           Returns true if the formatter would consider the next string passed to it as begin
           within a POD structure.  This is false immediately before any POD instigators ("=pod",
           "=head1", "=item", etc), true immediately after an instigator, throughout the POD and
           immediately before the POD terminator ("=cut"), and false immediately after the POD

           Returns true if the last line of the string just formatted was part of a POD
           structure.  This includes the "/^=\w+/" POD instigators and terminators.

           Returns true if the next string to be formatted will be inside the DATA section (ie,
           follows a "__DATA__" or "__END__" tag).

           Returns the number of lines processed by the formatter.

           Returns a reference to the substitution table used.  The substitution table is a hash
           whose keys are the strings to be replaced, and whose values are what to replace them

       define_substitution HASH_REF
       define_substitution LIST
           Allows user to define certain characters that will be substituted before formatting is
           done (but after they have been processed for meaning).

           If the first parameter is a reference to a hash, the formatter will replace it's own
           hash with the given one, and subsequent changes to the hash outside the formatter will
           be reflected.

           Otherwise, it will copy the arguments passed into it's own hash, and any substitutions
           already defined (but not in the parameter list) will be preserved. (ie, the new
           substitutions will be added, without destroying what was there already.)

       set_start_format HASH_REF
       set_start_format LIST
           Given either a list of keys/values, or a reference to a hash of keys/values, copy them
           into the object's Formats list.

       set_end_format HASH_REF
       set_end_format LIST
           Given either a list of keys/values, or a reference to a hash of keys/values, copy them
           into the object's Formats list.

       set_format LIST
           Sets the formatting string for one or more formats.

           You should pass a list of keys/values where the keys are the format names and the
           values are references to arrays containing the starting and ending formatting strings
           (in that order) for that format.

       get_start_format LIST
           Retrieve the string that is inserted to begin a given format type (starting format

           The names are looked for in the following order:

           First: Prefer the names joined by underscore, from most general to least.  For
           example, given ("Variable", "Scalar"): "Variable" then "Variable_Scalar".

           Second: Then try each name singly, in reverse order.  For example, "Scalar" then

           See "FORMAT TYPES" for more information.

       get_end_format LIST
           Retrieve the string that is inserted to end a given format type (ending format

           Returns a list of the names of all the Formats defined.

           Returns a list of the values of all the start Formats defined (in the same order as
           the names returned by "get_format_names_list()").

           Returns a list of the values of all the end Formats defined (in the same order as the
           names returned by "get_format_names_list()").

       format_string LIST
           Formats one or more strings of Perl code.  If no strings are specified, defaults to
           $_.  Returns the list of formatted strings (or the first string formatted if called in
           scalar context).

           Note:  The end of the string is considered to be the end of a line, regardless of
           whether or not there is a trailing line-break (but trailing line-breaks will not cause
           an extra, empty line).

           Another Note:  The function actually uses $/ to determine line-breaks, unless $/ is
           set to "\n" (newline).  If $/ is "\n", then it looks for the first match of
           "m/\r?\n|\n?\r/" in the string and uses that to determine line-breaks.  This is to
           make it easy to handle non-unix text.  Whatever characters it ends up using as line-
           breaks are preserved.

       format_token TOKEN, LIST
           Returns TOKEN wrapped in the start and end Formats corresponding to LIST (as would be
           returned by "get_start_format( LIST )" and "get_end_format( LIST )", respectively).

           No syntax checking is done on TOKEN but substitutions defined with
           "define_substitution()" are performed.


       ·   Barewords used as keys to a hash are formatted as strings.  This is Good.  They should
           not be, however, if they are not the only thing within the curly braces.  That can be

       ·   This version does not handle formats (see perlform(1)) very well.  It treats them as
           Here-Documents and ignores the rules for comment lines, as well as the fact that
           picture lines are not supposed to be interpolated.  Thus, your picture lines will look
           strange with the '@'s being formatted as array variables (albeit, invalid ones).
           Ideally, it would also treat value lines as normal Perl code and format accordingly.
           I think I'll get to the comment lines and non-interpolating picture lines first.
           If/When I do get this fixed, I will most likely add a format type of 'Format' or
           something, so that they can be formatted differently, if so desired.

       ·   This version does not handle Regular Expression significant characters.  It simply
           treats Regular Expressions as interpolated strings.

       ·   User-defined subroutines, called without parentheses, are formatted as barewords.
           This is because there is no way to tell them apart from barewords without parsing the
           code, and would require us to go as far as perl does when doing the "-c" check (ie,
           executing BEGIN and END blocks and the like).  That's not going to happen.

       ·   If you are indexing (subscripting) an array or hash, the formatter tries to figure out
           the "real" variable class by looking at how you index the variable.  However, if you
           do something funky (but legal in Perl) and put line-breaks or comments between the
           variable class character ($) and your identifier, the formatter will get confused and
           treat your variable as a scalar.  Until it finds the index character.  Then it will
           format the scalar class character ($) as a scalar and your identifier as the "correct"

       ·   If you put a line-break between your variable identifier and it's indexing character
           (see above), which is also legal in Perl, the formatter will never find it and treat
           your variable as a scalar.

       ·   If you put a line-break between a bareword hash-subscript and the hash variable, or
           between a bareword and its associated "=>" operator, the bareword will not be
           formatted correctly (as a string).  (Noticing a pattern here?)


       Bug reports are always welcome. Email me at b<>.


       David C.Y. Liu b<>

       based on code by Cory Johns

       Copyright (c) 2004 David C.Y. Liu.  This library is free software; you can redistribute
       and/or modify it under the same conditions as Perl itself.


       Note: This is Cory John's todo list, not mine. Currently none of these features are
       planned for the near future.

       1.  Improve handling of regular expressions.  Add support for regexp-special characters.
           Recognize the /e option to the substitution operator (maybe).

       2.  Improve handling of formats.  Don't treat format definitions as interpolating.  Handle
           format-comments.  Possibly format value lines as normal Perl code.

       3.  Create in-memory deep-copy routine to replace "eval(Data::Dumper)" deep-copy.

       4.  Generalize state transitions ("reset()" and, in the future, "copy_state()") to use
           non-hard-coded keys and values for state variables.  Probably will extrapolate them
           into an overloadable hash, and use the aforementioned deep-copy to assign them.

       5.  Create a method to save or copy states between objects ("copy_state()").  Would be
           useful for using this module in an editor.

       6.  Add support for greater-than-one length special characters.  Specifically, octal,
           hexadecimal, and control character codes.  For example, "\644", "\x1a4" or "\c[".


   05-03-2004  David C.Y. Liu (Version 1.01)
       ·   Added 'our' to the keywords list.

       ·   Fixed bug that prevented interpolation inside qq() quotes.

       ·   Renamed to Syntax::Highlight::Perl::Improved.

   04-04-2001  Cory Johns
       ·   Fixed problem with special characters not formatting inside of Here-Documents.

       ·   Fixed bug causing hash variables to format inside of Here-Documents.

   03-30-2001  Cory Johns
       ·   Fixed bug where quote-terminators were checked for inside of Here-Documents.

   03-29-2001  Cory Johns
       ·   Moved token processing tests from _format_line() into _process_token() (where they
           should've been all along), generally making _format_line() more logical.
           Contemplating extrapolating the tokenizing and token loop into its own subroutine to
           avoid all the recursive calls.

       ·   Fixed bug that caused special characters to be recognized outside of strings.

       ·   Added $VERSION variable.

       ·   Added support for different types of literal numbers: floating point, exponential
           notation (eg: 1.3e10), hexadecimal, and underscore-separated.

       ·   Added the "CodeTerm" and "DATA" Formats.

   03-27-2001  Cory Johns
       ·   Added was_pod() and updated the documentation for in_pod().

   03-20-2001  Cory Johns
       ·   Added support for Perl formats (ie, `"format = ..."').