Provided by: libhtml-tokeparser-simple-perl_3.16-4_all bug

NAME

       HTML::TokeParser::Simple - Easy to use "HTML::TokeParser" interface

SYNOPSIS

        use HTML::TokeParser::Simple;
        my $p = HTML::TokeParser::Simple->new( $somefile );

        while ( my $token = $p->get_token ) {
            # This prints all text in an HTML doc (i.e., it strips the HTML)
            next unless $token->is_text;
            print $token->as_is;
        }

DESCRIPTION

       "HTML::TokeParser" is an excellent module that's often used for parsing HTML.  However,
       the tokens returned are not exactly intuitive to parse:

        ["S",  $tag, $attr, $attrseq, $text]
        ["E",  $tag, $text]
        ["T",  $text, $is_data]
        ["C",  $text]
        ["D",  $text]
        ["PI", $token0, $text]

       To simplify this, "HTML::TokeParser::Simple" allows the user ask more intuitive (read:
       more self-documenting) questions about the tokens returned.

       You can also rebuild some tags on the fly.  Frequently, the attributes associated with
       start tags need to be altered, added to, or deleted.  This functionality is built in.

       Since this is a subclass of "HTML::TokeParser", all "HTML::TokeParser" methods are
       available.  To truly appreciate the power of this module, please read the documentation
       for "HTML::TokeParser" and "HTML::Parser".

CONTRUCTORS

   "new($source)"
       The constructor for "HTML::TokeParser::Simple" can be used just like "HTML::TokeParser"'s
       constructor:

         my $parser = HTML::TokeParser::Simple->new($filename);
         # or
         my $parser = HTML::TokeParser::Simple->new($filehandle);
         # or
         my $parser = HTML::TokeParser::Simple->new(\$html_string);

   "new($source_type, $source)"
       If you wish to be more explicit, there is a new style of constructor available.

         my $parser = HTML::TokeParser::Simple->new(file   => $filename);
         # or
         my $parser = HTML::TokeParser::Simple->new(handle => $filehandle);
         # or
         my $parser = HTML::TokeParser::Simple->new(string => $html_string);

       Note that you do not have to provide a reference for the string if using the string
       constructor.

       As a convenience, you can also attempt to fetch the HTML directly from a URL.

         my $parser = HTML::TokeParser::Simple->new(url => 'http://some.url');

       This method relies on "LWP::Simple".  If this module is not found or the page cannot be
       fetched, the constructor will "croak()".

PARSER METHODS

   get_token
       This method will return the next token that "HTML::TokeParser::get_token()" method would
       return.  However, it will be blessed into a class appropriate which represents the token
       type.

   get_tag
       This method will return the next token that "HTML::TokeParser::get_tag()" method would
       return.  However, it will be blessed into either the
       HTML::TokeParser::Simple::Token::Tag::Start or HTML::TokeParser::Simple::Token::Tag::End
       class.

   peek
       As of version 3.14, you can now "peek()" at the upcomings tokens without affecting the
       state of the parser.  By default, "peek()" will return the text of the next token, but
       specifying an integer $count will return the text of the next $count tokens.

       This is useful when you're trying to debug where you are in a document.

        warn $parser->peek(3); # show the next 3 tokens

ACCESSORS

       The following methods may be called on the token object which is returned, not on the
       parser object.

   Boolean Accessors
       These accessors return true or false.

       •   "is_tag([$tag])"

           Use this to determine if you have any tag.  An optional "tag type" may be passed.
           This will allow you to match if it's a particular tag.  The supplied tag is case-
           insensitive.

            if ( $token->is_tag ) { ... }

           Optionally, you may pass a regular expression as an argument.

       •   "is_start_tag([$tag])"

           Use this to determine if you have a start tag.  An optional "tag type" may be passed.
           This will allow you to match if it's a particular start tag.  The supplied tag is
           case-insensitive.

            if ( $token->is_start_tag ) { ... }
            if ( $token->is_start_tag( 'font' ) ) { ... }

           Optionally, you may pass a regular expression as an argument.  To match all header
           (h1, h2, ... h6) tags:

            if ( $token->is_start_tag( qr/^h[123456]$/ ) ) { ... }

       •   "is_end_tag([$tag])"

           Use this to determine if you have an end tag.  An optional "tag type" may be passed.
           This will allow you to match if it's a particular end tag.  The supplied tag is case-
           insensitive.

           When testing for an end tag, the forward slash on the tag is optional.

            while ( $token = $p->get_token ) {
              if ( $token->is_end_tag( 'form' ) ) { ... }
            }

           Or:

            while ( $token = $p->get_token ) {
              if ( $token->is_end_tag( '/form' ) ) { ... }
            }

           Optionally, you may pass a regular expression as an argument.

       •   "is_text()"

           Use this to determine if you have text.  Note that this is not to be confused with the
           "return_text" (deprecated) method described below!  "is_text" will identify text that
           the user typically sees display in the Web browser.

       •   "is_comment()"

           Are you still reading this?  Nobody reads POD.  Don't you know you're supposed to go
           to CLPM, ask a question that's answered in the POD and get flamed?  It's a rite of
           passage.

           Really.

           "is_comment" is used to identify comments.  See the HTML::Parser documentation for
           more information about comments.  There's more than you might think.

       •   "is_declaration()"

           This will match the DTD at the top of your HTML. (You do use DTD's, don't you?)

       •   "is_process_instruction()"

           Process Instructions are from XML.  This is very handy if you need to parse out PHP
           and similar things with a parser.

           Currently, there appear to be some problems with process instructions.  You can
           override "HTML::TokeParser::Simple::Token::ProcessInstruction" if you need to.

       •   "is_pi()"

           This is a shorthand for "is_process_instruction()".

   Data Accessors
       Some of these were originally "return_" methods, but that name was not only unwieldy, but
       also went against reasonable conventions.  The "get_" methods listed below still have
       "return_" methods available for backwards compatibility reasons, but they merely call
       their "get_" counterpart.  For example, calling "return_tag()" actually calls "get_tag()"
       internally.

       •   "get_tag()"

           Do you have a start tag or end tag?  This will return the type (lower case).  Note
           that this is not the same as the "get_tag()" method on the actual parser object.

       •   "get_attr([$attribute])"

           If you have a start tag, this will return a hash ref with the attribute names as keys
           and the values as the values.

           If you pass in an attribute name, it will return the value for just that attribute.

           Returns false if the token is not a start tag.

       •   "get_attrseq()"

           For a start tag, this is an array reference with the sequence of the attributes, if
           any.

           Returns false if the token is not a start tag.

       •   "return_text()"

           This method has been heavily deprecated (for a couple of years) in favor of "as_is".
           Programmers were getting confused over the difference between "is_text",
           "return_text", and some parser methods such as "HTML::TokeParser::get_text" and
           friends.

           Using this method still succeeds, but will now carp and will be removed in the next
           major release of this module.

       •   "as_is()"

           This is the exact text of whatever the token is representing.

       •   "get_token0()"

           For processing instructions, this will return the token found immediately after the
           opening tag.  Example:  For <?php, "php" will be the start of the returned string.

           Note that process instruction handling appears to be incomplete in "HTML::TokeParser".

           Returns false if the token is not a process instruction.

MUTATORS

       The "delete_attr()" and "set_attr()" methods allow the programmer to rewrite start tag
       attributes on the fly.  It should be noted that bad HTML will be "corrected" by this.
       Specifically, the new tag will have all attributes lower-cased with the values properly
       quoted.

       Self-closing tags (e.g. <hr />) are also handled correctly.  Some older browsers require a
       space prior to the final slash in a self-closed tag.  If such a space is detected in the
       original HTML, it will be preserved.

       Calling a mutator on an token type that does not support that property is a no-op.  For
       example:

        if ($token->is_comment) {
           $token->set_attr(foo => 'bar'); # does nothing
        }

       •   "delete_attr($name)"

           This method attempts to delete the attribute specified.  It will silently fail if
           called on anything other than a start tag.  The argument is case-insensitive, but must
           otherwise be an exact match of the attribute you are attempting to delete.  If the
           attribute is not found, the method will return without changing the tag.

            # <body bgcolor="#FFFFFF">
            $token->delete_attr('bgcolor');
            print $token->as_is;
            # <body>

           After this method is called, if successful, the "as_is()", "get_attr()" and
           "get_attrseq()" methods will all return updated results.

       •   "set_attr($name,$value)"

           This method will set the value of an attribute.  If the attribute is not found, then
           "get_attrseq()" will have the new attribute listed at the end.

            # <p>
            $token->set_attr(class => 'some_class');
            print $token->as_is;
            # <p class="some_class">

            # <body bgcolor="#FFFFFF">
            $token->set_attr('bgcolor','red');
            print $token->as_is;
            # <body bgcolor="red">

           After this method is called, if successful, the "as_is()", "get_attr()" and
           "get_attrseq()" methods will all return updated results.

       •   "set_attr($hashref)"

           Under the premise that "set_" methods should accept what their corresponding "get_"
           methods emit, the following works:

             $tag->set_attr($tag->get_attr);

           Theoretically that's a no-op and for purposes of rendering HTML, it should be.
           However, internally this calls "$tag->rewrite_tag", so see that method to understand
           how this may affect you.

           Of course, this is useless if you want to actually change the attributes, so you can
           do this:

             my $attrs = {
               class  => 'headline',
               valign => 'top'
             };
             $token->set_attr($attrs)
               if $token->is_start_tag('td') &&  $token->get_attr('class') eq 'stories';

       •   "rewrite_tag()"

           This method rewrites the tag.  The tag name and the name of all attributes will be
           lower-cased.  Values that are not quoted with double quotes will be.  This may be
           called on both start or end tags.  Note that both "set_attr()" and "delete_attr()"
           call this method prior to returning.

           If called on a token that is not a tag, it simply returns.  Regardless of how it is
           called, it returns the token.

            # <body alink=#0000ff BGCOLOR=#ffffff class='none'>
            $token->rewrite_tag;
            print $token->as_is;
            # <body alink="#0000ff" bgcolor="#ffffff" class="none">

           A quick cleanup of sloppy HTML is now the following:

            my $parser = HTML::TokeParser::Simple->new( string => $ugly_html );
            while (my $token = $parser->get_token) {
                $token->rewrite_tag;
                print $token->as_is;
            }

PARSER VERSUS TOKENS

       The parser returns tokens that are blessed into appropriate classes.  Some people get
       confused and try to call parser methods on tokens and token methods on the parser.  To
       prevent this, "HTML::TokeParser::Simple" versions 1.4 and above now bless all tokens into
       appropriate token classes.  Please keep this in mind while using this module (and many
       thanks to PodMaster <http://www.perlmonks.org/index.pl?node_id=107642> for pointing out
       this issue to me.)

EXAMPLES

   Finding comments
       For some strange reason, your Pointy-Haired Boss (PHB) is convinced that the graphics
       department is making fun of him by embedding rude things about him in HTML comments.  You
       need to get all HTML comments from the HTML.

        use strict;
        use HTML::TokeParser::Simple;

        my @html_docs = glob( "*.html" );

        open PHB, "> phbreport.txt" or die "Cannot open phbreport for writing: $!";

        foreach my $doc ( @html_docs ) {
            print "Processing $doc\n";
            my $p = HTML::TokeParser::Simple->new( file => $doc );
            while ( my $token = $p->get_token ) {
                next unless $token->is_comment;
                print PHB $token->as_is, "\n";
            }
        }

        close PHB;

   Stripping Comments
       Uh oh.  Turns out that your PHB was right for a change.  Many of the comments in the HTML
       weren't very polite.  Since your entire graphics department was just fired, it falls on
       you need to strip those comments from the HTML.

        use strict;
        use HTML::TokeParser::Simple;

        my $new_folder = 'no_comment/';
        my @html_docs  = glob( "*.html" );

        foreach my $doc ( @html_docs ) {
            print "Processing $doc\n";
            my $new_file = "$new_folder$doc";

            open PHB, "> $new_file" or die "Cannot open $new_file for writing: $!";

            my $p = HTML::TokeParser::Simple->new( $file => doc );
            while ( my $token = $p->get_token ) {
                next if $token->is_comment;
                print PHB $token->as_is;
            }
            close PHB;
        }

   Changing form tags
       Your company was foo.com and now is bar.com.  Unfortunately, whoever wrote your HTML
       decided to hardcode "http://www.foo.com/" into the "action" attribute of the form tags.
       You need to change it to "http://www.bar.com/".

        use strict;
        use HTML::TokeParser::Simple;

        my $new_folder = 'new_html/';
        my @html_docs  = glob( "*.html" );

        foreach my $doc ( @html_docs ) {
            print "Processing $doc\n";
            my $new_file = "$new_folder$doc";

            open FILE, "> $new_file" or die "Cannot open $new_file for writing: $!";

            my $p = HTML::TokeParser::Simple->new( file => $doc );
            while ( my $token = $p->get_token ) {
                if ( $token->is_start_tag('form') ) {
                    my $action = $token->get_attr(action);
                    $action =~ s/www\.foo\.com/www.bar.com/;
                    $token->set_attr('action', $action);
                }
                print FILE $token->as_is;
            }
            close FILE;
        }

CAVEATS

       For compatibility reasons with "HTML::TokeParser", methods that return references are
       violating encapsulation and altering the references directly will alter the state of the
       object.  Subsequent calls to "rewrite_tag()" can thus have unexpected results.  Do not
       alter these references directly unless you are following behavior described in these docs.
       In the future, certain methods such as "get_attr", "get_attrseq" and others may return a
       copy of the reference rather than the original reference.  This behavior has not yet been
       changed in order to maintain compatibility with previous versions of this module.  At the
       present time, your author is not aware of anyone taking advantage of this "feature," but
       it's better to be safe than sorry.

       Use of $HTML::Parser::VERSION which is less than 3.25 may result in incorrect behavior as
       older versions do not always handle XHTML correctly.  It is the programmer's
       responsibility to verify that the behavior of this code matches the programmer's needs.

       Note that "HTML::Parser" processes text in 512 byte chunks.  This sometimes will cause
       strange behavior and cause text to be broken into more than one token.  You can suppress
       this behavior with the following command:

        $p->unbroken_text( [$bool] );

       See the "HTML::Parser" documentation and http://www.perlmonks.org/index.pl?node_id=230667
       for more information.

BUGS

       There are no known bugs, but that's no guarantee.

       Address bug reports and comments to: <eop_divo_sitruc@yahoo.com>.  When sending bug
       reports, please provide the version of "HTML::Parser", "HTML::TokeParser",
       "HTML::TokeParser::Simple", the version of Perl, and the version of the operating system
       you are using.

       Reverse the name to email the author.

SUBCLASSING

       You may wish to change the behavior of this module.  You probably do not want to subclass
       "HTML::TokeParser::Simple".  Instead, you'll want to subclass one of the token classes.
       "HTML::TokeParser::Simple::Token" is the base class for all tokens.  Global behavioral
       changes should go there.  Otherwise, see the appropriate token class for the behavior you
       wish to alter.

SEE ALSO

       HTML::TokeParser::Simple::Token

       HTML::TokeParser::Simple::Token::Tag

       HTML::TokeParser::Simple::Token::Text

       HTML::TokeParser::Simple::Token::Comment

       HTML::TokeParser::Simple::Token::Declaration

       HTML::TokeParser::Simple::Token::ProcessInstruction

COPYRIGHT

       Copyright (c) 2004 by Curtis "Ovid" Poe.  All rights reserved.  This program is free
       software; you may redistribute it and/or modify it under the same terms as Perl itself

AUTHOR

       Curtis "Ovid" Poe <eop_divo_sitruc@yahoo.com>

       Reverse the name to email the author.