Provided by: tcllib_1.19-dfsg-2_all bug

NAME

       htmlparse - Procedures to parse HTML strings

SYNOPSIS

       package require Tcl  8.2

       package require struct::stack  1.3

       package require cmdline  1.1

       package require htmlparse  ?1.2.2?

       ::htmlparse::parse ?-cmd cmd? ?-vroot tag? ?-split n? ?-incvar var? ?-queue q? html

       ::htmlparse::debugCallback ?clientdata? tag slash param textBehindTheTag

       ::htmlparse::mapEscapes html

       ::htmlparse::2tree html tree

       ::htmlparse::removeVisualFluff tree

       ::htmlparse::removeFormDefs tree

_________________________________________________________________________________________________

DESCRIPTION

       The  htmlparse  package  provides  commands that allow libraries and applications to parse
       HTML in a string into a representation of their choice.

       The following commands are available:

       ::htmlparse::parse ?-cmd cmd? ?-vroot tag? ?-split n? ?-incvar var? ?-queue q? html
              This command is the basic parser for HTML. It takes an HTML string, parses  it  and
              invokes  a  command  prefix  for every tag encountered. It is not necessary for the
              HTML to be valid for this parser to function.  It  is  the  responsibility  of  the
              command  invoked for every tag to check this. Another responsibility of the invoked
              command  is  the  handling  of  tag  attributes  and  character  entities  (escaped
              characters).  The  parser provides the un-interpreted tag attributes to the invoked
              command to aid in the former, and the package at large provides a  helper  command,
              ::htmlparse::mapEscapes,  to  aid  in  the  handling of the latter. The parser does
              ignore leading DOCTYPE declarations and all valid HTML comments it encounters.

              All information beyond the HTML string itself is specified via options,  these  are
              explained below.

              To help understand the options, some more background information about the parser.

              It  is  capable  of detecting incomplete tags in the HTML string given to it. Under
              normal circumstances this will cause the parser to  throw  an  error,  but  if  the
              option -incvar is used to specify a global (or namespace) variable, the parser will
              store the incomplete part of the input into this variable instead.  This  will  aid
              greatly  in  the handling of incrementally arriving HTML, as the parser will handle
              whatever it can and defer the handling of the incomplete part until more  data  has
              arrived.

              Another  feature  of the parser are its two possible modes of operation. The normal
              mode is activated if the option -queue is not present on the command line  invoking
              the parser. If it is present, the parser will go into the incremental mode instead.

              The  main  difference  is  that a parser in normal mode will immediately invoke the
              command prefix for each tag it encounters. In incremental mode however  the  parser
              will  generate  a  number  of scripts which invoke the command prefix for groups of
              tags in the HTML string and then store these scripts in the specified queue. It  is
              then  the responsibility of the caller of the parser to ensure the execution of the
              scripts in the queue.

              Note: The queue object given to the parser has to provide the same interface as the
              queue defined in tcllib -> struct. This means, for example, that all queues created
              via that tcllib module can be immediately used here. Still, the queue doesn't  have
              to come from tcllib -> struct as long as the same interface is provided.

              In both modes the parser will return an empty string to the caller.

              The  -split option may be given to a parser in incremental mode to specify the size
              of the groups it creates. In other words, -split 5 means that each of the generated
              scripts will invoke the command prefix for 5 consecutive tags in the HTML string. A
              parser in normal mode will ignore this option and its value.

              The option -vroot specifies a virtual root tag. A parser in normal mode will invoke
              the command prefix for it immediately before and after it processes the tags in the
              HTML, thus simulating that the HTML  string  is  enclosed  in  a  <vroot>  </vroot>
              combination.  In  incremental  mode  however  the  parser  is unable to provide the
              closing virtual root as it never knows when the input is complete. In this case the
              first  script generated by each invocation of the parser will contain an invocation
              of the command prefix for the virtual root as its  first  command.   The  following
              options are available:

              -cmd cmd
                     The  command  prefix to invoke for every tag in the HTML string. Defaults to
                     ::htmlparse::debugCallback.

              -vroot tag
                     The virtual root tag to add around the HTML in normal mode.  In  incremental
                     mode  it  is  the first tag in each chunk processed by the parser, but there
                     will be no closing tags. Defaults to hmstart.

              -split n
                     The size of the groups produced by an incremental mode parser. Ignored  when
                     in normal mode. Defaults to 10. Values <= 0 are not allowed.

              -incvar var
                     The name of the variable where to store any incomplete HTML into. This makes
                     most sense for the incremental mode. The parser will throw an  error  if  it
                     sees  incomplete  HTML and has no place to store it to. This makes sense for
                     the normal mode. Only  incomplete  tags  are  detected,  not  missing  tags.
                     Optional, defaults to 'no variable'.

              Interface to the command prefix
                     In normal mode the parser will invoke the command prefix with four arguments
                     appended. See ::htmlparse::debugCallback for a description.

                     In incremental mode, however, the generated scripts will invoke the  command
                     prefix  with  five  arguments  appended. The last four of these are the same
                     which were mentioned above. The first is a placeholder string (@win@) for  a
                     clientdata  value  to  be  supplied later during the actual execution of the
                     generated scripts. This could be a tk window path, for example. This  allows
                     the  user of this package to preprocess HTML strings without committing them
                     to a specific window, object, whatever during parsing. This  connection  can
                     be  made  later.  This  also means that it is possible to cache preprocessed
                     HTML. Of course, nothing prevents the user of the parser from replacing  the
                     placeholder with an empty string.

       ::htmlparse::debugCallback ?clientdata? tag slash param textBehindTheTag
              This  command  is the standard callback used by the parser in ::htmlparse::parse if
              none was specified by the user. It simply dumps  its  arguments  to  stdout.   This
              callback can be used for both normal and incremental mode of the calling parser. In
              other words, it accepts four  or  five  arguments.  The  last  four  arguments  are
              described  below.  The optional fifth argument contains the clientdata value passed
              to the callback by a parser in incremental mode. All callbacks have to  follow  the
              signature  of  this  command  in  the  last  four  arguments, and callbacks used in
              incremental parsing have to follow this signature in the last five arguments.

              The first argument, clientdata, is optional and present only  if  this  command  is
              invoked  by  a  parser  in  incremental mode. It contains whatever the user of this
              package wishes.

              The second argument, tag, contains the name of the tag which is currently processed
              by the parser.

              The third argument, slash, is either empty or contains a slash character. It allows
              the callback to distinguish between opening  (slash  is  empty)  and  closing  tags
              (slash contains a slash character).

              The  fourth  argument, param, contains the un-interpreted list of parameters to the
              tag.

              The fifth and last argument, textBehindTheTag,  contains  the  text  found  by  the
              parser behind the tag named in tag.

       ::htmlparse::mapEscapes html
              This  command  takes  a  HTML  string,  substitutes all escape sequences with their
              actual characters and then returns the resulting string.  HTML strings which do not
              contain escape sequences are returned unchanged.

       ::htmlparse::2tree html tree
              This  command is a wrapper around ::htmlparse::parse which takes an HTML string (in
              html) and converts it into a tree containing the logical structure  of  the  parsed
              document.  The  name  of  the  tree  is given to the command as its second argument
              (tree). The command does not generate the tree  by  itself  but  expects  that  the
              caller  provided  it  with  an  existing  and  empty tree. It also expects that the
              specified tree object follows the same interface as the tree object  in  tcllib  ->
              struct.  It  doesn't have to be from tcllib -> struct, but it must provide the same
              interface.

              The internal callback does some basic  checking  of  HTML  validity  and  tries  to
              recover  from the most basic errors. The command returns the contents of its second
              argument. Side effects are the creation and manipulation of a tree object.

              Each node in the generated tree represent one tag in the input. The name of the tag
              is  stored  in  the attribute type of the node. Any html attributes coming with the
              tag are stored unmodified in the attribute data of the tag.  In  other  words,  the
              command does not parse html attributes into their names and values.

              If  a  tag contains text its node will have children of type PCDATA containing this
              text. The text will be stored in the attribute data of these children.

       ::htmlparse::removeVisualFluff tree
              This command walks a tree as generated by ::htmlparse::2tree and  removes  all  the
              nodes  which  represent  visual  tags  and  not structural ones. The purpose of the
              command is to make the tree easier to  navigate  without  getting  bogged  down  in
              visual information not relevant to the search. Its only argument is the name of the
              tree to cut down.

       ::htmlparse::removeFormDefs tree
              Like ::htmlparse::removeVisualFluff this command is here to cut down on the size of
              the  tree  as  generated  by  ::htmlparse::2tree. It removes all nodes representing
              forms and form elements. Its only argument is the name of the tree to cut down.

BUGS, IDEAS, FEEDBACK

       This document, and the package it describes,  will  undoubtedly  contain  bugs  and  other
       problems.    Please  report  such  in  the  category  htmlparse  of  the  Tcllib  Trackers
       [http://core.tcl.tk/tcllib/reportlist].  Please also report any ideas for enhancements you
       may have for either package and/or documentation.

       When proposing code changes, please provide unified diffs, i.e the output of diff -u.

       Note further that attachments are strongly preferred over inlined patches. Attachments can
       be made by going to the Edit form of the ticket immediately after its creation,  and  then
       using the left-most button in the secondary navigation bar.

SEE ALSO

       struct::tree

KEYWORDS

       html, parsing, queue, tree

CATEGORY

       Text processing