Ubuntu Manpage: page - Parser Generator

NAME

       page - Parser Generator

SYNOPSIS

       page ?options...? ?input ?output??

_________________________________________________________________________________________________

DESCRIPTION

The application described by this document, page, is actually not just a parser generator,
as the name implies, but a generic tool for the execution of arbitrary transformations on
texts.

Its genericity comes through the use of plugins for reading, transforming, and writing
data, and the predefined set of plugins provided by Tcllib is for the generation of
memoizing recursive descent parsers (aka packrat parsers) from grammar specifications
(Parsing Expression Grammars).

page is written on top of the package page::pluginmgr, wrapping its functionality into a
command line based application. All the other page::* packages are plugin and/or
supporting packages for the generation of parsers. The parsers themselves are based on the
packages grammar::peg, grammar::peg::interp, and grammar::mengine.

COMMAND LINE
page ?options...? ?input ?output??
This is general form for calling page. The application will read the contents of
the file input, process them under the control of the specified options, and then
write the result to the file output.

If input is the string - the data to process will be read from stdin instead of a
file. Analogously the result will be written to stdout instead of a file if output
is the string -. A missing output or input specification causes the application to
assume -.

The detailed specifications of the recognized options are provided in section
OPTIONS.

path input (in)
This argument specifies the path to the file to be processed by the
application, or -. The last value causes the application to read the text
from stdin. Otherwise it has to exist, and be readable. If the argument is
missing - is assumed.

path output (in)
This argument specifies where to write the generated text. It can be the
path to a file, or -. The last value causes the application to write the
generated documented to stdout.

If the file output does not exist then [file dirname $output] has to exist
and must be a writable directory, as the application will create the fileto
write to.

If the argument is missing - is assumed.

OPERATION
... reading ... transforming ... writing - plugins - pipeline ...

OPTIONS
This section describes all the options available to the user of the application. Options
are always processed in order. I.e. of both --help and --version are specified the option
encountered first has precedence.

Unknown options specified before any of the options -rd, -wr, or -tr will cause processing
to abort with an error. Unknown options coming in between these options, or after the last
of them are assumed to always take a single argument and are associated with the last
plugin option coming before them. They will be checked after all the relevant plugins, and
thus the options they understand, are known. I.e. such unknown options cause error if and
only if the plugin option they are associated with does not understand them, and was not
superceded by a plugin option coming after.

Default options are used if and only if the command line did not contain any options at
all. They will set the application up as a PEG-based parser generator. The exact list of
options is

-c peg

And now the recognized options and their arguments, if they have any:

--help

-h

-? When one of these options is found on the command line all arguments coming before
or after are ignored. The application will print a short description of the
recognized options and exit.

--version

-V When one of these options is found on the command line all arguments coming before
or after are ignored. The application will print its own revision and exit.

-P This option signals the application to activate visual feedback while reading the
input.

-T This option signals the application to collect statistics while reading the input
and to print them after reading has completed, before processing started.

-D This option signals the application to activate logging in the Safe base, for the
debugging of problems with plugins.

-r parser

-rd parser

--reader parser
These options specify the plugin the application has to use for reading the input.
If the options are used multiple times the last one will be used.

-w generator

-wr generator

--writer generator
These options specify the plugin the application has to use for generating and
writing the final output. If the options are used multiple times the last one will
be used.

-t process

-tr process

--transform process
These options specify a plugin to run on the input. In contrast to readers and
writers each use will not supersede previous uses, but add each chosen plugin to a
list of transformations, either at the front, or the end, per the last seen use of
either option -p or -a. The initial default is to append the new transformations.

-a

--append
These options signal the application that all following transformations should be
added at the end of the list of transformations.

-p

--prepend
These options signal the application that all following transformations should be
added at the beginning of the list of transformations.

--reset
This option signals the application to clear the list of transformations. This is
necessary to wipe out the default transformations used.

-c file

--configuration file
This option causes the application to load a configuration file and/or plugin. This
is a plugin which in essence provides a pre-defined set of commandline options.
They are processed exactly as if they have been specified in place of the option
and its arguments. This means that unknown options found at the beginning of the
configuration file are associated with the last plugin, even if that plugin was
specified before the configuration file itself. Conversely, unknown options coming
after the configuration file can be associated with a plugin specified in the file.

If the argument is a file which cannot be loaded as a plugin the application will
assume that its contents are a list of options and their arguments, separated by
space, tabs, and newlines. Options and argumentes containing spaces can be quoted
via double-quotes (") and quotes ('). The quote character can be specified within
in a quoted string by doubling it. Newlines in a quoted string are accepted as is.

PLUGINS
page makes use of four different types of plugins, namely: readers, writers,
transformations, and configurations. Here we provide only a basic introduction on how to
use them from page. The exact APIs provided to and expected from the plugins can be found
in the documentation for page::pluginmgr, for those who wish to write their own plugins.

Plugins are specified as arguments to the options -r, -w, -t, -c, and their equivalent
longer forms. See the section OPTIONS for reference.

Each such argument will be first treated as the name of a file and this file is loaded as
the plugin. If however there is no file with that name, then it will be translated into
the name of a package, and this package is then loaded. For each type of plugins the
package management searches not only the regular paths, but a set application- and type-
specific paths as well. Please see the section PLUGIN LOCATIONS for a listing of all paths
and their sources.

-c name
Configurations. The name of the package for the plugin name is
"page::config::name".

We have one predefined plugin:

peg It sets the application up as a parser generator accepting parsing
expression grammars and writing a packrat parser in Tcl. The actual
arguments it specifies are:

--reset
--append
--reader peg
--transform reach
--transform use
--writer me

-r name
Readers. The name of the package for the plugin name is "page::reader::name".

We have five predefined plugins:

peg Interprets the input as a parsing expression grammar (PEG) and generates a
tree representation for it. Both the syntax of PEGs and the structure of the
tree representation are explained in their own manpages.

hb Interprets the input as Tcl code as generated by the writer plugin hb and
generates its tree representation.

ser Interprets the input as the serialization of a PEG, as generated by the
writer plugin ser, using the package grammar::peg.

lemon Interprets the input as a grammar specification as understood by Richard
Hipp's LEMON parser generator and generates a tree representation for it.
Both the input syntax and the structure of the tree representation are
explained in their own manpages.

treeser
Interprets the input as the serialization of a struct::tree. It is validated
as such, but nothing else. It is not assumed to be the tree representation
of a grammar.

-w name
Writers. The name of the package for the plugin name is "page::writer::name".

We have eight predefined plugins:

identity
Simply writes the incoming data as it is, without making any changes. This
is good for inspecting the raw result of a reader or transformation.

null Generates nothing, and ignores the incoming data structure.

tree Assumes that the incoming data structure is a struct::tree and generates an
indented textual representation of all nodes, their parental relationships,
and their attribute information.

peg Assumes that the incoming data structure is a tree representation of a PEG
or other other grammar and writes it out as a PEG. The result is nicely
formatted and partially simplified (strings as sequences of characters). A
pretty printer in essence, but can also be used to obtain a canonical
representation of the input grammar.

tpc Assumes that the incoming data structure is a tree representation of a PEG
or other other grammar and writes out Tcl code defining a package which
defines a grammar::peg object containing the grammar when it is loaded into
an interpreter.

hb This is like the writer plugin tpc, but it writes only the statements which
define stat expression and grammar rules. The code making the result a
package is left out.

ser Assumes that the incoming data structure is a tree representation of a PEG
or other other grammar, transforms it internally into a grammar::peg object
and writes out its serialization.

me Assumes that the incoming data structure is a tree representation of a PEG
or other other grammar and writes out Tcl code defining a package which
implements a memoizing recursive descent parser based on the match engine
(ME) provided by the package grammar::mengine.

-t name
Transformers. The name of the package for the plugin name is
"page::transform::name".

We have two predefined plugins:

reach Assumes that the incoming data structure is a tree representation of a PEG
or other other grammar. It determines which nonterminal symbols and rules
are reachable from start-symbol/expression. All nonterminal symbols which
were not reached are removed.

use Assumes that the incoming data structure is a tree representation of a PEG
or other other grammar. It determines which nonterminal symbols and rules
are able to generate a finite sequences of terminal symbols (in the sense
for a Context Free Grammar). All nonterminal symbols which were not deemed
useful in this sense are removed.

PLUGIN LOCATIONS
The application-specific paths searched by page either are, or come from:

[1] The directory "~/.page/plugin"

[2] The environment variable PAGE_PLUGINS

[3] The registry entry HKEY_LOCAL_MACHINE\SOFTWARE\PAGE\PLUGINS

[4] The registry entry HKEY_CURRENT_USER\SOFTWARE\PAGE\PLUGINS

The type-specific paths searched by page either are, or come from:

[1] The directory "~/.page/plugin/<TYPE>"

[2] The environment variable PAGE_<TYPE>_PLUGINS

[3] The registry entry HKEY_LOCAL_MACHINE\SOFTWARE\PAGE\<TYPE>\PLUGINS

[4] The registry entry HKEY_CURRENT_USER\SOFTWARE\PAGE\<TYPE>\PLUGINS

Where the placeholder <TYPE> is always one of the values below, properly capitalized.

[1] reader

[2] writer

[3] transform

[4] config

The registry entries are specific to the Windows(tm) platform, all other platforms will
ignore them.

The contents of both environment variables and registry entries are interpreted as a list
of paths, with the elements separated by either colon (Unix), or semicolon (Windows).

BUGS, IDEAS, FEEDBACK

       This  document,  and  the  package  it  describes, will undoubtedly contain bugs and other
       problems.   Please  report  such  in  the   category   page   of   the   Tcllib   Trackers
       [http://core.tcl.tk/tcllib/reportlist].  Please also report any ideas for enhancements you
       may have for either package and/or documentation.

       When proposing code changes, please provide unified diffs, i.e the output of diff -u.

       Note further that attachments are strongly preferred over inlined patches. Attachments can
       be  made  by going to the Edit form of the ticket immediately after its creation, and then
       using the left-most button in the secondary navigation bar.

KEYWORDS

       parser generator, text processing

COPYRIGHT

       Copyright (c) 2005 Andreas Kupries <andreas_kupries@users.sourceforge.net>