Provided by: tcllib_1.21+dfsg-1_all 
NAME
docstrip_util - Docstrip-related utilities
SYNOPSIS
package require Tcl 8.4
package require docstrip ?1.2?
package require docstrip::util ?1.3.1?
pkgProvide name version terminals
pkgIndex ?terminal ...?
fileoptions ?option value ...?
docstrip::util::index_from_catalogue dir pattern ?option value ...?
docstrip::util::modules_from_catalogue target source ?option value ...?
docstrip::util::classical_preamble metaprefix message target ?source terminals ...?
docstrip::util::classical_postamble metaprefix message target ?source terminals ...?
docstrip::util::packages_provided text ?setup-script?
docstrip::util::ddt2man text
docstrip::util::guards subcmd text
docstrip::util::patch source-var terminals fromtext diff ?option value ...?
docstrip::util::thefile filename ?option value ...?
docstrip::util::import_unidiff diff-text ?warning-var?
_________________________________________________________________________________________________
DESCRIPTION
The docstrip::util package is meant for collecting various utility procedures that are
mainly useful at installation or development time. It is separate from the base package to
avoid overhead when the latter is used to source code.
PACKAGE INDEXING COMMANDS
Like raw ".tcl" files, code lines in docstrip source files can be searched for package
declarations and corresponding indices constructed. A complication is however that one
cannot tell from the code blocks themselves which will fit together to make a working
package; normally that information would be found in an accompanying ".ins" file, but
parsing one of those is not an easy task. Therefore docstrip::util introduces an
alternative encoding of such information, in the form of a declarative Tcl script: the
catalogue (of the contents in a source file).
The special commands which are available inside a catalogue are:
pkgProvide name version terminals
Declares that the code for a package with name name and version version is made up
from those modules in the source file which are selected by the terminals list of
guard expression terminals. This code should preferably not contain a package
provide command for the package, as one will be provided by the package loading
mechanisms.
pkgIndex ?terminal ...?
Declares that the code for a package is made up from those modules in the source
file which are selected by the listed guard expression terminals. The name and
version of this package is determined from package provide command(s) found in that
code (hence there must be such a command in there).
fileoptions ?option value ...?
Declares the fconfigure options that should be in force when reading the source;
this can usually be ignored for pure ASCII files, but if the file needs to be
interpreted according to some other -encoding then this is how to specify it. The
command should normally appear first in the catalogue, as it takes effect only for
commands following it.
Other Tcl commands are supported too — a catalogue is parsed by being evaluated in a safe
interpreter — but they are rarely needed. To allow for future extensions, unknown commands
in the catalogue are silently ignored.
To simplify distribution of catalogues together with their source files, the catalogue is
stored in the source file itself as a module selected by the terminal
'docstrip.tcl::catalogue'. This supports both the style of collecting all catalogue lines
in one place and the style of putting each catalogue line in close proximity of the code
that it declares.
Putting catalogue entries next to the code they declare may look as follows
% First there's the catalogue entry
% \begin{tcl}
%<docstrip.tcl::catalogue>pkgProvide foo::bar 1.0 {foobar load}
% \end{tcl}
% second a metacomment used to include a copyright message
% \begin{macrocode}
%<*foobar>
%% This file is placed in the public domain.
% \end{macrocode}
% third the package implementation
% \begin{tcl}
namespace eval foo::bar {
# ... some clever piece of Tcl code elided ...
% \end{tcl}
% which at some point may have variant code to make use of a
% |load|able extension
% \begin{tcl}
%<*load>
load [file rootname [info script]][info sharedlibextension]
%</load>
%<*!load>
# ... even more clever scripted counterpart of the extension
# also elided ...
%</!load>
}
%</foobar>
% \end{tcl}
% and that's it!
The corresponding set-up with pkgIndex would be
% First there's the catalogue entry
% \begin{tcl}
%<docstrip.tcl::catalogue>pkgIndex foobar load
% \end{tcl}
% second a metacomment used to include a copyright message
% \begin{tcl}
%<*foobar>
%% This file is placed in the public domain.
% \end{tcl}
% third the package implementation
% \begin{tcl}
package provide foo::bar 1.0
namespace eval foo::bar {
# ... some clever piece of Tcl code elided ...
% \end{tcl}
% which at some point may have variant code to make use of a
% |load|able extension
% \begin{tcl}
%<*load>
load [file rootname [info script]][info sharedlibextension]
%</load>
%<*!load>
# ... even more clever scripted counterpart of the extension
# also elided ...
%</!load>
}
%</foobar>
% \end{tcl}
% and that's it!
docstrip::util::index_from_catalogue dir pattern ?option value ...?
This command is a sibling of the standard pkg_mkIndex command, in that it adds
package entries to "pkgIndex.tcl" files. The difference is that it indexes
docstrip-style source files rather than raw ".tcl" or loadable library files. Only
packages listed in the catalogue of a file are considered.
The dir argument is the directory in which to look for files (and whose
"pkgIndex.tcl" file should be amended). The pattern argument is a glob pattern of
files to look into; a typical value would be *.dtx or *.{dtx,ddt}. Remaining
arguments are option-value pairs, where the supported options are:
-recursein dirpattern
If this option is given, then the index_from_catalogue operation will be
repeated in each subdirectory whose name matches the dirpattern. -recursein
* will cause the entire subtree rooted at dir to be indexed.
-sourceconf dictionary
Specify fileoptions to use when reading the catalogues of files (and also
for reading the packages if the catalogue does not contain a fileoptions
command). Defaults to being empty. Primarily useful if your system encoding
is very different from that of the source file (e.g., one is a two-byte
encoding and the other is a one-byte encoding). ascii and utf-8 are not very
different in that sense.
-options terminals
The terminals is a list of terminals in addition to docstrip.tcl::catalogue
that should be held as true when extracting the catalogue. Defaults to being
empty. This makes it possible to make use of "variant sections" in the
catalogue itself, e.g. gaurd some entries with an extra "experimental" and
thus prevent them from appearing in the index unless that is generated with
"experimental" among the -options.
-report boolean
If the boolean is true then the return value will be a textual, probably
multiline, report on what was done. Defaults to false, in which case there
is no particular return value.
-reportcmd commandPrefix
Every item in the report is handed as an extra argument to the command
prefix. Since index_from_catalogue would typically be used at a rather high
level in installation scripts and the like, the commandPrefix defaults to
"puts stdout". Use list to effectively disable this feature. The return
values from the prefix are ignored.
The package ifneeded scripts that are generated contain one package require
docstrip command and one docstrip::sourcefrom command. If the catalogue entry was
of the pkgProvide kind then the package ifneeded script also contains the package
provide command.
Note that index_from_catalogue never removes anything from an existing
"pkgIndex.tcl" file. Hence you may need to delete it (or have pkg_mkIndex recreate
it from scratch) before running index_from_catalogue to update some piece of
information, such as a package version number.
docstrip::util::modules_from_catalogue target source ?option value ...?
This command is an alternative to index_from_catalogue which creates Tcl Module
(".tm") files rather than "pkgIndex.tcl" entries. Since this action is more similar
to what docstrip classically does, it has features for putting pre- and postambles
on the generated files.
The source argument is the name of the source file to generate ".tm" files from.
The target argument is the directory which should count as a module path, i.e.,
this is what the relative paths derived from package names are joined to. The
supported options are:
-preamble message
A message to put in the preamble (initial block of comments) of generated
files. Defaults to a space. May be several lines, which are then separated
by newlines. Traditionally used for copyright notices or the like, but
metacomment lines provide an alternative to that.
-postamble message
Like -preamble, but the message is put at the end of the file instead of the
beginning. Defaults to being empty.
-sourceconf dictionary
Specify fileoptions to use when reading the catalogue of the source (and
also for reading the packages if the catalogue does not contain a
fileoptions command). Defaults to being empty. Primarily useful if your
system encoding is very different from that of the source file (e.g., one is
a two-byte encoding and the other is a one-byte encoding). ascii and utf-8
are not very different in that sense.
-options terminals
The terminals is a list of terminals in addition to docstrip.tcl::catalogue
that should be held as true when extracting the catalogue. Defaults to being
empty. This makes it possible to make use of "variant sections" in the
catalogue itself, e.g. gaurd some entries with an extra "experimental" guard
and thus prevent them from contributing packages unless those are generated
with "experimental" among the -options.
-formatpreamble commandPrefix
Command prefix used to actually format the preamble. Takes four additional
arguments message, targetFilename, sourceFilename, and terminalList and
returns a fully formatted preamble. Defaults to using classical_preamble
with a metaprefix of '##'.
-formatpostamble commandPrefix
Command prefix used to actually format the postamble. Takes four additional
arguments message, targetFilename, sourceFilename, and terminalList and
returns a fully formatted postamble. Defaults to using classical_postamble
with a metaprefix of '##'.
-report boolean
If the boolean is true (which is the default) then the return value will be
a textual, probably multiline, report on what was done. If it is false then
there is no particular return value.
-reportcmd commandPrefix
Every item in the report is handed as an extra argument to this command
prefix. Defaults to list, which effectively disables this feature. The
return values from the prefix are ignored. Use for example "puts stdout" to
get report items written immediately to the terminal.
An existing file of the same name as one to be created will be overwritten.
docstrip::util::classical_preamble metaprefix message target ?source terminals ...?
This command returns a preamble in the classical docstrip style
##
## This is `TARGET',
## generated by the docstrip::util package.
##
## The original source files were:
##
## SOURCE (with options: `foo,bar')
##
## Some message line 1
## line2
## line3
if called as
docstrip::util::classical_preamble {##}\
"\nSome message line 1\nline2\nline3" TARGET SOURCE {foo bar}
The command supports preambles for files generated from multiple sources, even
though modules_from_catalogue at present does not need that.
docstrip::util::classical_postamble metaprefix message target ?source terminals ...?
This command returns a postamble in the classical docstrip style
## Some message line 1
## line2
## line3
##
## End of file `TARGET'.
if called as
docstrip::util::classical_postamble {##}\
"Some message line 1\nline2\nline3" TARGET SOURCE {foo bar}
In other words, the source and terminals arguments are ignored, but supported for
symmetry with classical_preamble.
docstrip::util::packages_provided text ?setup-script?
This command returns a list where every even index element is the name of a package
provided by text when that is evaluated as a Tcl script, and the following odd
index element is the corresponding version. It is used to do package indexing of
extracted pieces of code, in the manner of pkg_mkIndex.
One difference to pkg_mkIndex is that the text gets evaluated in a safe
interpreter. package require commands are silently ignored, as are unknown commands
(which includes source and load). Other errors cause processing of the text to
stop, in which case only those package declarations that had been encountered
before the error will be included in the return value.
The setup-script argument can be used to customise the evaluation environment, if
the code in text has some very special needs. The setup-script is evaluated in the
local context of the packages_provided procedure just before the text is processed.
At that time, the name of the slave command for the safe interpreter that will do
this processing is kept in the local variable c. To for example copy the contents
of the ::env array to the safe interpreter, one might use a setup-script of
$c eval [list array set env [array get ::env]]
SOURCE PROCESSING COMMANDS
Unlike the previous group of commands, which would use docstrip::extract to extract some
code lines and then process those further, the following commands operate on text
consisting of all types of lines.
docstrip::util::ddt2man text
The ddt2man command reformats text from the general docstrip format to doctools
".man" format (Tcl Markup Language for Manpages). The different line types are
treated as follows:
comment and metacomment lines
The '%' and '%%' prefixes are removed, the rest of the text is kept as it
is.
empty lines
These are kept as they are. (Effectively this means that they will count as
comment lines after a comment line and as code lines after a code line.)
code lines
example_begin and example_end commands are placed at the beginning and end
of every block of consecutive code lines. Brackets in a code line are
converted to lb and rb commands.
verbatim guards
These are processed as usual, so they do not show up in the result but every
line in a verbatim block is treated as a code line.
other guards
These are treated as code lines, except that the actual guard is emphasised.
At the time of writing, no project has employed doctools markup in master source
files, so experience of what works well is not available. A source file could
however look as follows
% [manpage_begin gcd n 1.0]
% [keywords divisor]
% [keywords math]
% [moddesc {Greatest Common Divisor}]
% [require gcd [opt 1.0]]
% [description]
%
% [list_begin definitions]
% [call [cmd gcd] [arg a] [arg b]]
% The [cmd gcd] procedure takes two arguments [arg a] and [arg b] which
% must be integers and returns their greatest common divisor.
proc gcd {a b} {
% The first step is to take the absolute values of the arguments.
% This relieves us of having to worry about how signs will be treated
% by the remainder operation.
set a [expr {abs($a)}]
set b [expr {abs($b)}]
% The next line does all of Euclid's algorithm! We can make do
% without a temporary variable, since $a is substituted before the
% [lb]set a $b[rb] and thus continues to hold a reference to the
% "old" value of [var a].
while {$b>0} { set b [expr { $a % [set a $b] }] }
% In Tcl 8.3 we might want to use [cmd set] instead of [cmd return]
% to get the slight advantage of byte-compilation.
%<tcl83> set a
%<!tcl83> return $a
}
% [list_end]
%
% [manpage_end]
If the above text is fed through docstrip::util::ddt2man then the result will be a
syntactically correct doctools manpage, even though its purpose is a bit different.
It is suggested that master source code files with doctools markup are given the
suffix ".ddt", hence the "ddt" in ddt2man.
docstrip::util::guards subcmd text
The guards command returns information (mostly of a statistical nature) about the
ordinary docstrip guards that occur in the text. The subcmd selects what is
returned.
counts List the guard expression terminals with counts. The format of the return
value is a dictionary which maps the terminal name to the number of
occurencies of it in the file.
exprcount
List the guard expressions with counts. The format of the return value is a
dictionary which maps the expression to the number of occurencies of it in
the file.
exprerr
List the syntactically incorrect guard expressions (e.g. parentheses do not
match, or a terminal is missing). The return value is a list, with the
elements in no particular order.
expressions
List the guard expressions. The return value is a list, with the elements in
no particular order.
exprmods
List the guard expressions with modifiers. The format of the return value is
a dictionary where each index is a guard expression and each entry is a
string with one character for every guard line that has this expression. The
characters in the entry specify what modifier was used in that line: +, -,
*, /, or (for guard without modifier:) space. This is the most primitive
form of the information gathered by guards.
names List the guard expression terminals. The return value is a list, with the
elements in no particular order.
rotten List the malformed guard lines (this does not include lines where only the
expression is malformed, though). The format of the return value is a
dictionary which maps line numbers to their contents.
docstrip::util::patch source-var terminals fromtext diff ?option value ...?
This command tries to apply a diff file (for example a contributed patch) that was
computed for a generated file to the docstrip source. This can be useful if someone
has edited a generated file, thus mistaking it for being the source. This command
makes no presumptions which are specific for the case that the generated file is a
Tcl script.
patch requires that the source file to patch is kept as a list of lines in a
variable, and the name of that variable in the calling context is what goes into
the source-var argument. The terminals is the list of terminals used to extract
the file that has been patched. The diff is the actual diff to apply (in a format
as explained below) and the fromtext is the contents of the file which served as
"from" when the diff was computed. Options can be used to further control the
process.
The process works by "lifting" the hunks in the diff from generated to source file,
and then applying them to the elements of the source-var. In order to do this
lifting, it is necessary to determine how lines in the fromtext correspond to
elements of the source-var, and that is where the terminals come in; the source is
first extracted under the given terminals, and the result of that is then matched
against the fromtext. This produces a map which translates line numbers stated in
the diff to element numbers in source-var, which is what is needed to lift the
hunks.
The reason that both the terminals and the fromtext must be given is twofold.
First, it is very difficult to keep track of how many lines of preamble are
supplied some other way than by copying lines from source files. Second, a
generated file might contain material from several source files. Both make it
impossible to predict what line number an extracted file would have in the
generated file, so instead the algorithm for computing the line number map looks
for a block of lines in the fromtext which matches what can be extracted from the
source. This matching is affected by the following options:
-matching mode
How equal must two lines be in order to match? The supported modes are:
exact Lines must be equal as strings. This is the default.
anyspace
All sequences of whitespace characters are converted to single spaces
before comparing.
nonspace
Only non-whitespace characters are considered when comparing.
none Any two lines are considered to be equal.
-metaprefix string
The -metaprefix value to use when extracting. Defaults to "%%", but for Tcl
code it is more likely that "#" or "##" had been used for the generated
file.
-trimlines boolean
The -trimlines value to use when extracting. Defaults to true.
The return value is in the form of a unified diff, containing only those hunks
which were not applied or were only partially applied; a comment in the header of
each hunk specifies which case is at hand. It is normally necessary to manually
review both the return value from patch and the patched text itself, as this
command cannot adjust comment lines to match new content.
An example use would look like
set sourceL [split [docstrip::util::thefile from.dtx] \n]
set terminals {foo bar baz}
set fromtext [docstrip::util::thefile from.tcl]
set difftext [exec diff --unified from.tcl to.tcl]
set leftover [docstrip::util::patch sourceL $terminals $fromtext\
[docstrip::util::import_unidiff $difftext] -metaprefix {#}]
set F [open to.dtx w]; puts $F [join $sourceL \n]; close $F
return $leftover
Here, "from.dtx" was used as source for "from.tcl", which someone modified into
"to.tcl". We're trying to construct a "to.dtx" which can be used as source for
"to.tcl".
docstrip::util::thefile filename ?option value ...?
The thefile command opens the file filename, reads it to end, closes it, and
returns the contents (dropping a final newline if there is one). The option-value
pairs are passed on to fconfigure to configure the open file channel before
anything is read from it.
docstrip::util::import_unidiff diff-text ?warning-var?
This command parses a unified (diff flags -U and --unified) format diff into the
list-of-hunks format expected by docstrip::util::patch. The diff-text argument is
the text to parse and the warning-var is, if specified, the name in the calling
context of a variable to which any warnings about parsing problems will be
appended.
The return value is a list of hunks. Each hunk is a list of five elements "start1
end1 start2 end2 lines". start1 and end1 are line numbers in the "from" file of the
first and last respectively lines of the hunk. start2 and end2 are the
corresponding line numbers in the "to" file. Line numbers start at 1. The lines is
a list with two elements for each line in the hunk; the first specifies the type of
a line and the second is the actual line contents. The type is - for lines only in
the "from" file, + for lines that are only in the "to" file, and 0 for lines that
are in both.
SEE ALSO
docstrip, doctools, doctools_fmt
KEYWORDS
\.ddt, .dtx, LaTeX, Tcl module, catalogue, diff, docstrip, doctools, documentation,
literate programming, module, package indexing, patch, source
CATEGORY
Documentation tools
COPYRIGHT
Copyright (c) 2003–2010 Lars Hellström <Lars dot Hellstrom at residenset dot net>