Provided by: fcode-utils_1.0.2-7_amd64 
NAME
toke - OpenBIOS tokenizer
SYNOPSIS
toke [options] forth-file
DESCRIPTION
toke is an open-source FCode Tokenizer from the OpenBIOS project.
OPTIONS
-h, -? Print a brief help message and then exit.
-i, --ignore-errors
Generate a Binary Output even if errors were reported.
-l, --load-list
Collect the names of floaded files into an FLoad-List file. The names collected are
in the same form as they were presented in the fload statements.
The name of the FLoad-List File is derived from the name of the binary output file,
by replacing its extension with .fl , or, if the binary output file name had no
extension, merely appending the extension .fl.
The binary output file name used for this purpose is either the one specified on
the command line, or the one created by default.
-o, --output-name OutputFileName,
Direct the binary output (FCode result of Tokenization) to the named file instead
of to the default-named file. This option is not valid when multiple input files
are named.
-P, --dependencies
Collect the fully-resolved pathnames of floaded and ENCODEd files into a
dependency-list file. The names collected are in the form that is presented to the
host operating system: shell environment Variables and related expressions will be
fully expanded, and the directory within the include-list in which the file was
found will be attached.
The name of the dependency-list file will be the same as that of the FLoad-list
file, except that its extension will be .P instead of .fl.
-v, --verbose
Print additional messages (including advisories) during tokenization.
-I, --Include directory
This tokenizer supports the notion of an include-list. The user creates the
include-list by specifying a number of -I directory pairs on the command-line. All
file-reads, whether for an fload command or an encode-file directive, will involve
a search for the named file through the directories of the include-list, in the
order they were supplied on the command-line.
If no include-list is created, file-reads are relative to the current working
Directory. If an include-list is created, file-reads are restricted to the
directories within it. For the current working directory to be included in the
file-search, it must be specified explicitly. -I. will accomplish that quite
effectively.
-T, --Trace symbol
This tokenizer supports the notion of a trace-list. The User creates the trace-list
by specifying a number of -T symbol pairs on the command-line.
When a name is defined, whether as an FCode, an alias, a Macro or anything else,
either in normal tokenization mode or "Tokenizer Escape"‐mode, if it matches a
symbol that has been added to the trace list, a trace note message will be issued
indicating that a definition of that name has been created. Subsequent trace note
Messages will be issued when the definition of that name is invoked.
This trace-symbols feature can be helpful during maintenance of legacy code, for
instance, when multiple symbols carry the same name.
-d, --define Symbol[=Value]
Define a command-line symbol and optionally, assign a value to it. If you wish the
value to contain spaces or quotes, you can accomplish that using the shell escape
conventions. This sequence may be repeated. Once a Symbol is defined on the
command-line, it stays in effect for the duration of the entire batch of
tokenizations (i.e., if there are multiple input files named on the command line).
Command-line Symbols can be tested for purposes of conditional tokenization, or
their assigned values can be evaluated.
-f, --flag [no]<FlagName>
The tokenizer recognizes a specific set of special-feature flag-names; each is
associated with a specific non-standard variant behavior. Pass the flag-name as an
argument to the -f switch to enable the behavior; to disable it, precede the flag-
name with the optional string no.
The settings of the special-feature flags can also be changed or displayed from
within the source input file.
The special-feature flags are all initially set to be enabled, except where noted.
The flag-names and their associated special-features are as follows:
Local-Values
Support IBM-style Local Values ("LV"s). Initially disabled.
LV-Legacy-Separator
Allow Semicolon for Local Values Separator ("Legacy").
LV-Legacy-Message
Display a Warning Message when Semicolon is used as the Local Values
Separator.
ABORT-Quote
Allow ABORT" macro.
Sun-ABORT-Quote
ABORT" with implicit IF ... THEN
Abort-Quote-Throw
Use -2 THROW, rather than ABORT, in an Abort" phrase
String-remark-escape
Allow "\ (Quote-Backslash) to interrupt string parsing.
Hex-remark-escape
Allow \ (Backslash) to interrupt hex-sequence parsing within a string.
C-Style-string-escape
Allow the C-style String-Escape pairs \n \t and \xx\ to be treated as
special characters in string parsing.
Always-Headers
Override occurrences of the standard directive headerless in the source with
-- effectively -- headers to make all definitions have a header. Occurrences
of the directive external will continue to behave in the standard manner.
Initially disabled.
Always-External
All definitions will be made as though under the external directive;
occurrences of either Standard directive headerless or headers in the source
will be overridden. This special-feature flag will also override the
Always-Headers special-feature flag in the event that both have been
specified. Initially disabled.
Warn-if-Duplicate
Display a WARNING message whenever a definition is made whose name
duplicates that of an existing definition. Disabling this flag will suspend
the duplicate-names test globally, until it is re-enabled. A Directive is
supported that will suspend the test for the duration of only a single
definition, without affecting global behavior.
Obsolete-FCode-Warning
Display a WARNING message whenever an FCode function is invoked that the
Standard identifies as obsolete.
Trace-Conditionals
Issue advisory messages about the state of conditional tokenization.
(Remember that advisory messages are displayed only if the verbose option -v
is set.) Initially disabled.
Upper-Case-Token-Names
Lower-Case-Token-Names When outputting the names of headered functions
(token-names) to the binary output file, override the character-case in
which the names appeared in the source, and convert them to Upper- or Lower-
‐Case, respectively. (These flags do not affect text string sequences, whose
character-case is always preserved.) Initially disabled.
Big-End-PCI-Rev-Level
Save the Revision Level of the Vendor's ROM field of the PCI Header in big-
endian byte-order, rather than little-endian as per the general PCI Standard
convention. (This flag does not affect any other field
of the PCI Header). Initially disabled.
Ret-Stk-Interp
Allow return-stack operations during interpretation. While the standard
specifies that usage of the operators >r, r@, and r> while interpreting is
allowed, actual practice in the industry is inconsistent. Developers who
wish to take a more cautious approach to this question can disable this flag
so that any attempt to use the operators >r, r@, and r> in the interpreting
state will generate an ERROR Message.
Also, the pseudo-flag-name help will cause a list of the flag-names and their associated
special-features to be printed.
SEE ALSO
detok(1).
AUTHORS
Stefan Reinauer <stepan@openbios.org>
David L. Paktor <dlpaktor@us.ibm.com>
This manual page was written by Aurelien Jarno <aurel32@debian.org> for the Debian
GNU/Linux system (but may be used by others).
May 15, 2007 TOKE(1)