NAME

       toke - OpenBIOS tokenizer

SYNOPSIS

       toke [options] forth-file

DESCRIPTION

       toke is an open-source FCode Tokenizer from the OpenBIOS project.

OPTIONS

       -h, -? Print a brief help message and then exit.

       -i, --ignore-errors
              Generate a Binary Output even if errors were reported.

       -l, --load-list
              Collect  the  names  of floaded files into an FLoad-List file. The names collected are in the same
              form as they were presented in the fload statements.
              The name of the FLoad-List File is derived from the name of the binary output file,  by  replacing
              its extension with .fl , or, if the binary output file name had no extension, merely appending the
              extension .fl.
              The binary output file name used for this purpose is either the one specified on the command line,
              or the one created by default.

       -o, --output-name OutputFileName,
              Direct  the  binary  output  (FCode  result  of  Tokenization) to the named file instead of to the
              default-named file. This option is not valid when multiple input files are named.

       -P, --dependencies
              Collect the fully-resolved pathnames of floaded and ENCODEd files  into  a  dependency-list  file.
              The  names  collected  are  in  the  form  that  is  presented to the host operating system: shell
              environment Variables and related expressions will be fully expanded, and the directory within the
              include-list in which the file was found will be attached.
              The  name of the dependency-list file will be the same as that of the FLoad-list file, except that
              its extension will be .P instead of .fl.

       -v, --verbose
              Print additional messages (including advisories) during tokenization.

       -I, --Include directory
              This tokenizer supports the notion of an  include-list.  The  user  creates  the  include-list  by
              specifying  a  number  of  -I  directory pairs on the command-line. All file-reads, whether for an
              fload command or an encode-file directive, will involve a search for the named  file  through  the
              directories of the include-list, in the order they were supplied on the command-line.
              If  no  include-list  is  created, file-reads are relative to the current working Directory. If an
              include-list is created, file-reads are restricted to the directories within it.  For the  current
              working  directory  to be included in the file-search, it must be specified explicitly.  -I.  will
              accomplish that quite effectively.

       -T, --Trace symbol
              This tokenizer supports the notion of a trace-list. The User creates the trace-list by  specifying
              a number of -T symbol pairs on the command-line.
              When  a name is defined, whether as an FCode, an alias, a Macro or anything else, either in normal
              tokenization mode or "Tokenizer Escape"‐mode, if it matches a symbol that has been  added  to  the
              trace list, a trace note message will be issued indicating that a definition of that name has been
              created.  Subsequent trace note Messages will be issued  when  the  definition  of  that  name  is
              invoked.
              This  trace-symbols  feature  can be helpful during maintenance of legacy code, for instance, when
              multiple symbols carry the same name.

       -d, --define Symbol[=Value]
              Define a command-line symbol and optionally, assign a value to  it.  If  you  wish  the  value  to
              contain  spaces  or  quotes,  you  can  accomplish  that  using the shell escape conventions. This
              sequence may be repeated. Once a Symbol is defined on the command-line, it stays in effect for the
              duration  of  the  entire batch of tokenizations (i.e., if there are multiple input files named on
              the command line). Command-line Symbols can be tested for purposes of conditional tokenization, or
              their assigned values can be evaluated.

       -f, --flag [no]<FlagName>
              The  tokenizer  recognizes a specific set of special-feature flag-names; each is associated with a
              specific non-standard variant behavior. Pass the flag-name as an argument  to  the  -f  switch  to
              enable the behavior; to disable it, precede the flag-name with the optional string no.
              The  settings of the special-feature flags can also be changed or displayed from within the source
              input file.
              The special-feature flags are all initially set to be enabled, except where noted.
              The flag-names and their associated special-features are as follows:

              Local-Values
                     Support IBM-style Local Values ("LV"s). Initially disabled.

              LV-Legacy-Separator
                     Allow Semicolon for Local Values Separator ("Legacy").

              LV-Legacy-Message
                     Display a Warning Message when Semicolon is used as the Local Values Separator.

              ABORT-Quote
                     Allow ABORT" macro.

              Sun-ABORT-Quote
                     ABORT" with implicit IF ... THEN

              Abort-Quote-Throw
                     Use -2 THROW, rather than ABORT, in an Abort" phrase

              String-remark-escape
                     Allow "\ (Quote-Backslash) to interrupt string parsing.

              Hex-remark-escape
                     Allow \ (Backslash) to interrupt hex-sequence parsing within a string.

              C-Style-string-escape
                     Allow the C-style String-Escape pairs \n \t and \xx\ to be treated as special characters in
                     string parsing.

              Always-Headers
                     Override occurrences of the standard directive headerless in the source with -- effectively
                     -- headers to make all definitions have a header. Occurrences  of  the  directive  external
                     will continue to behave in the standard manner. Initially disabled.

              Always-External
                     All  definitions will be made as though under the external directive; occurrences of either
                     Standard directive headerless or headers in the source will be  overridden.  This  special-
                     feature  flag  will also override the Always-Headers special-feature flag in the event that
                     both have been specified. Initially disabled.

              Warn-if-Duplicate
                     Display a WARNING message whenever a definition is made whose name duplicates  that  of  an
                     existing  definition.   Disabling this flag will suspend the duplicate-names test globally,
                     until it is re-enabled. A Directive is  supported  that  will  suspend  the  test  for  the
                     duration of only a single definition, without affecting global behavior.

              Obsolete-FCode-Warning
                     Display  a  WARNING  message  whenever  an  FCode  function  is  invoked  that the Standard
                     identifies as obsolete.

              Trace-Conditionals
                     Issue advisory messages  about  the  state  of  conditional  tokenization.  (Remember  that
                     advisory messages are displayed only if the verbose option -v is set.) Initially disabled.

              Upper-Case-Token-Names
                     Lower-Case-Token-Names When outputting the names of headered functions (token-names) to the
                     binary output file, override the character-case in which the names appeared in the  source,
                     and  convert  them to Upper- or Lower- ‐Case, respectively. (These flags do not affect text
                     string sequences, whose character-case is always preserved.)  Initially disabled.

              Big-End-PCI-Rev-Level
                     Save the Revision Level of the Vendor's ROM field of the PCI  Header  in  big-endian  byte-
                     order,  rather  than  little-endian  as per the general PCI Standard convention. (This flag
                     does not affect any other field
                      of the PCI Header). Initially disabled.

              Ret-Stk-Interp
                     Allow return-stack operations during interpretation.  While  the  standard  specifies  that
                     usage of the operators >r, r@, and r> while interpreting is allowed, actual practice in the
                     industry is inconsistent.  Developers who wish to take a more  cautious  approach  to  this
                     question  can  disable this flag so that any attempt to use the operators >r, r@, and r> in
                     the interpreting state will generate an ERROR Message.

       Also, the pseudo-flag-name help will cause a list of the flag-names and their associated special-features
       to be printed.

SEE ALSO

       detok(1).

AUTHORS

       Stefan Reinauer <stepan@openbios.org>
       David L. Paktor <dlpaktor@us.ibm.com>

       This  manual page was written by Aurelien Jarno <aurel32@debian.org> for the Debian GNU/Linux system (but
       may be used by others).

                                                  May 15, 2007                                           TOKE(1)