Provided by: manpages-posix_2.16-1_all bug

NAME

       lex - generate programs for lexical tasks (DEVELOPMENT)

SYNOPSIS

       lex [-t][-n|-v][file ...]

DESCRIPTION

       The  lex  utility shall generate C programs to be used in lexical processing of character input, and that
       can be used as an interface to yacc. The C programs shall be generated from lex source code  and  conform
       to  the  ISO C  standard.  Usually,  the  lex  utility  shall  write the program it generates to the file
       lex.yy.c; the state of this file is unspecified if lex  exits  with  a  non-zero  exit  status.  See  the
       EXTENDED DESCRIPTION section for a complete description of the lex input language.

OPTIONS

       The  lex  utility  shall  conform  to  the Base Definitions volume of IEEE Std 1003.1-2001, Section 12.2,
       Utility Syntax Guidelines.

       The following options shall be supported:

       -n     Suppress the summary of statistics usually written with the -v  option.  If  no  table  sizes  are
              specified in the lex source code and the -v option is not specified, then -n is implied.

       -t     Write the resulting program to standard output instead of lex.yy.c.

       -v     Write  a  summary of lex statistics to the standard output. (See the discussion of lex table sizes
              in Definitions in lex .) If the -t option is specified and -n is not specified, this report  shall
              be  written  to standard error. If table sizes are specified in the lex source code, and if the -n
              option is not specified, the -v option may be enabled.

OPERANDS

       The following operand shall be supported:

       file   A pathname of an input file. If more  than  one  such  file  is  specified,  all  files  shall  be
              concatenated  to  produce  a  single  lex program. If no file operands are specified, or if a file
              operand is '-' , the standard input shall be used.

STDIN

       The standard input shall be used if no file operands are specified, or if a file operand  is  '-'  .  See
       INPUT FILES.

INPUT FILES

       The  input files shall be text files containing lex source code, as described in the EXTENDED DESCRIPTION
       section.

ENVIRONMENT VARIABLES

       The following environment variables shall affect the execution of lex:

       LANG   Provide a default value for the internationalization variables that are unset or  null.  (See  the
              Base  Definitions  volume of IEEE Std 1003.1-2001, Section 8.2, Internationalization Variables for
              the  precedence  of  internationalization  variables  used  to  determine  the  values  of  locale
              categories.)

       LC_ALL If  set  to  a  non-empty  string value, override the values of all the other internationalization
              variables.

       LC_COLLATE

              Determine the locale  for  the  behavior  of  ranges,  equivalence  classes,  and  multi-character
              collating  elements  within  regular expressions. If this variable is not set to the POSIX locale,
              the results are unspecified.

       LC_CTYPE
              Determine the locale for the interpretation of sequences of bytes of text data as characters  (for
              example,  single-byte  as  opposed to multi-byte characters in arguments and input files), and the
              behavior of character classes within regular expressions.  If this variable  is  not  set  to  the
              POSIX locale, the results are unspecified.

       LC_MESSAGES
              Determine  the locale that should be used to affect the format and contents of diagnostic messages
              written to standard error.

       NLSPATH
              Determine the location of message catalogs for the processing of LC_MESSAGES .

ASYNCHRONOUS EVENTS

       Default.

STDOUT

       If the -t option is specified, the text file of C source code output of lex shall be written to  standard
       output.

       If the -t option is not specified:

        * Implementation-defined  informational,  error,  and  warning  messages  concerning the contents of lex
          source code input shall be written to either the standard output or standard error.

        * If the -v option is specified and the -n option is not specified, lex statistics shall also be written
          to either the standard output or standard error, in an implementation-defined format. These statistics
          may also be generated if table sizes are specified with a '%' operator in the Definitions section,  as
          long as the -n option is not specified.

STDERR

       If  the  -t  option  is  specified,  implementation-defined  informational,  error,  and warning messages
       concerning the contents of lex source code input shall be written to the standard error.

       If the -t option is not specified:

        1. Implementation-defined informational, error, and warning messages  concerning  the  contents  of  lex
           source code input shall be written to either the standard output or standard error.

        2. If  the  -v  option  is  specified  and  the -n option is not specified, lex statistics shall also be
           written to either the standard output or standard error, in an implementation-defined  format.  These
           statistics  may also be generated if table sizes are specified with a '%' operator in the Definitions
           section, as long as the -n option is not specified.

OUTPUT FILES

       A text file containing C source code shall be written to lex.yy.c, or to the standard output  if  the  -t
       option is present.

EXTENDED DESCRIPTION

       Each input file shall contain lex source code, which is a table of regular expressions with corresponding
       actions in the form of C program fragments.

       When  lex.yy.c  is  compiled  and  linked  with  the  lex  library (using the -l l operand with c99), the
       resulting program shall read character input from the standard input and shall partition it into  strings
       that match the given expressions.

       When an expression is matched, these actions shall occur:

        * The  input  string  that was matched shall be left in yytext as a null-terminated string; yytext shall
          either be an external character array or a pointer to a character string. As explained in  Definitions
          in  lex  ,  the  type  can  be  explicitly selected using the %array or %pointer declarations, but the
          default is implementation-defined.

        * The external int yyleng shall be set to the length of the matching string.

        * The expression's corresponding program fragment, or action, shall be executed.

       During pattern matching, lex shall search the set of patterns for  the  single  longest  possible  match.
       Among rules that match the same number of characters, the rule given first shall be chosen.

       The general format of lex source shall be:

              Definitions
              %%
              Rules
              %%
              UserSubroutines

       The  first  "%%"  is  required  to mark the beginning of the rules (regular expressions and actions); the
       second "%%" is required only if user subroutines follow.

       Any line in the Definitions section beginning with a <blank> shall be assumed to be a C program  fragment
       and  shall  be  copied  to the external definition area of the lex.yy.c file.  Similarly, anything in the
       Definitions section included between delimiter lines containing only "%{" and "%}" shall also  be  copied
       unchanged to the external definition area of the lex.yy.c file.

       Any  such  input  (beginning  with  a  <blank>  or within "%{" and "%}" delimiter lines) appearing at the
       beginning of the Rules section before any rules are specified shall be  written  to  lex.yy.c  after  the
       declarations  of  variables  for the yylex() function and before the first line of code in yylex(). Thus,
       user variables local to yylex() can be declared here, as well as application code to execute  upon  entry
       to yylex().

       The  action  taken  by  lex  when encountering any input beginning with a <blank> or within "%{" and "%}"
       delimiter lines appearing in the Rules section but coming after one  or  more  rules  is  undefined.  The
       presence of such input may result in an erroneous definition of the yylex() function.

   Definitions in lex
       Definitions  appear  before the first "%%" delimiter. Any line in this section not contained between "%{"
       and "%}" lines and not beginning with a <blank> shall be assumed to define a lex substitution string. The
       format of these lines shall be:

              name substitute

       If a name does not meet the requirements for identifiers in the ISO C standard, the result is  undefined.
       The  string  substitute shall replace the string { name} when it is used in a rule. The name string shall
       be recognized in this context only when the braces are provided and when it  does  not  appear  within  a
       bracket expression or within double-quotes.

       In  the  Definitions  section,  any line beginning with a '%' (percent sign) character and followed by an
       alphanumeric word beginning with either 's' or 'S' shall define a  set  of  start  conditions.  Any  line
       beginning  with a '%' followed by a word beginning with either 'x' or 'X' shall define a set of exclusive
       start conditions. When the generated scanner is in a %s state, patterns with no state specified shall  be
       also  active;  in  a  %x  state, such patterns shall not be active. The rest of the line, after the first
       word, shall be considered to be one or more <blank>-separated names of start conditions. Start  condition
       names  shall be constructed in the same way as definition names. Start conditions can be used to restrict
       the matching of regular expressions to one or more states as described in Regular Expressions in lex .

       Implementations shall  accept  either  of  the  following  two  mutually-exclusive  declarations  in  the
       Definitions section:

       %array Declare the type of yytext to be a null-terminated character array.

       %pointer
              Declare the type of yytext to be a pointer to a null-terminated character string.

       The  default  type of yytext is implementation-defined. If an application refers to yytext outside of the
       scanner source file (that is, via an extern), the application shall include  the  appropriate  %array  or
       %pointer declaration in the scanner source file.

       Implementations  shall  accept declarations in the Definitions section for setting certain internal table
       sizes. The declarations are shown in the following table.

                                         Table: Table Size Declarations in lex
                            Declaration  Description                         Minimum Value
                            %p n         Number of positions                 2500
                            %n n         Number of states                    500
                            %a n         Number of transitions               2000
                            %e n         Number of parse tree nodes          1000
                            %k n         Number of packed character classes  1000
                            %o n         Size of the output array            3000

       In the table, n represents a positive decimal integer, preceded  by  one  or  more  <blank>s.  The  exact
       meaning  of  these  table  size  numbers is implementation-defined. The implementation shall document how
       these numbers affect the lex utility and how they are related to any output that may be generated by  the
       implementation  should  limitations  be  encountered during the execution of lex. It shall be possible to
       determine from this output which of the table  size  values  needs  to  be  modified  to  permit  lex  to
       successfully  generate  tables  for the input language.  The values in the column Minimum Value represent
       the lowest values conforming implementations shall provide.

   Rules in lex
       The rules in lex source files are a table in which the left column contains regular expressions  and  the
       right column contains actions (C program fragments) to be executed when the expressions are recognized.

              ERE action
              ERE action...

       The  extended  regular  expression  (ERE)  portion of a row shall be separated from action by one or more
       <blank>s. A regular expression containing <blank>s  shall  be  recognized  under  one  of  the  following
       conditions:

        * The entire expression appears within double-quotes.

        * The <blank>s appear within double-quotes or square brackets.

        * Each <blank> is preceded by a backslash character.

   User Subroutines in lex
       Anything in the user subroutines section shall be copied to lex.yy.c following yylex().

   Regular Expressions in lex
       The lex utility shall support the set of extended regular expressions (see the Base Definitions volume of
       IEEE Std 1003.1-2001,  Section  9.4,  Extended  Regular  Expressions),  with  the following additions and
       exceptions to the syntax:

       "..."  Any string enclosed in double-quotes shall represent the characters within  the  double-quotes  as
              themselves,  except  that  backslash  escapes  (which  appear  in  the  following  table) shall be
              recognized.  Any backslash-escape sequence shall be terminated by the closing quote. For  example,
              "\01" "1" represents a single string: the octal value 1 followed by the character '1' .

       <state>r, <state1,state2,...>r

              The  regular expression r shall be matched only when the program is in one of the start conditions
              indicated by state, state1, and so on; see Actions in lex . (As an exception to the  typographical
              conventions  of  the  rest  of  this volume of IEEE Std 1003.1-2001, in this case <state> does not
              represent a metavariable, but the literal angle-bracket  characters  surrounding  a  symbol.)  The
              start condition shall be recognized as such only at the beginning of a regular expression.

       r/x    The  regular  expression  r  shall  be  matched only if it is followed by an occurrence of regular
              expression x ( x is the instance of trailing context, further defined below).  The token  returned
              in  yytext shall only match r. If the trailing portion of r matches the beginning of x, the result
              is unspecified. The r expression cannot include further trailing context or the '$' (match-end-of-
              line) operator; x cannot include the '^' (match-beginning-of-line) operator, nor trailing context,
              nor the '$' operator. That is, only one occurrence of trailing context is allowed in a lex regular
              expression, and the '^' operator only can be used at the beginning of such an expression.

       {name} When name is one of the substitution symbols from the Definitions section, the  string,  including
              the  enclosing  braces,  shall  be replaced by the substitute value. The substitute value shall be
              treated in the extended regular expression as if it were enclosed in parentheses. No  substitution
              shall occur if { name} occurs within a bracket expression or within double-quotes.

       Within  an ERE, a backslash character shall be considered to begin an escape sequence as specified in the
       table in the Base Definitions volume of IEEE Std 1003.1-2001, Chapter 5, File Format Notation  (  '\\'  ,
       '\a'  ,  '\b'  ,  '\f'  , '\n' , '\r' , '\t' , '\v' ). In addition, the escape sequences in the following
       table shall be recognized.

       A literal <newline> cannot occur within an ERE; the escape sequence '\n'  can  be  used  to  represent  a
       <newline>. A <newline> shall not be matched by a period operator.

                                            Table: Escape Sequences in lex
                        Escape
                        Sequence Description                    Meaning
                        \digits  A backslash character followed The character whose encoding
                                 by the longest sequence of     is represented by the one,
                                 one, two, or three octal-digit two, or three-digit octal
                                 characters (01234567). If all  integer. If the size of a byte
                                 of the digits are 0 (that is,  on the system is greater than
                                 representation of the NUL      nine bits, the valid escape
                                 character), the behavior is    sequence used to represent a
                                 undefined.                     byte is implementation-
                                                                defined. Multi-byte characters
                                                                require multiple, concatenated
                                                                escape sequences of this type,
                                                                including the leading '\' for
                                                                each byte.
                        \xdigits A backslash character followed The character whose encoding
                                 by the longest sequence of     is represented by the
                                 hexadecimal-digit characters   hexadecimal integer.
                                 (01234567abcdefABCDEF). If all
                                 of the digits are 0 (that is,
                                 representation of the NUL
                                 character), the behavior is
                                 undefined.
                        \c       A backslash character followed The character 'c' , unchanged.
                                 by any character not described
                                 in this table or in the table
                                 in the Base Definitions volume
                                 of IEEE Std 1003.1-2001,
                                 Chapter 5, File Format
                                 Notation ( '\\' , '\a' , '\b'
                                 , '\f' , '\n' , '\r' , '\t' ,
                                 '\v' ).

       Note:  If  a  '\x' sequence needs to be immediately followed by a hexadecimal digit character, a sequence
              such as "\x1" "1" can be used, which represents a character containing the value  1,  followed  by
              the character '1' .

       The  order of precedence given to extended regular expressions for lex differs from that specified in the
       Base Definitions volume of IEEE Std 1003.1-2001, Section 9.4, Extended Regular Expressions. The order  of
       precedence for lex shall be as shown in the following table, from high to low.

       Note:  The escaped characters entry is not meant to imply that these are operators, but they are included
              in  the  table  to  show  their relationships to the true operators. The start condition, trailing
              context, and anchoring notations have been  omitted  from  the  table  because  of  the  placement
              restrictions described in this section; they can only appear at the beginning or ending of an ERE.

                                                 Table: ERE Precedence in lex
                                   Extended Regular Expression        Precedence
                                   collation-related bracket symbols  [= =] [: :] [. .]
                                   escaped characters                 \<special character>
                                   bracket expression                 [ ]
                                   quoting                            "..."
                                   grouping                           ( )
                                   definition                         {name}
                                   single-character RE duplication    * + ?
                                   concatenation
                                   interval expression                {m,n}
                                   alternation                        |

       The  ERE  anchoring operators '^' and '$' do not appear in the table. With lex regular expressions, these
       operators are restricted in their use: the '^' operator can only be used at the beginning  of  an  entire
       regular  expression,  and  the  '$'  operator  only at the end. The operators apply to the entire regular
       expression. Thus, for example, the pattern "(^abc)|(def$)" is undefined; it can instead be written as two
       separate rules, one with the regular expression "^abc" and one with "def$" , which share a common  action
       via  the  special '|' action (see below). If the pattern were written "^abc|def$" , it would match either
       "abc" or "def" on a line by itself.

       Unlike the general ERE rules, embedded anchoring is not allowed by most historical  lex  implementations.
       An  example  of  embedded  anchoring would be for patterns such as "(^| )foo( |$)" to match "foo" when it
       exists as a complete word. This functionality can be obtained using existing lex features:

              ^foo/[ \n]      |
              " foo"/[ \n]    /* Found foo as a separate word. */

       Note also that '$' is a form of trailing context (it is equivalent to "/\n" ) and as such cannot be  used
       with  regular  expressions  containing  another instance of the operator (see the preceding discussion of
       trailing context).

       The additional regular expressions trailing-context operator '/' can be used as an ordinary character  if
       presented  within  double-quotes,  "/"  ; preceded by a backslash, "\/" ; or within a bracket expression,
       "[/]" . The start-condition '<' and '>' operators shall be special only  in  a  start  condition  at  the
       beginning  of a regular expression; elsewhere in the regular expression they shall be treated as ordinary
       characters.

   Actions in lex
       The action to be taken when an ERE is matched can  be  a  C  program  fragment  or  the  special  actions
       described  below; the program fragment can contain one or more C statements, and can also include special
       actions. The empty C statement ';' shall be a valid action; any string in the lex.yy.c input that matches
       the pattern portion of such a rule is effectively ignored or skipped. However, the absence of  an  action
       shall not be valid, and the action lex takes in such a condition is undefined.

       The  specification  for  an action, including C statements and special actions, can extend across several
       lines if enclosed in braces:

              ERE <one or more blanks> { program statement
                                         program statement }

       The default action when a string in the input to a lex.yy.c program is  not  matched  by  any  expression
       shall  be to copy the string to the output. Because the default behavior of a program generated by lex is
       to read the input and copy it to the output, a minimal lex  source  program  that  has  just  "%%"  shall
       generate a C program that simply copies the input to the output unchanged.

       Four special actions shall be available:

              |   ECHO;   REJECT;   BEGIN

       |      The  action  '|'  means  that the action for the next rule is the action for this rule. Unlike the
              other three actions, '|' cannot be enclosed in braces or be semicolon-terminated; the  application
              shall ensure that it is specified alone, with no other actions.

       ECHO;  Write the contents of the string yytext on the output.

       REJECT;
              Usually only a single expression is matched by a given string in the input. REJECT means "continue
              to  the  next  expression  that  matches the current input", and shall cause whatever rule was the
              second choice after the current rule to be executed for the same input. Thus, multiple  rules  can
              be matched and executed for one input string or overlapping input strings.  For example, given the
              regular expressions "xyz" and "xy" and the input "xyz" , usually only the regular expression "xyz"
              would match. The next attempted match would start after z. If the last action in the "xyz" rule is
              REJECT,  both  this rule and the "xy" rule would be executed. The REJECT action may be implemented
              in such a fashion that flow of control does not continue after it, as if it were equivalent  to  a
              goto  to  another  part  of  yylex().  The  use of REJECT may result in somewhat larger and slower
              scanners.

       BEGIN  The action:

              BEGIN newstate;

       switches the state (start condition) to newstate. If the string newstate has not been declared previously
       as a start condition in the Definitions section, the  results  are  unspecified.  The  initial  state  is
       indicated by the digit '0' or the token INITIAL.

       The  functions  or  macros  described  below are accessible to user code included in the lex input. It is
       unspecified whether they appear in the C code output of lex, or are  accessible  only  through  the  -l l
       operand to c99 (the lex library).

       int  yylex(void)

              Performs lexical analysis on the input; this is the primary function generated by the lex utility.
              The  function  shall return zero when the end of input is reached; otherwise, it shall return non-
              zero values (tokens) determined by the actions that are selected.

       int  yymore(void)

              When called, indicates that when the next input string is recognized, it is to be appended to  the
              current  value  of  yytext  rather  than  replacing  it;  the  value  in  yyleng shall be adjusted
              accordingly.

       int  yyless(int  n)

              Retains n initial characters in yytext, NUL-terminated, and treats the remaining characters as  if
              they had not been read; the value in yyleng shall be adjusted accordingly.

       int  input(void)

              Returns the next character from the input, or zero on end-of-file.  It shall obtain input from the
              stream  pointer yyin, although possibly via an intermediate buffer. Thus, once scanning has begun,
              the effect of altering the value of yyin is undefined. The character read shall  be  removed  from
              the input stream of the scanner without any processing by the scanner.

       int  unput(int  c)

              Returns  the character 'c' to the input; yytext and yyleng are undefined until the next expression
              is matched. The result of using unput() for more characters than have been input is unspecified.

       The following functions shall appear only in the lex library accessible through the  -l l  operand;  they
       can therefore be redefined by a conforming application:

       int  yywrap(void)

              Called  by  yylex() at end-of-file; the default yywrap() shall always return 1. If the application
              requires yylex() to continue processing with another source of input,  then  the  application  can
              include  a function yywrap(), which associates another file with the external variable FILE * yyin
              and shall return a value of zero.

       int  main(int  argc, char *argv[])

              Calls yylex() to perform lexical analysis, then exits. The user code can contain main() to perform
              application-specific operations, calling yylex() as applicable.

       Except for input(), unput(), and main(), all external and static names generated by lex shall begin  with
       the prefix yy or YY.

EXIT STATUS

       The following exit values shall be returned:

        0     Successful completion.

       >0     An error occurred.

CONSEQUENCES OF ERRORS

       Default.

       The following sections are informative.

APPLICATION USAGE

       Conforming applications are warned that in the Rules section, an ERE without an action is not acceptable,
       but need not be detected as erroneous by lex. This may result in compilation or runtime errors.

       The  purpose of input() is to take characters off the input stream and discard them as far as the lexical
       analysis is concerned. A common use is to discard the body of a comment once the beginning of  a  comment
       is recognized.

       The  lex utility is not fully internationalized in its treatment of regular expressions in the lex source
       code or generated lexical analyzer. It would seem desirable to have the lexical  analyzer  interpret  the
       regular  expressions  given  in  the  lex  source according to the environment specified when the lexical
       analyzer is executed, but this is not possible with the current lex  technology.  Furthermore,  the  very
       nature  of  the lexical analyzers produced by lex must be closely tied to the lexical requirements of the
       input language being described, which is frequently locale-specific  anyway.  (For  example,  writing  an
       analyzer that is used for French text is not automatically useful for processing other languages.)

EXAMPLES

       The  following  is  an  example  of a lex program that implements a rudimentary scanner for a Pascal-like
       syntax:

              %{
              /* Need this for the call to atof() below. */
              #include <math.h>
              /* Need this for printf(), fopen(), and stdin below. */
              #include <stdio.h>
              %}

              DIGIT    [0-9]
              ID       [a-z][a-z0-9]*

              %%

              {DIGIT}+ {
                  printf("An integer: %s (%d)\n", yytext,
                      atoi(yytext));
                  }

              {DIGIT}+"."{DIGIT}*        {
                  printf("A float: %s (%g)\n", yytext,
                      atof(yytext));
                  }

              if|then|begin|end|procedure|function        {
                  printf("A keyword: %s\n", yytext);
                  }

              {ID}    printf("An identifier: %s\n", yytext);

              "+"|"-"|"*"|"/"        printf("An operator: %s\n", yytext);

              "{"[^}\n]*"}"    /* Eat up one-line comments. */

              [ \t\n]+        /* Eat up white space. */

              .  printf("Unrecognized character: %s\n", yytext);

              %%

              int main(int argc, char *argv[])
              {
                  ++argv, --argc;  /* Skip over program name. */
                  if (argc > 0)
                      yyin = fopen(argv[0], "r");
                  else
                      yyin = stdin;

                  yylex();
              }

RATIONALE

       Even though the -c option and references to the C language are retained in this description, lex  may  be
       generalized to other languages, as was done at one time for EFL, the Extended FORTRAN Language. Since the
       lex input specification is essentially language-independent, versions of this utility could be written to
       produce Ada, Modula-2, or Pascal code, and there are known historical implementations that do so.

       The  current  description  of  lex  bypasses  the issue of dealing with internationalized EREs in the lex
       source code or generated lexical analyzer. If it follows the model  used  by  awk  (the  source  code  is
       assumed  to  be  presented  in  the POSIX locale, but input and output are in the locale specified by the
       environment variables), then the tables in the lexical analyzer produced  by  lex  would  interpret  EREs
       specified  in  the  lex source in terms of the environment variables specified when lex was executed. The
       desired effect would be to have the lexical analyzer interpret the EREs given in the lex source according
       to the environment specified when the lexical analyzer is executed, but this is  not  possible  with  the
       current lex technology.

       The  description  of  octal  and hexadecimal-digit escape sequences agrees with the ISO C standard use of
       escape sequences. See the RATIONALE for ed for a discussion of bytes larger than 9 bits being represented
       by octal values.  Hexadecimal values can represent larger bytes and multi-byte characters directly, using
       as many digits as required.

       There is no detailed output format specification. The observed  behavior  of  lex  under  four  different
       historical  implementations was that none of these implementations consistently reported the line numbers
       for error and warning messages.  Furthermore, there was a desire that lex be allowed to output additional
       diagnostic messages. Leaving message formats unspecified avoids these formatting questions  and  problems
       with internationalization.

       Although the %x specifier for exclusive start conditions is not historical practice, it is believed to be
       a  minor change to historical implementations and greatly enhances the usability of lex programs since it
       permits an application to obtain the expected functionality with fewer statements.

       The %array and %pointer declarations were added as a compromise between historical systems. The System V-
       based lex copies the matched text to a yytext array. The flex program, supported in BSD and GNU  systems,
       uses a pointer. In the latter case, significant performance improvements are available for some scanners.
       Most  historical  programs  should  require  no  change in porting from one system to another because the
       string being referenced is null-terminated in both cases. (The method used by flex  in  its  case  is  to
       null-terminate  the  token  in place by remembering the character that used to come right after the token
       and replacing it before continuing on to the next scan.) Multi-file programs with external references  to
       yytext  outside the scanner source file should continue to operate on their historical systems, but would
       require one of the new declarations to be considered strictly portable.

       The description of EREs avoids unnecessary duplication of ERE details because their meanings within a lex
       ERE are the same as that for the ERE in this volume of IEEE Std 1003.1-2001.

       The reason for the undefined condition associated with text beginning with a <blank> or within  "%{"  and
       "%}" delimiter lines appearing in the Rules section is historical practice. Both the BSD and System V lex
       copy the indented (or enclosed) input in the Rules section (except at the beginning) to unreachable areas
       of the yylex() function (the code is written directly after a break statement). In some cases, the System
       V lex generates an error message or a syntax error, depending on the form of indented input.

       The  intention in breaking the list of functions into those that may appear in lex.yy.c versus those that
       only appear in libl.a is that only those functions in libl.a can be reliably redefined  by  a  conforming
       application.

       The  descriptions  of  standard output and standard error are somewhat complicated because historical lex
       implementations  chose  to  issue  diagnostic  messages  to  standard  output  (unless  -t  was   given).
       IEEE Std 1003.1-2001  allows this behavior, but leaves an opening for the more expected behavior of using
       standard error for diagnostics. Also, the System V behavior of writing  the  statistics  when  any  table
       sizes  are  given is allowed, while BSD-derived systems can avoid it. The programmer can always precisely
       obtain the desired results by using either the -t or -n options.

       The OPERANDS section does not mention the use of - as a synonym for standard input;  not  all  historical
       implementations support such usage for any of the file operands.

       A  description  of  the  translation table was deleted from early proposals because of its relatively low
       usage in historical applications.

       The change to the definition of the  input()  function  that  allows  buffering  of  input  presents  the
       opportunity for major performance gains in some applications.

       The  following  examples  clarify the differences between lex regular expressions and regular expressions
       appearing elsewhere in this volume of IEEE Std 1003.1-2001. For regular expressions of the form  "r/x"  ,
       the  string  matching  r  is  always  returned;  confusion  may arise when the beginning of x matches the
       trailing portion of r.  For example, given the regular expression  "a*b/cc"  and  the  input  "aaabcc"  ,
       yytext  would  contain  the string "aaab" on this match. But given the regular expression "x*/xy" and the
       input "xxxy" , the token xxx, not xx, is returned by some implementations because xxx matches "x*" .

       In the rule "ab*/bc" , the "b*" at the end of r extends r's match into  the  beginning  of  the  trailing
       context,  so  the  result  is unspecified. If this rule were "ab/bc" , however, the rule matches the text
       "ab" when it is followed by the text "bc" . In this latter case, the matching of r cannot extend into the
       beginning of x, so the result is specified.

FUTURE DIRECTIONS

       None.

SEE ALSO

       c99 , ed , yacc

COPYRIGHT

       Portions of this text are reprinted and reproduced in electronic form from IEEE Std 1003.1, 2003 Edition,
       Standard for Information Technology -- Portable Operating System Interface (POSIX), The Open  Group  Base
       Specifications Issue 6, Copyright (C) 2001-2003 by the Institute of Electrical and Electronics Engineers,
       Inc  and  The  Open Group. In the event of any discrepancy between this version and the original IEEE and
       The Open Group Standard, the original IEEE and The Open Group  Standard  is  the  referee  document.  The
       original Standard can be obtained online at http://www.opengroup.org/unix/online.html .

IEEE/The Open Group                                   2003                                                LEX(P)