Provided by: re2c_4.0.1-1_amd64 
NAME
re2ocaml - generate fast lexical analyzers for OCaml
SYNOPSIS
re2ocaml [ OPTIONS ] [ WARNINGS ] INPUT
Input can be either a file or - for stdin.
INTRODUCTION
re2ocaml works as a preprocessor. It reads the input file (which is usually a program in
OCaml, but can be anything) and looks for blocks of code enclosed in special-form
start/end markers. The text outside of these blocks is copied verbatim into the output
file. The contents of the blocks are processed by re2ocaml. It translates them to code in
OCaml and outputs the generated code in place of the block.
Here is an example of a small program that checks if a given string contains a decimal
number:
(* re2ocaml $INPUT -o $OUTPUT -i *)
open String
type state = {
yyinput: string;
mutable yycursor: int;
}
%{
re2c:YYFN = ["lex;bool", "yyrecord;state"];
re2c:yyfill:enable = 0;
[1-9][0-9]* { true }
* { false }
%}
let main () =
let st = {yyinput = "1234\x00"; yycursor = 0}
in if not (lex st) then raise (Failure "error")
let _ = main ()
In the output re2ocaml replaced the block in the middle with the generated code:
(* Generated by re2ocaml *)
(* re2ocaml $INPUT -o $OUTPUT -i *)
open String
type state = {
yyinput: string;
mutable yycursor: int;
}
let rec yy0 (yyrecord : state) : bool =
let yych = get yyrecord.yyinput yyrecord.yycursor in
yyrecord.yycursor <- yyrecord.yycursor + 1;
match yych with
| '1'..'9' -> (yy2 [@tailcall]) yyrecord
| _ -> (yy1 [@tailcall]) yyrecord
and yy1 (yyrecord : state) : bool =
false
and yy2 (yyrecord : state) : bool =
let yych = get yyrecord.yyinput yyrecord.yycursor in
match yych with
| '0'..'9' ->
yyrecord.yycursor <- yyrecord.yycursor + 1;
(yy2 [@tailcall]) yyrecord
| _ -> (yy3 [@tailcall]) yyrecord
and yy3 (yyrecord : state) : bool =
true
and lex (yyrecord : state) : bool =
(yy0 [@tailcall]) yyrecord
let main () =
let st = {yyinput = "1234\x00"; yycursor = 0}
in if not (lex st) then raise (Failure "error")
let _ = main ()
BASICS
A re2ocaml program consists of a sequence of blocks intermixed with code in the target
language. A block may contain definitions, configurations, rules and directives in any
order:
<name> = <regular expression>;
A definition binds a name to a regular expression. Names may contain alphanumeric
characters and underscore. The regular expressions section gives an overview of
re2ocaml syntax for regular expressions. Once defined, the name can be used in
other regular expressions and in rules. Recursion in named definitions is not
allowed, and each name should be defined before it is used. A block inherits named
definitions from the global scope. Redefining a name that exists in the current
scope is an error.
<configuration> = <value>;
A configuration allows one to change re2ocaml behavior and customize the generated
code. For a full list of configurations supported by re2ocaml see the
configurations section. Depending on a particular configuration, the value can be a
keyword, a nonnegative integer number or a one-line string which should be enclosed
in double or single quotes unless it consists of alphanumeric characters. A block
inherits configurations from the global scope and may redefine them or add new
ones. Configurations defined inside of a block affect the whole block, even if they
appear at the end of it.
<regular expression> { <code> }
A rule binds a regular expression to a semantic action (a block of code in the
target language). If the regular expression matches, the associated semantic action
is executed. If multiple rules match, the longest match takes precedence. If
multiple rules match the same string, the earliest one takes precedence. There are
two special rules: the default rule * and the end of input rule $. The default rule
should always be defined, it has the lowest priority regardless of its place in the
block, and it matches any code unit (not necessarily a valid character, see the
encoding support section). The end of input rule should be defined if the
corresponding method for handling the end of input is used. If start conditions are
used, rules have more complex syntax.
!<directive>;
A directive is one of the special predefined statements. Each directive has a
unique purpose. For example, the !use directive merges a rules block into the
current one (see the reusable blocks section), and the !include directive allows
one to include an outer file (see the include files section).
Blocks
Block start and end markers are either /*!re2c and */, or %{ and %} (both styles are
supported). Starting from version 2.2 blocks may have optional names that allow them to be
referenced in other blocks. There are different kinds of blocks:
/*!re2c[:<name>] ... */ or %{[:<name>] ... %}
A global block contains definitions, configurations, rules and directives.
re2ocaml compiles regular expressions associated with each rule into a
deterministic finite automaton, encodes it in the form of conditional jumps in the
target language and replaces the block with the generated code. Names and
configurations defined in a global block are added to the global scope and become
visible to subsequent blocks. At the start of the program the global scope is
initialized with command-line options.
/*!local:re2c[:<name>] ... */ or %{local[:<name>] ... %}
A local block is like a global block, but the names and configurations in it have
local scope (they do not affect other blocks).
/*!rules:re2c[:<name>] ... */ or %{rules[:<name>] ... %}
A rules block is like a local block, but it does not generate any code by itself,
nor does it add any definitions to the global scope -- it is meant to be reused in
other blocks. This is a way of sharing code (more details in the reusable blocks
section). Prior to re2ocaml version 2.2 rules blocks required -r --reusable option.
/*!use:re2c[:<name>] ... */ or %{use[:<name>] ... %}
A use block that references a previously defined rules block. If the name is
specified, re2ocaml looks for a rules blocks with this name. Otherwise the most
recent rules block is used (either a named or an unnamed one). A use block can add
definitions, configurations and rules of its own, which are added to those of the
referenced rules block. Prior to re2ocaml version 2.2 use blocks required -r
--reusable option.
/*!max:re2c[:<name1>[:<name2>...]] ... */ or %{max[:<name1>[:<name2>...]] ... %}
A block that generates YYMAXFILL definition. An optional list of block names
specifies which blocks should be included when computing YYMAXFILL value (if the
list is empty, all blocks are included). By default the generated code is a
macro-definition for C (#define YYMAXFILL <n>), or a global variable for Go (var
YYMAXFILL int = <n>). It can be customized with an optional configuration format
that specifies a template string where @@{max} (or @@ for short) is replaced with
the numeric value of YYMAXFILL.
/*!maxnmatch:re2c[:<name1>[:<name2>...]] ... */ or %{maxnmatch[:<name1>[:<name2>...]] ...
%}
A block that generates YYMAXNMATCH definition (it requires -P --posix-captures
option). An optional list of block names specifies which blocks should be included
when computing YYMAXNMATCH value (if the list is empty, all blocks are included).
By default the generated code is a macro-definition for C (#define YYMAXNMATCH
<n>), or a global variable for Go (var YYMAXNMATCH int = <n>). It can be customized
with an optional configuration format that specifies a template string where
@@{max} (or @@ for short) is replaced with the numeric value of YYMAXNMATCH.
/*!stags:re2c[:<name1>[:<name2>...]] ... */, /*!mtags:re2c[:<name1>[:<name2>...]] ... */
or %{stags[:<name1>[:<name2>...]] ... %}, %{mtags[:<name1>[:<name2>...]] ... %{
Blocks that specify a template piece of code that is expanded for each s-tag/m-tag
variable generated by re2ocaml. An optional list of block names specifies which
blocks should be included when computing the set of tag variables (if the list is
empty, all blocks are included). There are two optional configurations: format and
separator. Configuration format specifies a template string where @@{tag} (or @@
for short) is replaced with the name of each tag variable. Configuration separator
specifies a piece of code used to join the generated format pieces for different
tag variables.
/*!svars:re2c[:<name1>[:<name2>...]] ... */, /*!mvars:re2c[:<name1>[:<name2>...]] ... */
or %{svars[:<name1>[:<name2>...]] ... %}, %{mvars[:<name1>[:<name2>...]] ... %{
Blocks that specify a template piece of code that is expanded for each s-tag/m-tag
that is either explicitly mentioned by the rules (with --tags option) or implicitly
generated by re2ocaml (with --captvars or --posix-captvars options). An optional
list of block names specifies which blocks should be included when computing the
set of tags (if the list is empty, all blocks are included). There are two
optional configurations: format and separator. Configuration format specifies a
template string where @@{tag} (or @@ for short) is replaced with the name of each
tag. Configuration separator specifies a piece of code used to join the generated
format pieces for different tags.
/*!getstate:re2c[:<name1>[:<name2>...]] ... */ or %{getstate[:<name1>[:<name2>...]] ... %}
A block that generates conditional dispatch on the lexer state (it requires
--storable-state option). An optional list of block names specifies which blocks
should be included in the state dispatch. The default transition goes to the start
label of the first block on the list. If the list is empty, all blocks are
included, and the default transition goes to the first block in the file that has a
start label. This block type is incompatible with the --loop-switch option, as it
requires cross-block transitions that are unsupported without goto or function
calls.
/*!conditions:re2c[:<name1>[:<name2>...]] ... */, /*!types:re2c... */ or
%{conditions[:<name1>[:<name2>...]] ... %}, %{types... %}
A block that generates condition enumeration (it requires --conditions option). An
optional list of block names specifies which blocks should be included when
computing the set of conditions (if the list is empty, all blocks are included).
By default the generated code is an enumeration YYCONDTYPE. It can be customized
with optional configurations format and separator. Configuration format specifies
a template string where @@{cond} (or @@ for short) is replaced with the name of
each condition, and @@{num} is replaced with a numeric index of that condition.
Configuration separator specifies a piece of code used to join the generated format
pieces for different conditions.
/*!include:re2c <file> */ or %{include <file> %}
This block allows one to include <file>, which must be a double-quoted file path.
The contents of the file are literally substituted in place of the block, in the
same way as #include works in C/C++. This block can be used together with the
--depfile option to generate build system dependencies on the included files.
/*!header:re2c:on*/ or %{header:on %}
This block marks the start of header file. Everything after it and up to the
following header:off block is processed by re2ocaml and written to the header file
specified with -t --type-header option.
/*!header:re2c:off*/ or %{header:off %}
This block marks the end of header file started with header:on*/ block.
/*!ignore:re2c ... */ or %{ignore ... %}
A block which contents are ignored and removed from the output file.
Directives
Here is a full list of directives supported by re2ocaml:
!use:<name>;
An in-block use directive that merges a previously defined rules block with the
specified name into the current block. Named definitions, configurations and rules
of the referenced block are added to the current ones. Conflicts between
overlapping rules and configurations are resolved in the usual way: the first rule
takes priority, and the latest configuration overrides the preceding ones. One
exception is the special rules *, $ and <!> for which a block-local definition
always takes priority. A use directive can be placed anywhere inside of a block,
and multiple use directives are allowed.
!include <file>;
This directive is the same as include block: it inserts <file> contents verbatim in
place of the durective.
Regular expressions
re2ocaml uses the following syntax for regular expressions:
"foo" Case-sensitive string literal.
'foo' Case-insensitive string literal.
[a-xyz], [^a-xyz]
Character class (possibly negated).
. Any character except newline.
R \ S Difference of character classes R and S.
R* Zero or more occurrences of R.
R+ One or more occurrences of R.
R? Optional R.
R{n} Repetition of R exactly n times.
R{n,} Repetition of R at least n times.
R{n,m} Repetition of R from n to m times.
(R) Just R; parentheses are used to override precedence. If submatch extraction is
enabled, (R) is a capturing or a non-capturing group depending on --invert-captures
option.
(!R) If submatch extraction is enabled, (!R) is a non-capturing or a capturing group
depending on --invert-captures option.
R S Concatenation: R followed by S.
R | S Alternative: R or S.
R / S Lookahead: R followed by S, but S is not consumed.
name Regular expression defined as name (or literal string "name" in Flex compatibility
mode).
{name} Regular expression defined as name in Flex compatibility mode.
@stag An s-tag: saves the last input position at which @stag matches in a variable named
stag.
#mtag An m-tag: saves all input positions at which #mtag matches in a variable named
mtag.
Character classes and string literals may contain the following escape sequences: \a, \b,
\f, \n, \r, \t, \v, \\, octal escapes \ooo and hexadecimal escapes \xhh, \uhhhh and
\Uhhhhhhhh.
Configurations
Here is a full list of configurations supported by re2ocaml:
re2c:api, re2c:input
Same as the --api option.
re2c:api:sigil
Specify the marker ("sigil") that is used for argument placeholders in the API
primitives. The default is @@. A placeholder starts with sigil followed by the
argument name in curly braces. For example, if sigil is set to $, then placeholders
will have the form ${name}. Single-argument APIs may use shorthand notation without
the name in braces. This option can be overridden by options for individual API
primitives, e.g. re2c:YYFILL@len for YYFILL.
re2c:api:style
Specify API style. Possible values are functions (the default for C) and free-form
(the default for Go and Rust). In functions style API primitives are generated
with an argument list in parentheses following the name of the primitive. The
arguments are provided only for autogenerated parameters (such as the number of
characters passed to YYFILL), but not for the general lexer context, so the
primitives behave more like macros in C/C++ or closures in Go and Rust. In
free-form style API primitives do not have a fixed form: they should be defined as
strings containing free-form pieces of code with interpolated variables of the form
@@{var} or @@ (they correspond to arguments in function-like style). This
configuration may be overridden for individual API primitives, see for example
re2c:YYFILL:naked configuration for YYFILL.
re2c:bit-vectors, re2c:flags:bit-vectors, re2c:flags:b
Same as the --bit-vectors option, but can be configured on per-block basis.
re2c:captures, re2c:leftmost-captures
Same as the --leftmost-captures option, but can be configured on per-block basis.
re2c:captvars, re2c:leftmost-captvars
Same as the --leftmost-captvars option, but can be configured on per-block basis.
re2c:case-insensitive, re2c:flags:case-insensitive
Same as the --case-insensitive option, but can be configured on per-block basis.
re2c:case-inverted, re2c:flags:case-inverted
Same as the --case-inverted option, but can be configured on per-block basis.
re2c:case-ranges, re2c:flags:case-ranges
Same as the --case-ranges option, but can be configured on per-block basis.
re2c:computed-gotos, re2c:flags:computed-gotos, re2c:flags:g
Same as the --computed-gotos option, but can be configured on per-block basis.
re2c:computed-gotos:threshold, re2c:cgoto:threshold
If computed goto is used, this configuration specifies the complexity threshold
that triggers the generation of jump tables instead of nested if statements and
bitmaps. The default value is 9.
re2c:cond:abort
If set to a positive integer value, the default case in the generated condition
dispatch aborts program execution.
re2c:cond:goto
Specifies a piece of code used for the autogenerated shortcut rules :=> in
conditions. The default is goto @@;. The @@ placeholder is substituted with
condition name (see configurations re2c:api:sigil and re2c:cond:goto@cond).
re2c:cond:goto@cond
Specifies the sigil used for argument substitution in re2c:cond:goto definition.
The default value is @@. Overrides the more generic re2c:api:sigil configuration.
re2c:cond:divider
Defines the divider for condition blocks. The default value is /*
*********************************** */. Placeholders are substituted with
condition name (see re2c:api;sigil and re2c:cond:divider@cond).
re2c:cond:divider@cond
Specifies the sigil used for argument substitution in re2c:cond:divider definition.
The default is @@. Overrides the more generic re2c:api:sigil configuration.
re2c:cond:prefix, re2c:condprefix
Specifies the prefix used for condition labels. The default is yyc_.
re2c:cond:enumprefix, re2c:condenumprefix
Specifies the prefix used for condition identifiers. The default is yyc.
re2c:debug-output, re2c:flags:debug-output, re2c:flags:d
Same as the --debug-output option, but can be configured on per-block basis.
re2c:empty-class, re2c:flags:empty-class
Same as the --empty-class option, but can be configured on per-block basis.
re2c:encoding:ebcdic, re2c:flags:ecb, re2c:flags:e
Same as the --ebcdic option, but can be configured on per-block basis.
re2c:encoding:ucs2, re2c:flags:wide-chars, re2c:flags:w
Same as the --ucs2 option, but can be configured on per-block basis.
re2c:encoding:utf8, re2c:flags:utf-8, re2c:flags:8
Same as the --utf8 option, but can be configured on per-block basis.
re2c:encoding:utf16, re2c:flags:utf-16, re2c:flags:x
Same as the --utf16 option, but can be configured on per-block basis.
re2c:encoding:utf32, re2c:flags:unicode, re2c:flags:u
Same as the --utf32 option, but can be configured on per-block basis.
re2c:encoding-policy, re2c:flags:encoding-policy
Same as the --encoding-policy option, but can be configured on per-block basis.
re2c:eof
Specifies the sentinel symbol used with the end-of-input rule $. The default value
is -1 ($ rule is not used). Other possible values include all valid code units.
Only decimal numbers are recognized.
re2c:fn:sep
Specifies separator used in YYFN elements (defaults to semicolon).
re2c:header, re2c:flags:type-header, re2c:flags:t
Specifies the name of the generated header file relative to the directory of the
output file. Same as the --header option except that the file path is relative.
re2c:indent:string
Specifies the string used for indentation. The default is a single tab character
"\t". Indent string should contain whitespace characters only. To disable
indentation entirely, set this configuration to an empty string.
re2c:indent:top
Specifies the minimum amount of indentation to use. The default value is zero. The
value should be a non-negative integer number.
re2c:invert-captures
Same as the --invert-captures option, but can be configured on per-block basis.
re2c:label:prefix, re2c:labelprefix
Specifies the prefix used for DFA state labels. The default is yy.
re2c:label:start, re2c:startlabel
Controls the generation of a block start label. The default value is zero, which
means that the start label is generated only if it is used. An integer value
greater than zero forces the generation of start label even if it is unused by the
lexer. A string value also forces start label generation and sets the label name to
the specified string. This configuration applies only to the current block (it is
reset to default for the next block).
re2c:label:yyFillLabel
Specifies the prefix of YYFILL labels used with re2c:eof and in storable state
mode.
re2c:label:yyloop
Specifies the name of the label marking the start of the lexer loop with
--loop-switch option. The default is yyloop.
re2c:label:yyNext
Specifies the name of the optional label that follows YYGETSTATE switch in storable
state mode (enabled with re2c:state:nextlabel). The default is yyNext.
re2c:lookahead, re2c:flags:lookahead
Deprecated (see the deprecated --no-lookahead option).
re2c:monadic
If set to non-zero, the generated lexer will use monadic notation (this
configuration is specific to Haskell).
re2c:nested-ifs, re2c:flags:nested-ifs, re2c:flags:s
Same as the --nested-ifs option, but can be configured on per-block basis.
re2c:posix-captures, re2c:flags:posix-captures, re2c:flags:P
Same as the --posix-captures option, but can be configured on per-block basis.
re2c:posix-captvars
Same as the --posix-captvars option, but can be configured on per-block basis.
re2c:tags, re2c:flags:tags, re2c:flags:T
Same as the --tags option, but can be configured on per-block basis.
re2c:tags:expression
Specifies the expression used for tag variables. By default re2ocaml generates
expressions of the form yyt<N>. This might be inconvenient, for example if tag
variables are defined as fields in a struct. All occurrences of @@{tag} or @@ are
replaced with the actual tag name. For example, re2c:tags:expression = "s.@@";
results in expressions of the form s.yyt<N> in the generated code. See also
re2c:api:sigil configuration.
re2c:tags:negative
Specifies the constant expression that is used for negative tag value (typically
this would be -1 if tags are integer offsets in the input string, or null pointer
if they are pointers).
re2c:tags:prefix
Specifies the prefix for tag variable names. The default is yyt.
re2c:sentinel
Specifies the sentinel symbol used for the end-of-input checks (when bounds checks
are disabled with re2c:yyfill:enable = 0; and re2c:eof is not set). This
configuration does not affect code generation: its purpose is to verify that the
sentinel is not allowed in the middle of a rule, and ensure that the lexer won't
read past the end of buffer. The default value is -1` (in that case re2ocaml
assumes that the sentinel is zero, which is the most common case). Only decimal
numbers are recognized.
re2c:state:abort
If set to a positive integer value, the default case in the generated state
dispatch aborts program execution, and an explicit -1 case contains transition to
the start of the block.
re2c:state:nextlabel
Controls if the YYGETSTATE switch is followed by an yyNext label (the default value
is zero, which corresponds to no label). Alternatively one can use
re2c:label:start to generate a specific start label, or an explicit getstate block
to generate the YYGETSTATE switch separately from the lexer block.
re2c:unsafe, re2c:flags:unsafe
Same as the --no-unsafe option, but can be configured on per-block basis. If set
to zero, it suppresses the generation of unsafe wrappers around YYPEEK. The default
is non-zero (wrappers are generated). This configuration is specific to Rust.
re2c:YYBACKUP, re2c:define:YYBACKUP
Defines generic API primitive YYBACKUP.
re2c:YYBACKUPCTX, re2c:define:YYBACKUPCTX
Defines generic API primitive YYBACKUPCTX.
re2c:YYCONDTYPE, re2c:define:YYCONDTYPE
Defines API primitive YYCONDTYPE.
re2c:YYCTYPE, re2c:define:YYCTYPE
Defines API primitive YYCTYPE.
re2c:YYCTXMARKER, re2c:define:YYCTXMARKER
Defines API primitive YYCTXMARKER.
re2c:YYCURSOR, re2c:define:YYCURSOR
Defines API primitive YYCURSOR.
re2c:YYDEBUG, re2c:define:YYDEBUG
Defines API primitive YYDEBUG.
re2c:YYFILL, re2c:define:YYFILL
Defines API primitive YYFILL.
re2c:YYFILL@len, re2c:define:YYFILL@len
Specifies the sigil used for argument substitution in YYFILL definition. Defaults
to @@. Overrides the more generic re2c:api:sigil configuration.
re2c:YYFILL:naked, re2c:define:YYFILL:naked
Overrides the more generic re2c:api:style configuration for YYFILL. Zero value
corresponds to free-form API style.
re2c:YYFN
Defines API primitive YYFN.
re2c:YYINPUT
Defines API primitive YYINPUT.
re2c:YYGETCOND, re2c:define:YYGETCONDITION
Defines API primitive YYGETCOND.
re2c:YYGETCOND:naked, re2c:define:YYGETCONDITION:naked
Overrides the more generic re2c:api:style configuration for YYGETCOND. Zero value
corresponds to free-form API style.
re2c:YYGETSTATE, re2c:define:YYGETSTATE
Defines API primitive YYGETSTATE.
re2c:YYGETSTATE:naked, re2c:define:YYGETSTATE:naked
Overrides the more generic re2c:api:style configuration for YYGETSTATE. Zero value
corresponds to free-form API style.
re2c:YYGETACCEPT, re2c:define:YYGETACCEPT
Defines API primitive YYGETACCEPT.
re2c:YYLESSTHAN, re2c:define:YYLESSTHAN
Defines generic API primitive YYLESSTHAN.
re2c:YYLIMIT, re2c:define:YYLIMIT
Defines API primitive YYLIMIT.
re2c:YYMARKER, re2c:define:YYMARKER
Defines API primitive YYMARKER.
re2c:YYMTAGN, re2c:define:YYMTAGN
Defines generic API primitive YYMTAGN.
re2c:YYMTAGP, re2c:define:YYMTAGP
Defines generic API primitive YYMTAGP.
re2c:YYPEEK, re2c:define:YYPEEK
Defines generic API primitive YYPEEK.
re2c:YYRESTORE, re2c:define:YYRESTORE
Defines generic API primitive YYRESTORE.
re2c:YYRESTORECTX, re2c:define:YYRESTORECTX
Defines generic API primitive YYRESTORECTX.
re2c:YYRESTORETAG, re2c:define:YYRESTORETAG
Defines generic API primitive YYRESTORETAG.
re2c:YYSETCOND, re2c:define:YYSETCONDITION
Defines API primitive YYSETCOND.
re2c:YYSETCOND@cond, re2c:define:YYSETCONDITION@cond
Specifies the sigil used for argument substitution in YYSETCOND definition. The
default value is @@. Overrides the more generic re2c:api:sigil configuration.
re2c:YYSETCOND:naked, re2c:define:YYSETCONDITION:naked
Overrides the more generic re2c:api:style configuration for YYSETCOND. Zero value
corresponds to free-form API style.
re2c:YYSETSTATE, re2c:define:YYSETSTATE
Defines API primitive YYSETSTATE.
re2c:YYSETSTATE@state, re2c:define:YYSETSTATE@state
Specifies the sigil used for argument substitution in YYSETSTATE definition. The
default value is @@. Overrides the more generic re2c:api:sigil configuration.
re2c:YYSETSTATE:naked, re2c:define:YYSETSTATE:naked
Overrides the more generic re2c:api:style configuration for YYSETSTATE. Zero value
corresponds to free-form API style.
re2c:YYSETACCEPT, re2c:define:YYSETACCEPT
Defines API primitive YYSETACCEPT.
re2c:YYSKIP, re2c:define:YYSKIP
Defines generic API primitive YYSKIP.
re2c:YYSHIFT, re2c:define:YYSHIFT
Defines generic API primitive YYSHIFT.
re2c:YYCOPYMTAG, re2c:define:YYCOPYMTAG
Defines generic API primitive YYCOPYMTAG.
re2c:YYCOPYSTAG, re2c:define:YYCOPYSTAG
Defines generic API primitive YYCOPYSTAG.
re2c:YYSHIFTMTAG, re2c:define:YYSHIFTMTAG
Defines generic API primitive YYSHIFTMTAG.
re2c:YYSHIFTSTAG, re2c:define:YYSHIFTSTAG
Defines generic API primitive YYSHIFTSTAG.
re2c:YYSTAGN, re2c:define:YYSTAGN
Defines generic API primitive YYSTAGN.
re2c:YYSTAGP, re2c:define:YYSTAGP
Defines generic API primitive YYSTAGP.
re2c:yyaccept, re2c:variable:yyaccept
Defines API primitive yyaccept.
re2c:yybm, re2c:variable:yybm
Defines API primitive yybm.
re2c:yybm:hex, re2c:variable:yybm:hex
If set to nonzero, bitmaps for the --bit-vectors option are generated in
hexadecimal format. The default is zero (bitmaps are in decimal format).
re2c:yych, re2c:variable:yych
Defines API primitive yych.
re2c:yych:emit, re2c:variable:yych:emit
If set to zero, yych definition is not generated. The default is non-zero.
re2c:yych:conversion, re2c:variable:yych:conversion
If set to non-zero, re2ocaml automatically generates a conversion to YYCTYPE every
time yych is read. The default is to zero (no conversion).
re2c:yych:literals, re2c:variable:yych:literals
Specifies the form of literals that yych is matched against. Possible values are:
char (character literals in single quotes, non-printable ones use escape sequences
that start with backslash), hex (hexadecimal integers) and char_or_hex (a mixture
of both, character literals for printable characters and hexadecimal integers for
others).
re2c:yyctable, re2c:variable:yyctable
Defines API primitive yyctable.
re2c:yynmatch, re2c:variable:yynmatch
Defines API primitive yynmatch.
re2c:yypmatch, re2c:variable:yypmatch
Defines API primitive yypmatch.
re2c:yytarget, re2c:variable:yytarget
Defines API primitive yytarget.
re2c:yystable, re2c:variable:yystable
Deprecated.
re2c:yystate, re2c:variable:yystate
Defines API primitive yystate.
re2c:yyfill, re2c:variable:yyfill
Defines API primitive yyfill.
re2c:yyfill:check
If set to zero, suppresses the generation of pre-YYFILL check for the number of
input characters (the YYLESSTHAN definition in generic API and the YYLIMIT-based
comparison in C pointer API). The default is non-zero (generate the check).
re2c:yyfill:enable
If set to zero, suppresses the generation of YYFILL (together with the check). This
should be used when the whole input fits into one piece of memory (there is no need
for buffering) and the end-of-input checks do not rely on the YYFILL checks (e.g.
if a sentinel character is used). Use warnings (-W option) and re2c:sentinel
configuration to verify that the generated lexer cannot read past the end of input.
The default is non-zero (YYFILL is enabled).
re2c:yyfill:parameter
If set to zero, suppresses the generation of parameter passed to YYFILL. The
parameter is the minimum number of characters that must be supplied. Defaults to
non-zero (the parameter is generated). This configuration can be overridden with
re2c:YYFILL:naked or re2c:api:style.
Program interface
The generated code interfaces with the outer program with the help of primitives,
collectively referred to as the API. Which primitives should be defined for a particular
program depends on multiple factors, including the complexity of regular expressions,
input representation, buffering and the use of various features. All the necessary
primitives should be defined by the user in the form of macros, functions, variables or
any other suitable form that makes the generated code syntactically and semantically
correct. re2ocaml does not (and cannot) check the definitions, so if anything is missing
or defined incorrectly, the generated program may have compile-time or run-time errors.
This manual provides examples of API definitions in the most common cases.
re2ocaml has two API flavors that define the core set of primitives used by a program:
Record API
Record API is the default API for the OCaml backend. This API consists of a
variable yyrecord (the name can be overridden with re2c:yyrecord) that should be
defined as a record with fields _yyinput, _yycursor, _yymarker, _yyctxmarker,
_yylimit. Only the fields used by the generated code need to be defined, and their
names can be configured.
Generic API
This is the most flexible API. It is enabled with --api generic option or re2c:api
= generic configuration. It contains primitives for generic operations: YYPEEK,
YYSKIP, YYBACKUP, YYBACKUPCTX, YYSTAGP, YYSTAGN, YYMTAGP, YYMTAGN, YYRESTORE,
YYRESTORECTX, YYRESTORETAG, YYSHIFT, YYSHIFTSTAG, YYSHIFTMTAG, YYLESSTHAN.
Here is a full list of API primitives that may be used by the generated code in order to
interface with the outer program.
YYCTYPE
The type of the input characters (code units). For ASCII, EBCDIC and UTF-8
encodings it should be 1-byte unsigned integer. For UTF-16 or UCS-2 it should be
2-byte unsigned integer. For UTF-32 it should be 4-byte unsigned integer.
YYCURSOR
An l-value that stores the current input position (a pointer or an integer offset
in YYINPUT). Initially YYCURSOR should point to the first input character, and
later it is advanced by the generated code. When a rule matches, YYCURSOR position
is the one after the last matched character.
YYLIMIT
An r-value that stores the end of input position (a pointer or an integer offset in
YYINPUT). Initially YYLIMIT should point to the position after the last available
input character. It is not changed by the generated code. The lexer compares
YYCURSOR to YYLIMIT in order to determine if there are enough input characters
left.
YYMARKER
An l-value that stores the position of the latest matched rule (a pointer or an
integer offset in YYINPUT). It is used to restore the YYCURSOR position if the
longer match fails and the lexer needs to rollback. Initialization is not needed.
YYCTXMARKER
An l-value that stores the position of the trailing context (a pointer or an
integer offset in YYINPUT). No initialization is needed. YYCTXMARKER is needed only
if the lookahead operator / is used.
YYFILL A generic API primitive with one variable len. YYFILL should provide at least len
more input characters or fail. If re2c:eof is used, then len is always 1 and
YYFILL should always return to the calling function; zero return value indicates
success. If re2c:eof is not used, then YYFILL return value is ignored and it
should not return on failure. The maximum value of len is YYMAXFILL.
YYFN A primitive that defines function prototype in --recursive-functions code model.
Its value should be an array of one or more strings, where each string contains two
or three components separated by the string specified in re2c:fn:sep configuration
(typically a semicolon). The first array element defines function name and return
type (empty for a void function). Subsequent elements define function arguments:
first, the expression for the argument used in function body (usually just a name);
second, argument type; third, an optional formal parameter (it defaults to the
first component - usually both the argument and the parameter are the same
identifier).
YYINPUT
An r-value that stores the current input character sequence (string, buffer, etc.).
YYMAXFILL
An integral constant equal to the maximum value of the argument to YYFILL. It can
be generated with a max block.
YYLESSTHAN
A generic API primitive with one variable len. It should be defined as an r-value
of boolean type that equals true if and only if there are less than len input
characters left.
YYPEEK A generic API primitive with no variables. It should be defined as an r-value of
type YYCTYPE that is equal to the character at the current input position.
YYSKIP A generic API primitive that should advance the current input position by one code
unit.
YYBACKUP
A generic API primitive that should save the current input position (to be restored
with YYRESTORE later).
YYRESTORE
A generic API primitive that should restore the current input position to the value
saved by YYBACKUP.
YYBACKUPCTX
A generic API primitive that should save the current input position as the position
of the trailing context (to be restored with YYRESTORECTX later).
YYRESTORECTX
A generic API primitive that should restore the trailing context position saved
with YYBACKUPCTX.
YYRESTORETAG
A generic API primitive with one variable tag that should restore the trailing
context position to the value of tag.
YYSTAGP
A generic API primitive with one variable tag, where tag can be a pointer or an
offset in YYINPUT (see submatch extraction section for details). YYSTAGP should set
tag to the current input position.
YYSTAGN
A generic API primitive with one variable tag, where tag can be a pointer or an
offset in YYINPUT (see submatch extraction section for details). YYSTAGN should to
set tag to a value that represents non-existent input position.
YYMTAGP
A generic API primitive with one variable tag. YYMTAGP should append the current
position to the submatch history of tag (see the submatch extraction section for
details.)
YYMTAGN
A generic API primitive with one variable tag. YYMTAGN should append a value that
represents non-existent input position position to the submatch history of tag (see
the submatch extraction section for details.)
YYSHIFT
A generic API primitive with one variable shift that should shift the current input
position by shift characters (the shift value may be negative).
YYCOPYSTAG
A generic API primitive with two variables, lhs and rhs that should copy
right-hand-side s-tag variable rhs to the left-hand-side s-tag variable lhs. For
most languages this primitive has a default definition that assigns lhs to rhs.
YYCOPYMTAG
A generic API primitive with two variables, lhs and rhs that should copy
right-hand-side m-tag variable rhs to the left-hand-side m-tag variable lhs. For
most languages this primitive has a default definition that assigns lhs to rhs.
YYSHIFTSTAG
A generic API primitive with two variables, tag and shift that should shift tag by
shift code units (the shift value may be negative).
YYSHIFTMTAG
A generic API primitive with two variables, tag and shift that should shift the
latest value in the history of tag by shift code units (the shift value may be
negative).
YYMAXNMATCH
An integral constant equal to the maximal number of POSIX capturing groups in a
rule. It is generated with a maxnmatch block.
YYCONDTYPE
The type of the condition enum. It can be generated either with conditions block
or --header option.
YYGETACCEPT
A primitive with one variable var that stores numeric selector of the accepted
rule. For most languages this primitive has a default definition that reads from
var.
YYSETACCEPT
A primitive with two variables: var (an l-value that stores numeric selector of the
accepted rule), and val (the value of selector). For most languages this primitive
has a default definition that assigns var to val.
YYGETCOND
An r-value of type YYCONDTYPE that is equal to the current condition identifier.
YYSETCOND
A primitive with one variable cond that should set the current condition identifier
to cond.
YYGETSTATE
An r-value of integer type that is equal to the current lexer state. It should be
initialized to -1.
YYSETSTATE
A primitive with one variable state that should set the current lexer state to
state.
YYDEBUG
This primitive is generated only with -d, --debug-output option. Its purpose is to
add logging to the generated code (typical YYDEBUG definition is a print
statement). YYDEBUG statements are generated in every state and have two variables:
state (either a DFA state index or -1) and symbol (the current input symbol).
yyaccept
An l-value of unsigned integral type that stores the number of the latest matched
rule. User definition is necessary only with --storable-state option.
yybm A table containing compressed bitmaps for up to 8 transitions (used with the
--bitmaps option). The table contains 256 elements and is indexed by 1-byte code
units. Each 8-bit element combines boolean values for up to 8 transitions. k-Th bit
of n-th element is true iff n-th code unit is in the range of k-th transition. The
idea of this bitmap is to replace many if branches or switch cases with one check
of a single bit in the table.
yych An l-value of type YYCTYPE that stores the current input character. User
definition is necessary only with -f --storable-state option.
yyctable
Jump table generated for the initial condition dispatch (enabled with the
combination of --conditions and --computed-gotos options).
yyfill An l-value that stores the result of YYFILL call (this may be necessary for pure
functional languages, where YYFILL is a monadic function with complex return
value).
yynmatch
An l-value of unsigned integral type that stores the number of POSIX capturing
groups in the matched rule. Used only with -P --posix-captures option.
yypmatch
An array of l-values that are used to hold the tag values corresponding to the
capturing parentheses in the matching rule. Array length must be at least yynmatch
* 2 (usually YYMAXNMATCH * 2 is a good choice). Used only with -P --posix-captures
option.
yystable
Deprecated.
yystate
An l-value used with the --loop-switch option to store the current DFA state.
yytarget
Jump table that contains jump targets (label addresses) for all transitions from a
state. This table is local to each state. Generation of yytarget tables is enabled
with --computed-gotos option.
Options
Some of the options have corresponding configurations, others are global and cannot be
changed after re2c starts reading the input file. Debug options generally require
building re2c in debug configuration. Internal options are useful for experimenting with
the algorithms used in re2c.
-? --help -h
Show help message.
--api <simple | record | generic>
Specify the API used by the generated code to interface with used-defined code.
Option simple shold be used in simple cases when there's no need for buffer
refilling and storing lexer state. Option record should be used when lexer state
needs to be stored in a record (struct, class, etc.). Option generic should be
used in complex cases when the other two APIs are not flexible enough.
--bit-vectors -b
Optimize conditional jumps using bit masks. This option implies --nested-ifs.
--captures, --leftmost-captures
Enable submatch extraction with leftmost greedy capturing groups. The result is
collected into an array yybmatch of capacity 2 * YYMAXNMATCH, and yynmatch is set
to the number of groups for the matching rule.
--captvars, --leftmost-captvars
Enable submatch extraction with leftmost greedy capturing groups. The result is
collected into variables yytl<k>, yytr<k> for k-th capturing group.
--case-insensitive
Treat single-quoted and double-quoted strings as case-insensitive.
--case-inverted
Invert the meaning of single-quoted and double-quoted strings: treat single-quoted
strings as case-sensitive and double-quoted strings as case-insensitive.
--case-ranges
Collapse consecutive cases in a switch statements into a range of the form low ...
high. This syntax is a C/C++ language extension that is supported by compilers like
GCC, Clang and Tcc. The main advantage over using single cases is smaller generated
code and faster generation time, although for some compilers like Tcc it also
results in smaller binary size. This option is supported only for C.
--computed-gotos -g
Optimize conditional jumps using non-standard "computed goto" extension (which must
be supported by the compiler). re2ocaml generates jump tables only in complex cases
with a lot of conditional branches. Complexity threshold can be configured with
cgoto:threshold configuration. This option implies --bit-vectors. It is supported
only for C.
--conditions --start-conditions -c
Enable support of Flex-like "conditions": multiple interrelated lexers within one
block. This is an alternative to manually specifying different re2ocaml blocks
connected with goto or function calls.
--depfile FILE
Write dependency information to FILE in the form of a Makefile rule <output-file> :
<input-file> [include-file ...]. This allows one to track build dependencies in the
presence of include blocks/directives, so that updating include files triggers
regeneration of the output file. This option depends on the --output option.
--ebcdic --ecb -e
Generate a lexer that reads input in EBCDIC encoding. re2ocaml assumes that the
character range is 0 -- 0xFF and character size is 1 byte.
--empty-class <match-empty | match-none | error>
Define the way re2ocaml treats empty character classes. With match-empty (the
default) empty class matches empty input (which is illogical, but
backwards-compatible). With match-none empty class always fails to match. With
error empty class raises a compilation error.
--encoding-policy <fail | substitute | ignore>
Define the way re2ocaml treats Unicode surrogates. With fail re2ocaml aborts with
an error when a surrogate is encountered. With substitute re2ocaml silently
replaces surrogates with the error code point 0xFFFD. With ignore (the default)
re2ocaml treats surrogates as normal code points. The Unicode standard says that
standalone surrogates are invalid, but real-world libraries and programs behave in
different ways.
--flex-syntax -F
Partial support for Flex syntax: in this mode named definitions don't need the
equal sign and the terminating semicolon, and when used they must be surrounded
with curly braces. Names without curly braces are treated as double-quoted strings.
--goto-label
Use "goto/label" code model: encode DFA in form of labeled code blocks connected
with goto transitions across blocks. This is only supported for languages that have
a goto statement.
--header --type-header -t HEADER
Generate a HEADER file. The contents of the file can be specified using special
blocks header:on and header:off. If conditions are used, the generated header will
have a condition enum automatically appended to it (unless there is an explicit
conditions block).
-I PATH
Add PATH to the list of locations which are used when searching for include files.
This option is useful in combination with include block or directive. re2ocaml
looks for FILE in the directory of the parent file and in the include locations
specified with -I option.
--input <default | custom>
Deprecated alias for --api. Option default corresponds to simple (it is indeed the
default for most backends, but not for all). Option custom corresponds to generic.
--input-encoding <ascii | utf8>
Specify the way re2ocaml parses regular expressions. With ascii (the default)
re2ocaml handles input as ASCII-encoded: any sequence of code units is a sequence
of standalone 1-byte characters. With utf8 re2ocaml handles input as UTF8-encoded
and recognizes multibyte characters.
--invert-captures
Invert the meaning of capturing and non-capturing groups. By default (...) is
capturing and (! ...) is non-capturing. With this option (! ...) is capturing and
(...) is non-capturing.
--lang <none | c | d | go | haskell | java | js | ocaml | python | rust | v | zig>
Specify the target language. Supported languages are C, D, Go, Haskell, Java, JS,
OCaml, Python, Rust, V, Zig (more languages can be added via user-defined syntax
files, see the --syntax option). Option none disables default suntax configs, so
that the target language is undefined.
--location-format <gnu | msvc>
Specify location format in messages. With gnu locations are printed as
'filename:line:column: ...'. With msvc locations are printed as
'filename(line,column) ...'. The default is gnu.
--loop-switch
Use "loop/switch" code model: encode DFA in form of a loop over a switch statement,
where individual states are switch cases. State is stored in a variable yystate.
Transitions between states update yystate to the case label of the destination
state and continue execution to the head of the loop.
--nested-ifs -s
Use nested if statements instead of switch statements in conditional jumps. This
usually results in more efficient code with non-optimizing compilers.
--no-debug-info -i
Do not output line directives. This may be useful when the generated code is stored
in a version control system (to avoid huge autogenerated diffs on small changes).
--no-generation-date
Suppress date output in the generated file.
--no-version
Suppress version output in the generated file.
--no-unsafe
Do not generate unsafe wrapper over YYPEEK (this option is specific to Rust). For
performance reasons YYPEEK should avoid bounds-checking, as the lexer already
performs end-of-input checks in a more efficient way. The user may choose to
provide a safe YYPEEK definition, or a definition that is unsafe only in release
builds, in which case the --no-unsafe option helps to avoid warnings about
redundant unsafe blocks.
--output -o OUTPUT
Specify the OUTPUT file.
--posix-captures, -P
Enable submatch extraction with POSIX-style capturing groups. The result is
collected into an array yybmatch of capacity 2 * YYMAXNMATCH, and yynmatch is set
to the number of groups for the matching rule.
--posix-captvars
Enable submatch extraction with POSIX-style capturing groups. The result is
collected into variables yytl<k>, yytr<k> for k-th capturing group.
--recursive-functions
Use code model based on co-recursive functions, where each DFA state is a separate
function that may call other state-functions or itself.
--reusable -r
Deprecated since version 2.2 (reusable blocks are allowed by default now).
--skeleton -S
Ignore user-defined interface code and generate a self-contained "skeleton"
program. Additionally, generate input files with strings derived from the regular
grammar and compressed match results that are used to verify "skeleton" behavior on
all inputs. This option is useful for finding bugs in optimizations and code
generation. This option is supported only for C.
--storable-state -f
Generate a lexer which can store its inner state. This is useful in push-model
lexers which are stopped by an outer program when there is not enough input, and
then resumed when more input becomes available. In this mode users should
additionally define YYGETSTATE and YYSETSTATE primitives, and variables yych,
yyaccept and state should be part of the stored lexer state.
--syntax FILE
Load configurations from the specified FILE and apply them on top of the default
syntax file. Note that FILE can define only a few configurations (if it's used to
amend the default syntax file), or it can define a whole new language backend (in
the latter case it is recommended to use --lang none option).
--tags -T
Enable submatch extraction with tags.
--ucs2 --wide-chars -w
Generate a lexer that reads UCS2-encoded input. re2ocaml assumes that the character
range is 0 -- 0xFFFF and character size is 2 bytes. This option implies
--nested-ifs.
--utf8 --utf-8 -8
Generate a lexer that reads input in UTF-8 encoding. re2ocaml assumes that the
character range is 0 -- 0x10FFFF and character size is 1 byte.
--utf16 --utf-16 -x
Generate a lexer that reads UTF16-encoded input. re2ocaml assumes that the
character range is 0 -- 0x10FFFF and character size is 2 bytes. This option
implies --nested-ifs.
--utf32 --unicode -u
Generate a lexer that reads UTF32-encoded input. re2ocaml assumes that the
character range is 0 -- 0x10FFFF and character size is 4 bytes. This option
implies --nested-ifs.
--verbose
Output a short message in case of success.
--vernum -V
Show version information in MMmmpp format (major, minor, patch).
--version -v
Show version information.
--single-pass -1
Deprecated. Does nothing (single pass is the default now).
--debug-output -d
Emit YYDEBUG invocations in the generated code. This is useful to trace lexer
execution.
--dump-adfa
Debug option: output DFA after tunneling (in .dot format).
--dump-cfg
Debug option: output control flow graph of tag variables (in .dot format).
--dump-closure-stats
Debug option: output statistics on the number of states in closure.
--dump-dfa-det
Debug option: output DFA immediately after determinization (in .dot format).
--dump-dfa-min
Debug option: output DFA after minimization (in .dot format).
--dump-dfa-tagopt
Debug option: output DFA after tag optimizations (in .dot format).
--dump-dfa-tree
Debug option: output DFA under construction with states represented as tag history
trees (in .dot format).
--dump-dfa-raw
Debug option: output DFA under construction with expanded state-sets (in .dot
format).
--dump-interf
Debug option: output interference table produced by liveness analysis of tag
variables.
--dump-nfa
Debug option: output NFA (in .dot format).
--emit-dot -D
Instead of normal output generate lexer graph in .dot format. The output can be
converted to an image with the help of Graphviz (e.g. something like dot -Tpng
-odfa.png dfa.dot).
--dfa-minimization <moore | table>
Internal option: DFA minimization algorithm used by re2ocaml. The moore option is
the Moore algorithm (it is the default). The table option is the "table filling"
algorithm. Both algorithms should produce the same DFA up to states relabeling;
table filling is simpler and much slower and serves as a reference implementation.
--eager-skip
Internal option: make the generated lexer advance the input position eagerly --
immediately after reading the input symbol. This changes the default behavior when
the input position is advanced lazily -- after transition to the next state.
--no-lookahead
Internal option, deprecated. It used to enable TDFA(0) algorithm. Unlike TDFA(1),
TDFA(0) algorithm does not use one-symbol lookahead. It applies register operations
to the incoming transitions rather than the outgoing ones. Benchmarks showed that
TDFA(0) algorithm is less efficient than TDFA(1).
--no-optimize-tags
Internal option: suppress optimization of tag variables (useful for debugging).
--posix-closure <gor1 | gtop>
Internal option: specify shortest-path algorithm used for the construction of
epsilon-closure with POSIX disambiguation semantics: gor1 (the default) stands for
Goldberg-Radzik algorithm, and gtop stands for "global topological order"
algorithm.
--posix-prectable <complex | naive>
Internal option: specify the algorithm used to compute POSIX precedence table. The
complex algorithm computes precedence table in one traversal of tag history tree
and has quadratic complexity in the number of TNFA states; it is the default. The
naive algorithm has worst-case cubic complexity in the number of TNFA states, but
it is much simpler than complex and may be slightly faster in non-pathological
cases.
--stadfa
Internal option, deprecated. It used to enable staDFA algorithm, which differs
from TDFA in that register operations are placed in states rather than on
transitions. Benchmarks showed that staDFA algorithm is less efficient than TDFA.
--fixed-tags <none | toplevel | all>
Internal option: specify whether the fixed-tag optimization should be applied to
all tags (all), none of them (none), or only those in toplevel concatenation
(toplevel). The default is all. "Fixed" tags are those that are located within a
fixed distance to some other tag (called "base"). In such cases only the base tag
needs to be tracked, and the value of the fixed tag can be computed as the value of
the base tag plus a static offset. For tags that are under alternative or
repetition it is also necessary to check if the base tag has a no-match value (in
that case fixed tag should also be set to no-match, disregarding the offset). For
tags in top-level concatenation the check is not needed, because they always match.
Warnings
Warnings can be invividually enabled, disabled and turned into an error.
-W Turn on all warnings.
-Werror
Turn warnings into errors. Note that this option alone doesn't turn on any
warnings; it only affects those warnings that have been turned on so far or will be
turned on later.
-W<warning>
Turn on warning.
-Wno-<warning>
Turn off warning.
-Werror-<warning>
Turn on warning and treat it as an error (this implies -W<warning>).
-Wno-error-<warning>
Don't treat this particular warning as an error. This doesn't turn off the warning
itself.
-Wcondition-order
Warn if the generated program makes implicit assumptions about condition numbering.
One should use either --header option or conditions block to generate a mapping of
condition names to numbers and then use the autogenerated condition names.
-Wempty-character-class
Warn if a regular expression contains an empty character class. Trying to match an
empty character class makes no sense: it should always fail. However, for
backwards compatibility reasons re2ocaml permits empty character classes and treats
them as empty strings. Use the --empty-class option to change the default behavior.
-Wmatch-empty-string
Warn if a rule is nullable (matches an empty string). If the lexer runs in a loop
and the empty match is unintentional, the lexer may unexpectedly hang in an
infinite loop.
-Wswapped-range
Warn if the lower bound of a range is greater than its upper bound. The default
behavior is to silently swap the range bounds.
-Wundefined-control-flow
Warn if some input strings cause undefined control flow in the lexer (the faulty
patterns are reported). This is a dangerous and common mistake. It can be easily
fixed by adding the default rule * which has the lowest priority, matches any code
unit, and always consumes a single code unit.
-Wunreachable-rules
Warn about rules that are shadowed by other rules and will never match.
-Wuseless-escape
Warn if a symbol is escaped when it shouldn't be. By default, re2ocaml silently
ignores such escapes, but this may as well indicate a typo or an error in the
escape sequence.
-Wnondeterministic-tags
Warn if a tag has n-th degree of nondeterminism, where n is greater than 1.
-Wsentinel-in-midrule
Warn if the sentinel symbol occurs in the middle of a rule --- this may cause reads
past the end of buffer, crashes or memory corruption in the generated lexer. This
warning is only applicable if the sentinel method of checking for the end of input
is used. It is set to an error if re2c:sentinel configuration is used.
-Wundefined-syntax-config
Warn if the syntax file specified with --syntax option is missing definitions of
some configurations. This helps to maintain user-defined syntax files: if a new
release adds configurations, old syntax file will raise a warning, and the user
will be notified. If some configurations are unused and do not need a definition,
they should be explicitly set to <undefined>.
Syntax files
Support for different languages in re2c is based on the idea of syntax files. A syntax
file is a configuration file that defines syntax of the target language -- not the whole
language, but a small part of it that is used by the generated code. Syntax files make
re2c very flexible, but they should not be used as a replacement for re2c: configurations:
their purpose is to define syntax of the target language, not to customize one particular
lexer. All supported languages have default syntax files that are part of the distribution
(see include/syntax subdirectory); they are also embedded in the re2ocaml binary. Users
may provide a custom syntax file that overrides a few configurations for one of supported
languages, or they may choose to redefine all configurations (in that case --lang none
option should be used). Syntax files contain configurations of four different kinds:
feature lists, language configurations, inplace configurations and code templates.
Feature lists
A few list configurations define various features supported by a given backend, so that
re2ocaml may give a clear error if the user tries to enable an unsupported feature:
supported_apis
A list of supported APIs with possible elements simple, record, generic.
supported_api_styles
A list of supported API styles with possible elements functions, free-form.
supported_code_models
A list of supported code models with possible elements goto-label, loop-switch,
recursive-functions.
supported_targets
A list of supported codegen targets with possible elements code, dot, skeleton.
supported_features
A list of supported features with possible elements nested-ifs, bitmaps,
computed-gotos, case-ranges, monadic, unsafe, tags, captures, captvars.
Language configurations
A few boolean configurations describe features of the target language that affect
re2ocaml parser and code generator:
semicolons
Non-zero if the language uses semicolons after statements.
backtick_quoted_strings
Non-zero if the language has backtick-quoted strings.
single_quoted_strings
Non-zero if the language has single-quoted strings.
indentation_sensitive
Non-zero if the language is indentation sensitive.
wrap_blocks_in_braces
Non-zero if compound statements must be wrapped in curly braces.
Inplace configurations
Syntax files define initial values of all re2c: configurations, as they may differ for
different languages. See configurations section for a full list of all inplace
configurations and their meaning.
Code templates
Code templates define syntax of the target language. They are written in a simple
domain-specific language with the following formal grammar:
code-template ::
name '=' code-exprs ';'
| CODE_TEMPLATE ';'
| '<undefined>' ';'
code-exprs ::
<EMPTY>
| code-exprs code-expr
code-expr ::
STRING
| VARIABLE
| optional
| list
optional ::
'(' CONDITIONAL '?' code-exprs ')'
| '(' CONDITIONAL '?' code-exprs ':' code-exprs ')'
list ::
'[' VARIABLE ':' code-exprs ']'
| '[' VARIABLE '{' NUMBER '}' ':' code-exprs ']'
| '[' VARIABLE '{' NUMBER ',' NUMBER '}' ':' code-exprs ']'
A code template is a sequence of string literals, variables, optional elements and
lists, or a reference to another code template, or a special value <undefined>.
Variables are placeholders that are substituted during code generation phase. List
variables are special: when expanding list templates, re2ocaml repeats expressions the
right hand side of the column a few times, each time replacing occurrences of the list
variable with a value specific to this repetition. Lists have optional bounds (negative
values are counted from the end, e.g. -1 means the last element). Conditional names
start with a dot. Both conditionals and variables may be either local (specific to the
given code template) or global (allowed in all code templates). When re2ocaml reads
syntax file, it checks that each code template uses only the variables and conditionals
that are allowed in it.
For example, the following code template defines if-then-else construct for a C-like
language:
code:if_then_else =
[branch{0}: topindent "if " cond " {" nl
indent [stmt: stmt] dedent]
[branch{1:-1}: topindent "} else" (.cond ? " if " cond) " {" nl
indent [stmt: stmt] dedent]
topindent "}" nl;
Here branch is a list variable: branch{0} expands to the first branch (which is
special, as there is no else part), branch{1:-1} expands to all remaining branches (if
any). stmt is also a list variable: [stmt: stmt] is a nested list that expands to a
list of statements in the body of the current branch. topindent, indent, dedent and nl
are global variables, and .cond is a local conditional (their meaning is described
below). This code template could produce the following code:
if x {
// do something
} else if y {
// do something else
} else {
// don't do anything
}
Here's a list of all code templates supported by re2ocaml with their local variables
and conditionals. Note that a particular definition may, but does not have to use local
variables and conditionals. Any unused code templates should be set to <undefined>.
code:var_local
Declaration or definition of a local variable. Supported variables: type (the
type of the variable), name (its name) and init (initial value, if any).
Conditionals: .init (true if there is an initializer).
code:var_global
Same as code:var_local, except that it's used in top-level.
code:const_local
Definition of a local constant. Supported variables: type (the type of the
constant), name (its name) and init (initial value).
code:const_global
Same as code:const_local, except that it's used in top-level.
code:array_local
Definition of a local array (table). Supported variables: type (the type of
array elements), name (array name), size (its size), row (a list variable that
does not itself produce any code, but expands list expression as many times as
there are rows in the table) and elem (a list variable that expands to all table
elements in the current row -- it's meant to be nested in the row list).
code:array_global
Same as code:array_local, except that it's used in top-level.
code:array_elem
Reference to an element of an array (table). Supported variables: array (the
name of the array) and index (index of the element).
code:enum
Definition of an enumeration (it may be defined using a special language
construct for enumerations, or simply as a few standalone constants). Supported
variables are type (user-defined enumeration type or type of the constants),
elem (list variable that expands to the name of each member) and init
(initializer for each member). Conditionals: .init (true if there is an
initializer).
code:enum_elem
Enumeration element (a member of a user-defined enumeration type or a name of a
constant, depending on how code:enum is defined). Supported variables are name
(the name of the element) and type (its type).
code:assign
Assignment statement. Supported variables are lhs (left hand side) and rhs
(right hand side).
code:type_int
Signed integer type.
code:type_uint
Unsigned integer type.
code:type_yybm
Type of elements in the yybm table.
code:type_yytarget
Type of elements in the yytarget table.
code:cmp_eq
Operator "equals".
code:cmp_ne
Operator "not equals".
code:cmp_lt
Operator "less than".
code:cmp_gt
Operator "greater than"
code:cmp_le
Operator "less or equal"
code:cmp_ge
Operator "greater or equal"
code:if_then_else
If-then-else statement with one or more branches. Supported variables: branch (a
list variable that does not itself produce any code, but expands list expression
as many times as there are branches), cond (condition of the current branch) and
stmt (a list variable that expands to all statements in the current branch).
Conditionals: .cond (true if the current branch has a condition), .many (true if
there's more than one branch).
code:if_then_else_oneline
A specialization of code:if_then_else for the case when all branches have
one-line statements. If this is <undefined>, code:if_then_else is used instead.
code:switch
A switch statement with one or more cases. Supported variables: expr (the
switched-on expression) and case (a list variable that expands to all
cases-groups with their code blocks).
code:switch_cases
A group of switch cases that maps to a single code block. Supported variables
are case (a list variable that expands to all cases in this group) and stmt (a
list variable that expands to all statements in the code block.
code:switch_cases_oneline
A specialization of code:switch_cases for the case when the code block consists
of a single one-line statement. If this is <undefined>, code:switch_cases is
used instead.
code:switch_case_range
A single switch case that covers a range of values (possibly consisting of a
single value). Supported variable: val (a list variable that expands to all
values in the range). Supported conditionals: .many (true if there's more than
one value in the range) and .char_literals (true if this is a switch on
character literals -- some languages provide special syntax for this case).
code:switch_case_default
Default switch case.
code:loop
A loop that runs forever (unless interrupted from the loop body). Supported
variables: label (loop label), stmt (a list variable that expands to all
statements in the loop body).
code:continue
Continue statement. Supported variables: label (label from which to continue
execution).
code:goto
Goto statement. Supported variables: label (label of the jump target).
code:fndecl
Function declaration. Supported variables: name (function name), type (return
type), arg (a list variable that does not itself produce code, but expands list
expression as many times as there are function arguments), argname (name of the
current argument), argtype (type of the current argument). Conditional: .type
(true if this is a non-void function).
code:fndef
Like code:fndecl, but used for function definitions, so it has one additional
list variable stmt that expands to all statements in the function body.
code:fncall
Function call statement. Supported variables: name (function name), retval
(l-value where the return value is stored, if any) and arg (a list variable that
expands to all function arguments). Conditionals: .args (true if the function
has arguments) and .retval (true if return value needs to be saved).
code:tailcall
Tail call statement. Supported variables: name (function name), and arg (a list
variable that expands to all function arguments). Conditionals: .args (true if
the function has arguments) and .retval (true if this is a non-void function).
code:recursive_functions
Program body with --recursive-functions code model. Supported variables: fn (a
list variable that does not itself produce any code, but expands list expression
as many times as there are functions), fndecl (declaration of the current
function) and fndef (definition of the current function).
code:fingerprint
The fingerprint at the top of the generated output file. Supported variables:
ver (re2ocaml version that was used to generate this) and date (generation
date).
code:line_info
The format of line directives (if this is set to <undefined>, no directives are
generated). Supported variables: line (line number) and file (filename).
code:abort
A statement that aborts program execution.
code:yydebug
YYDEBUG statement, possibly specialized for different APIs. Supported
variables: YYDEBUG, yyrecord, yych (map to the corresponding re2c:
configurations), state (DFA state number).
code:yypeek
YYPEEK statement, possibly specialized for different APIs. Supported variables:
YYPEEK, YYCTYPE, YYINPUT, YYCURSOR, yyrecord, yych (map to the corresponding
re2c: configurations). Conditionals: .cast (true if re2c:yych:conversion is set
to non-zero).
code:yyskip
YYSKIP statement, possibly specialized for different APIs. Supported variables:
YYSKIP, YYCURSOR, yyrecord (map to the corresponding re2c: configurations).
code:yybackup
YYBACKUP statement, possibly specialized for different APIs. Supported
variables: YYBACKUP, YYCURSOR, YYMARKER, yyrecord (map to the corresponding
re2c: configurations).
code:yybackupctx
YYBACKUPCTX statement, possibly specialized for different APIs. Supported
variables: YYBACKUPCTX, YYCURSOR, YYCTXMARKER, yyrecord (map to the
corresponding re2c: configurations).
code:yyskip_yypeek
Combined code:yyskip and code:yypeek statement (defaults to code:yyskip followed
by code:yypeek).
code:yypeek_yyskip
Combined code:yypeek and code:yyskip statement (defaults to code:yypeek followed
by code:yyskip).
code:yyskip_yybackup
Combined code:yyskip and code:yybackup statement (defaults to code:yyskip
followed by code:yybackup).
code:yybackup_yyskip
Combined code:yybackup and code:yyskip statement (defaults to code:yybackup
followed by code:yyskip).
code:yybackup_yypeek
Combined code:yybackup and code:yypeek statement (defaults to code:yybackup
followed by code:yypeek).
code:yyskip_yybackup_yypeek
Combined code:yyskip, code:yybackup and code:yypeek statement (defaults
to``code:yyskip`` followed by code:yybackup followed by code:yypeek).
code:yybackup_yypeek_yyskip
Combined code:yybackup, code:yypeek and code:yyskip statement (defaults
to``code:yybackup`` followed by code:yypeek followed by code:yyskip).
code:yyrestore
YYRESTORE statement, possibly specialized for different APIs. Supported
variables: YYRESTORE, YYCURSOR, YYMARKER, yyrecord (map to the corresponding
re2c: configurations).
code:yyrestorectx
YYRESTORECTX statement, possibly specialized for different APIs. Supported
variables: YYRESTORECTX, YYCURSOR, YYCTXMARKER, yyrecord (map to the
corresponding re2c: configurations).
code:yyrestoretag
YYRESTORETAG statement, possibly specialized for different APIs. Supported
variables: YYRESTORETAG, YYCURSOR, yyrecord (map to the corresponding re2c:
configurations), tag (the name of tag variable used to restore position).
code:yyshift
YYSHIFT statement, possibly specialized for different APIs. Supported
variables: YYSHIFT, YYCURSOR, yyrecord (map to the corresponding re2c:
configurations), offset (the number of code units to shift the current
position).
code:yyshiftstag
YYSHIFTSTAG statement, possibly specialized for different APIs. Supported
variables: YYSHIFTSTAG, yyrecord, negative (map to the corresponding re2c:
configurations), tag (tag variable which needs to be shifted), offset (the
number of code units to shift). Conditionals: .nested (true if this is a nested
tag -- in this case its value may equal to re2c:tags:negative, which should not
be shifted).
code:yyshiftmtag
YYSHIFTMTAG statement, possibly specialized for different APIs. Supported
variables: YYSHIFTMTAG (maps to the corresponding re2c: configuration), tag (tag
variable which needs to be shifted), offset (the number of code units to shift).
code:yystagp
YYSTAGP statement, possibly specialized for different APIs. Supported
variables: YYSTAGP, YYCURSOR, yyrecord (map to the corresponding re2c:
configurations), tag (tag variable that should be updated).
code:yymtagp
YYMTAGP statement, possibly specialized for different APIs. Supported
variables: YYMTAGP (maps to the corresponding re2c: configuration), tag (tag
variable that should be updated).
code:yystagn
YYSTAGN statement, possibly specialized for different APIs. Supported
variables: YYSTAGN, negative, yyrecord (map to the corresponding re2c:
configurations), tag (tag variable that should be updated).
code:yymtagn
YYMTAGN statement, possibly specialized for different APIs. Supported
variables: YYMTAGN (maps to the corresponding re2c: configuration), tag (tag
variable that should be updated).
code:yycopystag
YYCOPYSTAG statement, possibly specialized for different APIs. Supported
variables: YYCOPYSTAG, yyrecord (map to the corresponding re2c: configurations),
lhs, rhs (left and right hand side tag variables of the copy operation).
code:yycopymtag
YYCOPYMTAG statement, possibly specialized for different APIs. Supported
variables: YYCOPYMTAG, yyrecord (map to the corresponding re2c: configurations),
lhs, rhs (left and right hand side tag variables of the copy operation).
code:yygetaccept
YYGETACCEPT statement, possibly specialized for different APIs. Supported
variables: YYGETACCEPT, yyrecord (map to the corresponding re2c:
configurations), var (maps to re2c:yyaccept configuration).
code:yysetaccept
YYSETACCEPT statement, possibly specialized for different APIs. Supported
variables: YYSETACCEPT, yyrecord (map to the corresponding re2c:
configurations), var (maps to re2c:yyaccept configuration) and val (numeric
value of the accepted rule).
code:yygetcond
YYGETCOND statement, possibly specialized for different APIs. Supported
variables: YYGETCOND, yyrecord (map to the corresponding re2c: configurations),
var (maps to re2c:yycond configuration).
code:yysetcond
YYSETCOND statement, possibly specialized for different APIs. Supported
variables: YYSETCOND, yyrecord (map to the corresponding re2c: configurations),
var (maps to re2c:yycond configuration) and val (numeric condition identifier).
code:yygetstate
YYGETSTATE statement, possibly specialized for different APIs. Supported
variables: YYGETSTATE, yyrecord (map to the corresponding re2c: configurations),
var (maps to re2c:yystate configuration).
code:yysetstate
YYSETSTATE statement, possibly specialized for different APIs. Supported
variables: YYSETSTATE, yyrecord (map to the corresponding re2c: configurations),
var (maps to re2c:yystate configuration) and val (state number).
code:yylessthan
YYLESSTHAN statement, possibly specialized for different APIs. Supported
variables: YYLESSTHAN, YYCURSOR, YYLIMIT, yyrecord (map to the corresponding
re2c: configurations), need (the number of code units to check against).
Conditional: .many (true if the need is more than one).
code:yybm_filter
Condition that is used to filter out yych values that are not covered by the
yybm table (used with --bitmaps option). Supported variable: yych (maps to
re2c:yych configuration).
code:yybm_match
The format of yybm table check (generated with --bitmaps option). Supported
variables: yybm, yych (map to the corresponding re2c: configurations), offset
(offset in the yybm table that needs to be added to yych) and mask (bit mask
that should be applied to the table entry to retrieve the boolean value that
needs to be checked)
Here's a list of all global variables that are allowed in syntax files:
nl A newline.
indent A variable that does not produce any code, but has a side-effect of increasing
indentation level.
dedent A variable that does not produce any code, but has a side-effect of decreasing
indentation level.
topindent
Indentation string for the current statement. Indentation level is tracked and
automatically updated by the code generator.
Here's a list of all global conditionals that are allowed in syntax files:
.api.simple
True if simple API is used (--api simple or re2c:api = simple).
.api.generic
True if generic API is used (--api generic or re2c:api = generic).
.api.record
True if record API is used (--api record or re2c:api = record).
.api_style.functions
True if function-like API style is used (re2c:api-style = functions).
.api_style.freeform
True if free-form API style is used (re2c:api-style = free-form).
.case_ranges
True if case ranges feature is enabled (--case-ranges or re2c:case-ranges = 1).
.code_model.goto_label
True if code model based on goto/label is used (--goto-label).
.code_model.loop_switch
True if code model based on loop/switch is used (--loop-switch).
.code_model.recursive_functions
True if code model based on recursive functions is used (--recursive-function).
.date True if the generated fingerprint should contain generation date.
.loop_label
True if re2ocaml generated loops must have a label (re2c:label:yyloop is set to
a nonempty string).
.monadic
True if the generated code should be monadic (re2c:monadic = 1). This is only
relevant for pure functional languages.
.start_conditions
True if start conditions are enabled (--start-conditions).
.storable_state
True if storable state is enabled (--storable-state).
.unsafe
True if re2ocaml should use "unsafe" blocks in order to generate faster code
(--unsafe, re2c:unsafe = 1). This is only relevant for languages that have
"unsafe" feature.
.version
True if the generated fingerprint should contain re2ocaml version.
HANDLING THE END OF INPUT
One of the main problems for the lexer is to know when to stop. There are a few
terminating conditions:
• the lexer may match some rule (including default rule *) and come to a final state
• the lexer may fail to match any rule and come to a default state
• the lexer may reach the end of input
The first two conditions terminate the lexer in a "natural" way: it comes to a state with
no outgoing transitions, and the matching automatically stops. The third condition, end of
input, is different: it may happen in any state, and the lexer should be able to handle
it. Checking for the end of input interrupts the normal lexer workflow and adds
conditional branches to the generated program, therefore it is necessary to minimize the
number of such checks. re2ocaml supports a few different methods for handling the end of
input. Which one to use depends on the complexity of regular expressions, the need for
buffering, performance considerations and other factors. Here is a list of methods:
• Sentinel. This method eliminates the need for the end of input checks altogether. It is
simple and efficient, but limited to the case when there is a natural "sentinel"
character that can never occur in valid input. This character may still occur in invalid
input, but it should not be allowed by the regular expressions, except perhaps as the
last character of a rule. The sentinel is appended at the end of input and serves as a
stop signal: when the lexer reads this character, it is either a syntax error or the end
of input. In both cases the lexer should stop. This method is used if YYFILL is disabled
with re2c:yyfill:enable = 0; and re2c:eof has the default value -1.
• Sentinel with bounds checks. This method is generic: it allows one to handle any input
without restrictions on the regular expressions. The idea is to reduce the number of end
of input checks by performing them only on certain characters. Similar to the "sentinel"
method, one of the characters is chosen as a "sentinel" and appended at the end of
input. However, there is no restriction on where the sentinel may occur (in fact, any
character can be chosen for a sentinel). When the lexer reads this character, it
additionally performs a bounds check. If the current position is within bounds, the
lexer resumes matching and handles the sentinel as a regular character. Otherwise it
invokes YYFILL (unless it is disabled). If more input is supplied, the lexer will
rematch the last character and continue as if the sentinel wasn't there. Otherwise it
must be the real end of input, and the lexer stops. This method is used when re2c:eof
has non-negative value (it should be set to the numeric value of the sentinel). YYFILL
is optional.
• Bounds checks with padding. This method is generic, and it may be faster than the
"sentinel with bounds checks" method, but it is also more complex. The idea is to
partition DFA states into strongly connected components (SCCs) and generate a single
check per SCC for enough characters to cover the longest non-looping path in this SCC.
This reduces the number of checks, but there is a problem with short lexemes at the end
of input, as the check requires enough characters to cover the longest lexeme. This can
be fixed by padding the input with a few fake characters that do not form a valid lexeme
suffix (so that the lexer cannot match them). The length of padding should be YYMAXFILL,
generated with a max block. If there is not enough input, the lexer invokes YYFILL which
should supply at least the required number of characters or not return. This method is
used if YYFILL is enabled and re2c:eof is -1 (this is the default configuration).
• Custom checks. Generic API allows one to override basic operations like reading a
character, which makes it possible to include the end-of-input checks as part of them.
This approach is error-prone and should be used with caution. To use a custom method,
enable generic API with --api custom or re2c:api = custom; and disable default bounds
checks with re2c:yyfill:enable = 0; or re2c:yyfill:check = 0;.
The following subsections contain an example of each method.
Sentinel
This example uses a sentinel character to handle the end of input. The program counts
space-separated words in a null-terminated string. The sentinel is null: it is the last
character of each input string, and it is not allowed in the middle of a lexeme by any of
the rules (in particular, it is not included in character ranges where it is easy to
overlook). If a null occurs in the middle of a string, it is a syntax error and the lexer
will match default rule *, but it won't read past the end of input or crash (use
-Wsentinel-in-midrule <https://re2c.org/manual/basics/warnings/warnings.html#wsentinel-in-
midrule>
warning and re2c:sentinel configuration to verify this). Configuration re2c:yyfill:enable
= 0; suppresses the generation of bounds checks and YYFILL invocations.
(* re2ocaml $INPUT -o $OUTPUT *)
open String
type state = {
yyinput: string;
mutable yycursor: int;
}
(* expect a null-terminated string *)
%{
re2c:YYFN = ["lex;int", "yyrecord;state", "count;int"];
re2c:yyfill:enable = 0;
* { -1 }
[\x00] { count }
[a-z]+ { lex yyrecord (count + 1) }
[ ]+ { lex yyrecord count }
%}
let test(yyinput, count) =
let st = {yyinput = yyinput; yycursor = 0}
in if not (lex st 0 = count) then raise (Failure "error")
let main () =
test("\x00", 0);
test("one two three\x00", 3);
test("f0ur\x00", -1)
let _ = main ()
Sentinel with bounds checks
This example uses sentinel with bounds checks to handle the end of input (this method was
added in version 1.2). The program counts space-separated single-quoted strings. The
sentinel character is null, which is specified with re2c:eof = 0; configuration. As in the
sentinel method, null is the last character of each input string, but it is allowed in the
middle of a rule (for example, 'aaa\0aa'\0 is valid input, but 'aaa\0 is a syntax error).
Bounds checks are generated in each state that matches an input character, but they are
scoped to the branch that handles null. Bounds checks are of the form YYLIMIT <= YYCURSOR
or YYLESSTHAN(1) with generic API. If the check condition is true, lexer has reached the
end of input and should stop (YYFILL is disabled with re2c:yyfill:enable = 0; as the input
fits into one buffer, see the YYFILL with sentinel section for an example that uses
YYFILL). Reaching the end of input opens three possibilities: if the lexer is in the
initial state it will match the end-of-input rule $, otherwise it may fallback to a
previously matched rule (including default rule *) or go to a default state, causing
-Wundefined-control-flow
<https://re2c.org/manual/basics/warnings/warnings.html#wundefined-control-flow> .
(* re2ocaml $INPUT -o $OUTPUT *)
open String
type state = {
yyinput: string;
mutable yycursor: int;
mutable yymarker: int;
yylimit: int;
}
(* expect a null-terminated string *)
%{
re2c:YYFN = ["lex;int", "yyrecord;state", "count;int"];
re2c:yyfill:enable = 0;
re2c:eof = 0;
str = ['] ([^'\\] | [\\][^])* ['];
* { -1 }
$ { count }
str { lex yyrecord (count + 1) }
[ ]+ { lex yyrecord count }
%}
let test(str, count) =
let st = {
yyinput = str;
yycursor = 0;
yymarker = 0;
yylimit = length str - 1; (* terminating null not included *)
}
in if not (lex st 0 = count) then raise (Failure "error")
let main () =
test("\x00", 0);
test("'qu\x00tes' 'are' 'fine: \\'' \x00", 3);
test("'unterminated\\'\x00", -1)
let _ = main ()
Bounds checks with padding
This example uses bounds checks with padding to handle the end of input (this method is
enabled by default). The program counts space-separated single-quoted strings. There is a
padding of YYMAXFILL null characters appended at the end of input, where YYMAXFILL value
is autogenerated with a max block. It is not necessary to use null for padding --- any
characters can be used as long as they do not form a valid lexeme suffix (in this example
padding should not contain single quotes, as they may be mistaken for a suffix of a
single-quoted string). There is a "stop" rule that matches the first padding character
(null) and terminates the lexer (note that it checks if null is at the beginning of
padding, otherwise it is a syntax error). Bounds checks are generated only in some states
that are determined by the strongly connected components of the underlying automaton.
Checks have the form (YYLIMIT - YYCURSOR) < n or YYLESSTHAN(n) with generic API, where n
is the minimum number of characters that are needed for the lexer to proceed (it also
means that the next bounds check will occur in at most n characters). If the check
condition is true, the lexer has reached the end of input and will invoke YYFILL(n) that
should either supply at least n input characters or not return. In this example YYFILL
always fails and terminates the lexer with an error (which is fine because the input fits
into one buffer). See the YYFILL with padding section for an example that refills the
input buffer with YYFILL.
(* re2ocaml $INPUT -o $OUTPUT *)
open String
exception Fill
type state = {
yyinput: string;
mutable yycursor: int;
yylimit: int;
}
%{max %}
%{
re2c:YYFN = ["lex;int", "yyrecord;state", "count;int"];
re2c:YYFILL = "raise Fill;";
str = ['] ([^'\\] | [\\][^])* ['];
[\x00] {
(* check that it is the sentinel, not some unexpected null *)
if yyrecord.yycursor = length yyrecord.yyinput - yymaxfill + 1 then count else -1
}
str { lex yyrecord (count + 1) }
[ ]+ { lex yyrecord count }
* { -1 }
%}
let test(str, count) =
let buf = cat str (make yymaxfill '\x00') in
let st = {yyinput = buf; yycursor = 0; yylimit = length buf} in
let result = try lex st 0 with Fill -> -1 in
if not (result = count) then raise (Failure "error")
let main () =
test("", 0);
test("'unterminated\\'", -1);
test("'qu\x00tes' 'are' 'fine: \\'' ", 3);
test("'unexpected \x00 null", -1)
let _ = main ()
Custom checks
This example uses a custom end-of-input handling method based on generic API. The program
counts space-separated single-quoted strings. It is the same as the sentinel example,
except that the input is not null-terminated. To cover up for the absence of a sentinel
character at the end of input, YYPEEK is redefined to perform a bounds check before it
reads the next input character. This is inefficient because checks are done very often.
If the check condition fails, YYPEEK returns the real character, otherwise it returns a
fake sentinel character.
(* re2ocaml $INPUT -o $OUTPUT *)
type state = {
str: string;
mutable cur: int;
lim: int;
}
(* expect a string without terminating null *)
%{
re2c:api = generic;
re2c:YYFN = ["lex;int", "st;state", "count;int"];
re2c:YYPEEK = "if st.cur < st.lim then st.str.[st.cur] else '\\x00'";
re2c:YYSKIP = "st.cur <- st.cur + 1;";
re2c:yyfill:enable = 0;
* { -1 }
[\x00] { count }
[a-z]+ { lex st (count + 1) }
[ ]+ { lex st count }
%}
let test(str, count) =
let st = {str = str; cur = 0; lim = String.length str}
in if not (lex st 0 = count) then raise (Failure "error")
let main () =
test("", 0);
test("one two three", 3);
test("f0ur", -1)
let _ = main ()
BUFFER REFILLING
The need for buffering arises when the input cannot be mapped in memory all at once:
either it is too large, or it comes in a streaming fashion (like reading from a socket).
The usual technique in such cases is to allocate a fixed-sized memory buffer and process
input in chunks that fit into the buffer. When the current chunk is processed, it is moved
out and new data is moved in. In practice it is somewhat more complex, because lexer state
consists not of a single input position, but a set of interrelated positions:
• cursor: the next input character to be read (YYCURSOR in C pointer API or YYSKIP/YYPEEK
in generic API)
• limit: the position after the last available input character (YYLIMIT in C pointer API,
implicitly handled by YYLESSTHAN in generic API)
• marker: the position of the most recent match, if any (YYMARKER in default API or
YYBACKUP/YYRESTORE in generic API)
• token: the start of the current lexeme (implicit in re2ocaml API, as it is not needed
for the normal lexer operation and can be defined and updated by the user)
• context marker: the position of the trailing context (YYCTXMARKER in C pointer API or
YYBACKUPCTX/YYRESTORECTX in generic API)
• tag variables: submatch positions (defined with stags and mtags blocks and generic API
primitives YYSTAGP/YYSTAGN/YYMTAGP/YYMTAGN)
Not all these are used in every case, but if used, they must be updated by YYFILL. All
active positions are contained in the segment between token and cursor, therefore
everything between buffer start and token can be discarded, the segment from token and up
to limit should be moved to the beginning of buffer, and the free space at the end of
buffer should be filled with new data. In order to avoid frequent YYFILL calls it is best
to fill in as many input characters as possible (even though fewer characters might
suffice to resume the lexer). The details of YYFILL implementation are slightly different
depending on which EOF handling method is used: the case of EOF rule is somewhat simpler
than the case of bounds-checking with padding. Also note that if -f --storable-state
option is used, YYFILL has slightly different semantics (described in the section about
storable state).
YYFILL with sentinel
If EOF rule is used, YYFILL is a function-like primitive that accepts no arguments and
returns a value which is checked against zero. YYFILL invocation is triggered by condition
YYLIMIT <= YYCURSOR in C pointer API and YYLESSTHAN() in generic API. A non-zero return
value means that YYFILL has failed. A successful YYFILL call must supply at least one
character and adjust input positions accordingly. Limit must always be set to one after
the last input position in buffer, and the character at the limit position must be the
sentinel symbol specified by re2c:eof configuration. The pictures below show the relative
locations of input positions in buffer before and after YYFILL call (sentinel symbol is
marked with #, and the second picture shows the case when there is not enough input to
fill the whole buffer).
<-- shift -->
>-A------------B---------C-------------D#-----------E->
buffer token marker limit,
cursor
>-A------------B---------C-------------D------------E#->
buffer, marker cursor limit
token
<-- shift -->
>-A------------B---------C-------------D#--E (EOF)
buffer token marker limit,
cursor
>-A------------B---------C-------------D---E#........
buffer, marker cursor limit
token
Here is an example of a program that reads input file input.txt in chunks of 4096 bytes
and uses EOF rule.
(* re2ocaml $INPUT -o $OUTPUT *)
open Bytes
let bufsize = 4096
type state = {
file: in_channel;
yyinput: bytes;
mutable yycursor: int;
mutable yymarker: int;
mutable yylimit: int;
mutable token: int;
mutable eof: bool;
}
type status = Ok | Eof | LongLexeme
let fill(st: state) : status =
if st.eof then Eof else
(* Error: lexeme too long. In real life could reallocate a larger buffer. *)
if st.token < 1 then LongLexeme else (
(* Shift buffer contents (discard everything up to the current token). *)
blit st.yyinput st.token st.yyinput 0 (st.yylimit - st.token);
st.yycursor <- st.yycursor - st.token;
st.yymarker <- st.yymarker - st.token;
st.yylimit <- st.yylimit - st.token;
st.token <- 0;
(* Fill free space at the end of buffer with new data from file. *)
let n = input st.file st.yyinput st.yylimit (bufsize - st.yylimit - 1) in (* -1 for sentinel *)
st.yylimit <- st.yylimit + n;
if n = 0 then
st.eof <- true; (* end of file *)
set st.yyinput st.yylimit '\x00'; (* append sentinel *)
Ok)
%{
re2c:YYFN = ["lex;int", "yyrecord;state", "count;int"];
re2c:YYFILL = "fill yyrecord = Ok";
re2c:eof = 0;
str = ['] ([^'\\] | [\\][^])* ['];
* { -1 }
$ { count }
str { lex_loop yyrecord (count + 1) }
[ ]+ { lex_loop yyrecord count }
%}
and lex_loop st count =
st.token <- st.yycursor;
lex st count
let main () =
let fname = "input" in
(* Prepare input file. *)
Out_channel.with_open_bin fname
(fun oc -> for i = 1 to bufsize do
output_string oc "'qu\x00tes' 'are' 'fine: \\'' "
done);
(* Run lexer on the prepared file. *)
In_channel.with_open_bin fname
(fun ic ->
let yylimit = bufsize - 1 in
let st = {
file = ic;
yyinput = create bufsize;
yycursor = yylimit;
yymarker = yylimit;
yylimit = yylimit;
token = yylimit;
eof = false;
} in if not (lex_loop st 0 = 3 * bufsize) then
raise (Failure "error"));
(* Cleanup. *)
Sys.remove fname
let _ = main ()
YYFILL with padding
In the default case (when EOF rule is not used) YYFILL is a function-like primitive that
accepts a single argument and does not return any value. YYFILL invocation is triggered
by condition (YYLIMIT - YYCURSOR) < n in C pointer API and YYLESSTHAN(n) in generic API.
The argument passed to YYFILL is the minimal number of characters that must be supplied.
If it fails to do so, YYFILL must not return to the lexer (for that reason it is best
implemented as a macro that returns from the calling function on failure). In case of a
successful YYFILL invocation the limit position must be set either to one after the last
input position in buffer, or to the end of YYMAXFILL padding (in case YYFILL has
successfully read at least n characters, but not enough to fill the entire buffer). The
pictures below show the relative locations of input positions in buffer before and after
YYFILL invocation (YYMAXFILL padding on the second picture is marked with # symbols).
<-- shift --> <-- need -->
>-A------------B---------C-----D-------E---F--------G->
buffer token marker cursor limit
>-A------------B---------C-----D-------E---F--------G->
buffer, marker cursor limit
token
<-- shift --> <-- need -->
>-A------------B---------C-----D-------E-F (EOF)
buffer token marker cursor limit
>-A------------B---------C-----D-------E-F###############
buffer, marker cursor limit
token <- YYMAXFILL ->
Here is an example of a program that reads input file input.txt in chunks of 4096 bytes
and uses bounds-checking with padding.
(* re2ocaml $INPUT -o $OUTPUT *)
open Bytes
%{max %}
let bufsize = 4096
exception Fill
type state = {
file: in_channel;
yyinput: bytes;
mutable yycursor: int;
mutable yymarker: int;
mutable yylimit: int;
mutable token: int;
mutable eof: bool;
}
type status = Ok | Eof | LongLexeme
let fill (st: state) (need: int) : status =
if st.eof then Eof else
(* Error: lexeme too long. In real life could reallocate a larger buffer. *)
if st.token < need then LongLexeme else (
(* Shift buffer contents (discard everything up to the current token). *)
blit st.yyinput st.token st.yyinput 0 (st.yylimit - st.token);
st.yycursor <- st.yycursor - st.token;
st.yymarker <- st.yymarker - st.token;
st.yylimit <- st.yylimit - st.token;
st.token <- 0;
(* Fill free space at the end of buffer with new data from file. *)
let n = input st.file st.yyinput st.yylimit (bufsize - st.yylimit - 1) in (* -1 for sentinel *)
st.yylimit <- st.yylimit + n;
(* If read zero characters, this is end of input => add zero padding
so that the lexer can access characters at the end of buffer. *)
if n = 0 then
st.eof <- true; (* end of file *)
for i = 0 to (yymaxfill - 1) do
set st.yyinput (st.yylimit + i) '\x00';
st.yylimit <- st.yylimit + yymaxfill
done;
Ok)
%{
re2c:YYFN = ["lex;int", "yyrecord;state", "count;int"];
re2c:YYFILL = "if not (fill yyrecord @@ = Ok) then raise Fill;";
str = ['] ([^'\\] | [\\][^])* ['];
[\x00] {
(* check that it is the sentinel, not some unexpected null *)
if yyrecord.token = yyrecord.yylimit - yymaxfill then count else -1
}
str { lex_loop yyrecord (count + 1) }
[ ]+ { lex_loop yyrecord count }
* { -1 }
%}
and lex_loop st count =
st.token <- st.yycursor;
try lex st count with Fill -> -1
let main () =
let fname = "input" in
(* Prepare input file. *)
Out_channel.with_open_bin fname
(fun oc -> for i = 1 to bufsize do
output_string oc "'qu\x00tes' 'are' 'fine: \\'' "
done);
(* Run lexer on the prepared file. *)
In_channel.with_open_bin fname
(fun ic ->
let yylimit = bufsize - yymaxfill in
let st = {
file = ic;
yyinput = create bufsize;
yycursor = yylimit;
yymarker = yylimit;
yylimit = yylimit;
token = yylimit;
eof = false;
} in if not (lex_loop st 0 = 3 * bufsize) then
raise (Failure "error"));
(* Cleanup. *)
Sys.remove fname
let _ = main ()
FEATURES
Multiple blocks
Sometimes it is necessary to have multiple interrelated lexers (for example, if there is a
high-level state machine that transitions between lexer modes). This can be implemented
using multiple connected re2ocaml blocks. Another option is to use start conditions.
The implementation of connections between blocks depends on the target language. In
languages that have goto statement (such as C/C++ and Go) one can have all blocks in one
function, each of them prefixed with a label. Transition from one block to another is a
simple goto. In languages that do not have goto (such as Rust) it is necessary to use a
loop with a switch on a state variable, similar to the yystate loop/switch generated by
re2ocaml, or else wrap each block in a function and use function calls.
The example below uses multiple blocks to parse binary, octal, decimal and hexadecimal
numbers. Each base has its own block. The initial block determines base and dispatches to
other blocks. Common configurations are defined in a separate block at the beginning of
the program; they are inherited by the other blocks.
(* re2ocaml $INPUT -o $OUTPUT -i *)
open Int64
open Option
open String
type state = {
yyinput: string;
mutable yycursor: int;
mutable yymarker: int;
}
let add (num: int option) (dgt: int) (base: int) : int option =
match num with
| None -> None
| Some n ->
let n' = add (mul (of_int n) (of_int base)) (of_int dgt)
in if n' > (of_int32 Int32.max_int) then None else Some (to_int n')
%{
re2c:yyrecord = "st";
re2c:yyfill:enable = 0;
%}
%{local
re2c:YYFN = ["parse_bin;int option", "st;state", "num;int option"];
[01] { parse_bin st (add num (Char.code st.yyinput.[st.yycursor - 1] - 48) 2) }
* { num }
%}
%{local
re2c:YYFN = ["parse_oct;int option", "st;state", "num;int option"];
[0-7] { parse_oct st (add num (Char.code st.yyinput.[st.yycursor - 1] - 48) 8) }
* { num }
%}
%{local
re2c:YYFN = ["parse_dec;int option", "st;state", "num;int option"];
[0-9] { parse_dec st (add num (Char.code st.yyinput.[st.yycursor - 1] - 48) 10) }
* { num }
%}
%{local
re2c:YYFN = ["parse_hex;int option", "st;state", "num;int option"];
[0-9] { parse_hex st (add num (Char.code st.yyinput.[st.yycursor - 1] - 48) 16) }
[a-f] { parse_hex st (add num (Char.code st.yyinput.[st.yycursor - 1] - 87) 16) }
[A-F] { parse_hex st (add num (Char.code st.yyinput.[st.yycursor - 1] - 55) 16) }
* { num }
%}
%{local
re2c:YYFN = ["parse;int option", "st;state"];
'0b' / [01] { parse_bin st (Some 0) }
"0" { parse_oct st (Some 0) }
"" / [1-9] { parse_dec st (Some 0) }
'0x' / [0-9a-fA-F] { parse_hex st (Some 0) }
* { None }
%}
let test (yyinput: string) (result: int option) =
let st = {yyinput = yyinput; yycursor = 0; yymarker = 0} in
if not (parse st = result) then raise (Failure "error")
let main () =
test "\x00" None;
test "1234567890\x00" (Some 1234567890);
test "0b1101\x00" (Some 13);
test "0x7Fe\x00" (Some 2046);
test "0644\x00" (Some 420);
test "9999999999\x00" None
let _ = main ()
Start conditions
Start conditions are enabled with --start-conditions option. They provide a way to encode
multiple interrelated automata within the same re2ocaml block.
Each condition corresponds to a single automaton and has a unique name specified by the
user and a unique internal number defined by re2ocaml. The numbers are used to switch
between conditions: the generated code uses YYGETCOND and YYSETCOND primitives to get the
current condition or set it to the given number. Use conditions block, --header option or
re2c:header configuration to generate numeric condition identifiers. Configuration
re2c:cond:enumprefix specifies the generated identifier prefix.
In condition mode every rule must be prefixed with a list of comma-separated condition
names in angle brackets, or a wildcard <*> to denote all conditions. The rule syntax is
extended as follows:
< cond-list > regexp action
A rule that is merged to every condition on the cond-list. It matches regexp
and executes the associated action.
< cond-list > regexp => cond action
A rule that is merged to every condition on the cond-list. It matches regexp,
sets the current condition to cond and executes the associated action.
< cond-list > regexp :=> cond
A rule that is merged to every condition on the cond-list. It matches regexp
and immediately transitions to cond (there is no semantic action).
<! cond-list > action
The action is prepended to semantic actions of all rules for every condition on
the cond-list. This may be used to deduplicate common code.
< > action
A rule that is merged to a special entry condition with number zero and name
"0". It matches empty string and executes the action.
< > => cond action
A rule that is merged to a special entry condition with number zero and name
"0". It matches empty string, sets the current condition to cond and executes
the action.
< > :=> cond
A rule that is merged to a special entry condition with number zero and name
"0". It matches empty string and immediately transitions to cond.
The code re2ocaml generates for conditions depends on whether re2ocaml uses goto/label
approach or loop/switch approach to encode the automata.
In languages that have goto statement (such as C/C++ and Go) conditions are naturally
implemented as blocks of code prefixed with labels of the form yyc_<cond>, where cond is a
condition name (label prefix can be changed with re2c:cond:prefix). Transitions between
conditions are implemented using goto and condition labels. Before all conditions re2ocaml
generates an initial switch on YYGETSTATE that jumps to the start state of the current
condition. The shortcut rules :=> bypass the initial switch and jump directly to the
specified condition (re2c:cond:goto can be used to change the default behavior). The rules
with semantic actions do not automatically jump to the next condition; this should be done
by the user-defined action code.
In languages that do not have goto (such as Rust) re2ocaml reuses the yystate variable to
store condition numbers. Each condition gets a numeric identifier equal to the number of
its start state, and a switch between conditions is no different than a switch between DFA
states of a single condition. There is no need for a separate initial condition switch.
(Since the same approach is used to implement storable states, YYGETCOND/YYSETCOND are
redundant if both storable states and conditions are used).
The program below uses start conditions to parse binary, octal, decimal and hexadecimal
numbers. There is a single block where each base has its own condition, and the initial
condition is connected to all of them. User-defined variable cond stores the current
condition number; it is initialized to the number of the initial condition generated with
conditions block.
(* re2ocaml $INPUT -o $OUTPUT -ci *)
open Int64
open Option
open String
%{conditions %}
type state = {
yyinput: string;
mutable yycursor: int;
mutable yymarker: int;
mutable yycond: yycondtype;
}
let add (num: int option) (dgt: int) (base: int) : int option =
match num with
| None -> None
| Some n ->
let n' = add (mul (of_int n) (of_int base)) (of_int dgt)
in if n' > (of_int32 Int32.max_int) then None else Some (to_int n')
%{
re2c:YYFN = ["parse;int option", "st;state", "num;int option"];
re2c:yyrecord = "st";
re2c:yyfill:enable = 0;
<init> '0b' / [01] :=> bin
<init> "0" :=> oct
<init> "" / [1-9] :=> dec
<init> '0x' / [0-9a-fA-F] :=> hex
<init> * { None }
<bin> [01] { yyfnbin st (add num (Char.code st.yyinput.[st.yycursor - 1] - 48) 2) }
<oct> [0-7] { yyfnoct st (add num (Char.code st.yyinput.[st.yycursor - 1] - 48) 8) }
<dec> [0-9] { yyfndec st (add num (Char.code st.yyinput.[st.yycursor - 1] - 48) 10) }
<hex> [0-9] { yyfnhex st (add num (Char.code st.yyinput.[st.yycursor - 1] - 48) 16) }
<hex> [a-f] { yyfnhex st (add num (Char.code st.yyinput.[st.yycursor - 1] - 87) 16) }
<hex> [A-F] { yyfnhex st (add num (Char.code st.yyinput.[st.yycursor - 1] - 55) 16) }
<bin, oct, dec, hex> * { num }
%}
let test (yyinput: string) (result: int option) =
let st = {yyinput = yyinput; yycursor = 0; yymarker = 0; yycond = YYC_init} in
if not (parse st (Some 0) = result) then raise (Failure "error")
let main () =
test "\x00" None;
test "1234567890\x00" (Some 1234567890);
test "0b1101\x00" (Some 13);
test "0x7Fe\x00" (Some 2046);
test "0644\x00" (Some 420);
test "9999999999\x00" None
let _ = main ()
Storable state
With --storable-state option re2ocaml generates a lexer that can store its current state,
return to the caller, and later resume operations exactly where it left off. The default
mode of operation in re2ocaml is a "pull" model, in which the lexer "pulls" more input
whenever it needs it. This may be unacceptable in cases when the input becomes available
piece by piece (for example, if the lexer is invoked by the parser, or if the lexer
program communicates via a socket protocol with some other program that must wait for a
reply from the lexer before it transmits the next message). Storable state feature is
intended exactly for such cases: it allows one to generate lexers that work in a "push"
model. When the lexer needs more input, it stores its state and returns to the caller.
Later, when more input becomes available, the caller resumes the lexer exactly where it
stopped. There are a few changes necessary compared to the "pull" model:
• Define YYSETSTATE() and YYGETSTATE(state) primitives.
• Define yych, yyaccept (if used) and state variables as a part of persistent lexer state.
The state variable should be initialized to -1.
• YYFILL should return to the outer program instead of trying to supply more input. Return
code should indicate that lexer needs more input.
• The outer program should recognize situations when lexer needs more input and respond
appropriately.
• Optionally use getstate block to generate YYGETSTATE switch detached from the main
lexer. This only works for languages that have goto (not in --loop-switch mode).
• Use re2c:eof and the sentinel with bounds checks method to handle the end of input.
Padding-based method may not work because it is unclear when to append padding: the
current end of input may not be the ultimate end of input, and appending padding too
early may cut off a partially read greedy lexeme. Furthermore, due to high-level
program logic getting more input may depend on processing the lexeme at the end of
buffer (which already is blocked due to the end-of-input condition).
Here is an example of a "push" model lexer that simulates reading packets from a socket.
The lexer loops until it encounters the end of input and returns to the calling function.
The calling function provides more input by "sending" the next packet and resumes lexing.
This process stops when all the packets have been sent, or when there is an error.
(* re2ocaml $INPUT -o $OUTPUT -fi *)
open Bytes
(* Use a small buffer to cover the case when a lexeme doesn't fit.
In real world use a larger buffer. *)
let bufsize = 10
let debug = false
let log format = (if debug then Printf.eprintf else Printf.ifprintf stderr) format
type state = {
file: in_channel;
yyinput: bytes;
mutable yycursor: int;
mutable yymarker: int;
mutable yylimit: int;
mutable token: int;
mutable yystate: int;
mutable recv: int;
}
type status = End | Ready | Waiting | BadPacket | BigPacket
let fill(st: state) : status =
(* Error: lexeme too long. In real life could reallocate a larger buffer. *)
if st.token < 1 then BigPacket else (
(* Shift buffer contents (discard everything up to the current token). *)
blit st.yyinput st.token st.yyinput 0 (st.yylimit - st.token);
st.yycursor <- st.yycursor - st.token;
st.yymarker <- st.yymarker - st.token;
st.yylimit <- st.yylimit - st.token;
st.token <- 0;
(* Fill free space at the end of buffer with new data from file. *)
let n = In_channel.input st.file st.yyinput st.yylimit (bufsize - st.yylimit - 1) in
st.yylimit <- st.yylimit + n;
set st.yyinput st.yylimit '\x00'; (* append sentinel *)
Ready)
%{
re2c:YYFN = ["lex;status", "yyrecord;state"];
re2c:YYFILL = "Waiting";
re2c:eof = 0;
packet = [a-z]+[;];
* { BadPacket }
$ { End }
packet { yyrecord.recv <- yyrecord.recv + 1; lex_loop yyrecord }
%}
and lex_loop st =
st.token <- st.yycursor;
lex st
let test (packets: string list) (sts: status) =
let fname = "pipe" in
let oc = Out_channel.open_bin fname in
let ic = In_channel.open_bin fname in
let yylimit = bufsize - 1 in
let st = {
file = ic;
(* Sentinel (at `yylimit` offset) is set to null, which triggers YYFILL. *)
yyinput = create bufsize;
yycursor = yylimit;
yymarker = yylimit;
yylimit = yylimit;
token = yylimit;
yystate = -1;
recv = 0;
} in
let rec loop packets = match lex_loop st with
| End ->
log "done: got %d packets\n" st.recv;
End
| Waiting ->
log "waiting...\n";
let packets' = match packets with
| [] -> []
| p :: ps ->
log "sent packet '%s'\n" p;
Out_channel.output_string oc p;
Out_channel.flush oc; (* without `flush` write happens too late *)
ps
in (match fill st with
| BigPacket ->
log "error: packet too big\n";
BigPacket
| Ready -> loop packets'
| _ -> raise (Failure "unexpected status after fill"))
| BadPacket ->
log "error: ill-formed packet\n";
BadPacket
| _ -> raise (Failure "unexpected status")
in if not (loop packets = sts) then
raise (Failure "error");
In_channel.close ic;
Out_channel.close oc;
Sys.remove fname
let main () =
test [] End;
test ["zero;"; "one;"; "two;"; "three;"; "four;"] End;
test ["zer0;"] BadPacket;
test ["goooooooooogle;"] BigPacket
let _ = main ()
Reusable blocks
Reusable blocks of the form /*!rules:re2c[:<name>] ... */ or %{rules[:<name>] ... %} can
be reused any number of times and combined with other re2ocaml blocks. The <name> is
optional. A rules block can be used in a use block or directive. The code for a rules
block is generated at every point of use.
Use blocks are defined with /*!use:re2c[:<name>] ... */ or %{use[:<name>] ... %}. The
<name> is optional: if it's not specified, the associated rules block is the most recent
one (whether named or unnamed). A use block can add named definitions, configurations and
rules of its own. An important use case for use blocks is a lexer that supports multiple
input encodings: the same rules block is reused multiple times with encoding-specific
configurations (see the example below).
In-block use directive !use:<name>; can be used from inside of a re2ocaml block. It merges
the referenced block <name> into the current one. If some of the merged rules and
configurations overlap with the previously defined ones, conflicts are resolved in the
usual way: the earliest rule takes priority, and latest configuration overrides preceding
ones. One exception are the special rules *, $ and (in condition mode) <!>, for which a
block-local definition overrides any inherited ones. Use directive allows one to combine
different re2ocaml blocks together in one block (see the example below).
Named blocks and in-block use directive were added in re2ocaml version 2.2. Since that
version reusable blocks are allowed by default (no special option is needed). Before
version 2.2 reuse mode was enabled with -r --reusable option. Before version 1.2 reusable
blocks could not be mixed with normal blocks.
Example of a !use directive
(* re2ocaml $INPUT -o $OUTPUT *)
(* This example shows how to combine reusable re2c blocks: two blocks
('colors' and 'fish') are merged into one. The 'salmon' rule occurs
in both blocks; the 'fish' block takes priority because it is used
earlier. Default rule * occurs in all three blocks; the local (not
inherited) definition takes priority. *)
open String
type answer = Color | Fish | Dunno
type state = {
yyinput: string;
mutable yycursor: int;
mutable yymarker: int;
}
%{rules:colors
* { raise (Failure "ah"); }
"red" | "salmon" | "magenta" { Color }
%}
%{rules:fish
* { raise (Failure "oh"); }
"haddock" | "salmon" | "eel" { Fish }
%}
%{
re2c:YYFN = ["lex;answer", "yyrecord;state"];
re2c:yyfill:enable = 0;
!use:fish;
!use:colors;
* { Dunno } // overrides inherited '*' rules
%}
let test(str, ans) =
let st = {yyinput = str; yycursor = 0; yymarker = 0}
in if not (lex st = ans) then raise (Failure "error")
let main () =
test("salmon", Fish);
test("what?", Dunno)
let _ = main ()
Example of a /*!use:re2c ... */ block
(* re2ocaml $INPUT -o $OUTPUT --input-encoding utf8 *)
(* This example supports multiple input encodings: UTF-8 and UTF-32.
Both lexers are generated from the same rules block, and the use
blocks add only encoding-specific configurations. *)
open Array
type 'a state = {
yyinput: 'a array;
mutable yycursor: int;
mutable yymarker: int;
}
%{rules
re2c:yyfill:enable = 0;
"∀x ∃y" { Some yyrecord.yycursor }
* { None }
%}
%{use
re2c:YYFN = ["lex8;int option", "yyrecord;char state"];
re2c:encoding:utf8 = 1;
%}
%{use
re2c:YYFN = ["lex32;int option", "yyrecord;int state"];
re2c:encoding:utf32 = 1;
%}
let main() =
let st8 = {
yyinput = [|'\xe2'; '\x08'; '\x80'; '\x78'; '\x20'; '\xe2'; '\x88'; '\x83'; '\x79'|];
yycursor = 0;
yymarker = 0;
} in if not (lex8 st8 = Some (Array.length st8.yyinput)) then raise (Failure "error");
let st32 = {
yycursor = 0;
yymarker = 0;
yyinput = [|0x2200; 0x78; 0x20; 0x2203; 0x79|];
} in if not (lex32 st32 = Some (Array.length st32.yyinput)) then raise (Failure "error");
Submatch extraction
re2ocaml has two options for submatch extraction.
Tags The first option is to use standalone tags of the form @stag or #mtag, where stag
and mtag are arbitrary used-defined names. Tags are enabled with -T --tags option
or re2c:tags = 1 configuration. Semantically tags are position markers: they can be
inserted anywhere in a regular expression, and they bind to the corresponding
position (or multiple positions) in the input string. S-tags bind to the last
matching position, and m-tags bind to a list of positions (they may be used in
repetition subexpressions, where a single position in a regular expression
corresponds to multiple positions in the input string). All tags should be defined
by the user, either manually or with the help of svars and mvars blocks. If there
is more than one way tags can be matched against the input, ambiguity is resolved
using leftmost greedy disambiguation strategy.
Captures
The second option is to use capturing groups. They are enabled with --captures
option or re2c:captures = 1 configuration. There are two flavours for different
disambiguation policies, --leftmost-captures (the default) is for leftmost greedy
policy, and, --posix-captures is for POSIX longest-match policy. In this mode all
parenthesized subexpressions are considered capturing groups, and a bang can be
used to mark non-capturing groups: (! ... ). With --invert-captures option or
re2c:invert-captures = 1 configuration the meaning of bang is inverted. The number
of groups for the matching rule is stored in a variable yynmatch (the whole regular
expression is group number zero), and submatch results are stored in yypmatch
array. Both yynmatch and yypmatch should be defined by the user, and yypmatch size
must be at least [yynmatch * 2]. Use maxnmatch block to define YYMAXNMATCH, a
constant that equals to the maximum value of yynmatch among all rules.
Captvars
Another way to use capturing groups is the --captvars option or re2c:captvars = 1
configuration. The only difference with --captures is in the way the generated code
stores submatch results: instead of yynmatch and yypmatch re2ocaml generates
variables yytl<k> and yytr<k> for k-th capturing group (the user should declare
these using an svars block). Captures with variables support two disambiguation
policies: --leftmost-captvars or re2c:leftmost-captvars = 1 for leftmost greedy
policy (the default one) and --posix-captvars or re2c:posix-captvars for POSIX
longest-match policy.
Under the hood all these options translate into tags and Tagged Deterministic Finite
Automata with Lookahead <https://arxiv.org/abs/1907.08837> . The core idea of TDFA is to
minimize the overhead on submatch extraction. In the extreme, if there're no tags or
captures in a regular expression, TDFA is just an ordinary DFA. If the number of tags is
moderate, the overhead is barely noticeable. The generated TDFA uses a number of tag
variables which do not map directly to tags: a single variable may be used for different
tags, and a tag may require multiple variables to hold all its possible values. Eventually
ambiguity is resolved, and only one final variable per tag survives. Tag variables should
be defined using stags or mtags blocks. If lexer state is stored, tag variables should be
part of it. They also need to be updated by YYFILL.
S-tags support the following operations:
• save input position to an s-tag: t = YYCURSOR with C pointer API or a user-defined
operation YYSTAGP(t) with generic API
• save default value to an s-tag: t = NULL with C pointer API or a user-defined operation
YYSTAGN(t) with generic API
• copy one s-tag to another: t1 = t2
M-tags support the following operations:
• append input position to an m-tag: a user-defined operation YYMTAGP(t) with both default
and generic API
• append default value to an m-tag: a user-defined operation YYMTAGN(t) with both default
and generic API
• copy one m-tag to another: t1 = t2
S-tags can be implemented as scalar values (pointers or offsets). M-tags need a more
complex representation, as they need to store a sequence of tag values. The most naive and
inefficient representation of an m-tag is a list (array, vector) of tag values; a more
efficient representation is to store all m-tags in a prefix-tree represented as array of
nodes (v, p), where v is tag value and p is a pointer to parent node.
Here is a simple example of using s-tags to parse semantic versions consisting of three
numeric components: major, minor, patch (the latter is optional). See below for a more
complex example that uses YYFILL.
(* re2ocaml $INPUT -o $OUTPUT *)
open String
type state = {
yyinput: string;
mutable yycursor: int;
mutable yymarker: int;
(* Final tag variables available in semantic action. *)
%{svars format = "\n\tmutable @@{tag}: int;"; %}
(* Intermediate tag variables used by the lexer (must be autogenerated). *)
%{stags format = "\n\tmutable @@{tag}: int;"; %}
}
type semver = {
major: int;
minor: int;
patch: int;
}
let s2n (str: string) (i1: int) (i2: int) : int =
let rec f s i j n =
if i >= j then n else f s (i + 1) j (n * 10 + Char.code s.[i] - 48)
in f str i1 i2 0
%{local
re2c:YYFN = ["parse;semver option", "st;state"];
re2c:yyrecord = "st";
re2c:tags = 1;
re2c:yyfill:enable = 0;
num = [0-9]+;
@t1 num @t2 "." @t3 num @t4 ("." @t5 num)? [\x00] {
Some {
major = s2n st.yyinput st.t1 st.t2;
minor = s2n st.yyinput st.t3 st.t4;
patch = if st.t5 = -1 then 0 else s2n st.yyinput st.t5 (st.yycursor - 1)
}
}
* { None }
%}
let test (str: string) (result: semver option) =
let st = {
yyinput = str;
yycursor = 0;
yymarker = 0;
%{svars format = "\n\t\t@@{tag} = -1;"; %}
%{stags format = "\n\t\t@@{tag} = -1;"; %}
}
in if not (parse st = result) then raise (Failure "error")
let main () =
test "23.34\x00" (Some {major = 23; minor = 34; patch = 0});
test "1.2.99999\x00" (Some {major = 1; minor = 2; patch = 99999});
test "1.a\x00" None
let _ = main ()
Here is a more complex example of using s-tags with YYFILL to parse a file with
newline-separated semantic versions. Tag variables are part of the lexer state, and they
are adjusted in YYFILL like other input positions. Note that it is necessary for s-tags
because their values are invalidated after shifting buffer contents. It may not be
necessary in a custom implementation where tag variables store offsets relative to the
start of the input string rather than the buffer, which may be the case with m-tags.
(* re2ocaml $INPUT -o $OUTPUT *)
open Bytes
let bufsize = 4096
type state = {
file: in_channel;
yyinput: bytes;
mutable yycursor: int;
mutable yymarker: int;
mutable yylimit: int;
mutable token: int;
mutable eof: bool;
(* Final tag variables available in semantic action. *)
%{svars format = "\n\tmutable @@{tag}: int;"; %}
(* Intermediate tag variables used by the lexer (must be autogenerated). *)
%{stags format = "\n\tmutable @@{tag}: int;"; %}
}
type status = Ok | Eof | LongLexeme
type semver = {
major: int;
minor: int;
patch: int;
}
let s2n (str: bytes) (i1: int) (i2: int) : int =
let rec f s i j n =
if i >= j then n else f s (i + 1) j (n * 10 + Char.code (get s i) - 48)
in f str i1 i2 0
let fill(st: state) : status =
if st.eof then Eof else
(* Error: lexeme too long. In real life could reallocate a larger buffer. *)
if st.token < 1 then LongLexeme else (
(* Shift buffer contents (discard everything up to the current token). *)
blit st.yyinput st.token st.yyinput 0 (st.yylimit - st.token);
st.yycursor <- st.yycursor - st.token;
st.yymarker <- st.yymarker - st.token;
st.yylimit <- st.yylimit - st.token;
%{stags format = "\n\tst.@@ <- if st.@@ = -1 then -1 else st.@@ - st.token;"; %}
st.token <- 0;
(* Fill free space at the end of buffer with new data from file. *)
let n = input st.file st.yyinput st.yylimit (bufsize - st.yylimit - 1) in (* -1 for sentinel *)
st.yylimit <- st.yylimit + n;
if n = 0 then
st.eof <- true; (* end of file *)
set st.yyinput st.yylimit '\x00'; (* append sentinel *)
Ok)
%{
re2c:YYFN = ["lex;(semver list) option", "st;state", "vers;semver list"];
re2c:YYFILL = "fill st = Ok";
re2c:yyrecord = "st";
re2c:tags = 1;
re2c:eof = 0;
num = [0-9]+;
@t1 num @t2 "." @t3 num @t4 ("." @t5 num)? [\n] {
let ver = {
major = s2n st.yyinput st.t1 st.t2;
minor = s2n st.yyinput st.t3 st.t4;
patch = if st.t5 = -1 then 0 else s2n st.yyinput st.t5 (st.yycursor - 1)
} in lex_loop st (ver :: vers)
}
$ { Some (List.rev vers) }
* { None }
%}
and lex_loop st vers =
st.token <- st.yycursor;
lex st vers
let main () =
let fname = "input" in
(* Prepare input file. *)
Out_channel.with_open_bin fname
(fun oc -> for i = 1 to bufsize do
output_string oc "1.22.333\n"
done);
(* Construct the expected result to compare against. *)
let expect = Some (List.init bufsize
(fun _ -> {major = 1; minor = 22; patch = 333;})) in
(* Run lexer on the prepared file. *)
In_channel.with_open_bin fname
(fun ic ->
let yylimit = bufsize - 1 in
let st = {
file = ic;
yyinput = create bufsize;
yycursor = yylimit;
yymarker = yylimit;
yylimit = yylimit;
token = yylimit;
eof = false;
%{svars format = "\n\t\t@@{tag} = -1;"; %}
%{stags format = "\n\t\t@@{tag} = -1;"; %}
} in if (lex_loop st [] <> expect) then
raise (Failure "error"));
(* Cleanup. *)
Sys.remove fname
let _ = main ()
Here is an example of using capturing groups to parse semantic versions.
(* re2ocaml $INPUT -o $OUTPUT *)
open String
type state = {
yyinput: string;
mutable yycursor: int;
mutable yymarker: int;
(* Final tag variables available in semantic action. *)
%{svars format = "\n\tmutable @@{tag}: int;"; %}
(* Intermediate tag variables used by the lexer (must be autogenerated). *)
%{stags format = "\n\tmutable @@{tag}: int;"; %}
}
type semver = {
major: int;
minor: int;
patch: int;
}
let s2n (str: string) (i1: int) (i2: int) : int =
let rec f s i j n =
if i >= j then n else f s (i + 1) j (n * 10 + Char.code s.[i] - 48)
in f str i1 i2 0
%{local
re2c:YYFN = ["parse;semver option", "st;state"];
re2c:yyrecord = "st";
re2c:captvars = 1;
re2c:yyfill:enable = 0;
num = [0-9]+;
(num) "." (num) ("." num)? [\x00] {
Some {
major = s2n st.yyinput st.yytl1 st.yytr1;
minor = s2n st.yyinput st.yytl2 st.yytr2;
patch = if st.yytl3 = -1 then 0 else s2n st.yyinput (st.yytl3 + 1) st.yytr3
}
}
* { None }
%}
let test (str: string) (result: semver option) =
let st = {
yyinput = str;
yycursor = 0;
yymarker = 0;
%{svars format = "\n\t\t@@{tag} = -1;"; %}
%{stags format = "\n\t\t@@{tag} = -1;"; %}
}
in if not (parse st = result) then raise (Failure "error")
let main () =
test "23.34\x00" (Some {major = 23; minor = 34; patch = 0});
test "1.2.99999\x00" (Some {major = 1; minor = 2; patch = 99999});
test "1.a\x00" None
let _ = main ()
Here is an example of using m-tags to parse a version with a variable number of
components. Tag variables are stored in a trie.
(* re2ocaml $INPUT -o $OUTPUT *)
open String
type state = {
yyinput: string;
mutable yycursor: int;
mutable yymarker: int;
(* Final tag variables available in semantic action. *)
%{svars format = "\n\tmutable @@{tag}: int;"; %}
%{mvars format = "\n\tmutable @@{tag}: int list;"; %}
(* Intermediate tag variables used by the lexer (must be autogenerated). *)
%{stags format = "\n\tmutable @@{tag}: int;"; %}
%{mtags format = "\n\tmutable @@{tag}: int list;"; %}
}
let s2n (str: string) (i1: int) (i2: int) : int =
let rec f s i j n =
if i >= j then n else f s (i + 1) j (n * 10 + Char.code s.[i] - 48)
in f str i1 i2 0
%{local
re2c:YYFN = ["parse;(int list) option", "st;state"];
re2c:YYMTAGP = "@@ <- st.yycursor :: @@;";
re2c:YYMTAGN = ""; // alternatively could add `-1` to the list
re2c:yyrecord = "st";
re2c:tags = 1;
re2c:yyfill:enable = 0;
num = [0-9]+;
@t1 num @t2 ("." #t3 num #t4)* [\x00] {
let x = s2n st.yyinput st.t1 st.t2 in
let xs = List.rev (List.map2 (fun x y -> s2n st.yyinput x y) st.t3 st.t4) in
Some (x :: xs)
}
* { None }
%}
let test (str: string) (result: (int list) option) =
let st = {
yyinput = str;
yycursor = 0;
yymarker = 0;
%{svars format = "\n\t\t@@{tag} = -1;"; %}
%{mvars format = "\n\t\t@@{tag} = [];"; %}
%{stags format = "\n\t\t@@{tag} = -1;"; %}
%{mtags format = "\n\t\t@@{tag} = [];"; %}
}
in if not (parse st = result) then raise (Failure "error")
let main () =
test "1\x00" (Some [1]);
test "1.2.3.4.5.6.7\x00" (Some [1; 2; 3; 4; 5; 6; 7;]);
test "1.2.\x00" None
let _ = main ()
Encoding support
It is necessary to understand the difference between code points and code units. A code
point is a numeric identifier of a symbol. A code unit is the smallest unit of storage in
the encoded text. A single code point may be represented with one or more code units. In a
fixed-length encoding all code points are represented with the same number of code units.
In a variable-length encoding code points may be represented with a different number of
code units. Note that the "any" rule [^] matches any code point, but not necessarily any
code unit (the only way to match any code unit regardless of the encoding is the default
rule *). The generated lexer works with a stream of code units: yych stores a code unit,
and YYCTYPE is the code unit type. Regular expressions, on the other hand, are specified
in terms of code points. When re2ocaml compiles regular expressions to automata it
translates code points to code units. This is generally not a simple mapping: in
variable-length encodings a single code point range may get translated to a complex code
unit graph. The following encodings are supported:
• ASCII (enabled by default). It is a fixed-length encoding with code space [0-255] and
1-byte code points and code units.
• EBCDIC (enabled with --ebcdic or re2c:encoding:ebcdic). It is a fixed-length encoding
with code space [0-255] and 1-byte code points and code units.
• UCS2 (enabled with --ucs2 or re2c:encoding:ucs2). It is a fixed-length encoding with
code space [0-0xFFFF] and 2-byte code points and code units.
• UTF8 (enabled with --utf8 or re2c:encoding:utf8). It is a variable-length Unicode
encoding. Code unit size is 1 byte. Code points are represented with 1 -- 4 code units.
• UTF16 (enabled with --utf16 or re2c:encoding:utf16). It is a variable-length Unicode
encoding. Code unit size is 2 bytes. Code points are represented with 1 -- 2 code units.
• UTF32 (enabled with --utf32 or re2c:encoding:utf32). It is a fixed-length Unicode
encoding with code space [0-0x10FFFF] and 4-byte code points and code units.
Include file include/unicode_categories.re provides re2ocaml definitions for the standard
Unicode categories.
Option --input-encoding specifies source file encoding, which can be used to enable
Unicode literals in regular expressions. For example --input-encoding utf8 tells re2ocaml
that the source file is in UTF8 (it differs from --utf8 which sets input text encoding).
Option --encoding-policy specifies the way re2ocaml handles Unicode surrogates (code
points in range [0xD800-0xDFFF]).
Below is an example of a lexer for UTF8 encoded Unicode identifiers.
(* re2ocaml $INPUT -o $OUTPUT --utf8 -i *)
open String
%{include "unicode_categories.re" %}
type state = {
yyinput: string;
mutable yycursor: int;
mutable yymarker: int;
mutable yyaccept: int;
}
%{
re2c:YYFN = ["lex;bool", "yyrecord;state"];
re2c:yyfill:enable = 0;
// Simplified "Unicode Identifier and Pattern Syntax"
// (see https://unicode.org/reports/tr31)
id_start = L | Nl | [$_];
id_continue = id_start | Mn | Mc | Nd | Pc | [\u200D\u05F3];
identifier = id_start id_continue*;
identifier { true }
* { false }
%}
let main () =
let st = {
yyinput = "_Ыдентификатор\x00";
yycursor = 0;
yymarker = 0;
yyaccept = 0;
}
in if not (lex st) then raise (Failure "error")
let _ = main ()
Include files
re2ocaml allows one to include other files using a block of the form /*!include:re2c FILE
*/ or %{include FILE %}, or an in-block directive !include FILE ;, where FILE is a path to
the file to be included. re2ocaml looks for include files in the directory of the
including file and in include locations, which can be specified with the -I option.
Include blocks/directives in re2ocaml work in the same way as C/C++ #include: FILE
contents are copy-pasted verbatim in place of the block/directive. Include files may have
further includes of their own. Use --depfile option to track build dependencies of the
output file on include files. re2ocaml provides some predefined include files that can be
found in the include/ subdirectory of the project. These files contain definitions that
may be useful to other projects (such as Unicode categories) and form something like a
standard library for re2ocaml. Below is an example of using include files.
Include file 1 (definitions.ml)
type number = Int | Float | NaN
%{
number = [1-9][0-9]*;
%}
Include file 2 (extra_rules.re.inc)
// floating-point numbers
frac = [0-9]* "." [0-9]+ | [0-9]+ ".";
exp = 'e' [+-]? [0-9]+;
float = frac exp? | [0-9]+ exp;
float { Float }
Input file
(* re2ocaml $INPUT -o $OUTPUT -i *)
open String
%{include "definitions.ml" %}
type state = {
yyinput: string;
mutable yycursor: int;
mutable yymarker: int;
mutable yyaccept: int;
}
%{
re2c:YYFN = ["lex;number", "yyrecord;state"];
re2c:yyfill:enable = 0;
* { NaN }
number { Int }
!include "extra_rules.re.inc";
%}
let test(str, num) =
let st = {yyinput = str; yycursor = 0; yymarker = 0; yyaccept = 0}
in if not (lex st = num) then raise (Failure "error")
let main () =
test("123\x00", Int);
test("123.4567\x00", Float)
let _ = main ()
Header files
re2ocaml allows one to generate header file from the input .re file using --header option
or re2c:header configuration and block pairs of the form /*!header:re2c:on*/ and
/*!header:re2c:off*/, or %{header:on%} and %{header:off%}. The first block marks the
beginning of header file, and the second block marks the end of it. Everything between
these blocks is processed by re2ocaml, and the generated code is written to the file
specified with --header option or re2c:header configuration (or stdout if neither option
nor configuration is used). Autogenerated header file may be needed in cases when re2ocaml
is used to generate definitions that must be visible from other translation units.
Here is an example of generating a header file that contains definition of the lexer state
with tag variables (the number variables depends on the regular grammar and is unknown to
the programmer).
Input file
(* re2ocaml $INPUT -o $OUTPUT --header lexer/state.ml -i *)
open State
open String
%{header:on %}
type state = {
yyinput: string;
mutable yycursor: int;
mutable tag: int;
%{stags format = "mutable @@: int;"; %}
}
%{header:off %}
%{
re2c:YYFN = ["lex;int", "yyrecord;State.state"];
re2c:tags = 1;
re2c:yyfill:enable = 0;
re2c:header = "lexer/state.ml";
[a]* @tag [b]* { yyrecord.tag }
%}
let main () =
let st = {
yyinput = "ab\x00";
yycursor = 0;
tag = 0;
%{stags format = "\n\t@@ = 0;"; %}
}
in if not (lex st = 1) then raise (Failure "error")
let _ = main ()
Header file
(* Generated by re2c *)
type state = {
yyinput: string;
mutable yycursor: int;
mutable tag: int;
mutable yyt1: int;
}
Skeleton programs
With the -S, --skeleton option, re2ocaml ignores all non-re2ocaml code and generates a
self-contained C program that can be further compiled and executed. The program consists
of lexer code and input data. For each constructed DFA (block or condition) re2ocaml
generates a standalone lexer and two files: an .input file with strings derived from the
DFA and a .keys file with expected match results. The program runs each lexer on the
corresponding .input file and compares results with the expectations. Skeleton programs
are very useful for a number of reasons:
• They can check correctness of various re2ocaml optimizations (the data is generated
early in the process, before any DFA transformations have taken place).
• Generating a set of input data with good coverage may be useful for both testing and
benchmarking.
• Generating self-contained executable programs allows one to get minimized test cases
(the original code may be large or have a lot of dependencies).
The difficulty with generating input data is that for all but the most trivial cases the
number of possible input strings is too large (even if the string length is limited).
re2ocaml solves this difficulty by generating sufficiently many strings to cover almost
all DFA transitions. It uses the following algorithm. First, it constructs a skeleton of
the DFA. For encodings with 1-byte code unit size (such as ASCII, UTF-8 and EBCDIC)
skeleton is just an exact copy of the original DFA. For encodings with multibyte code
units skeleton is a copy of DFA with certain transitions omitted: namely, re2ocaml takes
at most 256 code units for each disjoint continuous range that corresponds to a DFA
transition. The chosen values are evenly distributed and include range bounds. Instead of
trying to cover all possible paths in the skeleton (which is infeasible) re2ocaml
generates sufficiently many paths to cover all skeleton transitions, and thus trigger the
corresponding conditional jumps in the lexer. The algorithm implementation is limited by
~1Gb of transitions and consumes constant amount of memory (re2ocaml writes data to file
as soon as it is generated).
Visualization and debug
With the -D, --emit-dot option, re2ocaml does not generate code. Instead, it dumps the
generated DFA in DOT format. One can convert this dump to an image of the DFA using
Graphviz or another library. Note that this option shows the final DFA after it has gone
through a number of optimizations and transformations. Earlier stages can be dumped with
various debug options, such as --dump-nfa, --dump-dfa-raw etc. (see the full list of
options).
SEE ALSO
You can find more information about re2c at the official website: <http://re2c.org> .
Similar programs are flex(1), lex(1), quex( <http://quex.sourceforge.net> ).
AUTHORS
re2ocaml was originally written by Peter Bumbulis ( <peter@csg.uwaterloo.ca> ) in 1993.
Marcus Boerger and Dan Nuffer spent several years to turn the original idea into a
production ready code generator. Since then it has been maintained and developed by
multiple volunteers, most notably, Brian Young ( <bayoung@acm.org> ), Marcus Boerger
<https://github.com/helly25> , Dan Nuffer ( <nuffer@users.sourceforge.net> ), Ulya
Trofimovich <https://github.com/skvadrik>
( <skvadrik@gmail.com> ), Serghei Iakovlev <https://github.com/sergeyklay> , Sergei
Trofimovich <https://github.com/trofi> , Petr Skocik <https://github.com/pskocik> ,
<ligfx>
<raekye> and <PolarGoose> .
RE2OCAML(1)