/ 
MarpaX-ESLIF-5.0.6-TRIAL
River stage one • 2 direct dependents • 4 total dependents
/ MarpaX::ESLIF::BNF

NAME

MarpaX::ESLIF::BNF - MarpaX::ESLIF's BNF

VERSION

version 5.0.6

DESCRIPTION

MarpaX::ESLIF is a Scanless Interface expressed in a BNF format, that is using marpaWrapper, itself being a thin interface on top of libmarpa parser.

CONVENTIONS

The MarpaX::ESLIF BNF is composed of unicode characters, in any encoding supported by the underlying convertor (ICU or iconv, in order of preference). Unsignificant whitespaces, Perl-like comments and C++-like comments are discarded.

Symbol names

They consist of bare names, or can be enclosed in angle brackets if whitespace if desired. They are case sensitive, and can be composed only of ASCII characters. There is no attempt to discard any leading, trailing, or repeated whitespace in angle brackets version, i.e. all the followings are different symbol names:

this
<this >
< this
  >
Levels

The grammar can contain multiple levels, the level syntax being:

::=           # Alias for level 0
  ~           # Alias for level 1
:\[[\d]+\]:=  # General form

The level 0 must exist. We will use only ::= and/or ~ in the rest of this document for conveniene, though we are adressing any possible level.

Rules

A rule consist of a left symbol, followed by the level, followed by one or more symbols, or a single quantified symbol:

leftsymbol1 ::=  # Nothing
leftsymbol2 ::= rightSymbol
leftsymbol3 ::= quantifiedSymbol*
Terminals

There are four types of explicit terminals.

  • Pseudo terminals

    :eof

    A zero-length terminal that matches only at the of the stream.

    :eol

    A zero-length terminal that matches only after a newline.

    Note that the newline is hardcoded to be any unicode newline, i.e. (*BSR_UNICODE)\R in PCRE2 terminology. The presence of :eol pseudo-terminal anywhere in the grammar enforces the newlineb flag in recognizers.

  • Strings

    They can be single ('), double-quoted ("), or LEFT DOUBLE QUOTATION MARK (“) then RIGHT DOUBLE QUOTATION MARK (”) enclosed. The content is any valid unicode character, and the \ character can be used to escape the expected right-side quote character (i.e. ', ", or ”) or \ itself. The i modifier can be used to force case-insensitive match:

    'string'
'string':i
'string\'s'
"string\"s\\"
“str'"\”ing”:i

    and the modifier c to force unicode character mode. Internally a string is nothing else but a regular expression, so the exact implementation of the i and c modifiers correspond to the PCRE2 flags listed below in the Regular expression section. A string created with no modifier is faster when doing comparison (internally MarpaX::ESLIF will switch to a brutal memory comparison instead of calling the PCRE2 engine). Please note this is really a quoted string, not a string terminal. I.e. everything inside the quote is taken as-is, with no interpretation.

  • Character classes

    They are always enclosed with left and right brackets []. Modifiers can start after a : character. A character class class is nothing else but a lexically restricted regular expression.

  • Regular expression

    They are always enclosed within slashes //, and the content must be valid as per the PCRE2 Perl Compatible Regular Expression library. Modifiers can start after the slash on the right. Regular expression patterns are by default anchored. The slash character itself must be preceeded by a backslash, i.e. \/ in the string seen by the parser (so, in practice, it is coded like this: "\\/").

    Regular expressions must be used with care in the two following scenarios:

    Quantifiers at the end

    If the regular expression ends with an unlimited quantifier at the end, i.e. * or +, it is very likely that the data will match partially until the whole input is read, effectively forcing ESLIF to read the entire input. This can break the streaming nature of your implementation.

    Negative lookahead at the end

    If the regular expression ends with a negative lookahead, it can match when you think it should not. This is because negative lookahead does not trigger a partial match. In such a case, you should ensure that your regular expression forces a minimum number of characters in the subject string.

The PCRE2 syntax is supported in its entirety, this include any PCRE2 add-on. Character classes and regular expression share the same set of modifiers, executed in order of appearance, that are:

----------------------------------------------------------------
Modifiers   Explanation
----------------------------------------------------------------
e           Unset back-references in the pattern will match to empty strings
i           Case-insensitive
j           \u, \U and \x and unset back-references will act as JavaScript standard
m           Multi-line regex
n           Enable Unicode properties and extend meaning of meta-characters
s           A dot meta-character in the pattern matches all characters, including newlines
x           Enable comments. This has some limitation due MarpaX::ESLIF semantics
D           A dollar meta-character matches only at the end of the subject string
J           Allow duplicate names for sub-patterns
U           Inverts the "greediness" of the quantifiers
a           Meta-characters will be limited to their ASCII equivalent
u           Forces support of large codepoints
b           Could mean "forced binary" mode
c           Could mean "forced unicode character" mode
A           Remove the systematic anchoring
----------------------------------------------------------------

Internally this correspond to this set of options in PCRE2:

----------------------------------------------------------------
Modifiers         PCRE2 flag unset   PCR2 flag set
----------------------------------------------------------------
e                                    PCRE2_MATCH_UNSET_BACKREF
i                                    PCRE2_CASELESS
j                                    PCRE2_ALT_BSUX|PCRE2_MATCH_UNSET_BACKREF
m                                    PCRE2_MULTILINE
n                                    PCRE2_UCP
s                                    PCRE2_DOTALL
x                                    PCRE2_EXTENDED
D                                    PCRE2_DOLLAR_ENDONLY
J                                    PCRE2_DUPNAMES
U                                    PCRE2_UNGREEDY
a                 PCRE2_UTF
N                 PCRE2_UCP
u                                    PCRE2_UTF
b                 PCRE2_UTF          PCRE2_NEVER_UTF
c                 PCRE2_NEVER_UTF    PCRE2_UTF
A                 PCRE2_ANCHORED
----------------------------------------------------------------
Lexemes and Terminals

Lexemes are meta-symbols that does appear as the LHS symbol anywhere within the current grammar. Therefore they behave like terminals, except that their definition is not in the current grammar. By default such meta-symbol is looked up at the next level. For example:

rule      ::= something
something   ~ [\d]

say that symbol something at grammar level 0 is a reference to something at grammar level 1.

In contrast a terminal is an explicit quoted string, character class or regular expression, e.g.:

the_rhs_is_a_terminal ::= '"'

Lexemes can be references:

implicitely

Without any indication, a lexeme is always assumed to be at the grammar of the next level

explicity by grammar description
X ::= Y@'Grammar Description'

This is working because a grammar description is unique across all sub-grammars. Note that when accessing a grammar by description, the later must have been declared before, using the :desc rule. Therefore a good practice is to forward declare all grammars at the beginning, e.g.:

:desc    ::= 'Main Grammar'
:desc      ~ 'Sub Grammar 1'
:desc :[2]:= 'Sub Grammar 2'
:desc :[3]:= 'Sub Grammar 3'
explicitely by relative level
X ::= Y@+1
X ::= Y@-2
X ::= Y@3

The signed integer is interpreted as a delta with current grammar level.

explicitely by absolute level
X ::= Y@=1

The unsigned integer is interpreted as an explicit grammar level.

Discard

Everytime expected terminals cannot be match, MarpaX::ESLIF will try to match the special rule :discard. The :discard rule also have precedence if it matches longer than the longest acceptable lexeme or terminal. and can not be ambiguous (else discard silently fail).

Grammar meta settings

Start rule

By default, the first symbol of a grammar of level n is its start symbol. This can be set once with e.g.:

:start ::= symbolname
Grammar description

By default, a grammar of level n has the description Grammar level n. This can be set once with e.g.:

:desc ::= 'A single-quoted string'
Settings sub grammar

Any setting consist of a reserved keyword, followed by => or the Rightwards Double Arrow UTF-8 character ⇒, followed by a setting-specific value:

action => myAction
action ⇒ myAction

When you use the ⇒ UTF-8 character, it is recommended to say that the grammar itself is encoded in UTF-8 (c.f. marpaESLIFGrammarOption structure).

Defaults

By default, symbol action is ::transfer and rule action is ::concat, i.e. the parse tree value of a grammar is a binary concatenation of every input representation (see the representation section below), without the eventual discard. Stack manipulation may require the trigger of a free function, and this has no default. Only expected terminals or lexemes are looked up, this is the Longest Acceptable Token Match (LATM) setting, defaulting to a true value. You should not change that. Defaults can be set once, for example like this:

:default ::= action            => defaultRuleAction
             latm              => 1
             symbol-action     => defaultSymbolAction
             default-encoding  => UTF-8
             fallback-encoding => UTF-16

Predefined actions are available for rules and symbols. Please refer the API documentation to know more about value types.

The symbol-action adverb is how a match within a sub-grammar is transfered.

The default-encoding adverb gives default encoding when the recognizer runs in character mode and the end-user gave no encoding.

The fallback-encoding adverb gives fallback encoding when the recognizer runs in character mode and the end-user gave no encoding nor default-encoding. Then MarpaX::ESLIF will try to guess the encoding, and if this fail, will fallback to this setting. This is the desired setting when you grammar accept input in different encoding, and defaults to a given encoding if the guess fail. Please note that, when guessing an encoding, MarpaX::ESLIF takes into account an eventual BOM, retreiving the later from input characters if it exists.

Meta actions
::undef

Creates a value of type UNDEF.

Meaningful for both rule and symbol actions.

::ascii

Creates a value of type STRING, with encoding "ASCII", from the right-hand side representation, guaranteed to be a NUL byte terminated sequence of ASCII characters, or UNDEF if representation is empty. Please refer to the representation section below.

Meaningful for both rule and symbol actions.

::convert[[^]]+]

Creates a value of type STRING from the right-hand side representation encoded in the charset specified within the brackets, or UNDEF if representation is empty. iconv convention is used for the charset, i.e. a charset name, followed by eventual options like //TRANSLIT and/or //IGNORE. Any other option depend on how the tconv library is built, and may probably not be supported. Note that using the iconv notation does not mean that this is iconv running behind.

Please refer to the representation section below.

Meaningful for both rule and symbol actions.

::concat

Creates a value of type ARRAY from the binary concatenatation all the RHS's representation, or UNDEF if representation is empty.

Please refer to the representation section below.

Meaningful for both rule and symbol actions, and is the default rule action.

::copy[x]

Copies the RHS number x (first RHS is at indice 0), putting UNDEF if it does not exist. This action is the only one that guarantees that the nature of the RHS value is unchanged.

Meaningful only for rule actions.

::shift

Alias for ::copy[0].

::transfer

Copies the single RHS number value. This action guarantees that the nature of the RHS value is unchanged.

Meaningful only for symbol actions, and is the default symbol action.

::true

Creates a value of type BOOL, containing a true value.

Meaningful for both rule and symbol actions.

::false

Creates a value of type BOOL, containing a false value.

Meaningful for both rule and symbol actions.

::json

Creates a value of type STRING in the UTF-8 encoding, containing a strict JSON string as per original JSON specification, i.e. it is using UTF-16 surrogates to describe characters above 0xFFFF.

Meaningful for both rule and symbol actions.

::jsonf

Creates a value of type STRING in the UTF-8 encoding, containing a JSON string as per original JSON specification plus infinity and nan extensions. Without these extensions, infinity and nan are writen as "null".

Meaningful for both rule and symbol actions.

::row

Creates a value of type ROW, that contains all RHS's values.

Meaningful only for rule actions.

::table

Creates a value of type TABLE, that contains all RHS's values. The number of RHS must be odd.

Meaningful only for rule actions.

::ast

Creates a value of type TABLE, where the single key is a string containing the lhs name, and the single value is a row containing all RHS's values, or UNDEF if rule is nullable.

Meaningful only for rule actions.

Discard

The :discard symbol, despite belonging to a given grammar, is not accessible directly, and can only be set as a meta setting. An event can be associated upon discard completion, there can be multiple :discard statements:

:discard ::= symbolname1 event => discard_symbolname1$
:discard ::= symbolname2 event => discard_symbolname2$

Note than when an event is set, this will be triggered only on the :discard's RHS completion, therefore the RHS of the :discard must be an LHS in the same grammar when there is an event setting and when :discard refers to a symbol.

An explicit terminal can also be set directly, e.g.:

:discard ::= /[\s]+/                                                                        event => whitespace$
:discard ::= /(?:(?:#)(?:[^\n]*)(?:\n|\z))/u                                                event => perl_comment$
:discard ::= /(?:(?:(?://)(?:[^\n]*)(?:\n|\z))|(?:(?:/\*)(?:(?:[^\*]+|\*(?!/))*)(?:\*/)))/u event => cplusplus_comment$

this form is highly recommended, because ESLIF recognizes the case where :discard consist only of explicit terminals, and applies an optimization that prevent the intanciation of an internal parse.

Events

  • Event names

    They are composed of a restricted set of the ASCII graph characters. Special cases are:

    :symbol

    Transformed to the symbol name for which the event is triggered.

    :discard[on]

    Hook that is disabling :discard rule for the current recognizer. Equivalent to a call to MarpaX::ESLIF::recognizer_hook_discardb(1). Not propagated. Take care, this is a permanent setting.

    :discard[off]

    Hook that is enabling :discard rule for the current recognizer. Equivalent to a call to MarpaX::ESLIF::recognizer_hook_discardb(0). Not propagated. Take care, this is a permanent setting.

    :discard[switch]

    Hook that is switching :discard rule for the current recognizer. Equivalent to a call to MarpaX::ESLIF::recognizer_hook_discard_switchb(). Not propagated. Take care, this is a permanent setting.

    Please note that the :discard[on], :discard[off] and :discard[switch] events will always happen if specified in the grammar with an true initial state and if associated to lexemes or terminals. No callback to the end user will happen. This is because these are internal events, categorized as parsing hooks.

  • Event initializers

    By default, events are on, this is equivalent to appending =on after the event name. The =off characters are putting event off at startup.

Lexemes are different than non-lexeme symbols because they are treated in the grammar as terminals, others are not.

Lexeme events

Meta symbols that are lexemes can have pause events, before mean that the scanning recognized them, after mean they have been consumed, e.g.:

:lexeme ::= symbolname1 pause => before event => ^symbolname1
:lexeme ::= symbolname2 pause => after  event =>  symbolname2$

It is not allowed to set a lexeme event on a symbol that is not a lexeme. You cannot declare two pause events in the same line.

Terminal events

Explicit terminals can have pause events, before mean that the scanning recognized them, after mean they have been consumed, e.g.:

:terminal ::= '"' pause => before event => ^beforeDquote
:terminal ::= "'" pause => after  event =>  afterSquote$

Only an explicit terminal can be set using the :terminal keyword.

Lexeme and terminal specific actions

The grammar's symbol-action can be overwritten by setting such entry in :lexeme and :terminal settings, e.g.:

:lexeme   ::= symbolname1 symbol-action => ::u8"Custom String"
:lexeme   ::= symbolname2 symbol-action => Custom_action
:terminal ::= 'ThisTerminal' symbol-action => ::lua->lua_custom_action

A lexeme or a terminal can have an if-action setting, that applies only when the lexeme or a terminal is discovered via the automatic scan. The if-action must refer to a callback in user-land's recognizer interface or in embedded lua interpreter, and must return a boolean:

:lexeme   ::= symbolname1 if-action => Custom_action1
:lexeme   ::= symbolname2 if-action => lua::->Custom_action2

Please note that the if-action will always be called with an argument of type ARRAY (i.e. up to the implementation to convert that to a multibyte string, if any).

A lexeme or a terminal can have a regex-action setting, that always applies, regardless if the parsing is done using the automatic scan or not. regex-action is a transversal grammar setting; and can be set only via ::default meta rule:

::default ::= regex-action => Custom_regexaction1
::default   ~ regex-action => lua::->Custom_regexaction2

If a regular expression has a callout in the form (?CX) (where X is a number> or (?C"string"), the regex-action is called with a block argument. This <block> argument is a hash in perl, a table in lua, and an ESLIFRegexCallout instance in java.

Non-lexeme events

Completion, predicted or nulled events are supported, targetting a symbol name. These are grammar events.

For example:

event a     = completed  symbolname
event b=off = nulled     symbolname
event c=on  = predicted ^symbolname
Generic event callback

The grammar, in :default meta rule, can have an event action callback that will end into user-land's recognizer interface or in the embedded lua interpreter, e.g.

:default event-action => Custom_action1

or

:default event-action => lua::->Custom_action2

<luascript>

  -- In the embedded lua, globals are automatically
  -- created. During parsing phase, they are:
  -- marpaESLIF
  -- marpaESLIFGrammar
  -- marpaESLIFRecognizer

  function Custom_action2(events)
    -- "events" is a table, exactly like the output
    -- of marpaESLIFRecognizer.events()
    -- This function must return a boolean
    return true
  end

</luascript>

The event callback must return a boolean, a false value indicates that the parsing failed.

Autoranking

Rules can be autoranked, the higest of a set of alternative having the highest rank, default is off:

autorank is on by default
autorank is off by default
Inaccessible statements

Inaccessible statements can generate warnings, can be ignored, or be error on demand, default is to ignore them:

inaccessible is warn by default
inaccessible is ok by default
inaccessible is fatal by default

Statements

A statement have a symbol name on the left-hand side (LHS) and zero or more symbol names, or terminals, on the right-hand side (RHS):

LHS ::= RHS1 RHS2 etc...

There are two exceptions:

The exception statement

Its semantic is a single symbol name following by another single symbol name, with - in the middle:

LHS ::= RHS1 - RHS2

Constraints are:

Both RHS1 and RHS2 must be lexemes
Both RHS1 and RHS2 must not contain sub-lexemes

This mean that the LHS of an exception statement can never be nullable. You must explicitely declare so if this is wanted, i.e.:

LHS ::=

Note that managing an exception can be consuming, eventually reading the whole data if it is writen without special care. Trying to minimize the number of characters needed and/or using well-thinked regular expressions often lead to the same result with better performance. Internally it is implemented like this: First the longest RHS1 is matched, then ESLIF rollbacks to every RHS1's start completion until RHS2 does not match.

The sequence statement

This is a single symbol name following by the * or the + character:

LHS1 ::= RHS1*
LHS2 ::= RHS2+

Empty rule have no RHS:

EMPTYRULE ::=

Eventual ambiguities in the grammar itself may be solved by adding the ; character at the end of a rule, or by enclosing zero or more statements within { and } characters:

EMPTYRULE ::= ;
{
  LHS1 ::= RHS1
  LHS2 ::= RHS2 - RHS3
}
Alternatives

There are two types of alternatives: the standard | meaning this is an or, or the loosen character || meaning that this is an alternative starting a prioritized group of alternatives, for example the calculator grammar is:

Expression  ::=  /[\d]+/
              | '(' Expression ')'              assoc => group
              ||    Expression '**' Expression  assoc => right
              ||    Expression  '*' Expression
              |     Expression  '/' Expression
              ||    Expression  '+' Expression
              |     Expression  '-' Expression

which is strictly equivalent, in traditional BNF syntax to:

Expression  ::= Expression0
Expression0 ::= Expression1
Expression1 ::= Expression2
Expression2 ::= Expression3

Expression3 ::= /[\d]+/
              | '(' Expression0 ')'
Expression2 ::=  Expression3 '**' Expression2
Expression1 ::=  Expression1  '*' Expression2
              |  Expression1  '/' Expression2
Expression0 ::=  Expression0  '+' Expression1
Expression0 ::=  Expression0  '-' Expression1

As you can see statements has been grouped at every occurence of || operator. Therefore the loosen operator || is a convenience operator, it is always possible to write an equivalent grammar without it, though this can become quite tedious. The assoc adverb has a meaning only in the presence of prioritized alternatives, else it has no effect.

The following is copied almost verbatim from the Marpa::R2 section on precedence:

In prioritized statements, every alternative has an arity. The arity is the number of times an operand appears on the RHS. A RHS symbol is an operand if and only if it is the same as the LHS symbol. Anything else is considered as an operator. When the arity is 0, precedence and associativy are meaningless and ignored. When the arity is 1, precedence has effect, but not left nor right associativity.

If arity is 2 or more and the alternative is left associative, the leftmost operand associates and operands after the first will have the next-tighest priority level. If arity is 2 or more and the alternative is right associative, the last operand associates and operands before the last will have the next-tighest priority level. In group associativity, all operands associate at the lowest priority.

Adverbs

Any rule can be followed by zero or more of these adverbs, if an adverb appears more than once, the latest is the winner:

Action

During valuation, a specific action can be associated to a rule:

action => my_action

It is possible to set a hardcoded UTF-8 string as result, using a string literal:

action => ::u8"string literal, supporting \x{0D}, \u{0972} and \U{0001F600}"

where \x{hh} will translate to a byte having the hexadecimal value hh, and \u{uuuu} and \U{uuuuuuuu} will translate to the UTF-8 version of uuuu unicode code point, uuuuuuuu is for the very large, less common, code points.

Left association

In a prioritized statement, associate with the left-most operand:

assoc => left
Right association

In a prioritized statement, associate with the right-most operand:

assoc => right
Group association

All operands associate at the lowest priority:

assoc => group
Separator

Sequence rules can have a separator, that can be a symbol name, a string, a character class or a regular expression.

separator => comma
separator => ','
separator => [,]
separator => /,/

Modifiers are allowed after string, character class or regular expressions.

Proper specification

Sequence rules can be proper, i.e. without trailing separator:

proper => 1
Hiding separator specification

Default for sequence rules actions is to always include the separator in the stack. This may be changed in the grammar using:

hide-separator => 1
Rank specification

During valuation, rules can have a rank to get prioritized. Rank is a signed integer and default to 0:

rank => -2

Any other value but 0 is not allowed if autoranking is set to a true value.

Null-ranking specification

Nulling symbols can rank high low, the default is low.

null-ranking => 'low'
null-ranking => 'high'
Priority specification

Lexemes and terminals can be prioritized, using a signed integer:

priority => 15
Pause specification

Scanner can be paused before a lexeme or a terminal is recognized, or just after it has been completed:

pause => before
pause => after
Event specification

Events can be specified, with an eventual initializer, given that default initialization is =on:

event => eventName
event => eventName=on
event => eventName=off
Naming

A name can be associated to the rule, in the form:

name => something
name => 'quoted string literal'  # No modifier is allowed after the string
name => "quoted string literal"  # No modifier is allowed after the string

NAME

BNF

MarpaX::ESLIF BNF can be expressed in itself:

/*
 * **********************
 * Meta-grammar settings:
 * **********************
 */
:start                         ::= statements
:desc                          ::= 'G1'
 # whitespace
:discard                       ::= /[\s]+/
 # perl comment
:discard                       ::= /(?:(?:#)(?:[^\n]*)(?:\n|\z))/u
 # C++ comment
:discard                       ::= /(?:(?:(?:\/\/)(?:[^\n]*)(?:\n|\z))|(?:(?:\/\*)(?:(?:[^\*]+|\*(?!\/))*)(?:\*\/)))/u

/*
 * ***************
 * Event settings:
 * ***************
 */
event :discard[off]=on = nulled <discard off>
event :discard[on]=on  = nulled <discard on>

/*
 * ******
 * Rules:
 * ******
 */
<statements>                   ::= <statement>*
<statement>                    ::= <start rule>
                                 | <desc rule>
                                 | <empty rule>
                                 | <null statement>
                                 | <statement group>
                                 | <priority rule>
                                 | <quantified rule>
                                 | <discard rule>
                                 | <default rule>
                                 | <lexeme rule>
                                 | <completion event declaration>
                                 | <nulled event declaration>
                                 | <prediction event declaration>
                                 | <inaccessible statement>
                                 | <exception statement>
                                 | <autorank statement>
                                 | <lua script statement>
                                 | <terminal rule>
<start rule>                   ::= ':start' <op declare> <symbol>
<desc rule>                    ::= ':desc' <op declare> <quoted string literal>
<empty rule>                   ::= <lhs> <op declare> <adverb list>
<null statement>               ::= ';'
<statement group>              ::= '{' <statements> '}'
<priority rule>                ::= <lhs> <op declare> <priorities>
<quantified rule>              ::= <lhs> <op declare> <rhs primary> <quantifier> <adverb list>
<discard rule>                 ::= ':discard' <op declare> <rhs primary> <adverb list>
<default rule>                 ::= ':default' <op declare> <adverb list>
<lexeme rule>                  ::= ':lexeme' <op declare> <symbol> <adverb list>
<terminal rule>                ::= ':terminal' <op declare> <terminal> <adverb list>
<completion event declaration> ::= 'event' <event initialization> '=' 'completed' <symbol>
                                 | 'event' <event initialization> <op declare> 'completed' <symbol>
<nulled event declaration>     ::= 'event' <event initialization> '=' 'nulled' <symbol>
                                 | 'event' <event initialization> <op declare> 'nulled' <symbol>
<prediction event declaration> ::= 'event' <event initialization> '=' 'predicted' <symbol>
                                 | 'event' <event initialization> <op declare> 'predicted' <symbol>
<inaccessible statement>       ::= 'inaccessible' 'is' <inaccessible treatment> 'by' 'default'
<inaccessible treatment>       ::= 'warn'
                                 | 'ok'
                                 | 'fatal'
<exception statement>          ::= <lhs> <op declare> <rhs primary> '-' <rhs primary> <adverb list>
<autorank statement>           ::= 'autorank' 'is' <on or off> 'by' 'default'
<op declare>                   ::= <op declare top grammar>
                                 | <op declare lex grammar>
                                 | <op declare any grammar>
<priorities>                   ::= <alternatives>+ separator => <op loosen> proper => 1 hide-separator => 1
<alternatives>                 ::= <alternative>+ separator => <op equal priority> proper => 1 hide-separator => 1
<alternative>                  ::= <rhs> <adverb list>
<adverb list>                  ::= <adverb list items>
<adverb list items>            ::= <adverb item>*
<adverb item>                  ::= <action>
                                 | <left association>
                                 | <right association>
                                 | <group association>
                                 | <separator specification>
                                 | <proper specification>
                                 | <rank specification>
                                 | <null ranking specification>
                                 | <priority specification>
                                 | <pause specification>
                                 | <latm specification>
                                 | <naming>
                                 | <null adverb>
                                 | <symbol action>
                                 | <free action>
                                 | <event specification>
                                 | <hide separator specification>
                                 | <if action>
                                 | <event action>
                                 | <default encoding>
                                 | <fallback encoding>
                                 | <regex action>
<action>                       ::= 'action' /=>|\x{21D2}/u <action name>
                                 | 'action' /=>|\x{21D2}/u <string literal>
                                 | 'action' /=>|\x{21D2}/u <quoted string literal>
<left association>             ::= 'assoc' /=>|\x{21D2}/u 'left'
<right association>            ::= 'assoc' /=>|\x{21D2}/u 'right'
<group association>            ::= 'assoc' /=>|\x{21D2}/u 'group'
<separator specification>      ::= 'separator' /=>|\x{21D2}/u <rhs primary>
<proper specification>         ::= 'proper' /=>|\x{21D2}/u <false>
                                 | 'proper' /=>|\x{21D2}/u <true>
<hide separator specification> ::= 'hide-separator' /=>|\x{21D2}/u <false>
                                 | 'hide-separator' /=>|\x{21D2}/u <true>
<rank specification>           ::= 'rank' /=>|\x{21D2}/u <signed integer>
<null ranking specification>   ::= 'null-ranking' /=>|\x{21D2}/u <null ranking constant>
                                 | 'null' 'rank' /=>|\x{21D2}/u <null ranking constant>
<null ranking constant>        ::= 'low'
                                 | 'high'
<priority specification>       ::= 'priority' /=>|\x{21D2}/u <signed integer>
<pause specification>          ::= 'pause' /=>|\x{21D2}/u 'before'
                                 | 'pause' /=>|\x{21D2}/u 'after'
<event specification>          ::= 'event' /=>|\x{21D2}/u <event initialization>
<event initialization>         ::= <event name> <event initializer>
<event initializer>            ::= '=' <on or off>
<on or off>                    ::= 'on'
                                 | 'off'
<event initializer>            ::=
<latm specification>           ::= 'latm' /=>|\x{21D2}/u <false>
                                 | 'latm' /=>|\x{21D2}/u <true>
<naming>                       ::= 'name' /=>|\x{21D2}/u <alternative name>
<null adverb>                  ::= ','
<symbol action>                ::= 'symbol-action' /=>|\x{21D2}/u <symbol action name>
                                 | 'symbol-action' /=>|\x{21D2}/u <string literal>
                                 | 'symbol-action' /=>|\x{21D2}/u <quoted string literal>
<if action>                    ::= 'if-action' /=>|\x{21D2}/u <if action name>
<regex action>                 ::= 'regex-action' /=>|\x{21D2}/u <regex action name>
<event action>                 ::= 'event-action' /=>|\x{21D2}/u <event action name>
<default encoding>             ::= 'default-encoding' /=>|\x{21D2}/u <default encoding name>
<fallback encoding>            ::= 'fallback-encoding' /=>|\x{21D2}/u <fallback encoding name>
<alternative name>             ::= <standard name>
                                 | <quoted string literal>
<event name>                   ::= <restricted ascii graph name>
                                 | ':symbol'
                                 | ':discard[on]'
                                 | ':discard[off]'
                                 | ':discard[switch]'
<lhs>                          ::= <symbol name>
<rhs>                          ::= <rhs alternative>+
<rhs alternative>              ::= <rhs primary>
                                 | '(-' <priorities> '-)'
                                 | '(' <priorities> ')'
                                 | '(-' <rhs primary> '-' <rhs primary> <adverb list> '-)'
                                 | '(' <rhs primary> '-' <rhs primary> <adverb list> ')'
                                 | '(-' <rhs primary> <quantifier> <adverb list> '-)'
                                 | '(' <rhs primary> <quantifier> <adverb list> ')'
<rhs primary>                  ::= <single symbol>
                                 | <symbol> '@' <grammar reference>
<single symbol>                ::= <symbol>
                                 | <terminal>
<terminal>                     ::= <character class>
                                 | <regular expression>
                                 | <quoted string>
                                 | ':eof'
                                 | ':eol'
<symbol>                       ::= <symbol name>
<symbol name>                  ::= <bare name>
                                 | <bracketed name>
<action name>                  ::= <restricted ascii graph name>
                                 | '::shift'
                                 | '::undef'
                                 | '::ascii'
                                 | /::convert\[[^\]]+\]/
                                 | '::concat'
                                 | /::copy\[\d+\]/
                                 | <lua action name>
                                 | '::true'
                                 | '::false'
                                 | '::json'
                                 | '::jsonf'
                                 | '::row'
                                 | '::table'
                                 | '::ast'
<free name>                    ::= <restricted ascii graph name>
<symbol action name>           ::= <restricted ascii graph name>
                                 | '::transfer'
                                 | '::undef'
                                 | '::ascii'
                                 | /::convert\[[^\]]+\]/
                                 | '::concat'
                                 | <lua action name>
                                 | '::true'
                                 | '::false'
<if action name>               ::= <restricted ascii graph name>
                                 | <lua action name>
<regex action name>            ::= <restricted ascii graph name>
                                 | <lua action name>
<event action name>            ::= <restricted ascii graph name>
                                 | <lua action name>
<default encoding name>        ::= <graph ascii name>
<fallback encoding name>       ::= <graph ascii name>
<quantifier>                   ::= '*'
                                 | '+'
<signed integer>               ::= /[+-]?\d+/
<unsigned integer>             ::= /\d+/
<grammar reference>            ::= <quoted string>
                                 | <signed integer>
                                 | '=' <unsigned integer>
<string literal>               ::= <string literal unit>+ proper => 1
<string literal unit>          ::= '::u8"' <discard off> <string literal inside any> '"' <discard on>
<discard off>                  ::=
<discard on>                   ::=
<string literal inside any>    ::= <string literal inside>* proper => 1
<string literal inside>        ::= /[^"\\\n]/
                                 | '\\' /["'?\\abfnrtve]/
                                 | '\\' /x\{[a-fA-F0-9]{2}\}/
                                 | '\\' /u\{[a-fA-F0-9]{4}\}/
                                 | '\\' /U\{[a-fA-F0-9]{8}\}/
<lua script statement>         ::= '<luascript>' <discard off> <lua script source> '</luascript>' <discard on>
<lua script source>            ::= /[\s\S]/*

<op declare any grammar>         ~ /:\[[\d]+\]:=/
<op declare top grammar>         ~ '::='
<op declare lex grammar>         ~ '~'
<op loosen>                      ~ '||'
<op equal priority>              ~ '|'
<true>                           ~ '1'
<false>                          ~ '0'
<word character>                 ~ /[\w]/
<one or more word characters>    ~ <word character>+ proper => 1
<zero or more word characters>   ~ <word character>* proper => 1
<restricted ascii graph name>    ~ /[!#$%&*+.\/;?\[\\\]^_`~A-Za-z0-9]+/
<graph ascii name>               ~ /[[:graph:]]+/
<lua action name>                ~ /::lua->[a-zA-Z_][a-zA-Z0-9_]*/
<bare name>                      ~ <word character>+ proper => 1
<standard name>                  ~ /[a-zA-Z]/ <zero or more word characters>
<bracketed name>                 ~ '<' <bracketed name string> '>'
<bracketed name string>          ~ /[\s\w]+/
<quoted string>                  ~ /(?:(?|(?:')(?:[^\\']*(?:\\.[^\\']*)*)(?:')|(?:")(?:[^\\"]*(?:\\.[^\\"]*)*)(?:")|(?:\x{201C})(?:[^\\\x{201D}]*(?:\\.[^\\\x{201D}]*)*)(?:\x{201D})))/su
                                 | /(?:(?|(?:')(?:[^\\']*(?:\\.[^\\']*)*)(?:')|(?:")(?:[^\\"]*(?:\\.[^\\"]*)*)(?:")|(?:\x{201C})(?:[^\\\x{201D}]*(?:\\.[^\\\x{201D}]*)*)(?:\x{201D})))/su ':' /ic?/
<quoted string literal>          ~ /(?:(?|(?:')(?:[^\\']*(?:\\.[^\\']*)*)(?:')|(?:")(?:[^\\"]*(?:\\.[^\\"]*)*)(?:")|(?:\x{201C})(?:[^\\\x{201D}]*(?:\\.[^\\\x{201D}]*)*)(?:\x{201D})))/su
<character class>                ~ /((?:\[(?:(?>[^\[\]]+)|(?-1))*\]))/
                                 | /((?:\[(?:(?>[^\[\]]+)|(?-1))*\]))/ ':' /[eijmnsxDJUuaNbcA]+/
<regular expression>             ~ /(?:(?|(?:\/(?![*\/]))(?:[^\\\/]*(?:\\.[^\\\/]*)*)(?:\/)))/su
                                 | /(?:(?|(?:\/(?![*\/]))(?:[^\\\/]*(?:\\.[^\\\/]*)*)(?:\/)))/su /[eijmnsxDJUuaNbcA]+/

NOTES

    The embedded lua language have the following globals when executing its actions:

    marpaESLIF

    Lua object representing current MarpaX::ESLIF instance.

    marpaESLIFGrammar

    Lua object representing current marpaESLIFGrammar instance.

    marpaESLIFRecognizer

    Lua object representing current marpaESLIFRecognizer instance.

    marpaESLIFValue

    Lua object representing current marpaESLIFValue instance, when doing valuation.

    Grammar events have a cost, and when possible lexeme or terminal events should be prefered.

SEE ALSO

marpaESLIF, tconv, ICU

AUTHOR

Jean-Damien Durand <jeandamiendurand@free.fr>

COPYRIGHT AND LICENSE

This software is copyright (c) 2017 by Jean-Damien Durand.

This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.