Antlr - Lexer Rule (Token names|Lexical Rule)

> Antlr (ANother Tool for Language Recognition)

1 - About

Compiler - Lexer rule (Token names, Lexical rule, Token name) in Antlr.

They are rules that defines tokens.

They are written generally in the grammar but may be written in a lexer grammar file

Advertising

3 - Type

  • standard
  • fragment - A fragment is a special type of lexer rule that does not result in creation of tokens. They are only present to introduce logical expression that simplify the grammar.

4 - Example

CommonLexerRules.g4
ID  :   [a-zA-Z]+ ;      // match lowercase and uppercase letters from A to Z
INT :   [0-9]+ ;         // match a serie of digit from 0 to 9
DIGITS : [0-9] +; // same
NEWLINE:'\r'? '\n' ;     // match/return newlines to parser (end-statement signal)
WS  :   [ \t]+ -> skip ; // toss out whitespace and tab
HEX : ('%' [a-fA-F0-9] [a-fA-F0-9])+ ; // hexadecimal
STRING : ([a-zA-Z~] |HEX) ([a-zA-Z0-9.-] | HEX)*; // lexer rule can use other lexer rule

5 - Syntax

Basically the same syntax than parser rules except that lexer rules:

  • cannot have arguments,
  • cannot return values, or local variables.

Lexer rule names (known als as Token name) must begin with an uppercase letter whereas parser rule names begin with a lowercase letter.

A lexer rule can be associated with:

  • a single literal string expected in the input
  • a selection of literal strings that may be found
  • a sequence of specific characters and ranges of characters using the cardinality indicators ?, * and +

A lexer rule:

  • cannot be associated with a regular expression.
  • can refer to other lexer rules.
Advertising

6 - Precedence

Grammar - (Order of (operations|precedence)|operator precedence): The lexer chooses the rule that matches the most characters. If there is a tie then the first one is used.