Antlr - Token

> Antlr

1 - About

The Token definition section of the grammar for which there is no associated lexical rule. The tokens section defines a set of tokens to add to the overall set.

3 - Syntax

The basic syntax is:

tokens { Token1, ..., TokenN }
Advertising

4 - Usage

Most of the time, the tokens section is used to define token types needed by actions in the grammar

Example: explicitly define keyword token types to avoid implicit definition warnings

tokens { BEGIN, END, IF, THEN, WHILE }
 
@lexer::members { // keywords map used in lexer to assign token types
Map<String,Integer> keywords = new HashMap<String,Integer>() {{
    put("begin", KeywordsParser.BEGIN);
    put("end", KeywordsParser.END);
    ...
}};
}

5 - Tokens in the target language

What becomes the tokens defined in the token section..

$ cat Tok.g4
grammar Tok;
tokens { A, B, C }
a : X ;
antlr4 Tok.g4
warning(125): Tok.g4:3:4: implicit definition of token X in parser
cat Tok.tokens
A=1
B=2
C=3
X=4

6 - Documentation / Reference

Advertising