Lexical Analysis - (Token|Lexical unit|Lexeme|Symbol|Word)
Table of Contents
1 - About
A token is symbols of the vocabulary of the language.
Each token is a single atomic unit of the language.
A token is:
- a string of characters,
- categorized with a lexeme's type.
The process of finding and categorizing tokens from an input stream is called “tokenizing” and is performed by a Lexer (Lexical analyzer).
Token represents symbols of the vocabulary of a language.
See also Natural Language - Token
2 - Articles Related
3 - Lexeme Type
A token might be:
- a literal (number, …)
- an operator (Assignment,Addition,…)
- a comment
- (Delimiters|End of Statement) (simple and compound symbols)
- (keyword|Identifiers), which include reserved words
- a keyword,
Consider the following programming expression:
sum = 3 + 2;
Tokenized in the following table: