Lexical Analysis is the first phase of a Compiler.
In simple words:
Lexical Analysis reads the program character by character and converts it into meaningful units called tokens.
The program that performs lexical analysis is called a Lexical Analyzer or Lexer.
1. Simple Example
Suppose we have this C statement:
int sum = a + 10;
The lexical analyzer breaks it into tokens:
| Token | Type |
|---|---|
int | Keyword |
sum | Identifier |
= | Operator |
a | Identifier |
+ | Operator |
10 | Number/Constant |
; | Separator |
So:
Source Program → Lexical Analyzer → Tokens
2. What is a Token?
A token is a meaningful unit of a program.
Common types of tokens are:
1. Keywords
Reserved words of a programming language.
Examples:
int, float, if, else, while, return
2. Identifiers
Names given to variables, functions, classes, etc.
Examples:
sum
total
studentName
3. Operators
Symbols used to perform operations.
Examples:
+
-
*
/
=
==
<
>
4. Constants/Literals
Fixed values.
Examples:
10
25.5
"Hello"
'A'
5. Separators/Punctuation
Used to separate program elements.
Examples:
;
,
(
)
{
}
3. Main Functions of Lexical Analyzer
The lexical analyzer performs several important tasks:
1. Reads the source program
It reads the program from left to right.
2. Identifies tokens
It recognizes keywords, identifiers, operators, constants, etc.
3. Removes white spaces
Spaces, tabs and unnecessary line breaks are generally ignored.
Example:
int x = 10;
and
int x=10;
have the same tokens.
4. Removes comments
Comments are generally ignored during lexical analysis.
Example:
// This is a comment
int x = 10;
5. Detects lexical errors
It identifies invalid characters or invalid token patterns.
Example:
int x = @10;
@10 may produce a lexical error depending on the language.
6. Sends tokens to the parser
After identifying tokens, the lexical analyzer passes them to the Syntax Analyzer (Parser).
4. Lexical Analysis in Compiler
The basic compiler structure is:
Source Program
↓
Lexical Analyzer
↓
Tokens
↓
Syntax Analyzer
↓
Semantic Analyzer
↓
Intermediate Code
↓
Code Optimization
↓
Code Generation
So remember:
Lexical Analysis comes FIRST.
5. Lexical Analyzer and Regular Languages
This is particularly important because Lexical Analysis is related to Regular Languages.
Tokens can generally be described using Regular Expressions and recognized using Finite Automata.
For example:
Identifier:
letter (letter | digit)*
This pattern can be represented using a Regular Expression and recognized by a Finite Automaton.
Therefore:\[ \boxed{\text{Regular Expression → Finite Automaton → Token Recognition}} \]
6. Lexeme, Token and Pattern
These three terms are important for exams.
| Term | Meaning | Example |
|---|---|---|
| Token | Category/type | Identifier |
| Lexeme | Actual sequence in program | total |
| Pattern | Rule describing valid lexemes | letter(letter|digit)* |
Example:
int total = 100;
For total:
- Token: Identifier
- Lexeme:
total - Pattern: valid identifier pattern
7. Lexical Errors
A lexical error occurs when the sequence of characters cannot form a valid token.
Example:
int x = 10 @ 20;
If @ is not a valid operator/symbol in that language, the lexer reports a lexical error.
Other examples:
- Invalid character
- Incorrect number format
- Unterminated string
- Invalid identifier
8. Lexical Analyzer vs Syntax Analyzer
| Lexical Analyzer | Syntax Analyzer |
|---|---|
| First phase | Second phase |
| Produces tokens | Produces parse tree |
| Checks valid tokens | Checks grammar/syntax |
| Uses regular expressions/finite automata | Uses CFG/parser |
Example: identifies int as keyword | Checks whether statement structure is correct |
Easy Example
int x = ;
Lexical analysis may recognize:
int → x → = → ;
All are valid tokens.
But the syntax analyzer detects that the statement is incomplete because a value is missing after =.
⭐ Exam Answer — 5–10 Marks
Lexical Analysis is the first phase of a compiler. It reads the source program character by character and groups the characters into meaningful units called tokens. The program that performs this task is called a Lexical Analyzer or Lexer.
For example:
int sum = a + 10;
is converted into:
int → Keyword
sum → Identifier
= → Operator
a → Identifier
+ → Operator
10 → Constant
; → Separator
The major functions of lexical analysis are:
- Reading the source program.
- Identifying tokens.
- Removing white spaces and comments.
- Detecting lexical errors.
- Passing tokens to the syntax analyzer.
- Maintaining information about identifiers and their attributes.
Lexical analysis is closely related to regular languages because tokens can generally be described using regular expressions and recognized using finite automata.
🧠 Fast Memory Trick
Lexical Analysis = Character → Token
Remember:
“Lexer reads, groups, removes, checks and sends tokens.”
And the compiler order:
Lexical → Syntax → Semantic → Intermediate Code → Optimization → Code Generation
