Build a Lexer From Scratch in Python: Nyravelle Part 3

10 min read · Last updated: October 24, 2026

To build a lexer from scratch, you write a loop that reads source code one character at a time and groups those characters into tokens: keywords, names, strings, numbers and punctuation. Each token also records its type and its line number. In this tutorial you'll write about 120 lines of plain Python that do exactly that for Nyravelle workflow files.

Came here from Instagram, LinkedIn, X or YouTube? Here is the full source list you came for. Everything below is built from these, nothing else.

The series

• Building a Lexer from Scratch, Part 3 of 24 in "Building Nyravelle, a Programming Language for AI Workflows" (Stackademic, accessed October 24, 2026)

The code

• Nyravelle on GitHub: "A language for orchestrating intelligent systems" (4 commits, October 18 to 20, 2026)
• nyra.py, the bootstrap validator (commit "Adding basic syntax", October 20, 2026)
• examples/documentreview.nyra, the sample workflow used below

The reference

• Crafting Interpreters by Robert Nystrom, chapter 4: Scanning

Key takeaways

• A lexer turns a flat string into a list of tokens. Each token carries its type, raw text, value, line and column.
• The whole lexer below is about 120 lines of Python with zero dependencies. You need only the standard library.
• Nyravelle's current syntax needs just 3 keywords (workflow, input, say) and 3 punctuation marks ({, }, :).
• Keywords are scanned as identifiers first, then looked up in a table. That one trick prevents the classic sayHello bug.
• A good lexer records an error and keeps going, so you see every bad character in one run instead of one per run.
• The longest match wins. Crafting Interpreters calls this "maximal munch".

What you'll build, and why a language needs a lexer first

A lexer, also called a scanner or tokenizer, is the first stage of every compiler and interpreter. It reads raw characters and produces tokens, so the parser never has to think about spaces, quotes or line breaks. You'll build one for Nyravelle that tokenizes a full workflow file and reports errors with line and column.

Nyravelle is a small language for describing AI workflows. Its GitHub repo describes it as "a language for orchestrating intelligent systems". The series covering it runs to 24 parts on Stackademic, and Part 3 is the lexer.

Here is a real file from the repo, examples/documentreview.nyra:

workflow DocumentReview {
    input document: string
    input reviewer: string
    say "Review started."
}

Right now the repo checks files like this with nyra.py, a line-by-line validator. It confirms the file opens with workflow Name {, accepts input name: type where type is string, number or boolean, accepts say "...", and flags everything else as unsupported.

That works for 3 statement types. It stops working the moment the language grows. Line-by-line string matching can't handle a string that contains a brace, or two statements that share a prefix. A lexer fixes that by cutting the source into clean pieces once, so every later stage works with tokens instead of text.

By the end you'll have this output for the file above:

1:1   WORKFLOW  'workflow'
1:10  IDENT     'DocumentReview'
1:25  LBRACE    '{'
1:26  NEWLINE   '\n'
2:5   INPUT     'input'
2:11  IDENT     'document'
2:19  COLON     ':'
2:21  IDENT     'string'
...

This is my own walkthrough, written against the syntax in the public repo. For the series author's version and reasoning, read the Stackademic post linked above.

What is the difference between a lexeme and a token?

A lexeme is the raw slice of source text, like workflow or "Review started.". A token is that lexeme bundled with extra data: its type, its parsed value and its position. Crafting Interpreters puts it simply: when you bundle the lexeme with that other data, "the result is a token."

Term	Example from the file	What it holds
Character	w	One character of source
Lexeme	"Review started."	The exact text, quotes included
Token type	STRING	The category the parser checks
Value	Review started.	The cleaned value, quotes removed
Position	line 4, column 9	Where to point when something breaks

The parser in Part 4 and beyond will only look at types and values. The lexeme and position exist for one reason: good error messages.

Prerequisites

You need very little to follow along.

• Python 3.10 or newer. The code uses dataclasses and enum, both in the standard library.
• Comfort with classes and string indexing in Python.
• Optional: pytest if you want to run the tests at the end.
• No parser generator, no regex library, no framework.

If compilers feel abstract, our piece on programming languages and the languages used to build them gives the bigger picture of how one language gets built inside another.

How to build a lexer from scratch, step by step

You build a lexer in six steps: define the token types, define the token shape, set up a cursor over the source, write the main loop, then handle the two multi-character cases (strings, and words or numbers). Every step below is real code. Put it all in one file called lexer.py.

Step 1: Define the token types

List every kind of token the language has. Nyravelle needs very few right now.

from dataclasses import dataclass
from enum import Enum, auto


class TokenType(Enum):
    LBRACE = auto()
    RBRACE = auto()
    COLON = auto()
    IDENT = auto()
    STRING = auto()
    NUMBER = auto()
    WORKFLOW = auto()
    INPUT = auto()
    SAY = auto()
    NEWLINE = auto()
    EOF = auto()


KEYWORDS = {
    "workflow": TokenType.WORKFLOW,
    "input": TokenType.INPUT,
    "say": TokenType.SAY,
}

SINGLE = {
    "{": TokenType.LBRACE,
    "}": TokenType.RBRACE,
    ":": TokenType.COLON,
}

Two choices here are worth explaining.

NEWLINE is a real token. The Nyravelle examples end each statement at the end of the line, with no semicolons. So the line break carries meaning, and the parser needs to see it.

string, number and boolean are not keywords. In input document: string, the word string comes out as a plain IDENT. The lexer doesn't decide what counts as a valid type. The parser does. Keep the lexer dumb and the type list stays easy to change.

I've also added NUMBER now, even though the current examples have no number literals. The validator already accepts number as an input type, so literals are coming, and it costs 8 lines.

Step 2: Define what a token looks like

@dataclass(frozen=True)
class Token:
    type: TokenType
    lexeme: str
    value: object
    line: int
    col: int

frozen=True makes tokens immutable. Nothing downstream should ever edit a token, and now nothing can.

Step 3: Set up the cursor

The lexer is a cursor walking over a string. It needs to know where the current token started, where it is now, and which line and column it's on.

class Lexer:
    def __init__(self, source):
        self.src = source
        self.start = 0
        self.pos = 0
        self.line = 1
        self.col = 1
        self.start_col = 1
        self.tokens = []
        self.errors = []

    def peek(self):
        return self.src[self.pos] if self.pos < len(self.src) else "\0"

    def advance(self):
        ch = self.src[self.pos]
        self.pos += 1
        self.col += 1
        return ch

    def add(self, type_, value=None):
        lexeme = self.src[self.start:self.pos]
        self.tokens.append(Token(type_, lexeme, value, self.line, self.start_col))

    def error(self, msg):
        self.errors.append(f"line {self.line}, col {self.start_col}: {msg}")

peek() looks at the next character without consuming it. advance() consumes it. Almost every decision in a lexer is "peek, then maybe advance". Returning "\0" at the end of input means you never index past the string.

Notice that error() appends to a list instead of raising. That matters in step 4.

Step 4: Write the main loop

This is the heart of the lexer. Each pass marks where a token starts, then scans one token.

    def tokenize(self):
        while self.pos < len(self.src):
            self.start = self.pos
            self.start_col = self.col
            self.scan_token()
        self.start = self.pos
        self.start_col = self.col
        self.add(TokenType.EOF)
        return self.tokens

    def scan_token(self):
        ch = self.advance()
        if ch in " \t\r":
            return
        if ch == "\n":
            self.add(TokenType.NEWLINE)
            self.line += 1
            self.col = 1
            return
        if ch == "#":
            while self.peek() not in ("\n", "\0"):
                self.advance()
            return
        if ch in SINGLE:
            self.add(SINGLE[ch])
            return
        if ch == '"':
            self.string()
            return
        if ch.isdigit():
            self.number()
            return
        if ch.isalpha() or ch == "_":
            self.identifier()
            return
        self.error(f"unexpected character {ch!r}")

What each branch does:

1. Spaces, tabs and carriage returns are skipped. No token.
2. A newline emits NEWLINE, bumps the line counter and resets the column.
3. # starts a comment that runs to the end of the line. The repo's validator doesn't support comments yet, so this is my addition. Crafting Interpreters makes the point well: comments are lexemes, but the parser doesn't want them, so you emit nothing.
4. Single characters {, } and : map straight to a token.
5. A quote, digit or letter hands off to a helper that reads more than one character.
6. Anything else is an error. You record it and keep going.

That last point is the difference between a toy and a useful tool. Crafting Interpreters says it directly: "we keep scanning. There may be other errors later in the program." If a file has 3 typos, you want 3 messages from one run, not 3 runs.

EOF at the end gives the parser a clean stop signal, so it never has to check list length.

Step 5: Scan strings, with escapes

Strings are the trickiest token because they can contain anything, including the quote character itself.

    def string(self):
        chars = []
        while self.peek() != '"':
            if self.peek() in ("\n", "\0"):
                self.error("unterminated string")
                return
            ch = self.advance()
            if ch == "\\":
                nxt = self.advance() if self.peek() != "\0" else ""
                chars.append({"n": "\n", "t": "\t", '"': '"', "\\": "\\"}.get(nxt, nxt))
            else:
                chars.append(ch)
        self.advance()
        self.add(TokenType.STRING, "".join(chars))

The repo's validator already allows escaped quotes inside say "...", so the lexer must too. A backslash consumes the next character and translates it, so \" becomes a literal quote instead of ending the string.

The lexeme keeps the raw text with its quotes. The value holds the clean string. Your error messages can show the original, and your runtime gets the real text.

Strings also stop at a newline. I made say strings single-line on purpose: an unclosed quote then fails on its own line instead of eating the rest of the file.

Step 6: Scan numbers, identifiers and keywords

    def number(self):
        while self.peek().isdigit():
            self.advance()
        if self.peek() == "." and self.pos + 1 < len(self.src) and self.src[self.pos + 1].isdigit():
            self.advance()
            while self.peek().isdigit():
                self.advance()
        self.add(TokenType.NUMBER, float(self.src[self.start:self.pos]))

    def identifier(self):
        while self.peek().isalnum() or self.peek() == "_":
            self.advance()
        text = self.src[self.start:self.pos]
        self.add(KEYWORDS.get(text, TokenType.IDENT), text)

number() only takes a . if a digit follows it. That keeps 3. from swallowing a dot that might mean something else later.

identifier() is the important one. It reads the whole word first, then checks the keyword table. This is maximal munch: when two rules could match, the longest one wins. Crafting Interpreters uses the example orchid, which must scan as one identifier and not as the keyword or plus chid. In Nyravelle the same trap is sayHello. Read the whole word and it's an IDENT. Match say first and you've got a bug.

As the book puts it, "a reserved word is an identifier, it's just one that has been claimed by the language for its own use."

Step 7: Run it

Add a small entry point at the bottom of lexer.py:

if __name__ == "__main__":
    import sys

    lexer = Lexer(open(sys.argv[1]).read())
    for t in lexer.tokenize():
        print(f"{t.line}:{t.col:<3} {t.type.name:<9} {t.lexeme!r}")
    for e in lexer.errors:
        print("error:", e)
    sys.exit(1 if lexer.errors else 0)

Then run it on the example file:

python lexer.py examples/documentreview.nyra

The exit code follows the convention the repo's validator already uses: 1 when there are errors, 0 when the file is clean. That makes the lexer usable in scripts and CI from day one.

How do you test a lexer?

You test a lexer by feeding it short strings and asserting the exact list of token types it returns. Each test covers one rule: a keyword, an escape, an error, a comment. Lexers are pure functions from text to tokens, so they're the easiest part of a language to test well.

Save this as test_lexer.py next to lexer.py and run pytest:

from lexer import Lexer, TokenType as T


def types(src):
    return [t.type for t in Lexer(src).tokenize()]


def test_keyword_is_not_a_prefix_match():
    assert types("say sayHello") == [T.SAY, T.IDENT, T.EOF]


def test_input_line():
    assert types("input document: string") == [T.INPUT, T.IDENT, T.COLON, T.IDENT, T.EOF]


def test_escaped_quote():
    tok = Lexer(r'say "a \"quoted\" word"').tokenize()[1]
    assert tok.value == 'a "quoted" word'


def test_unterminated_string_reports_position():
    lexer = Lexer('workflow A {\n  say "oops\n}')
    lexer.tokenize()
    assert lexer.errors == ["line 2, col 7: unterminated string"]


def test_bad_character_keeps_scanning():
    lexer = Lexer('say @ "hi"')
    assert [t.type for t in lexer.tokenize()] == [T.SAY, T.STRING, T.EOF]
    assert len(lexer.errors) == 1

The repo already has a tests/test_nyra.py for the validator, so a lexer test file fits the project's existing layout.

Should you hand-write a lexer or use a generator?

For a small language like Nyravelle, hand-write it. A hand-written lexer is short, has no dependencies, and gives you full control over error messages. Generators and regex tables pay off when the token set is large, but they make good errors harder.

Approach	Good for	Cost
Hand-written loop (this tutorial)	Small to medium languages, custom error messages	You write every rule yourself
One big regex with named groups	Quick prototypes	Error positions and escapes get messy
Lexer generator tool	Large grammars with many token types	A new dependency and a build step

Crafting Interpreters takes the hand-written route for the same reason: you learn what a lexer actually does, and you can change any rule in a minute.

Common mistakes when you build a lexer from scratch

Most lexer bugs come from 5 places: matching keywords before reading the whole word, forgetting to track lines, stopping at the first error, mishandling escapes, and not emitting an end-of-file token. Each one is cheap to avoid if you know it's coming.

1. Matching keywords as prefixes. If you check startswith("say"), then sayHello breaks. Read the whole identifier, then look it up.
2. Losing line numbers. Reset the column on every \n, and record start_col before you scan, not after. Otherwise every error points to the end of the token.
3. Raising on the first error. Collect errors in a list. One run should show every problem.
4. Ending strings at an escaped quote. Handle \\ before you check for the closing ".
5. Skipping EOF. Without it, the parser has to check "am I past the end?" everywhere.
6. Putting grammar rules in the lexer. Whether string is a valid type is the parser's job. The lexer only says "this is a word".

If you're learning this with an AI assistant, ask it to break your lexer with edge cases rather than write it for you. Our guide on how to use AI to learn programming 3x faster covers that habit in more detail.

What comes after the lexer?

After the lexer comes the parser. It reads the token list and checks that the tokens appear in a valid order, like INPUT IDENT COLON IDENT NEWLINE, and builds a tree from them. That tree, usually called an abstract syntax tree, is what the rest of the language works on.

With tokens in hand, the parser for Nyravelle's current syntax is short. workflow must be followed by an IDENT and LBRACE. Each body line is either input or say. The file ends at RBRACE then EOF. Every check that nyra.py does today with string matching becomes a check on token types.

Things to add to your lexer as the language grows:

• More keywords as new statements land. Each one is a single line in KEYWORDS.
• Two-character operators like == or ->. Peek one character ahead before you emit.
• Booleans true and false. Add them as keywords with a value.
• Indentation tracking, only if the language ever drops braces.

The hard part of a language is rarely the code. It's deciding what the syntax should mean, a point we made in code was never the hard part.

FAQ

What does a lexer do in a programming language?

A lexer reads raw source code and splits it into tokens such as keywords, names, strings, numbers and punctuation. It also drops things the parser doesn't need, like whitespace and comments. Every token records its position, so later errors can point to the exact line and column.

Is a lexer the same as a tokenizer or a scanner?

Yes. Lexer, scanner and tokenizer all name the same stage. Crafting Interpreters uses "scanning" and "lexing" interchangeably. The output is always a flat list of tokens that the parser consumes next.

How long does it take to build a lexer from scratch?

For a small language like Nyravelle, the lexer in this tutorial is about 120 lines of Python, and you can write and test it in an afternoon. Larger languages take longer mainly because of more operators and literal formats, not because the core loop changes.

Why are keywords scanned as identifiers first?

Because the longest match must win. If you match keywords as prefixes, a name like sayHello would split into the keyword say and the name Hello. Reading the full word and then looking it up in a keyword table avoids that bug in one line.

Type the code in, run it on documentreview.nyra, then break it on purpose with a missing quote. While you're building, browse the free tools on Art of Code, and if you're still picking your path, read our AI-first roadmap for learning programming in 2026.

Written by Adithya Guttha, Founder of Art of Code.