The joy of Compiler Development

2026-08-31

Most of my side projects involve, in some form or another, writing compilers. They lie in the middle ground between something very complex and very simple at the same time, and in case you get bored or tired of implementing a particular feature, you can most certainly switch to something new, or complement an existing piece you wrote before. It is, for me, the equivalent to doomscrolling for programming (1) (1)Although, unlike endlessly scrolling, you actually have to use your brain. .

Compilers as language transformation tools #

A compiler’s task is to take in some kind of code (mostly in human-readable text form) and produce a program that your computer can execute. This “code” can be anything you want: a known language – like plain English –, a highly technical set of rules for defining an algorithm with specific keywords, a set of distinct integers, or a bunch of symbols next to each other. It really doesn’t matter, for as long as you can define a grammar for it (2) (2)There is technicality in regards to what can form a language and what cannot, but I won’t dive into it, mainly because it is not a topic I’m particularly knowledgeable about! – i.e., a set of rules that describe how its elements can be interpreted.

Here’s the fun bit: since you can define your own rules, and use those rules to turn whatever thing you like into code that can be executed in your machine, you can create almost like a virtual playground with its own inner world that can be used to write your own programs in. ✍ I should really stop living in fairy tales. If you want to create your own programming language with its own syntax and cool operators and whatnot, you are more than capable of doing so. And the best part is that it’s not as hard as it sounds.

When I started programming, I thought programming languages were some kind of magic that spawned into this world out of nowhere. I didn’t understand how you’re able to run a language. Running a language… Do you not have to speak a language in order to use it?

After a couple of years in college some pieces started to make sense. Languages are implemented by something called a compiler and a compiler does magic on your code to transform it into a program. Easy. And more so, you are taught how to write one yourself.

First add a few bricks, then the whole ceiling #

Many kinds of software need lots of tiny subsystems to be written first in order to see actual progress. For example, databases require you to design how you’re going to store data, which file formats you’re going to use (if any existing, otherwise you’ll have to define your own), what your access patterns are going to look like, and how you’re going to grab that data: from disk, network, cache, etc. Operating Systems require a lot of assembly code in order to see your first “Hello, world” running on a VM. Compilers, on the other hand, allow you to see progress really early during development.

They are – at least in my experience – similar to writing videogames. It doesn’t take much until you start seeing your first tokens being printed from a piece of code you came with. In games, it is like seeing your red rectangle moving for the first time.

Compilers are modular, which means they are made up of various components, usually one consumed by the other. The main ones are the lexer (sometimes referred to as “tokenizer”, “scanner”) and parser(s), which make up the frontend (3) (3)There are lots of tools that help when implementing parsers and lexers: bison, lex, yacc are some examples. , while the code generation module (and many others that may exist in the middle, such as optimization phases, AST reconstruction, etc.) makes up the backend (4) (4)Similarly to frontends, there are tools for code generation, such as the well-known LLVM. . Implementing these modules is not hard (though, they may be for an actual production-grade compiler), and it’s very rewarding once you see they begin to work together.

Simply put, a lexer translates a string of text into “word types” (or tokens). For example, the following text can be tokenized as follows:

constant   the_answer_to_the_world :    42            ;
========   ======================= =    ==            =
[ConstKwd] [Identifier]            [Eq] [NumLiteral]  [Semicolon]

In a non-programming language, such as English, one would categorize words as “verbs”, “adjectives” or “nouns”. In programming languages, this is no different: we put similar associations to words (which form the syntax of the language) and produce a list of tokens to be used later. These tokens are used by the parser to construct an abstract syntax tree (AST), an abstract representation of your program. For example, the above statement can be shown as:

[ValueAssignment]
    [Identifier] the_answer_to_the_world
    [Expression] 42

Notice how we got rid of certain tokens (like the constant keyword, equal (':') symbol and semicolon (';')) and only saved the important pieces: the variable name and value. ✍ This representation can then take many shapes: some compilers produce something called an intermediate representation or IR, others get further rewritten when optimizing, and others operate directly on this form. This is the essence of the AST and simplifies the following phases a lot. The following phases use this AST in some way or another to construct the final executable (5) (5)While it is possible to construct an executable by skipping this step, it is very convenient to have it since the AST can then be used to introduce optimizations and even allow for the usage of tools like LSP servers, not mentioning the fact that it gives you an easier way to look at a program. .

What’s interesting in this approach is that you can build things incrementally: adding support for value assignments does not mean you have to finish support for floating point numbers first. You can do it in any order you like (6) (6)Some restrictions apply: you may not be able to implement for loops until you have boolean values working. , and if you feel tired of getting something working, you can always leave it for the moment and come back to it later. But one thing is sure: having control over your language is fun. If you want to add a token for a symbol that only exists in the cyrillic alphabet, you can. If you want to introduce a maybe keyword that has a 50% chance of being true and 50% of being false, you also can. The sky is the limit.

Building a small programming language for the sake of it #

I spent the last week implementing a small language inspired by simple pseudocode / Python style syntax. It looks like the following:

algorithm is_prime(n) is
    if n < 2 then
        return false
    end

    variable i := 2
    for i * i < n + 1 do
        variable divided := n // i
        variable remainder := n - divided * i
        if remainder == 0 then
            return false
        end
        i := i + 1
    end

    return true
end

algorithm count_primes(limit) is
    variable count := 0
    variable n := 2
    for n < limit + 1 do
        if is_prime(n) then
            count := count + 1
        end
        n := n + 1
    end
    return count
end

variable result := count_primes(50)
print(result)

It is refreshingly simple: you have variable assignments, for loops, if checks and functions. No big features like borrow checks, compile-time execution, or linear types (as the current trends of new programming languages seem to have!). Of course, this is a toy language, written for the sake of having fun, and it is actually somewhat usable!

The development was so smooth I spent the past week staring at my screen for a total of 30 hours without noticing. I spent quite some time writing the lexer and parsers myself (without using any frontend tools – I just like writing things myself) and quickly jumped into running the produced code.

The language is interpreted, which means that the program itself runs on top of another program written in another programming language. In this case, it is C++. Interpreted languages generally have trade-offs compared to compiled ones (such as Rust, C, Zig…) (7) (7)They’re usually slower, since they have a layer of indirection between your code and whatever’s running to support it (which can prevent optimizations). but are easier to write, since they can reuse the existing functionality found in the programming language you’re using to write the interpreter. Following the same set of examples above, interpreting a variable assignment during execution can be thought as follows:

func interpret_assignment(value_assignment_ast):
    var_name = value_ast.get_var_name()
    var_value = to_integer(value_ast.get_var_value())

    program.variables[var_name] = var_value
    return ok

“Interpreting” a value assignment would then mean keeping track of the variable’s name along with its current value. Whenever the variable changes, its value must be updated.

This is the heart of compiler development. With lots of bite-sized chunks of work you can implement anything, and I must admit it is quite addicting to see your language grow over time. If you want to have a great time, have a few weekends to spare and want to turn your favorite football player into a programming language, you should try writing a compiler.

If you want to take a look (and build, or possibly contribute) at the language itself, it’s open source in GitHub. For now, I’ll go ahead and add new things to it!