An LLM has never seen a letter in its life

Open figure

It reads integers. Everything you type becomes a list of numbers before the model sees a thing - and that translation layer, the tokenizer, quietly sets your cost, your context window, and half your weird bugs. Here is the whole thing, from bytes up, the way Stanford's CS336 teaches it in lecture one.

It reads integers, not text#

A model does math on numbers, not characters. So step zero of every LLM is a reversible map: string → list of integers → string. The tokenizer owns that map.

The number that decides everything is the compression ratio: bytes ÷ tokens. Higher means each token carries more text, which means shorter sequences. And sequence length is not free - attention scales with the square of it. Halve your tokens and you roughly quarter the compute, while fitting twice the context in the same window. The tokenizer silently sets the ceiling on both price and memory.

Attempt 1 - one token per character#

Map every Unicode code point to an integer. Clean and intuitive. The problem is the vocabulary: Unicode has about 150,000 code points, most of them rare characters and emoji that almost never appear - yet each one still burns a row in the embedding table. You pay for a giant vocabulary and use a sliver of it, and common words still fragment into many tokens.

Attempt 2 - one token per byte#

Encode the text as UTF-8. Now the vocabulary is exactly 256. Tiny, universal, and there is no such thing as an unknown character. But the compression ratio collapses to about 1 byte per token, so sequences get brutally long and attention cost explodes with them. Worth knowing: UTF-8 is variable width - English is 1 byte per character, but many scripts are 3 - so non-English text is penalized before the tokenizer does anything clever.

Attempt 3 - one token per word#

Split on whitespace and punctuation. Great compression, one token per word. But the vocabulary is now unbounded. New words, typos, product names, code identifiers - all collapse to a single "unknown" token, and that unknown is information you can never recover. Good ratio, fatal edge cases.

The answer - Byte-Pair Encoding#

Sennrich and colleagues introduced BPE for translation in 2016; GPT-2 made it the default. Instead of picking a rule, it learns the vocabulary from data. Common sequences like "the", "ing" and "tion" get merged early into single tokens; rare words stay split into subword pieces. The result is the best of every earlier attempt - a bounded vocabulary, strong compression, and zero unknowns. The whole training loop is four steps:

• Start from raw bytes (vocabulary = 256).
• Count every adjacent pair of tokens in the corpus.
• Merge the single most frequent pair into one new token (first new id = 256).
• Repeat until you hit the target vocabulary size.

What GPT-2's tokenizer actually is#

Its vocabulary is 50,257 tokens, and that number isn't arbitrary: 256 base bytes, plus 50,000 learned merges, plus 1 special end-of-text marker. On English it compresses to roughly 4 bytes per token. That "1 token ≈ 4 characters" rule of thumb on every pricing page? This is where it comes from.

Encoding at inference is just: bytes → apply the learned merges in order → token ids. Decoding walks the merges backward to reconstruct the exact bytes. Fully reversible.

Open figure

The details that break real systems#

This is the part most explainers skip, and it is where the bookmarks live.

• The space is part of the token. "the" and " the" are different tokens - in GPT-2, id 1169 versus id 262. The space-prefixed form is so much more common that it earned the smaller number. This is why prompts are whitespace-sensitive and why a stray leading space quietly changes behavior.
• Numbers get chopped. GPT-2 splits "1234567" into "123 · 45 · 67" - three ragged tokens. That misalignment is a real reason models fumble arithmetic: the digits never arrive as clean, place-aligned units.
• The multilingual tax is real. English runs about 4 bytes per token; many other languages need two to three times more tokens for the same meaning. Same paragraph, more tokens, higher cost, less usable context. The gap is baked into the tokenizer.
• Glitch tokens. BPE merged some strings into a single token - the infamous " SolidGoldMagikarp" is one real GPT-2 token, id 43453 - that then almost never appeared in training. The embedding was created but never learned, so feeding it makes models hallucinate or break. A ghost left behind by the merge step.

Open figure

The mental model to keep#

Tokenization is a compression problem wearing a linguistics costume. Every design choice trades between three things:

• Vocabulary size - embedding parameters.
• Sequence length - attention cost and context.
• Robustness - never emit an unknown, handle anything.

Character optimizes robustness and kills compression. Word optimizes compression and kills robustness. Byte is bulletproof but slow. BPE is the negotiated peace - which is why a 2016 translation trick still runs inside models built today.

Why this is worth a whole lecture#

You can't reason about context windows, API cost, why your non-English users pay more, or why the model can't add - until you accept that none of it is text to the model. It is integers, and the tokenizer chose them. Learn this one layer and half the "weird LLM behavior" you filed under magic turns into arithmetic.

The full lecture is free: Stanford CS336, Lecture 1. If you want it to truly stick, their Assignment 1 has you implement BPE by hand.

Which of these gotchas has bitten you in production?

Source: Stanford CS336 - Language Modeling from Scratch, Lecture 1. Diagrams drawn from the lecture by @sushant_p18; every token id verified against the GPT-2 tokenizer.