What the conversion layer actually does
— text in, integers out, and the same route back again
Raw text
The blue whale is a massive creature. It thrives on small ocean shrimp that can grow no more than 2 inches, known as krill.
1 · SPLIT INTO TOKENS — the boundaries are the tokenizer's decision, not the language's
The blue whale is a massive creature.
2 · LOOK EACH ONE UP IN THE VOCABULARY
9769861103574382261189654694913
3 · EACH INTEGER INDEXES ONE ROW OF THE EMBEDDING TABLE
-0.10.40.9⋮0.70.20.5⋮-0.20.30.6⋮0.50.80.8⋮…only from here on does the model do arithmetic
The map is reversible. Nothing above is lossy: the integers
976 · 9861 · 103574 · … decode back to exactly the bytes that went in, which is
why the tokenizer can be a hard boundary rather than a guess. The only choice being made is
where the cuts fall — and every design in this article is a different answer to that.
Eight tokens for eight English words, which is close to the best case. The same sentence in a script the
tokenizer was not built for would produce the same three steps and a
very different number of boxes in row 2 — and rows 2 and 3 are where every cost
in the model is counted.
Token ids are real, from OpenAI o200k_base·@sushant_p18