Rules apply in this order at encode time. Equal-frequency pairs break by a fixed order — training is deterministic.
Startraw bytes
the␣the␣then
12
tokens
+256merge t·h
the␣the␣then
9
tokens
+257merge th·e
the␣the␣then
6
tokens
+258merge the·␣
the␣258the␣258the257n
4
tokens
THE PAYOFFFrequent sequences collapse to one token · rare words fall back to subwords · nothing is ever UNK. Same loop, 50,000 merges over real text = GPT-2's vocabulary.
FIG. 2 — 3 MERGES: 12 → 4 TOKENS, VOCAB +3 · GPT-2 DOES THIS 50,000×