Criterion 1 — byte-pair encoding
— merge the pair you see most often, then do it again
# start from single characters
splits = {w: list(w) for w in corpus}
vocab = set(all_characters)
for step in range(n_merges):
# count every adjacent pair in the corpus,
# weighted by how often each word occurs
counts = pair_counts(splits)
best = max(counts, key=counts.get) # ← the criterion
# fuse that pair everywhere it appears
apply_merge(splits, best)
vocab.add(best)
# vocabulary size = base characters + n_merges
# nothing here looks at meaning, only at counts
Three rounds on the toy corpus hug ×10 · pug ×5 · pun ×12 · bun ×4 · hugs ×5
STEP 1
ug:20 pu:17 un:16 hu:15 gs:5 bu:4
STEP 2
un:16 hug:15 pu:12 pug:5 ugs:5 bu:4
STEP 3
hug:15 pun:12 pug:5 ugs:5 bun:4
Frequency is the only thing consulted. After three rounds the corpus needs 8 tokens where it
needed 19 characters, and “hug” has become a single unit — not because it means anything, but because
those letters kept turning up together. Whichever language dominates the corpus is the
one whose words get absorbed first, and that is the entire origin of the tax.
Every count produced by vocab_algorithms.py, published with this article·@sushant_p18