# start DELIBERATELY too large: characters # plus many frequent substrings probs = {t: c / total for t, c in seed.items()} def loss(vocab): # a word can be cut many ways; Viterbi # finds the most probable one return sum(freq * best_split_cost(w, vocab) for w, freq in corpus.items()) base = loss(probs) for tok in candidates: delta[tok] = loss(probs - tok) - base # ← the criterion # drop the cheapest 10–20%, then repeat drop(sorted(delta)[:k])

What each candidate costs to delete

TOKENLOSS +

ug0.00pug re-splits as pu·g, same cost

pu0.00p·u is just as good

un0.00pun and bun are whole words already

hu0.00never used in any best split

gs0.00hugs is a whole word already

bu0.00never used

bun10.59keep

hugs13.56keep

hug24.25keep

pun27.60keep

A token is worth keeping only if the words containing it get worse without it. Deleting ug costs exactly nothing, because “pug” simply re-splits as pu·g at an identical score. Deleting pun costs 27.6, because there is no comparable way left to write it.

Byte-pair encoding and WordPiece grow a vocabulary upward from characters. Unigram shrinks one downward from an oversized pool — and because it is a probability model rather than a merge procedure, a word keeps a distribution over splits rather than one answer. The likeliest is used at inference; sampling the alternatives during training acts as a regulariser.