# identical to byte-pair encoding, except
# for how "best" is chosen
pairs = pair_counts(splits)
syms = symbol_counts(splits) # each half alone
# syms[a] * syms[b] is how often the pair
# would land together by coincidence, so
# dividing by it measures surprise, not
# frequency
scores = {(a, b): n / (syms[a] * syms[b])
for (a, b), n in pairs.items()}
best = max(scores, key=scores.get)
apply_merge(splits, best)
Every candidate pair, scored on the same corpus
PAIRSCORE=
g s5 ÷ (20 × 5)
0.0500wins
u g20 ÷ (36 × 20)
0.0278
p u17 ÷ (17 × 36)
0.0278
u n16 ÷ (36 × 16)
0.0278
h u15 ÷ (15 × 36)
0.0278
b u4 ÷ (4 × 36)
0.0278
Every pair containing u scores identically, because “u” appears in all
thirty-six letter positions — merging it absorbs a character that turns up everywhere, which tells
the model almost nothing. “s” occurs five times and never anywhere except after “g”. That
pair is perfectly predictive, so it wins on a quarter of the raw frequency.