One regular expression, decided before any learning
— the same three numbers, cut by two different pre-tokenizers
Group digits in threes
GPT-4 and GPT-4o encodings
1234567
1234567
234567
234567
34567
34567
0 tokens in common
Drop a single leading digit and the grouping shifts underneath everything. A “4” sitting in the
thousands column is a different token depending on how long the number happens to be, so
place value never arrives intact.
Split every digit
Sarvam-1, and several other current models
1234567
▁1234567
234567
▁234567
34567
▁34567
every digit stable
A “4” is the same token wherever it appears. The cost is real — eight positions instead of three
for one number — and it is paid deliberately, because arithmetic needs place value more than it
needs compression.
Neither is a bug. They are opposite answers to the same question, and notice where the question
was settled: not in the merge criterion, not in the vocabulary size, but in a
regular expression that ran before a single merge was learned. Much of what
gets blamed on a tokenizer is really a property of its pre-tokenizer.
Token ids measured on tiktoken o200k_base and the Sarvam-1 tokenizer·@sushant_p18