The same three words, sliced four ways
— each row is what the model would actually receive
नई शिक्षा नीति
“new education policy” · 3 words · 14 characters · 38 UTF-8 bytes
नईशिक्षानीति
3 tokens
Shortest possible sequence — but any word never seen before becomes a
single unknown token, and that loss is permanent.
Character
vocabulary ~150,000
नई␣
शिक्
षा␣
नीति
14 tokens
Nothing is ever unknown — but one row of the embedding table is spent on
every code point in Unicode, most of which never appear.
e0a4a8e0a48820e0a4b6e0a4bfe0a495e0a58de0a4b7e0a4be20e0a4a8e0a580e0a4a4e0a4bf
38 tokens
The most closed vocabulary there is, and the longest sequence. Devanagari
takes three bytes per character where Latin takes one — the gap starts here, before any
design decision is made.
▁नई▁शिक्षा▁नीति
3 tokens
A bounded vocabulary and a short sequence — because these three pieces
were learned from a corpus rather than fixed by a rule. Which is the catch: whichever corpus
it learned from.
Every fixed rule buys one goal by giving up the other. Word tokenization has the shortest sequence
and an unbounded vocabulary; byte tokenization has the smallest vocabulary and the longest sequence.
Only the learned option gets both — and it pays for that by inheriting the
statistics of whatever text it was trained on.
Sequences produced with the Sarvam-1 tokenizer·@sushant_p18