The hidden language tax

Open figure

"8,192 tokens" reads like a fixed amount of room. It isn't. That same window holds roughly 7,600 Hindi words under one tokenizer and about 1,600 under another - same model, same advertised context length, four to five times the difference in how much language fits.

Nothing about the model explains that. Not the parameter count, not the attention mechanism, not the training budget. The whole difference is produced by one small file you can download and read, written by decisions taken before training began.

This is a cost that does not appear on any dashboard, and it is diagnosable. The way to learn to read it is to build the thing that causes it.

Designing a tokenizer#

A model does arithmetic on numbers, not on text. So we need a conversion layer between model and text, which we do by mapping each unique unit of text in the vocabulary to a unique integer - and that is what a tokeniser is. Every language model begins with a reversible map: a string becomes a list of integers, and that list can become the string again.

Open figure

If we were to design it, we would be optimising two things that pull against each other. We want the integer sequence to be short, because sequence length is what attention cost and context capacity are priced in. And we want the vocabulary to be small, because every entry in it costs a row in an embedding table; comprehensive, because it has to be able to represent anything we might be handed; and closed, because we can never emit an integer we did not plan for.

It helps to think of this as inventing a writing system from scratch. We choose how many distinct symbols exist, and how they combine into words. Every natural language has already made that choice differently - an idea that takes one short word in one language can take a long compound in another - and it is exactly that compactness we are trying to design for.

But the two costs are not the same bill. A shorter sequence is cheaper at the attention layer, which scales with sequence length. A larger vocabulary is expensive somewhere else entirely: it costs parameters in the embedding tables, and compute in the output projection, which multiplies against a matrix as wide as the vocabulary itself. Every design below pays one of those to reduce the other.

To compare designs we need a number. The conventional one is fertility, the average number of tokens a word turns into:

fertility = tokens ÷ words

Lower is better: each token is carrying more language. It is the unit the industry quotes, so it is the one this article leads with - but it has a weakness worth naming now, because it will matter later. Fertility depends on what counts as a word, and languages disagree about that. Indic morphology packs more meaning into one whitespace-delimited word than English does, which quietly flatters those languages.

So the second unit is bytes per token - how much raw text each integer carries:

bytes per token = UTF-8 bytes ÷ tokens

It needs no notion of a word at all, which makes it unarguable, and it is the compression ratio in the literal sense. Where the two units disagree, the second one is telling the truth.

Let us try the obvious designs and watch where each one breaks.

Attempt 1: one token per word#

Split on whitespace and punctuation, then number the results. Fertility is 1.0 by construction, which is as good as compression can get.

The vocabulary is the problem. It has no bound. Every new product name, every typo, every inflected form and code identifier is either another entry or it is nothing - and "nothing" means a single catch-all unknown token. That token is not merely unhelpful. It is lossy in a way nothing downstream can repair: two different unknown words become the same integer, and the model has no route back to what was written.

Good compression, unrecoverable edge cases.

Attempt 2: one token per character#

Now the vocabulary is bounded by the character set, and nothing is ever unknown. This fixes exactly the thing that broke.

But the sequence got long, and a character carries far less meaning than a word. The model now spends layers reassembling words before it can begin reasoning with them, and attention pays for every extra position.

There is a subtler problem. Unicode has roughly 150,000 code points. Almost all of them are rare, and each one still occupies a row in the embedding table whether or not it ever appears.

Attempt 3: one token per byte#

Encode the text as UTF-8 and use the raw bytes. The vocabulary is now exactly 256, which is as closed as a vocabulary can be, and every string that can exist is representable.

This is the most robust design available and the worst compressor. Roughly one byte per token means the longest sequences of any option.

It also introduces the asymmetry this article is about, and it is worth being precise about where it comes from. UTF-8 is variable width. Latin characters take one byte; Devanagari, Bengali, Tamil and Kannada characters take three. Before any clever design decision is made, Indic text is already three times longer than equivalent English text.

What the three attempts tell us#

Each attempt fixed its predecessor's flaw and inherited a new one. Laid side by side, the pattern is clear: we are trading vocabulary size against sequence length, and picking any fixed rule forces us to give up one of them.

Open figure

So the design has to satisfy both ends at once. It must be able to represent a word it has never seen before - which Attempt 1 could not - without stretching the sequence so far that meaning is spread too thin, which is what Attempt 2 did.

No fixed rule reaches that, because a fixed rule has to decide the units in advance, before it has seen a single sentence. So we stop deciding, and we analyse the data instead: keep frequent words whole, break rare ones into pieces, and let a corpus tell us which is which. That is subword tokenization, and it converts our design problem into a single sharper question:

Which pieces should the vocabulary contain, and by what criterion do we choose them?

Every algorithm in use today is an answer to that question. They differ in the criterion, and the criterion is the design.

Criterion 1: merge what is frequent#

Byte-pair encoding was not invented for language models at all. It was a data-compression algorithm, published by Philip Gage in 1994 to squeeze repeated byte sequences out of a file.[8] Sennrich, Haddow and Birch repurposed it for machine translation in 2015,[6] and it is now the default nearly everywhere. The lineage is worth holding onto: a tokenizer is a compression scheme, and fertility is simply its compression ratio measured in words.

The procedure is short. Start from single characters. Count every adjacent pair in the corpus. Merge the most frequent pair into one new entry. Repeat until the vocabulary reaches its target size.

The vocabulary is therefore just the base entries plus the number of merges performed - GPT's 40,478 was 478 base characters plus 40,000 merges. Common sequences like "the", "ing" and "tion" get absorbed early; rare words stay in pieces.

Open figure

Criterion 2: merge what is surprising#

WordPiece, introduced by Schuster and Nakajima in 2012 for Japanese and Korean voice search, later used by BERT.[6] Structurally it is byte-pair encoding. One line differs: rather than the most frequent pair, it takes the pair that most increases the likelihood of the corpus, scoring each candidate as

freq(ab) ÷ ( freq(a) × freq(b) )

The denominator is the whole trick. If two pieces had nothing to do with one another, they would still end up side by side sometimes - roughly in proportion to how common each one is on its own. So the product freq(a) × freq(b) is how often the pair would occur by coincidence. Dividing by it stops the algorithm asking "how often do these two appear together?" and makes it ask "do they appear together more often than coincidence explains?"

The effect is to punish pairs whose halves are individually popular. In the five-word corpus above, "u" appears 36 times and "g" 20 times, and they sit together 20 times - which scores 20 ÷ (36 × 20), or 0.028. The pair "g s" occurs only five times. But "s" is rare, and it never appears anywhere except directly after "g", so it scores 5 ÷ (20 × 5) = 0.050, and it wins.

Hugging Face's course puts the intuition well: WordPiece will not merge "un" and "able" however often that pair occurs, because both halves turn up in plenty of other words - but "hu" and "gging" merge early, because neither half is common on its own.[7]

That ratio has a name - it is pointwise mutual information, unnormalised - but the intuition carries it without the term. Byte-pair encoding asks which pair it sees most often. WordPiece asks which pair surprises it most. Same corpus, same starting characters, different vocabulary - from one changed line.

Open figure

Criterion 3: start too big and prune#

Unigram, from Kudo in 2018 and used by T5, never merges anything.[6] It runs in the opposite direction entirely: begin with a vocabulary that is deliberately far too large, then delete from it until it is the right size.

It can work that way because it is a probability model rather than a procedure. Four steps:

• Start oversized. Take every base character plus a large pool of frequent substrings - many more candidates than the finished vocabulary will hold.
• Give every candidate a probability: its frequency divided by the total frequency of all candidates. If "ug" occurs 20 times in a corpus of 210 token occurrences, its probability is 20 ÷ 210.
• Score each word by its best split. A word can be cut many ways, and each way has a probability - the product of its pieces. The model keeps the most probable one, found with a Viterbi pass. Adding up −log of those probabilities across every word gives the corpus a single loss.
• Delete whatever is not earning its place. For each candidate, ask how much that loss would rise if it were removed. A token whose words can be split almost as well without it is doing no work, so it goes - typically the lowest 10 to 20 per cent at a time.

Those last two steps repeat until the vocabulary is the right size. Nothing is ever merged - the vocabulary only ever gets smaller.[7]

Two consequences are worth carrying forward. Because the model is defined over probabilities of whole segmentations, a word does not have one correct split - it has a distribution over splits, and the likeliest is used at inference. And because the alternatives genuinely exist, they can be sampled during training instead of always taking the best one, which acts as a regulariser: the model sees the same word cut several different ways.

Open figure

Which is the real distinction between the three: the first two differ only in which pair they reach for, while Unigram is not reaching at all.

A change of assumption: SentencePiece#

Kudo and Richardson, 2018.[6] This is not a fourth criterion but a reframing of what the criteria operate on.

Byte-pair encoding and WordPiece both assume whitespace separates words, because both are applied after something has already split the text on it. SentencePiece drops the assumption: treat the input as one raw stream, put the space character itself into the vocabulary as the symbol ▁, then run byte-pair encoding or Unigram over that.

The effect is easiest to see by running one. Under Sarvam-1's tokenizer, which is built this way, "the cat" becomes two tokens - ▁the and ▁cat - with the space carried inside each token rather than thrown away between them. Give it a double space and it produces ▁the, ▁▁, cat: the extra space becomes a token of its own, and decoding returns the original string exactly, both spaces intact.

That is the real gain. Whitespace stops being a delimiter the tokenizer consumes and becomes data it carries, so the map stays reversible on any input - leading spaces, double spaces, none at all.

It was built for Chinese and Japanese, which do not delimit words with spaces. It matters as much for Indic scripts, where whitespace is a weak guide to where morphemes actually begin and end - and it is why a tokenizer in this lineage can be pointed at a language whose writing system it was never shown.

Choosing where to start: byte-level base vocabularies#

Return to the choice between Attempt 2 and Attempt 3, because learned merges let us have it both ways.

If the base vocabulary is every Unicode code point, we inherit the 150,000-entry problem. If it is the 256 byte values, then every possible string is reachable, the vocabulary is closed, and the unknown token stops existing as a concept - while learned merges recover the compression that raw bytes gave up.

GPT-2's vocabulary is the worked example, and the arithmetic is the entire design:

50,257 = 256 byte values + 50,000 learned merges + 1 end-of-text marker

This combination - byte-level base, learned merges - is now the dominant choice. Llama, Gemma and Qwen all use it.[6]

The decision that gets overlooked: pre-tokenization#

Before any merging happens, the text is cut into chunks: on whitespace, on punctuation, at digit boundaries, by a regular expression. Merges can never cross those cuts.

This makes pre-tokenization quietly decisive, because it does not shape the outcome so much as bound it. A pre-tokenizer that splits on whitespace can never learn a token spanning two words, however often the phrase occurs.

Numbers are the clearest case, and the tokenizers here disagree completely about them. Hand each of them 1234567: the GPT-4 and GPT-4o encodings return three tokens, grouping digits left to right as 123, 456, 7. Sarvam-1 returns every digit separately.

Open figure

Grouping is the better compressor - a long number costs a third of the positions. But the grouping shifts with length. Drop the leading digit and 234567 shares not one token with 1234567, so a digit never means the same thing in the same place. Splitting every digit gives up that compression entirely and buys back place value; several model families now do exactly this, and arithmetic is the reason.

Which means the behaviour was settled before a single merge was learned.

The decision that produces the tax: whose text gets counted#

Now look back at the three criteria. Each one is defined over a corpus. Byte-pair encoding merges what is frequent in that corpus. WordPiece measures surprise relative to that corpus. Unigram prunes what is cheap given that corpus.

Which means choosing the data mix is not preparation for designing the vocabulary. It is designing the vocabulary.

And merges are a finite budget. In an English-heavy corpus, English pairs are the frequent ones, so English words are absorbed into whole-word tokens and the budget is spent there. Text in other scripts is left to be assembled out of whatever fragments happen to remain.

The inequality is not policy, malice, or oversight. It is corpus statistics becoming architecture, produced faithfully by a criterion that was never asked to care about coverage.

The result is more severe than "more tokens per word" suggests. Consider the Hindi word सरकार, "government" - five characters, fifteen UTF-8 bytes. Under a tokenizer built for Indic scripts it is a single token, id 7243. Under the current OpenAI encoding it is two. Under the GPT-4-era encoding it is five, and the last two are the ones to look at: one token holds a complete vowel sign plus the first two bytes of the following character, and the next holds the single remaining byte, 0xB0.

The final character is split across two tokens, mid-codepoint. The model is not receiving a fragmented word. It is receiving fragments of a single character, and must reassemble a letter before it can start on a word.

How large should the vocabulary be?#

The last knob, and the only one that has moved recently - always in the same direction.

Llama 2 shipped a 32,000-entry SentencePiece vocabulary. Llama 3 replaced it with 128,256 entries and switched to a byte-level, tiktoken-style tokenizer at the same time. OpenAI's own encodings went the same way: cl100k_base holds 100,277 entries, and o200k_base that succeeded it holds 200,019. In four years the going rate roughly sextupled.

Both of those moves were made for the same stated reason, and it is the reason this article has been circling: efficiency on code and on non-English text. A bigger vocabulary is the most direct way to buy back the fertility that an English-heavy merge budget takes away - it simply leaves room for other scripts to earn whole-word tokens too.

The underlying trade has not changed: a larger vocabulary buys shorter sequences and pays for them in the embedding matrices. What changed is the price the field is willing to pay.

Measured across four thousand words of real prose per language, the spread these decisions produce is enormous. Indic fertility runs from 1.65 to 15.11 depending on nothing but which tokenizer is used.

Open figure

That spread is the hidden language tax. It is not concealed by anyone; it is hidden by units. You write words, you are billed tokens, and the unit is where the difference disappears.

Open figure

What the fragmentation costs#

Three consequences, all following from sequence length rather than from anything linguistic.

Context. A window is denominated in tokens, so its capacity in language moves with fertility. The honest metric is usable content per window, not window size.

Training-token counts stop being comparable. CS336's first-order estimate for training compute is C ≈ 6ND, where N is parameters and D is training tokens.[4] The same document at fertility 6 consumes roughly 4.3 times the budget it does at fertility 1.4, so "trained on N trillion tokens" describes different quantities of actual text for different tokenizers.

Serving. The KV cache is a fixed cost per cached token, so fragmentation converts directly into memory.

Open figure

Per-token pricing then passes that same multiplier to whoever sent the request - which is why cost per word, or per completed task, is a more informative unit than cost per token for anything multilingual.

Reading the decisions off a real tokenizer#

Every decision above was made by someone and written down, which means a shipped tokenizer can be read back.

Sarvam-1 is a 2-billion-parameter model for ten Indian languages plus English.[1] Taking its tokenizer file and going through the same list of decisions in order:

• Criterion: byte-pair encoding - frequency, not Unigram's pruning or WordPiece's surprise score.
• Framing: a Metaspace pre-tokenizer using ▁, with splitting disabled. The SentencePiece lineage, explicitly declining to impose a whitespace-first split.
• Base vocabulary: byte fallback enabled, so no input is ever unrepresentable.
• Normalisation: none configured.

The design is legible without weights, benchmarks, or a GPU. One file answers every question we raised while building our own.

The vocabulary is a budget, and it decomposes exactly. 68,096 entries: 63,741 learned tokens, 256 raw byte values for the fallback path, 4,096 slots reserved for future expansion, and 3 special tokens. The reserved block is deliberate forward-compatibility - room to add entries later without disturbing existing ids.

How that budget was divided is the design. Classifying every learned entry by script gives roughly Latin 26%, Devanagari 18%, then Telugu, Kannada, Bengali, Gujarati, Tamil and Gurmukhi at 7 to 9% each, Malayalam 5% and Oriya 3%.

Open figure

The shares track the corpus: Sarvam states Hindi is roughly 20% of their Indic data, and English tokens were added in near-equal measure for training.[1] Someone divided a finite budget across eleven scripts, and every fertility number in this article is the arithmetic consequence of how they divided it.

Specialisation has a measurable price. Because the budget went to Indic scripts and Latin, everything else falls through the byte-fallback path. Cyrillic, Chinese, Arabic and emoji cost two to four tokens per character here - the Russian word Москва, six characters, becomes thirteen tokens. That is not a defect. It is the same finite budget seen from the other side: strong on ten languages and weak everywhere else, by construction rather than by accident.

And low fertility is paid for in parameters. Every vocabulary entry needs an embedding row:

P = V × d × n

Sarvam-1's configuration leaves input and output embeddings untied, so n is 2 and the vocabulary is charged once going in and once coming out.[3]

Open figure

That comes to about 278.9 million parameters, roughly 11% of the model's approximately 2.52 billion, spent entirely on representation. The trade we hit in Attempt 3 never disappears. It moves.

The vendor's number depends on the corpus, and so does everyone else's. Sarvam reports fertility of 1.4 to 2.1 across its supported languages.[1] On a single matched paragraph the tokenizer measures 1.40 - the exact bottom of that range, a tidy corroboration. On four thousand words of real prose per language it measures 2.26, just outside it.

Neither number is wrong. They are different corpora, and that is the point: a fertility figure without a corpus attached is not a fact about a tokenizer. The vendor's range may well hold on whatever text they measured; there is no way to check, because the text is not published.

Two more things the corpus shows that the paragraph hid. Sarvam's headline comparison is 4 to 8 tokens per word for existing multilingual models,[1] and the GPT-4-era encoding is worse than that - 10.64 on average, 15.11 on Kannada. But the current OpenAI encoding averages 2.84 on the same text, so Sarvam's real lead over a current frontier tokenizer is about 1.3×, not 4 to 8×. The tax is severe, and the severe version of it belongs to a superseded generation.

And the specialisation shows up in the other direction too. On English, Sarvam manages 1.69 tokens per word against o200k's 1.35, and packs 3.68 bytes into a token against o200k's 4.60. A vocabulary that spends its budget on Indic scripts is measurably worse at English - which is not a flaw, it is the budget.

Which is why none of it can be changed afterwards. Token ids are what the embedding table spent an entire training run learning. Change the tokenizer and every id changes meaning and every training example re-segments. A vocabulary can be extended - new entries, new rows, continued pretraining - but the additions arrive without the representations the original entries earned. So it is settled before the first training step, which is what makes it structural.

What this all costs, and who pays it#

A tokenizer is where a model's language priorities stop being a claim and become numbers. A handful of decisions, one vocabulary file, and the cost of carrying your language is fixed before training starts. Sarvam's contribution is not that they built a tokenizer for Indian languages - it is that they treated tokenizer design as systems design, with a cost they published and anyone can audit.

If you ship in a language that isn't English, what fertility does your model actually give you?

Sources#

[1] Sarvam AI - "Sarvam 1: The first Indian language LLM" - https://www.sarvam.ai/blogs/sarvam-1

[2] Sarvam AI - "Open-Sourcing Sarvam 30B and 105B" - https://www.sarvam.ai/blogs/sarvam-30b-105b

[3] Sarvam-1 model configuration and tokenizer files - https://huggingface.co/sarvamai/sarvam-1

[4] Stanford CS336, Lecture 1 - "Overview, tokenization", Percy Liang - https://stanford-cs336.github.io/spring2025/

[5] Sarvam-1 model card - https://huggingface.co/sarvamai/sarvam-1

[6] Hugging Face Transformers - "Tokenization algorithms" - https://huggingface.co/docs/transformers/en/tokenizer_summary

[8] Philip Gage - "A New Algorithm for Data Compression", C Users Journal 12(2), February 1994 - the original compression algorithm later repurposed as BPE

[7] Hugging Face LLM Course, Chapter 6 - WordPiece and Unigram tokenization, with full worked examples - https://huggingface.co/learn/llm-course/chapter6/6

Fertility and compression measured by running the Sarvam-1 tokenizer alongside OpenAI's tiktoken encodings over roughly four thousand words of Wikipedia prose per language. One domain, measured openly - indicative of magnitude, not a benchmark ranking. Measurement scripts, the sampled corpus and the full claim ledger are published with this article. Diagrams: @sushant_p18