# 4,000 words of real prose per language, # sampled from random Wikipedia articles corpus[lang] = fetch_random_articles(lang, n=4000) for lang, text in corpus.items(): words = len(text.split()) bytes = len(text.encode("utf-8")) # the same text through all three tokenizers n_sarvam = len(sarvam.encode(text).ids) n_o200k = len(o200k.encode(text)) n_cl100k = len(cl100k.encode(text)) fertility = n_tokens / words # needs "word" bytes_per_tok = bytes / n_tokens # needs nothing # Sarvam counts exclude the BOS token — including it # is the commonest way to get these numbers wrong

Why it is built this way

Real prose, not a written example

A hand-written sentence can be accidentally easy. Random encyclopedia articles bring names, numbers, loanwords and punctuation — the things that actually fragment.

The same text through every tokenizer

No tokenizer sees text the others did not. Any difference in the output is a property of the vocabulary, not of the sample.

Two units, because one of them is arguable

Fertility depends on what counts as a word, and Indic morphology packs more into one whitespace-delimited word than English does. Bytes per token has no such dependency.

WHAT THIS CANNOT TELL YOU

Wikipedia is one domain, and an unusually formal one. These figures do not transfer to chat logs, code, or transliterated text, and they are not a benchmark ranking — they are one corpus, measured openly, with the script published. That is the whole point: a fertility number without a corpus attached is not a fact about a tokenizer.

The measurement is the argument. Every fertility figure quoted in this article — including the vendor's own — belongs to some corpus, and most of the time nobody says which. Changing the corpus moved Sarvam's measured fertility from 1.40 to 2.26, which is the difference between comfortably inside its published range and just outside it.