What a large vocabulary costs in parameters — the counterweight to low fertility
Pemb=
V
vocabulary
68,096
×
d
hidden size
2,048
×
n
matrices
2
68,096 × 2,048 × 2 = 278,921,216
278.9M parameters, on representation alone
why n is 2, and not 1
The vocabulary is charged once on the way in and once on the way out — the input embedding table and the output projection are separate matrices.

Sarvam's write-up does not say whether they share weights. The shipped config does:

tie_word_embeddings: false

Untied. So two matrices of 139.5M each, not one.
Low fertility is bought, not free. Shorter sequences are paid for in embedding parameters, and at this vocabulary size the bill is about a ninth of the entire model. The trade that appeared in the very first design attempt never disappears — it only moves.
Read from the published Sarvam-1 config.json·@sushant_p18