The memory price of one cached token
— fixed by the architecture, paid once per token in the sequence
Mkv=
2
keys + values
2
·
L
layers
28
·
Hkv
kv heads
8
·
dhead
head dim
128
·
b
bytes / element
2 · bf16
2 · 28 · 8 · 128 · 2 = 114,688 bytes
112 KiB per token
the full 8,192-token window
896 MiB
one sequence, batch size 1
a 1,000-word Hindi prompt, Sarvam-1
153–230 MiB
1,400–2,100 tokens
the same prompt at 4–8 tokens per word
438–875 MiB
4,000–8,000 tokens — same sentences
Cache memory is charged per token, never per word. So fragmentation is not only a longer sequence —
it is a larger resident footprint for identical content, which is what decides
how many requests fit on one GPU.
Derived from the published Sarvam-1 config.json·@sushant_p18