All tracks / Track I
Core LLM
What enters a language model, how tokenization shapes it, and where the costs begin.
2 of 2 chapters written.
Chapters in Core LLM
2/2 written- 01An LLM has never seen a letter in its lifeA visual tour from raw bytes to BPE: how one invisible translation layer sets model cost, context capacity, and many of the bugs that look like model failures.
- 02The hidden language taxThe same 8,192-token window holds roughly 7,600 Hindi words under one tokenizer and about 1,600 under another. The whole difference comes from one small file, written by decisions taken before training began.