Tokens, Context Windows & Sampling
What a language model actually does on every call: split text into tokens, predict one next token at a time, fit everything into a fixed context window, and pick each token by sampling — with interactive temperature and context-budget explorers.
Last updated
Foundations9 min read
After this section you can
- Estimate the tokens, cost and latency of a request from its size
- Explain how sampling picks each token, what temperature and top-p do, and why the newest Claude models use effort instead
- Budget a context window across system prompt, tools, history, documents and the answer
01
Tokens, Context Windows & Sampling
Every call to a language model does the same thing: turn text into tokens, predict one next token, repeat. Learn that loop and pricing, latency, context limits and hallucination stop being mysterious.
Key idea
A language model does one job. Given the tokens so far, it scores every possible next token, picks one, appends it, and does it again. Chat, tool calls and agents are all built on that loop.
One step of generation. Every output token is one trip round this loop.