Inference Optimization
The single-GPU arithmetic of LLM serving: prefill and decode as two clocks, the memory-bandwidth ceiling, KV cache sizing, batching, quantization and speculative decoding, and how to match a lever to the bottleneck.
Last updated
After this section you can
- Decompose a latency complaint into time to first token and per-token decode.
- Size the KV cache for a concurrency and context target.
- Choose the lever that matches the bottleneck, and say what it costs.
Inference Optimization
Read a slow reply as two clocks, size the KV cache for a concurrency target, and pick the lever that matches the bottleneck.
Decode reads every weight for every output token, so one user's speed is memory bandwidth divided by the bytes moved per token. Prefill sets time to first token; decode sets the rest. Each lever either moves fewer bytes per token or shares one weight read across more tokens.
Blue input, amber model, green output, purple KV state. Batch 1: attention reads the whole KV cache each step. Pans sideways on a phone.