Inference Optimization
LLM inference optimization: batching, quantization, KV-cache, speculative decoding, and hardware selection.
Last updated
After this section you can
- Decompose any LLM latency complaint into TTFT vs steady-state decode, and explain why prefill is compute-bound while decode is memory-bandwidth-bound
- Choose the right optimization lever — KV/prefix caching, speculative decoding, continuous batching, quantization, or parallelism — for a given bottleneck, and name what it costs
- Compute the utilization-adjusted $/token of a self-hosted GPU and argue the self-host vs API break-even with real numbers
- Compare serving frameworks (vLLM, TGI, TensorRT-LLM, SGLang) against throughput, latency, and operational constraints
LLM Inference Optimization
Where the milliseconds and the money go: prefill vs decode, KV caches, speculative decoding, quantization, parallelism — and the self-host vs API break-even math.
Latency and cost are the two axes every LLM system gets graded on, and both are decided at inference time. This section takes apart where the milliseconds go — prefill vs decode, TTFT vs throughput — walks the levers that move them (KV caching, speculative decoding, continuous batching, quantization, parallelism), and ends where production decisions actually end: the break-even math between renting GPUs and paying per token.
Before the math: the territory in plain English
Everything in this section follows from one mechanical fact: an LLM never generates an answer — it generates one token, then starts over. A 250-token reply means running the model 250 times. The first of those runs is different in kind from the other 249, and that difference is the map for every optimization below.