Inference Optimization
LLM inference optimization: batching, quantization, KV-cache, speculative decoding, and hardware selection.
Last updated
LLM Inference Optimization
Where the milliseconds and the money go: prefill vs decode, KV caches, speculative decoding, quantization, parallelism — and the self-host vs API break-even math.
Latency and cost are the two axes every LLM system gets graded on, and both are decided at inference time. This section takes apart where the milliseconds go — prefill vs decode, TTFT vs throughput — walks the levers that move them (KV caching, speculative decoding, continuous batching, quantization, parallelism), and ends where production decisions actually end: the break-even math between renting GPUs and paying per token.
Before the math: the territory in plain English
Everything in this section follows from one mechanical fact: an LLM never generates an answer — it generates one token, then starts over. A 250-token reply means running the model 250 times. The first of those runs is different in kind from the other 249, and that difference is the map for every optimization below.
Related
More in System Design
System Design 101
FreeSystem design fundamentals for AI engineers: client-server, APIs, latency percentiles, caching, load balancing, databases, and queues — each explained from zero, then mapped to how LLM systems change it.
AI System Design Vocabulary
FreeThe 60-term plain-English glossary for AI system design: LLM basics, retrieval, agents, infrastructure, reliability, scaling, cost, and safety — with deep-dive links into every guide section.
Your First Agentic System
FreeBuild a support bot end to end: six iterations from one API call to a production-shaped architecture with retrieval, caching, model routing, guardrails, and observability — runnable code at every step.
The Paradigm Shift
FreeTraditional vs agentic system design: the 7 dimensions that transform, anatomy of an agentic system, control flow paradigms, failure modes, and when to go agentic.
Get full access to all 87+ sections with code examples, diagrams, and interactive animations.
Unlock Premium