1:1 mentoring with Big Tech AI engineers
AgenticPrep.aiAgenticPrep.ai
Interview PlanRoadmapLLM & AgenticRAGMCPSystem DesignInterview PrepBlogDiscussAI System Design Studio
Cost & OptimizationMedium

LLM Inference Cost vs Latency Tradeoffs

You are building an AI-powered product and must choose between a premium model and an efficient model. How would you decide?

Cost OptimizationModel RoutingLatencyProduction
04

Cost vs Latency

Separate the wait from the bill, price a task per completed request, then choose an operating point.

Key idea

Cost and latency are two meters on one request. Free levers move one meter or both without changing quality; trade-offs buy the rest with quality. Read the meter your service level names, and pull the free levers first.

Every lever moves the wait, the bill, or both
ONE REQUEST, LEFT TO RIGHT end to end = time to first token + output time time to first token billed as input · 1x billed as output · 5x, thinking included network and queue prefill input tokens thinking when thinking is on decode one token after another Cache hit prefill shorter, reads 0.05x wait ↓ · bill ↓ Lower effort less thinking and output wait ↓ · bill ↓ Fast mode decode up to 2.5x faster wait ↓ · bill ↑ 2x Batch, for work nobody waits on: the whole request leaves the clock (most within an hour) · wait ↑ · bill ↓ 0.5x

Blue input, amber thinking, green output, teal levers. Cache reads: 0.05x on Opus 5.5. Pans on a phone.

Next

Preventing Private Code Leakage in Coding Agents