Scaling 10k to 1M
Scale AI systems from 10K to 1M users: caching, sharding, async processing, and infrastructure evolution strategies.
Last updated
Scaling 10k to 1M
At 10K users the model bill breaks, at 100K the provider rate limit breaks, at 1M the architecture itself breaks. The scaling journey, with every number derived on screen.
Every earlier section optimized one request; this one multiplies it by a million. The interview version is a single escalation — “now what if we have 1M users?” — and the wrong answer is one architecture drawn bigger. The right answer is a sequence of three different systems, each justified by what actually breaks at that scale, each priced with the same cost model, and each checked against the provider rate limit nobody mentions until it pages you. This section derives that model once, then walks 10K, 100K, and 1M users with the arithmetic on screen.
The per-query economics come from SD-8 · Model Routing (the 70/30 Haiku/Sonnet mix, the $1/$5 and $3/$15 prices) and the 40% hit rate from SD-7 · Semantic Caching. This section assumes both, scales them, and keeps the same canonical workload the rate-limiting section bills against — so the numbers agree everywhere in the guide.