Scaling the Agent Runtime
Why CPU autoscaling is the wrong signal for a 40-second agent run: in-flight-run scaling, queueing-theory capacity math, the derived timeout ladder, connection pooling, a five-rung degradation ladder, token-rate limiting, and a load-test recipe.
Last updated
After this section you can
- Pick an autoscaling signal that reflects agent load instead of CPU
- Size instances and concurrency with Little's law, and find the real ceiling
- Build a timeout ladder where every budget is derived from the one above it
- Prevent connection-pool exhaustion caused by holding resources across model calls
- Design a degradation ladder with the user-visible message at every rung
- Treat provider token-rate limits as the primary constraint and scale around them
- Run a load test for an agent and state what passing means
Scaling the Agent Runtime
A request that takes 40 seconds, makes 12 model calls, and spends 95% of its life waiting. Almost everything you know about scaling web services points the wrong way here.
Scaling from 10k to 1M Users covers the cost model and what the architecture looks like at each stage. This section covers the runtime: which signal to autoscale on, how much concurrency one instance can really hold, where the timeouts go, what to shed first when you cannot keep up, and why the wall you eventually hit is your model provider rather than your compute bill. As in SD-30, every part is layered: plain English first, then the mechanics with code, then a staff-level callout.
Traffic multiplies and things break in a predictable order. Nobody hits all six at once, and almost nobody hits them out of order.