1:1 mentoring with Big Tech AI engineers
System Design

Scaling the Agent Runtime

Why CPU autoscaling is the wrong signal for a 40-second agent run: in-flight-run scaling, queueing-theory capacity math, the derived timeout ladder, connection pooling, a five-rung degradation ladder, token-rate limiting, and a load-test recipe.

Last updated

Production30 min readFirst readScaling 10k to 1MDeploy Your First Agent

After this section you can

  • Pick an autoscaling signal that reflects agent load instead of CPU
  • Size instances and concurrency with Little's law, and find the real ceiling
  • Build a timeout ladder where every budget is derived from the one above it
  • Prevent connection-pool exhaustion caused by holding resources across model calls
  • Design a degradation ladder with the user-visible message at every rung
  • Treat provider token-rate limits as the primary constraint and scale around them
  • Run a load test for an agent and state what passing means
SD-33

Scaling the Agent Runtime

A request that takes 40 seconds, makes 12 model calls, and spends 95% of its life waiting. Almost everything you know about scaling web services points the wrong way here.

Scaling from 10k to 1M Users covers the cost model and what the architecture looks like at each stage. This section covers the runtime: which signal to autoscale on, how much concurrency one instance can really hold, where the timeouts go, what to shed first when you cannot keep up, and why the wall you eventually hit is your model provider rather than your compute bill. As in SD-30, every part is layered: plain English first, then the mechanics with code, then a staff-level callout.

THE ONE-SCREEN VERSION

Traffic multiplies and things break in a predictable order. Nobody hits all six at once, and almost nobody hits them out of order.

Six walls, in the order traffic makes you hit them
SIX WALLS — ALMOST NOBODY HITS THEM OUT OF ORDER1Autoscaling does not reactlatency triples while CPU sits at 12% and no pod appearsin-flight runs, not CPU2Request timeouts504s for runs that completed — and were billedone derived ladder3Database connectionstoo many clients, and unrelated features fail toonever hold one across a model call4Provider rate limitsbursts of 429s, and the retries make it worsetoken-rate budgeting + lanes5No backpressureeverything degrades for everyone, all at oncea ladder with a shed rung6Cost per run drifts upspend grows faster than volume doeswatch tokens per runTHE ORDER YOU HIT THEM IS THE ORDER OF THIS SECTION

Related

More in System Design

Get full access to all 74+ sections with code examples, diagrams, and interactive animations.

Unlock Premium