Scaling the Agent Runtime
Autoscale on runs in flight, size slots with Little's law on the measured mean, keep connections out of model calls, and find the provider's walls.
Last updated
After this section you can
- Choose runs in flight as the autoscaling signal, and say why CPU misleads
- Size slots and the database pool with Little's law and a tail count
- Name which provider wall binds first, and what each lever does to it
36
Scaling the Agent Runtime
Why CPU misleads an autoscaler, how many slots a pod needs, and which provider limit binds first.
Key idea
An agent pod is full when its slots are full, even at low CPU. Scale on runs in flight, size slots with Little's law on the measured mean, and never hold a connection across a model call. Pods are elastic; the provider's token meter and spend cap are not.
A full pod can run at 8 percent CPU
Teal: the run slots, the thing to scale on. Grey: CPU, which adds no pod. Pans sideways on a phone.