How to Answer
“The users are waiting synchronously, so this is a latency-under-throughput problem, not a throughput problem — and that rules out the obvious design.
Static batching is wrong here: you wait for a batch to fill, run it, and everyone pays for the longest sequence in it. What I want is continuous batching — the scheduler admits new requests at each decoding step and evicts finished ones, so a short request never waits behind a long one.
Concretely: an admission queue, a scheduler that runs every decoding step, and a paged KV cache so capacity is bounded by memory pages rather than by the longest sequence anyone might send. The batch isn’t 100 because someone wrote 100 — it is whatever fits in KV memory at the context length I actually serve.
The knob that matters is how much queueing delay I’ll accept. I’d set a maximum wait, something like ten milliseconds, and a maximum batch, and fire on whichever comes first. And I’d shed load rather than let the queue grow, because an unbounded queue makes everyone’s latency bad instead of failing a few requests quickly.”