Rate Limiting & Cost
How an LLM gateway enforces three currencies (requests, tokens, dollars) with four limiter algorithms, reserve-then-settle accounting, a four-layer stack, a 429 contract a client can act on, and per-tenant ceilings.
Last updated
After this section you can
- Explain why LLM limiting needs three currencies and three responses
- Choose a limiter and state what it remembers
- Design a 429 contract a client can act on
18
Rate Limiting & Cost
How to limit an LLM gateway in requests, tokens and dollars, and what each refusal should tell the caller to do next.
Key idea
Requests and tokens refill, so a no means wait or shrink. Dollars do not, so a no means stop or degrade. Check the cheapest meter first, and give every no a next step.
Check the cheapest meter first, and give every no a next step
Blue: the request. Purple: counters. Amber: a decision. Pans sideways on a phone.