Rate Limiting & Cost
Rate limiting and cost management for LLM APIs: token budgets, per-user quotas, and cost optimization strategies.
Last updated
After this section you can
- Explain why LLM rate limiting needs three currencies — requests, tokens, dollars — and how providers enforce each
- Implement token bucket, leaky bucket, fixed window, and sliding window limiters and choose the right one for a given traffic shape
- Design a 4-layer enforcement stack with per-tenant quotas, hard cost ceilings, and model-level fallback
- Price the five cost levers against a canonical workload and defend the savings in an interview
Rate Limiting & Cost
Requests, tokens, and dollars are three separate currencies — any one of them can veto a call. Four algorithms, a four-layer enforcement stack, and the five levers that shrink what you need to limit.
A classic API gateway counts requests per second and calls it done. An LLM gateway cannot: one request can carry 50 tokens or 500,000, cost $0.0003 or $3, and the provider meters you on all three axes at once, per model. So rate limiting for LLM systems is really budget enforcement in three currencies, with a fallback plan for every “no.” This section builds the real thing: the four classic algorithms with running code, a four-layer stack that enforces per-tenant quotas, and the cost math that decides how much limiting you even need.