LLM Gateway / Model Router
Design an internal gateway every product team calls instead of hitting model providers directly.
Key Requirements
- 01Capability-aware routing and/or cheap→escalate cascade
- 02Provider failover (aware that swaps change behavior)
- 03Per-team budgets, chargeback, rate limiting, fairness
- 04Semantic/prompt caching with correctness bounds
- 05Central tracing, cost attribution, circuit breakers
Review me as:
Draw your design on the canvas before submitting.
Build your design, then submit for an AI-powered review with dimension scores, strengths, gaps, and actionable suggestions.
