Model Routing
Tiered model routing: route queries to the right model (GPT-4, Claude, Haiku) based on complexity, cost, and latency requirements.
Last updated
Model Routing
Send every query to the cheapest model that can actually answer it — and know exactly what happens when that model is down, throttled, or wrong.
One model for every query is the most expensive architecture there is, and the slowest to recover when a provider has a bad day. Model routing puts a cheap decision-maker in front of your model fleet: easy questions pay easy prices, hard questions get the firepower they need. This assumes the L0 foundations — System Design 101 and the first agentic system build-along — and picks up where Semantic Caching left off: the cache routes around the model, the router picks which model answers the misses.
Model routing is per-request selection of which model answers a query, made by a classifier (any cheap mechanism that predicts how hard the query is). A fallback chain is the ordered list of what to try when the chosen model fails: retry, fail over to a second model, or degrade to a static answer. Routing optimizes the happy path; the fallback chain owns the unhappy path.