Model Routing
Tiered model routing: route queries to the right model (GPT-4, Claude, Haiku) based on complexity, cost, and latency requirements.
Last updated
After this section you can
- Explain the cost/latency/quality triangle with real per-MTok prices for three model tiers, and compute the monthly bill for a given traffic split (60/25/15) against all-Sonnet and all-Opus baselines
- Choose between rules, a small classifier model, and embedding-similarity routing for a given workload, defending the choice with latency, cost, and accuracy numbers
- Design a three-level fallback chain — retry with exponential backoff and jitter, fail over per error type, degrade gracefully — and explain how the misclassification asymmetry should bias the classifier's operating point
Model Routing
Send every query to the cheapest model that can actually answer it — and know exactly what happens when that model is down, throttled, or wrong.
One model for every query is the most expensive architecture there is, and the slowest to recover when a provider has a bad day. Model routing puts a cheap decision-maker in front of your model fleet: easy questions pay easy prices, hard questions get the firepower they need. This assumes the L0 foundations — System Design 101 and the first agentic system build-along — and picks up where Semantic Caching left off: the cache routes around the model, the router picks which model answers the misses.
Model routing is per-request selection of which model answers a query, made by a classifier (any cheap mechanism that predicts how hard the query is). A fallback chain is the ordered list of what to try when the chosen model fails: retry, fail over to a second model, or degrade to a static answer. Routing optimizes the happy path; the fallback chain owns the unhappy path.