Model Routing
Choosing a model tier per request: the tier table at current prices, the bill by mix, the operating point where a cheap wrong answer costs more than the price gap, the cascade's escalation break-even and the fail-over target.
Last updated
After this section you can
- Price a routed mix from one tier table
- Find the operating point and the cascade break-even
- Choose a fail-over target by the cause of the failure
08
Model Routing
Price a routed mix, find where a cheaper tier stops paying, and pick the fail-over target by cause.
Key idea
Spend the cheaper levers first: cache, then effort. Route down only while the extra wrong answers cost less than the price saved.
One router, four tiers: the first step up costs 20 times
Per call at 800 input and 250 output tokens, list price. Thinking left out. Pans sideways on a phone.