Design an LLM Gateway / Model Router & Inference Proxy

Design the internal gateway every product team calls instead of calling model providers directly: one place for keys, budgets, routing, fail-over and logs.

InfrastructureCost OptimizationReliabilityRouting

Key Requirements

Route requests to the right model tier by cost and difficulty
Per-team rate limits, budgets and cost attribution
Caching that cuts cost without serving wrong answers
Fail-over when a provider or model is down, without silent behaviour changes
Central logging, auditing and usage dashboards

Interviewer Follow-ups

Q1How do you decide which model to route a request to?
Q2How do you handle a provider outage without dropping requests?
Q3How do you prevent one team from exhausting the shared budget?

Design a Deep Research Agent

Design an Intelligent Document Processing & Workflow Agent