1:1 Mentoring with Big Tech AI Engineers
Back
Infra · AI System Designsenior

LLM Gateway / Model Router

LLM Gateway / Model Router

Design an internal gateway every product team calls instead of hitting model providers directly.

Key Requirements

  • Capability-aware routing and/or cheap→escalate cascade
  • Provider failover (aware that swaps change behavior)
  • Per-team budgets, chargeback, rate limiting, fairness
  • Semantic/prompt caching with correctness bounds
  • Central tracing, cost attribution, circuit breakers

AI Review

0/5

Review me as:

Draw your design on the canvas before submitting.

Build your design, then submit for an AI-powered review with dimension scores, strengths, gaps, and actionable suggestions.



Comments (0)

Sign in to leave a comment

Loading comments...