1:1 mentoring with Big Tech AI engineers
AgenticPrep.ai
Interview PlanRoadmapLLM & AgenticRAGMCPSystem DesignInterview PrepBlogDiscussAI System Design Studio
Q9Premium

Latency SLA is 2 seconds but your agent needs 3 tool calls. How do you meet it?

Tradeoffs & Scenarios

LatencyOptimizationCachingModel Routing

Commonly asked at Google · Stripe · Sierra

How to answer — what you would actually say

"Four levers:

  • (1)Parallel tool calls — if tools are independent, call them simultaneously. 3 sequential x 500ms = 1.5s → 3 parallel = 500ms.
  • (2)Semantic cache — if this query was asked before, serve from cache in <100ms.
  • (3)Model tiering — use Flash/Haiku for the routing step, only escalate to Pro for the final synthesis. Flash is 5-10x faster.
  • (4)Streaming — start sending tokens to the user while the last tool call is still running. Perceived latency drops dramatically."

The deep dive — diagrams, tradeoff tables, and the follow-up trap

Loading…
Previous

When would you not use an agent framework and write the loop yourself?

Next

The customer wants the agent to send emails automatically. You're worried about blast radius. How do you handle it?