Commonly asked at Anthropic · Google · Glean
How to answer — what you would actually say
"If the task is single-turn retrieval + generation — user asks a question, you find the answer in docs — RAG is cheaper, faster, and more predictable. I'd reach for agents only when:
The deep dive — diagrams, tradeoff tables, and the follow-up trap
Ask them in order and stop at the first yes. Most “we need an agent” requests fail all three — they are a retrieval problem wearing a loop.
| Dimension | RAG pipeline | Agent |
|---|---|---|
| Control flow | Fixed in your code: retrieve → stuff → generate | Chosen by the model at every step |
| Latency | One round trip, p50 ≈ 0.8s | 3–8 round trips, seconds |
| Cost | 1× baseline | 3–10× — history is re-sent every iteration |
| Reproducibility | Same input, same path, every time | Path varies per run — you debug traces, not code |
| Failure mode | Wrong answer; the user reads it and moves on | Wrong action; the refund is already issued |
| Eval effort | Retrieval hit rate + answer accuracy | Outcome and trajectory scoring, per task type |
Between the two there is a pipeline that keeps its fixed shape but lets the model make one bounded decision:
An internal docs assistant handles ~60K questions/month. Classification showed 82% are single-hop lookups — those run as plain RAG at ~$0.002 and ~0.8s. The remaining 18% (“compare our Q2 and Q3 policy and tell me what changed for contractors”) route to an agent at ~$0.04 and ~7s. Running everything through the agent would have cost ~$2.4K/month for ~$430 of actual work, and pushed p50 latency from 0.8s to 7s for the 82% who never needed it.
“Retrieval keeps missing the answer — won’t an agent fix that?” — No. An agent that searches a broken index just pays 5× to fail more slowly, and now it fails non-deterministically so you cannot reproduce the bug. Fix chunking, add hybrid keyword + vector search, and measure recall@5 first. Reach for the loop when recall is good and the task still needs several steps — not when retrieval is bad.
