How to Answer
"If the task is single-turn retrieval + generation — user asks a question, you find the answer in docs — RAG is cheaper, faster, and more predictable. I'd reach for agents only when:
- (1)the task requires multiple steps
- (2)it needs tool use (write operations, calculations, API calls), or
- (3)the solution path is not known upfront and requires reasoning. An agent adds 3-10x the cost and latency of RAG. The tradeoff is autonomy vs predictability."
The three gates
Ask them in order and stop at the first yes. Most “we need an agent” requests fail all three — they are a retrieval problem wearing a loop.
What you give up when you add the loop
| Dimension | RAG pipeline | Agent |
|---|---|---|
| Control flow | Fixed in your code: retrieve → stuff → generate | Chosen by the model at every step |
| Latency | One round trip, p50 ≈ 0.8s | 3–8 round trips, seconds |
| Cost | 1× baseline | 3–10× — history is re-sent every iteration |
| Reproducibility | Same input, same path, every time | Path varies per run — you debug traces, not code |
| Failure mode | Wrong answer; the user reads it and moves on | Wrong action; the refund is already issued |
| Eval effort | Retrieval hit rate + answer accuracy | Outcome and trajectory scoring, per task type |
The middle ground most teams skip
Between the two there is a pipeline that keeps its fixed shape but lets the model make one bounded decision:
- Conditional retrieval — the model decides whether to search before answering. One extra call, no loop.
- Query rewriting — one rewrite pass before retrieval; fixes most “retrieval missed it” complaints for ~$0.0002.
- Bounded re-query — if the top chunk scores below threshold, retrieve once more with a different query. Hard cap of two.
- Read-only tools — a loop with lookups but no writes. You get multi-step reasoning with none of the blast radius.
An internal docs assistant handles ~60K questions/month. Classification showed 82% are single-hop lookups — those run as plain RAG at ~$0.002 and ~0.8s. The remaining 18% (“compare our Q2 and Q3 policy and tell me what changed for contractors”) route to an agent at ~$0.04 and ~7s. Running everything through the agent would have cost ~$2.4K/month for ~$430 of actual work, and pushed p50 latency from 0.8s to 7s for the 82% who never needed it.
“Retrieval keeps missing the answer — won’t an agent fix that?” — No. An agent that searches a broken index just pays 5× to fail more slowly, and now it fails non-deterministically so you cannot reproduce the bug. Fix chunking, add hybrid keyword + vector search, and measure recall@5 first. Reach for the loop when recall is good and the task still needs several steps — not when retrieval is bad.