How to Answer
"A chatbot is a single LLM call — input in, text out, stateless. An agent is an LLM inside a loop. The loop gives it tools, memory, and the ability to take actions in the world. The agent decides what to do next based on observations. The key difference is autonomy — an agent can reason, act, observe, and iterate until a task is complete. The 'agent' is actually the while-loop your code runs around the LLM, not the LLM itself."
What the loop actually looks like
Same model in both rows. The only difference is what your code does with the model's output — return it, or feed it back in.
Side-by-side, the way an interviewer wants it
| Dimension | Chatbot | Agent |
|---|---|---|
| Control flow | Fixed by your code: one call, text out | Chosen by the model each iteration — it decides the next step |
| State | Stateless (or raw chat history re-sent) | Working memory carried across steps |
| Tools | None, or retrieval only | Calls APIs, databases, code — can write to the world |
| Latency / cost | ~600ms, ~$0.001 per question | 3–8 loop iterations, seconds, cents per task |
| Failure mode | Wrong answer — user reads it and moves on | Wrong action — refund issued, email sent. Needs guardrails |
| Ship it when… | Q&A, FAQ deflection, drafting | Multi-step tasks with side effects |
A support copilot runs both, tiered. The FAQ tier is a chatbot — one call, ~400 tokens, ~600ms, ~$0.001/question — it deflects “what’s your refund policy?”. The resolution tier is an agent: for “refund my last order” it loops lookup_order → check_policy → issue_refund → send_confirmation — ~5 iterations, ~6K tokens, ~8s, ~$0.03 — but it closes the ticket end-to-end. Same base model; the wrapper decides which one it is. Routing between the two tiers is where most of the money is saved.
“So is RAG an agent?” — No. Vanilla RAG is a fixed pipeline: retrieve → stuff → generate, same path every time. It becomes agentic when the model decides whether to retrieve, what to retrieve, and when to stop. Quick litmus test: if your control flow lives in Python if statements, it’s a pipeline; if it lives in the model’s next-token distribution, it’s an agent.