"A chatbot is a single LLM call — input in, text out, stateless. An agent is an LLM inside a loop. The loop gives it tools, memory, and the ability to take actions in the world. The agent decides what to do next based on observations. The key difference is autonomy — an agent can reason, act, observe, and iterate until a task is complete. The 'agent' is actually the while-loop your code runs around the LLM, not the LLM itself."
Q1
What's the difference between a chatbot and an agent?
AgentsArchitectureTool Use
Commonly asked at Anthropic · OpenAI · Sierra
How to answer — what you would actually say
The deep dive — diagrams, tradeoff tables, and the follow-up trap
What the loop actually looks like
Same model in both rows. The only difference is what your code does with the model's output — return it, or feed it back in.
Chatbot vs agent — the difference is the loop, not the model
Side-by-side, the way an interviewer wants it
| Dimension | Chatbot | Agent |
|---|---|---|
| Control flow | Fixed by your code: one call, text out | Chosen by the model each iteration — it decides the next step |
| State | Stateless (or raw chat history re-sent) | Working memory carried across steps |
| Tools | None, or retrieval only | Calls APIs, databases, code — can write to the world |
| Latency / cost | ~600ms, ~$0.001 per question | 3–8 loop iterations, seconds, cents per task |
| Failure mode | Wrong answer — user reads it and moves on | Wrong action — refund issued, email sent. Needs guardrails |
| Ship it when… | Q&A, FAQ deflection, drafting | Multi-step tasks with side effects |
REAL SYSTEM
A support copilot runs both, tiered. The FAQ tier is a chatbot — one call, ~400 tokens, ~600ms, ~$0.001/question — it deflects “what’s your refund policy?”. The resolution tier is an agent: for “refund my last order” it loops lookup_order → check_policy → issue_refund → send_confirmation — ~5 iterations, ~6K tokens, ~8s, ~$0.03 — but it closes the ticket end-to-end. Same base model; the wrapper decides which one it is. Routing between the two tiers is where most of the money is saved.
FOLLOW-UP TRAP
“So is RAG an agent?” — No. Vanilla RAG is a fixed pipeline: retrieve → stuff → generate, same path every time. It becomes agentic when the model decides whether to retrieve, what to retrieve, and when to stop. Quick litmus test: if your control flow lives in Python if statements, it’s a pipeline; if it lives in the model’s next-token distribution, it’s an agent.