Fundamentals & Architecture

"A chatbot is a single LLM call — input in, text out, stateless. An agent is an LLM inside a loop. The loop gives it tools, memory, and the ability to take actions in the world. The agent decides what to do next based on observations. The key difference is autonomy — an agent can reason, act, observe, and iterate until a task is complete. The 'agent' is actually the while-loop your code runs around the LLM, not the LLM itself."

What the loop actually looks like

Same model in both rows. The only difference is what your code does with the model's output — return it, or feed it back in.

Side-by-side, the way an interviewer wants it

"If the task is single-turn retrieval + generation — user asks a question, you find the answer in docs — RAG is cheaper, faster, and more predictable. I'd reach for agents only when:

• (1)the task requires multiple steps
• (2)it needs tool use (write operations, calculations, API calls), or
• (3)the solution path is not known upfront and requires reasoning. An agent adds 3-10x the cost and latency of RAG. The tradeoff is autonomy vs predictability."

The three gates

Ask them in order and stop at the first yes. Most “we need an agent” requests fail all three — they are a retrieval problem wearing a loop.

What you give up when you add the loop

The middle ground most teams skip

Between the two there is a pipeline that keeps its fixed shape but lets the model make one bounded decision:

• Conditional retrieval — the model decides whether to search before answering. One extra call, no loop.
• Query rewriting — one rewrite pass before retrieval; fixes most “retrieval missed it” complaints for ~$0.0002.
• Bounded re-query — if the top chunk scores below threshold, retrieve once more with a different query. Hard cap of two.
• Read-only tools — a loop with lookups but no writes. You get multi-step reasoning with none of the blast radius.

"Decision tree: ReAct when the task is exploratory and the path isn't known upfront (research, diagnosis). Planner-Executor when the task has clear phases and I want auditability — the plan is a human-readable artifact I can approve before execution. Great for compliance-heavy workflows. Multi-Agent only when there are genuinely separable domains of expertise — e.g., a researcher who searches the web and an analyst who runs SQL shouldn't share a context window. Multi-agent is a tool, not a default — it adds coordination overhead and debugging complexity."

The three shapes, side by side

The pattern is not a taste question — it follows from whether the path is known upfront and whether anyone must approve it.

Choose on the failure you can live with

Why the plan artifact is the whole point

Planner-Executor wins compliance reviews because the plan is inspectable before anything happens — a diff a human can approve, not a trace they read afterwards:

{
  "task": "close out the Q3 vendor invoice dispute",
  "plan": [
    { "step": 1, "tool": "fetch_invoices", "args": { "vendor_id": "V-8841" },
      "reads": ["billing.invoices"], "writes": [] },
    { "step": 2, "tool": "diff_against_contract", "args": { "contract_id": "C-119" },
      "reads": ["legal.contracts"], "writes": [] },
    { "step": 3, "tool": "open_credit_memo", "args": { "amount_cap_usd": 5000 },
      "writes": ["billing.credit_memos"], "requires_approval": true }
  ],
  "approval": { "because": "step 3 moves money", "approver_role": "ap_manager" }
}

Every step declares what it reads and writes, so the approval gate is derived rather than remembered. ReAct cannot produce this document: step 3 does not exist until step 2 returns.

"I'd diagnose in this order:

• (1)Check tool descriptions — are they ambiguous? If the agent can't tell which tool to use, it tries them all. Fix the descriptions.
• (2)Check if it's re-calling the same tool — add deduplication.
• (3)Check context bloat — after 10 calls, the context is so long the model loses track. Add observation summarization.
• (4)Consider a planner step — if the agent is wandering, a plan upfront constrains the path.
• (5)Check the system prompt — add explicit guidance like 'you should need at most 5 tool calls for this type of task.'"

Read the trace before you touch the prompt

Fifteen calls is not one bug. Pull ten traces and count what each call contributed — the shape of the waste names the fix.

Five shapes of waste and what each one means

Two guards, both in your code

def step(state, call):
    sig = (call.tool, json.dumps(call.args, sort_keys=True))

    if sig in state.seen:                       # never pay twice for one question
        return Obs(state.seen[sig], note="identical args — reusing the earlier result")

    if state.calls >= HARD_CAP:                 # a cap the model cannot argue with
        return Obs(None, note="tool budget spent — answer from what you have")

    state.seen[sig] = execute(call)
    state.calls += 1
    return summarize(state.seen[sig], max_lines=3)

The note matters as much as the guard — told only “no result”, the model retries with a cosmetic argument change.

“Three levers, and they fix different failures. Prompting fixes instruction-following — the model can do the task, it just isn’t doing it your way. Retrieval fixes missing knowledge — the model doesn’t know your data, or your data changes daily. Fine-tuning fixes consistency — the model knows how, but drifts on format or tone across thousands of calls, or you need a smaller model to hit a latency and cost target.

The order matters: prompt, then retrieval, then fine-tune. Most teams reach for fine-tuning first because it sounds like the serious answer, spend six weeks building a dataset, and find out the real problem was a vague system prompt.

In production they combine. Fine-tune for shape, retrieve for facts. What I never do is fine-tune facts into a model, because then every fact change is a retraining job.”

Match the lever to the failure

The symptom tells you which one you need

The mistake is almost always reaching one rung too high.

What a fine-tune actually buys

The honest case for fine-tuning is rarely quality. It is holding quality while the model gets small enough to be fast and cheap.

“Six stages — chunk, embed, index, retrieve, rerank, generate. The interesting decisions are at the two ends.

Chunking: I start at about 512 tokens with 10–15% overlap for prose, but I don’t chunk structured documents on a token count at all. I chunk them on their own boundaries — a section, a whole table, a function. The failure everyone hits once is splitting a table so that neither half means anything.

Retrieval: dense alone misses exact strings — part numbers, error codes, names. So hybrid. BM25 and vector search in parallel, fused with reciprocal rank fusion, then a cross-encoder rerank over the top 50 down to the five that go in the prompt.

And the number I actually track isn’t similarity score, it’s whether the chunk containing the answer made it into the context. I measure recall on a labelled set before I tune anything else.”

Where quality is lost

Chunk on the boundary, not on the number

512 tokens is a default for prose and wrong for everything else.

Retrieve wide, rerank narrow

Each stage buys recall; the rerank spends a little of it back to buy prompt space.

The last row gives up three points of recall and cuts prompt tokens roughly tenfold. That is the trade, and it is usually worth taking.

“First, don’t ask for JSON in prose and hope. Use the provider’s constrained decoding — attach a schema to the request so invalid tokens are masked at sampling time and the output parses by construction.

That buys you syntactic validity, not semantic validity. The model will happily return a perfectly-shaped object with a date in the wrong format, an enum value it invented, or line items that don’t add up to the total it also gave you. So there are two layers: constrained decoding for shape, then my own validator for meaning.

When validation fails, I retry exactly once with the validation error echoed back — ‘currency must be one of USD, EUR, GBP; you sent US Dollars’. That fixes most of them. If the second attempt fails, it goes to a fallback path or a human. It does not loop.”

Two gates, two different failures

What still goes wrong after the JSON parses

The retry that works

One repair attempt, carrying the specific error. Not a loop, and not a bare retry of the same prompt — that just resamples the same mistake.

def extract(doc, attempt=1):
    resp = client.messages.create(model=MODEL, tools=[INVOICE_SCHEMA],
                                  messages=build(doc))
    if resp.stop_reason == "max_tokens":
        raise Truncated(doc.id)          # parses fine, is missing rows

    obj = resp.content[0].input          # shape already guaranteed
    errors = check_business_rules(obj)   # enums, units, totals, ids
    if not errors:
        return obj
    if attempt == 2:
        return escalate(doc, errors)     # human queue; do not keep looping
    return extract(doc, attempt + 1, repair_hint=render(errors))

“The agent loop is about forty lines — call the model, if it asked for tools run them, append the results, call again, stop on a final answer or a step cap. That’s not the hard part, and adopting a framework doesn’t remove the hard part.

What a framework does give you is the machinery around the loop: retries, streaming, tracing, checkpointing, human-in-the-loop pauses, durable state. Writing all of that yourself is real work, and that’s the honest argument for using one.

What it costs you is a layer between you and the model. When tool selection goes wrong you need to see the exact request that went over the wire, and that’s harder through three abstractions — especially when a version bump quietly edits the prompt underneath you.

So: own the loop and own the prompt, adopt libraries for the plumbing. If a framework won’t show me what the model was actually sent, that’s disqualifying.”

What is actually in the loop

Decide per concern, not per framework

This is not an all-or-nothing choice, and treating it as one is how teams end up rewriting a workflow engine badly.

The loop, in full

def run(messages, tools, max_steps=8):
    for _ in range(max_steps):
        reply = model.call(messages, tools=tools)   # your prompt, your schemas
        messages.append(reply)
        if not reply.tool_calls:
            return reply.text
        for call in reply.tool_calls:
            messages.append(execute(call))          # timeouts and retries here
    raise StepBudgetExceeded(max_steps)             # never spin forever