Fundamentals & Architecture

"A chatbot is a single LLM call — input in, text out, stateless. An agent is an LLM inside a loop. The loop gives it tools, memory, and the ability to take actions in the world. The agent decides what to do next based on observations. The key difference is autonomy — an agent can reason, act, observe, and iterate until a task is complete. The 'agent' is actually the while-loop your code runs around the LLM, not the LLM itself."

What the loop actually looks like

Same model in both rows. The only difference is what your code does with the model's output — return it, or feed it back in.

Side-by-side, the way an interviewer wants it

Learn it in full: The Agent Loop: Think → Act → Observe — the loop in fifteen lines, why messages[] only grows, and the guards that stop it.

"If the task is single-turn retrieval + generation — user asks a question, you find the answer in docs — RAG is cheaper, faster, and more predictable. I'd reach for agents only when:

• (1)the task requires multiple steps
• (2)it needs tool use (write operations, calculations, API calls), or
• (3)the solution path is not known upfront and requires reasoning. An agent adds 3-10x the cost and latency of RAG. The tradeoff is autonomy vs predictability."

The three gates

Ask them in order and stop at the first yes. Most “we need an agent” requests fail all three — they are a retrieval problem wearing a loop.

What you give up when you add the loop

The middle ground most teams skip

Between the two there is a pipeline that keeps its fixed shape but lets the model make one bounded decision:

• Conditional retrieval — the model decides whether to search before answering. One extra call, no loop.
• Query rewriting — one rewrite pass before retrieval; fixes most “retrieval missed it” complaints for ~$0.0001 on Haiku 5.5.
• Bounded re-query — if the top chunk scores below threshold, retrieve once more with a different query. Hard cap of two.
• Read-only tools — a loop with lookups but no writes. You get multi-step reasoning with none of the blast radius.

Learn it in full: The Agentic Spectrum: Workflows to Autonomy — chaining, routing and orchestrator-workers between the two, and how to choose the lowest level that works.

"Decision tree: ReAct when the task is exploratory and the path isn't known upfront (research, diagnosis). Planner-Executor when the task has clear phases and I want auditability — the plan is a human-readable artifact I can approve before execution. Great for compliance-heavy workflows. Multi-Agent only when there are genuinely separable domains of expertise — e.g., a researcher who searches the web and an analyst who runs SQL shouldn't share a context window. Multi-agent is a tool, not a default — it adds coordination overhead and debugging complexity."

The three shapes, side by side

Choose on the failure you can live with

Why the plan artifact is the whole point

It wins compliance reviews: the plan is inspectable before anything runs — a diff to approve, not a trace to read later:

{
  "task": "close out the Q3 vendor invoice dispute",
  "plan": [
    { "step": 1, "tool": "fetch_invoices", "args": { "vendor_id": "V-8841" },
      "reads": ["billing.invoices"], "writes": [] },
    { "step": 2, "tool": "diff_against_contract", "args": { "contract_id": "C-119" },
      "reads": ["legal.contracts"], "writes": [] },
    { "step": 3, "tool": "open_credit_memo", "args": { "amount_cap_usd": 5000 },
      "writes": ["billing.credit_memos"], "requires_approval": true }
  ],
  "approval": { "because": "step 3 moves money", "approver_role": "ap_manager" }
}

Every step declares what it reads and writes, so the approval gate is derived, not remembered. ReAct cannot write this: step 3 does not exist until step 2 returns.

Learn it in full: Planner-Executor: Divide & Conquer — the plan as checkable data, and when ReAct fits.

"I'd diagnose in this order:

• (1)Check tool descriptions — are they ambiguous? If the agent can't tell which tool to use, it tries them all. Fix the descriptions.
• (2)Check if it's re-calling the same tool — add deduplication.
• (3)Check context bloat — after 10 calls, the context is so long the model loses track. Add observation summarization.
• (4)Consider a planner step — if the agent is wandering, a plan upfront constrains the path.
• (5)Check the system prompt — add explicit guidance like 'you should need at most 5 tool calls for this type of task.'"

Read the trace before you touch the prompt

Fifteen calls is not one bug. Pull ten traces and count what each call contributed — the shape of the waste names the fix.

Five shapes of waste and what each one means

Two guards, both in your code

def step(state, call):
    sig = (call.tool, json.dumps(call.args, sort_keys=True))

    if sig in state.seen:                       # never pay twice for one question
        return Obs(state.seen[sig], note="identical args — reusing the earlier result")

    if state.calls >= HARD_CAP:                 # a cap the model cannot argue with
        return Obs(None, note="tool budget spent — answer from what you have")

    state.seen[sig] = execute(call)
    state.calls += 1
    return summarize(state.seen[sig], max_lines=3)

The note matters as much as the guard — told only “no result”, the model retries with a cosmetic argument change.

Learn it in full: Agent Failure Modes — naming a failing run from its trace, the detector for each failure, and fixing by blast radius.

“Three levers, and they fix different failures. Prompting fixes instruction-following — the model can do the task, it just isn’t doing it your way. Retrieval fixes missing knowledge — the model doesn’t know your data, or your data changes daily. Fine-tuning fixes consistency — the model knows how, but drifts on format or tone across thousands of calls, or you need a smaller model to hit a latency and cost target.

The order matters: prompt, then retrieval, then fine-tune. Most teams reach for fine-tuning first because it sounds like the serious answer, spend six weeks building a dataset, and find out the real problem was a vague system prompt.

In production they combine. Fine-tune for shape, retrieve for facts. What I never do is fine-tune facts into a model, because then every fact change is a retraining job.”

Match the lever to the failure

The symptom tells you which one you need

The mistake is almost always reaching one rung too high.

What a fine-tune actually buys

The honest case for fine-tuning is rarely quality. It is holding quality while the model gets small enough to be fast and cheap.

Learn it in full: When to Fine-Tune: The Decision Framework — the prompt → retrieval → fine-tune ladder, the three questions that settle it, and why facts stay out of the weights.

“Six stages — chunk, embed, index, retrieve, rerank, generate. The interesting decisions are at the two ends.

Chunking: I start at about 512 tokens with 10–15% overlap for prose, but I don’t chunk structured documents on a token count at all. I chunk them on their own boundaries — a section, a whole table, a function. The failure everyone hits once is splitting a table so that neither half means anything.

Retrieval: dense alone misses exact strings — part numbers, error codes, names. So hybrid. BM25 and vector search in parallel, fused with reciprocal rank fusion, then a cross-encoder rerank over the top 50 down to the five that go in the prompt.

And the number I actually track isn’t similarity score, it’s whether the chunk containing the answer made it into the context. I measure recall on a labelled set before I tune anything else.”

Where quality is lost

Chunk on the boundary, not on the number

512 tokens is a default for prose and wrong for everything else.

Retrieve wide, rerank narrow

Each stage buys recall; the rerank spends a little of it back to buy prompt space.

The last row gives up three points of recall and cuts prompt tokens roughly tenfold. That is the trade, and it is usually worth taking.

Learn it in full: Chunking: Cut on Structure, Size for the Embedder — cutting on the document’s own units, fitting heading path and overlap in the embedder’s window, and measuring the cut by recall@k.

“First, don’t ask for JSON in prose and hope. Use the provider’s constrained decoding — attach a schema to the request so invalid tokens are masked at sampling time and the output parses by construction.

That buys you syntactic validity, not semantic validity. The model will happily return a perfectly-shaped object with a date in the wrong format, an enum value it invented, or line items that don’t add up to the total it also gave you. So there are two layers: constrained decoding for shape, then my own validator for meaning.

When validation fails, I retry exactly once with the validation error echoed back — ‘currency must be one of USD, EUR, GBP; you sent US Dollars’. That fixes most of them. If the second attempt fails, it goes to a fallback path or a human. It does not loop.”

Two gates, two different failures

What still goes wrong after the JSON parses

The retry that works

One repair attempt, carrying the specific error. Not a loop, and not a bare retry of the same prompt — that just resamples the same mistake.

def extract(doc, attempt=1, repair_hint=None):
    resp = client.messages.create(
        model=MODEL, max_tokens=16000,
        output_config={"format": {"type": "json_schema",
                                  "schema": INVOICE_SCHEMA}},
        messages=build(doc, repair_hint))
    if resp.stop_reason in ("refusal", "max_tokens"):
        raise Incomplete(doc.id, resp.stop_reason)   # may not match the schema

    text = "".join(b.text for b in resp.content if b.type == "text")
    obj = json.loads(text)                  # shape already guaranteed
    errors = check_business_rules(obj, doc) # enums, units, totals, ids, row count
    if not errors:
        return obj
    if attempt == 2:
        return escalate(doc, errors)        # human queue; do not keep looping
    return extract(doc, attempt + 1, repair_hint=render(errors))

Learn it in full: Structured Output: JSON Every Time — output_config.format, messages.parse(), strict tools, and what a schema cannot say.

“The agent loop is about forty lines — call the model, if it asked for tools run them, append the results, call again, stop on a final answer or a step cap. That’s not the hard part, and adopting a framework doesn’t remove the hard part.

What a framework does give you is the machinery around the loop: retries, streaming, tracing, checkpointing, human-in-the-loop pauses, durable state. Writing all of that yourself is real work, and that’s the honest argument for using one.

What it costs you is a layer between you and the model. When tool selection goes wrong you need to see the exact request that went over the wire, and that’s harder through three abstractions — especially when a version bump quietly edits the prompt underneath you.

So: own the loop and own the prompt, adopt libraries for the plumbing. If a framework won’t show me what the model was actually sent, that’s disqualifying.”

What is actually in the loop

Decide per concern, not per framework

This is not an all-or-nothing choice, and treating it as one is how teams end up rewriting a workflow engine badly.

The loop, in full

def run(messages, tools, max_steps=8):
    for _ in range(max_steps):
        reply = model.call(messages, tools=tools)   # your prompt, your schemas
        messages.append(reply)
        if not reply.tool_calls:
            return reply.text
        for call in reply.tool_calls:
            messages.append(execute(call))          # timeouts and retries here
    raise StepBudgetExceeded(max_steps)             # never spin forever

Learn it in full: Who Owns the Loop: Harness vs Deployment — harness versus deployment, the two different things called “the SDK”, and why “I need control” rarely means writing the loop.

“The loop is the small part. Call the model, run whatever tools it asked for, append the results, call it again, stop when it says it’s done or I hit a cap. That’s fifteen lines, every team’s version looks the same, and the Tool Runner or the Agent SDK will write it for me.

The harness is the loop plus everything that decides what the model sees, what it can touch and when it’s allowed to stop: the instructions and context it loads, the tools and permissions, the guardrails on risky calls, the checks that tell it whether it actually finished, the state it leaves for the next run, and the turn and budget limits.

That’s the part I engineer, because the model is the same for everybody and the harness is where the results come from. When an agent declares victory early, the fix is a test that decides ‘done’, not a better prompt. When it forgets conventions, it’s standing instructions. Each piece answers a failure I’ve actually seen.

And I don’t hand-write the loop just to get control — approval gates, result rewriting and run limits are already hooks in the SDKs.”

The loop sits inside the harness

Same model, same fifteen lines for every team. What differs is the boundary around them.

Build each part from a failure you have seen

Add a part when the failure shows up, not before, and re-test them all when you change models. The last column names where each part lives in the Agent SDK and Claude Code.

Learn it in full: Who Owns the Loop: Harness vs Deployment — harness versus deployment, and which of the four ways to build gives you what. Then read Harness Engineering: Building the System Around a Coding Agent — the harness parts in Claude Code, file by file.

“Most of the time, honestly. A multi-agent system is a context-management strategy: workers burn their own windows on the reading and hand the lead a short report. You pay for that in tokens — Anthropic measured roughly four times a chat for one agent and fifteen times for a team — so it only wins when the work is broad, parallel and worth that bill.

It’s worse when the steps are coupled. Every action carries implicit decisions, so two workers editing the same module make conflicting ones, and the merge is worse than either. It’s worse when the task fits in one window anyway, and worse when the task is cheap, because a router in front would have sent it down a one-call path.

It’s also worse to operate. Every hop loses context, results come back partial, retries multiply per layer, and a run that says ‘success’ at the root can have two failed children under it.

So I start with one agent and only split when an eval shows context pollution or tool confusion in it — and then a star, two levels, parallel reads, serialised writes.”

Two questions decide it: is the work separable, and is it worth 15×?

Where a team loses to one agent

Price it per finished task, not per call

Illustrative. Lead: 45K in, 5K out = $0.28. Each worker: 40K in, 4K out = $0.12. The team is 33% more likely to finish and still 1.8× the price per finished task — worth it only when a miss costs more than the gap.

Learn it in full: Hierarchical Delegation: Multi-Agent Teams — context isolation, the delegation brief, the 4× and 15× token bill and the coordination failures.