Your First Agentic System

Your First Agentic System, End to End

Six versions, six real fires. Start with a five-line script and end with a support bot you could defend in a design review — and in an interview.

Most real AI systems are not designed in one heroic whiteboard session. They start as an embarrassing little script, something breaks in the real world, and a new piece gets added to fix exactly that break. This section rebuilds that journey — for a customer-support bot — one honest failure at a time.

The scenario: SupportBot for Acme

You are the first engineer at Acme, a fictional 40-person SaaS company. Two support agents handle ~300 conversations a week, and roughly 40% of the questions are the same ten questions — password resets, invoices, “do you support SSO?” The founders ask: can we wire an LLM into the contact form? An LLM (large language model — here, Anthropic’s Claude) is a brilliant new hire who has read the whole internet but knows nothing about your company. You say yes.

v1 · The naive version: one call, question in, answer out

Friday afternoon, five lines. At the demo the founder types “how do I export my data?”, a fluent answer appears three seconds later, and the room applauds. You ship to five beta customers on Monday. And honestly? Shipping v1 was correct — it proves the channel works, and everything missing from it is about to announce itself with an incident.

# supportbot_v1.py -- question in, answer out. That is the whole system.
def llm(question):
    """The ONE place that talks to the model. Mocked so this file runs offline."""
    # REAL VERSION (pip install anthropic, export ANTHROPIC_API_KEY):
    #   import anthropic
    #   r = anthropic.Anthropic().messages.create(
    #       model="claude-sonnet-4-5", max_tokens=400,
    #       messages=[{"role": "user", "content": question}])
    #   return r.content[0].text
    return f"[mock sonnet] You asked: {question!r} -- here is a fluent answer."

print(llm("How do I reset my password?"))   # one call; the answer appears

Architecturally, v1 is a function with an HTTPS call inside: no memory, no knowledge of Acme, no limits, no logs — every later version deletes one item from that list. And it will answer any question on any topic, inventing policies with identical confidence whether right or wrong. Not a bug in the model; a missing component in your system. Onto v2.

v2 · A system prompt: quality by instruction

The incident, week two. A customer asks “what’s your refund policy?” and the model — trained on an internet where some companies offer 90 days — answers “90 days, no questions asked!” Acme’s real policy is 30. The support lead forwards the screenshot with subject line “???”, plus a second ticket in which the bot wrote a poem about a competitor.

The new component is a system prompt: a standing block of instructions sent with every API call, separate from the user’s message — who the model is, what it may discuss, what to do when it doesn’t know. It is your first guardrail, and editing behavior is now a text edit, not a retraining run.

# supportbot_v2.py -- standing instructions the model reads before every answer.
SYSTEM_PROMPT = """You are SupportBot for Acme, a project-management SaaS.
Rules:
1. Answer ONLY Acme questions. Otherwise decline, point to support@acme.dev.
2. Never invent policy. Refunds are 30 days. If unsure, say "I don't know".
3. Three sentences max. Plain words."""

def llm(question, system=""):
    # REAL: same call as v1, plus system=system in messages.create(...)
    return f"[mock] saw {len(system)} chars of rules | q: {question!r}"

# Same API as v1 -- one new argument. The rules now travel with every call.
print(llm("Do you offer 90-day refunds?", system=SYSTEM_PROMPT))  # rule 2
print(llm("Write a poem about Jira", system=SYSTEM_PROMPT))       # rule 1

Two caveats. A system prompt is a soft guardrail: “ignore your previous instructions” can sometimes talk the model around it — hard enforcement needs input/output classifiers (see Guardrails & Safety). And those rules add ~200–400 tokens to every call, which the bill in v4 will notice.

v3 · Retrieval: answers from your docs, not its memory

The incident, week four. A prospect asks “do you support SSO?” The model’s training data predates Acme shipping SSO, so it says — fluently, confidently — “no, but it’s on the roadmap.” The prospect churns. The answer sounds perfect; nothing looks wrong. That is hallucination, and the system prompt cannot fix it: “don’t invent” does not hand the model facts it has never seen.

The fix is retrieval: before calling the model, find the relevant help docs and paste them into the prompt — the model stops remembering and starts reading. Our retriever is a deliberately simple keyword matcher: score each doc by shared content words, keep the top two.

# supportbot_v3.py -- look the answer up in OUR docs before asking the model.
import re
SYSTEM_PROMPT = "You are SupportBot for Acme. Answer ONLY from the docs below."
DOCS = [  # production: thousands of help-center pages, chunked and indexed
    "Password reset: Settings, Security, Reset password. Link expires in 1 hour.",
    "SSO: available on the Team plan and up. We support Okta and Google Workspace.",
    "Refunds: full refund within 30 days of purchase, via support@acme.dev.",
    "API rate limits: 100 req/min on Free, 1,000 on Team, 10,000 on Enterprise.",
]

def llm(prompt, system=""):
    return f"[mock] answer grounded in {prompt.count(chr(10)) + 1} lines of prompt"

def retrieve(question, k=2):
    """Keyword retriever: score each doc by shared content words. Teaching stand-in!"""
    stop = {"how", "do", "i", "the", "a", "is", "does", "what", "your", "you"}
    words = set(re.findall(r"[a-z]+", question.lower())) - stop
    scored = sorted(((sum(w in d.lower() for w in words), d) for d in DOCS), reverse=True)
    return [d for s, d in scored[:k] if s > 0]

def answer(question):
    context = "\n".join(retrieve(question)) or "No docs matched."
    return llm(f"Docs:\n{context}\n\nQuestion: {question}", system=SYSTEM_PROMPT)

print(answer("Does Acme support SSO?"))   # finds the SSO doc instead of guessing

Be plain about the stand-in: keyword matching cannot know that “sign in” and “log in” mean the same thing — zero shared words, zero score. Production retrieval converts text into embedding vectors (numeric fingerprints of meaning) and searches a vector database for the closest — its own discipline, covered in RAG. The pipeline shape you just wrote (retrieve → stuff → answer) does not change when you make it.

v4 · A cache: stop paying twice for the same answer

The incident, month two. Finance asks why the model bill tripled while ticket volume grew 20%. You dig in (no logs yet — you add a print) and find it: 40% of questions are near-duplicates — “How do I reset my password?”, “how do i reset my password”, “password reset??” Each paid full price and waited ~2.5 seconds for an answer you already owned.

The new component is a cache: a map from question to answer, checked before paying for a new one. You want two flavors: exact match for identical strings, and semantic match for near-duplicates — here difflib scores string similarity, and 0.92 is our “same question” line, standing in for embedding similarity.

# supportbot_v4.py -- identical (and near-identical) questions get answered once.
import difflib
SYSTEM_PROMPT = "You are SupportBot for Acme."
def llm(prompt, system="", model="sonnet"):
    return f"[mock {model}] answer"
def answer(q):
    return llm(q, system=SYSTEM_PROMPT)   # stand-in for the full v3 pipeline

CACHE = {}                 # exact cache: question text -> answer
SIMILAR_ENOUGH = 0.92      # above this similarity, treat it as the same question

def cached_answer(question):
    if question in CACHE:                        # 1) exact hit: free, ~0 ms
        return CACHE[question], "exact hit"
    for old in CACHE:                            # 2) near-duplicate wording?
        if difflib.SequenceMatcher(None, question.lower(), old.lower()).ratio() > SIMILAR_ENOUGH:
            return CACHE[old], "semantic hit"
    CACHE[question] = answer(question)           # 3) miss: pay for the model once
    return CACHE[question], "miss (paid)"

for q in ["How do I reset my password?",
          "how do i reset my password?",     # case differs -> semantic hit
          "What is your refund policy?"]:
    ans, how = cached_answer(q)
    print(f"{how:13s} | {q}")

Production swaps the dict for Redis and difflib for real embeddings — GPTCache packages exactly this (see Semantic Caching). One warning: a cache amplifies mistakes — a wrong answer used to embarrass you once, now it is served to everyone until eviction. Threshold high, TTL set, invalidate on doc changes.

v5 · Model routing and retries: right-sized cost, absorbed errors

Two incidents in one week. First, the CFO again: cost per answer is double your estimate — “hi” and “how do I reset my password” all ran on the biggest, priciest model. Second, Tuesday 9:07 a.m.: the API returns 529 overloaded for four minutes, v4 lets the exception fly straight through, and thirty customers meet an error page.

Two components, one theme — match the machinery to the job. A router classifies each question as easy or hard and picks a model accordingly; the price gap is real money (per Anthropic’s pricing page, July 2026):

Model	Input / MTok	Output / MTok	Latency (illustrative)	Route to it for
Claude Haiku 4.5	$1	$5	~0.9 s	Short, factual, low-stakes — ~70% of support traffic
Claude Sonnet	$3	$15	~2.5 s	Messy, emotional, multi-step — refunds, disputes

And a retry with exponential backoff: transient errors (429, 5xx/529) usually clear in seconds, so wait 1 s, then 2 s, then 4 s, then give up gracefully. Exponential because each failure hints the API is overloaded, so back off harder each time — plus jitter in production so clients don’t retry in lockstep. More in Tiered Model Routing.

# supportbot_v5.py -- small model for easy asks, big for hard ones, plus retries.
import time
SYSTEM_PROMPT = "You are SupportBot for Acme."
class TransientAPIError(Exception): pass   # 429 rate-limit, 500/529 overload: retryable

def llm(prompt, system="", model="sonnet"):       # MOCK: never fails, canned text
    # REAL: anthropic.Anthropic().messages.create(model=model, ...) -- SDK raises APIStatusError
    return f"[mock {model}] answer"

HARD = {"refund", "billing", "charged", "cancel", "angry", "lawyer"}
def pick_model(q):
    """Cheap heuristic router: short + safe -> haiku ($1/$5 per MTok), else sonnet."""
    easy = len(q) < 60 and not any(w in q.lower() for w in HARD)
    return "claude-haiku-4-5" if easy else "claude-sonnet-4-5"

def ask(q, tries=3):
    for attempt in range(tries):
        try:
            return llm(q, system=SYSTEM_PROMPT, model=pick_model(q))
        except TransientAPIError:
            if attempt == tries - 1: raise        # 3 strikes -> real error page
            time.sleep(2 ** attempt)              # backoff 1s, 2s, 4s (+ jitter in prod)

print(ask("hi, how do I reset my password?"))      # -> haiku (cheap, fast)
print(ask("I was charged twice and I am furious")) # -> sonnet (careful)

v6 · Logging, PII redaction, rate limits: operable, not just working

The 3 a.m. page. The bot was down forty minutes and nobody can say why — there are no logs. Poking around, you find a customer pasted their card number into chat and it flowed straight into the prompt. And overnight, someone’s broken script hit the endpoint 500 times a minute; each hit was a paid model call.

Three small components make the system operable. Structured logging: one JSON object per request — who asked, what (redacted), which model, how long, cache hit or miss — so the next page comes with evidence. PII redaction: personally identifiable information is masked before the model and logs ever see it (our regex catches cards and emails; production uses a real detector such as Presidio). And a rate limiter: a token bucket per user — it holds a few tokens, refills steadily, each request spends one; empty bucket, 429 — slow down. More in Rate Limiting & Quotas.

# supportbot_v6.py -- logs you can grep, PII out of prompts, per-user speed limits.
import json, re, time
def answer(q, model="haiku"):
    return f"[mock {model}] answer"   # stand-in for the full v5 pipeline

def redact(text):   # naive PII removal; production: Presidio / NER detectors
    text = re.sub(r"\b\d{12,16}\b", "[CARD]", text)              # card numbers
    return re.sub(r"[\w.+-]+@[\w-]+\.[\w.]+", "[EMAIL]", text)  # emails

class TokenBucket:  # `burst` requests at once, then refills at `rate` per second
    def __init__(self, rate=1.0, burst=3):
        self.rate, self.burst, self.tokens, self.ts = rate, burst, float(burst), time.time()
    def allow(self):
        now = time.time()
        self.tokens = min(self.burst, self.tokens + (now - self.ts) * self.rate); self.ts = now
        if self.tokens < 1: return False          # empty bucket -> reject
        self.tokens -= 1; return True

buckets = {}   # production: Redis, shared across servers -- not an in-memory dict
def handle(user, question):
    if not buckets.setdefault(user, TokenBucket()).allow(): return "429 -- slow down"
    q = redact(question)                 # nothing downstream ever sees raw PII
    print(json.dumps({"user": user, "q": q, "ms": 118, "model": "haiku"}))  # structured log
    return answer(q)
handle("u1", "my card 4242424242424242 was charged, email me at a@b.co")

The whole system, on one canvas

Six versions in, still under 150 lines of Python — with a quality, knowledge, cost, and operations layer. Walk one question through it: the rate limiter admits it, redaction strips cards and emails, the cache answers repeats in ~5 ms, and on a miss the router picks a model sized to the question — with the retriever pasting in docs, the system prompt bounding behavior, retries absorbing a flaky API, and the logger recording all of it.

What it costs and how fast it answers: v1 → v6

Version	What drives the cost	Cost / 1,000 questions	Typical latency	What you got for it
v1	One Sonnet call, no context	$2.85	~2.5 s	It talks. Cheapest — and the one that lies.
v2	+ system prompt (~300 tokens)	$3.75	~2.5 s	On-rails, on-brand answers
v3	+ retrieved docs (~800 tokens)	$6.15	~2.6 s	Correct, current answers
v4	40% of questions served free	$3.69	~1.5 s avg	Repeats in ~5 ms
v5	70% of paid calls → Haiku	$1.97	~0.8 s avg	Right-sized spend, errors absorbed
v6	+ logs, redaction, limits	~$1.97 (+ pennies)	~0.8 s (+ ~2 ms)	Operable at 3 a.m.

Every number in that table is illustrative, not measured. Assumptions: 1,000 questions/day; ~200 input + 150 output tokens per call in v1, +300 system-prompt tokens (v2), +800 retrieved-doc tokens (v3); 40% near-duplicates (v4); 70% of remaining calls route to Haiku (v5); Sonnet $3/$15, Haiku $1/$5 per MTok (July 2026); latencies ~2.5 s Sonnet, ~0.9 s Haiku, ~5 ms cache hit. The trend is the lesson: v3 is the most expensive version — quality and knowledge cost tokens — and every version after it claws the spend back down, until v6 delivers correct, grounded, on-rails answers for roughly the per-question cost of the naive bot that made things up. Cost and quality are not a seesaw; they are a sequence.

What we have NOT built yet

• Queues and async. Everything here is synchronous; one slow call blocks the web request. A queue goes between the front door and the pipeline → Event-Driven & Async.
• Evals. We tuned by anecdote. Real teams keep a graded set of questions and run it on every change → Agentic Eval.
• Multi-tenancy. One global cache means customer A can get customer B’s cached answer. Every layer gets scoped by tenant.
• Memory. Each call is stateless; follow-ups like “and the annual plan?” need conversation history.
• Actions, not just answers. Real bots look up orders and issue refunds through tools — a bot that can act changes the whole risk picture → How LLMs Call Tools, Guardrails & Safety.
• Human escalation. The Klarna lesson: define what the bot may handle, and make “talk to a human” a first-class path, not a failure state.

Failure modes & trade-offs

Component	Fixes	New failure mode it introduces	Numbers to watch
System prompt (v2)	Off-rails, invented answers	Soft enforcement — “ignore your instructions” can talk around it; rules can contradict docs	+200–400 input tokens per call (~$0.001 at Sonnet rates)
Keyword retrieval (v3)	Hallucinated, outdated answers	Misses synonyms (“sign in” vs “log in”); stale docs = confidently stale answers	Recall well below embedding search — swap to a vector DB
Cache (v4)	Repeat cost and latency	Amplifies a wrong answer to every future asker; low threshold merges distinct questions	Hit ~5 ms vs miss ~2,500 ms; threshold 0.92, TTL, invalidate on doc change
Router (v5)	Overpaying for easy questions	A hard question misrouted to the small model fails confidently — and cheaply	Haiku $1/$5 vs Sonnet $3/$15 per MTok (Jul 2026); watch the misroute rate
Retry + backoff (v5)	Transient 429/529 crashes	Retries multiply spend on persistent failures; lockstep retries hammer a sick API	3 tries = up to 3× tokens for a failing request; cap, jitter, alert
Logs + redaction + limits (v6)	Blindness, leaks, abuse	Regex redaction misses names and addresses; in-memory buckets reset per server	~1 KB of log per request (~1 GB per 1M); bucket 1 rps + burst 3 per user

Checkpoint: test yourself

1. Why not just build v6 on day one?

Model answer: Each component exists to fix a failure you have actually observed — you cannot pick a cache threshold before you know your duplicate rate, or tune a router before you know your traffic mix. Ship the smallest thing that works, instrument it, let reality order your backlog.

2. The cache threshold is 0.92. A user asks “how do I delete my account?” and gets the cached answer to “how do I delete a project?” What happened — and name two fixes.

Model answer: A semantic false positive: the questions share most words, similarity cleared the threshold, and the cache confidently served the wrong answer. Fixes (any two): raise the threshold and/or scope cache keys by intent; TTL plus invalidate on doc changes; route high-risk intents (delete, refund, billing) around the cache.

3. The system prompt says “refunds are 30 days”, but a retrieved doc says 60. What does the model do, and how do you fix the system?

Model answer: It receives two conflicting authorities and picks one — unpredictably. The fix is a single source of truth: policy numbers live in the docs (versioned, reviewed), and the system prompt defers to them (“answer policy questions only from the provided docs”). Add a freshness check on docs — a stale doc is now your only source of this bug.

4. 3 a.m. page: model spend doubled overnight, but answers look fine. First three checks in v6’s logs?

Model answer: (1) Cache hit rate — a doc deploy or cache flush can drop it toward zero and re-charge you for every repeat. (2) Route mix — did “easy” traffic start landing on the big model? (3) Per-user request counts — one broken script can double volume alone; check whether the rate limiter actually capped it.