Your First Agentic System

Your First Agentic System

Build one support bot six times, each version added because a failure broke the one before, and price every version per question.

You are the first engineer at Acme, a fictional 40-person SaaS company. The launch plan is 50,000 questions a month, about 1,667 a day, and every number below uses that volume. Terms are in AI Vocabulary.

The code ran offline against a stub that answers like the API. Replies, usage, ids and latency are stubs; the SDK is real.

Each version adds one part, with two exceptions: v5 adds a router and a retry wrapper, and v6 adds a rate limit, redaction and logging.

v1: one call, the reply read by type

v1 is one call, and shipping it was correct: it proves the channel works.

{"max_tokens":2000,"messages":[{"role":"user","content":"How do I reset my password?"}],"model":"claude-sonnet-5-5","thinking":{"type":"between_tools"}}

AttributeError: 'ThinkingBlock' object has no attribute 'text'

MODELS = {"cheap": "claude-haiku-5-5", "main": "claude-sonnet-5-5",
          "hard": "claude-opus-5-5"}  # checked 2026-10-08
TIER = {  # each tier: its model, and its off-switch for up-front thinking
    "main": {"model": MODELS["main"], "thinking": {"type": "between_tools"}},
    "cheap": {"model": MODELS["cheap"], "thinking": {"type": "disabled"}},
}
client = anthropic.Anthropic()

class NoFullAnswer(Exception):
    """HTTP 200, but not a full answer."""

def llm(question: str, tier: str = "main"):
    cfg = TIER[tier]
    return client.messages.create(
        model=cfg["model"],
        max_tokens=2000,                  # thinking counts toward this cap
        thinking=cfg["thinking"],
        messages=[{"role": "user", "content": question}],
    )

def text_of(msg) -> str:
    if msg.stop_reason != "end_turn":     # max_tokens, refusal: no full answer
        raise NoFullAnswer(msg.stop_reason)
    return "".join(b.text for b in msg.content if b.type == "text")

text_of reads the text blocks by type and refuses any stop reason but end_turn: max_tokens and refusal both arrive as HTTP 200. Thinking counts toward max_tokens, so a cap of 300 can run out before the first word.

The new failure you just bought: a 200 response with no answer, which the caller must treat as a failure.

v2: a system prompt sets the rules

Week two: a customer was told 90 days, no questions asked. Acme’s policy is 30. The system prompt now travels with every call.

SYSTEM_PROMPT = """You are SupportBot for Acme, a project-management SaaS.
1. Answer only Acme questions. Otherwise decline and point to support@acme.dev.
2. Never invent policy. Refunds are 30 days. If unsure, say "I don't know".
3. Three sentences at most. Plain words."""

llm passes system=SYSTEM_PROMPT on every call. The four lines cost about 130 input tokens (an assumption). A system prompt is a soft guardrail, not enforcement: a user can argue the model around it. The prompt also restates the refund number (checkpoint question 1).

The new failure you just bought: a number in the prompt that goes stale when the policy changes.

v3: answers come from your docs

Week four: a prospect asked about SSO, and the model said it was on the roadmap. A system prompt cannot supply facts the model never saw; retrieval pastes in the relevant pages.

DOCS = {  # production: thousands of pages, indexed
    "password": "Password reset: Settings, Security, Reset password.",
    "sso": "SSO: available on the Team plan and up. We support Okta and Google Workspace.",
    "refunds": "Refunds: full refund within 30 days of purchase, via support@acme.dev.",
    "api-limits": "API rate limits: 100 a minute on Free, 1,000 on Team.",
}

STOP = {"how", "do", "i", "the", "a", "is", "does", "what", "you", "your", "my", "to", "on"}

def words(text: str) -> set:
    return set(re.findall(r"[a-z]+", text.lower())) - STOP

def retrieve(question: str, k: int = 2):
    q = words(question)     # whole words, not substrings
    scored = sorted(((len(q & words(DOCS[key])), key) for key in DOCS), reverse=True)
    return [(key, s) for s, key in scored[:k] if s > 0]

HANDOFF = "Not in our help pages. I passed it to support@acme.dev."

def answer(question: str) -> dict:
    hits = retrieve(question)
    if not hits:                          # nothing relevant: hand off, skip the model
        return {"text": HANDOFF, "hits": [], "model": None}
    docs = "\n".join(f"[{key}] {DOCS[key]}" for key, _ in hits)
    msg = llm(f"Docs:\n{docs}\n\nQuestion: {question}")
    return {"text": text_of(msg), "hits": hits, "model": msg.model}

Docs: [sso] SSO: available on the Team plan ... [refunds] Refunds: ...

Question: Does Acme support SSO?

Words count as whole words: a draft that counted substrings scored “in” as a hit inside “Settings”. “Do you ship hardware?” shares no words, so the bot hands off and makes no model call. Embedding search matches meaning instead: Retrieval & Reranking.

The new failure you just bought: an off-topic or stale page answered with confidence. Retrieval also lifts the request to 1,040 input tokens, from 260.

v4: a cache stops paying twice

Month two: the bill nearly doubled while tickets grew 20%. Retrieval had raised the cost per question 1.52 times, and 40% of questions were repeats (an assumption).

SIMILAR_ENOUGH = 0.92      # a starting point to measure
TTL_SECONDS = 24 * 3600    # our choice
DOCS_VERSION = "docs-v12"  # bump when the help pages change
CACHE = {}                 # key -> (answer, stored_at, docs_version)

def norm_key(question: str) -> str:
    return " ".join(question.lower().split())

def cached_answer(question: str) -> dict:
    key, now = norm_key(question), time.time()
    for old, (out, stored, version) in CACHE.items():
        fresh = now - stored < TTL_SECONDS and version == DOCS_VERSION
        if fresh and (old == key or difflib.SequenceMatcher(None, key, old).ratio() > SIMILAR_ENOUGH):
            return {"text": out["text"], "hits": out["hits"], "cache": "hit", "err": None}
    out = answer(question)
    if out.get("err") is None and out["model"]:   # answers only, not handoffs
        CACHE[key] = (out, now, DOCS_VERSION)
    return {**out, "cache": "miss"}

The new failure you just bought: one wrong answer, once cached, reaches every similar question until it expires. Asked “Do you support SSH?” after “Do you support SSO?”, the cache returned the SSO answer with zero model calls. Semantic depth is Semantic Caching.

v5: a router picks the tier

Week five: “hi” ran on the biggest model, and a four-minute 529 outage hit customers. The router swaps the whole request, because the thinking setting differs by tier.

HARD = {"refund", "billing", "charged", "cancel", "angry", "lawyer"}
DEGRADED = "We could not answer just now. A person will reply by email."

def pick_tier(question: str) -> str:
    words_ = set(re.findall(r"[a-z]+", question.lower()))
    return "cheap" if len(question) < 60 and not (words_ & HARD) else "main"

def answer(question: str) -> dict:
    hits = retrieve(question)
    if not hits:
        return {"text": HANDOFF, "hits": [], "model": None, "err": None}
    docs = "\n".join(f"[{key}] {DOCS[key]}" for key, _ in hits)
    try:
        msg = llm(f"Docs:\n{docs}\n\nQuestion: {question}", pick_tier(question))
    except anthropic.APIError as e:           # 529 or 5xx, after the SDK's retries
        return {"text": DEGRADED, "hits": hits, "model": None, "err": type(e).__name__}
    out = {"model": msg.model, "hits": hits, "stop": msg.stop_reason, "err": None,
           "request_id": msg._request_id, "in_tok": msg.usage.input_tokens,
           "out_tok": msg.usage.output_tokens}
    out["cost_usd"] = cost_usd(msg.model, out["in_tok"], out["out_tok"])
    try:
        return {**out, "text": text_of(msg)}
    except NoFullAnswer as e:
        return {**out, "text": DEGRADED, "err": "NoFullAnswer"}

The client is anthropic.Anthropic(max_retries=2, timeout=20.0). The SDK’s retries absorb blips. A 1, 2, 4 second loop waits seven seconds in total, against a four-minute outage. An outage needs a deadline, a second tier, a degraded answer that says so, and a breaker (Event-Driven Async).

outage, 529 each attempt: 3 calls, 1.23 s, degraded (OverloadedError)
one blip, 529 then 200: 2 calls, answered
HTTP 200, stop_reason refusal: 1 call, degraded (NoFullAnswer)

The new failure you just bought: a hard question on the small tier fails confidently and cheaply. A retry loop on top of the SDK makes nine calls per tier.

v6: the front door: limit, redact, log

Week six: forty minutes down with no logs, a card in the prompt, and a script at 500 requests a minute. Three parts go in front.

1. Rate limitper user
2. Redactcards, emails
3. Cacherepeats
4. Routethe tier
5. Calldocs, model
6. Logone line

BUCKETS = {}   # production: a shared store

def allow(user: str) -> bool:             # holds 3, refills 1 a second
    b = BUCKETS.setdefault(user, {"tokens": 3.0, "at": time.monotonic()})
    now = time.monotonic()
    b["tokens"] = min(3.0, b["tokens"] + (now - b["at"]))
    b["at"] = now
    if b["tokens"] < 1:
        return False
    b["tokens"] -= 1
    return True

CARD_RE = re.compile(r"\b(?:\d[ -]?){12,18}\d\b")   # 13 to 19 digits, spaces or dashes allowed
EMAIL_RE = re.compile(r"[\w.+-]+@[\w-]+\.[\w.]+")

def luhn_ok(digits: str) -> bool:
    total = 0
    for i, ch in enumerate(reversed(digits)):
        d = int(ch) * (2 if i % 2 else 1)
        total += d - 9 if d > 9 else d
    return total % 10 == 0

def redact(text: str) -> str:
    def card(m):
        digits = re.sub(r"\D", "", m.group(0))
        return "[CARD]" if 13 <= len(digits) <= 19 and luhn_ok(digits) else m.group(0)
    return EMAIL_RE.sub("[EMAIL]", CARD_RE.sub(card, text))

my card 4242424242424242 was charged, email a@b.co  ->  my card [CARD] was charged, email [EMAIL]
card 4242 4242 4242 4242 and 4242-4242-4242-4242  ->  card [CARD] and [CARD]
Amex 3782 822463 10005 on file  ->  Amex [CARD] on file
order 123456789012 is late  ->  order 123456789012 is late
ref 1234 5678 9012 3456  ->  ref 1234 5678 9012 3456
Jane Doe, 12 Elm Street, Leeds  ->  Jane Doe, 12 Elm Street, Leeds

Cards are found by digit count and the Luhn check. Names and addresses pass (the Jane Doe line), and Luhn passes one random digit string in ten: PII Detection.

log = logging.getLogger("supportbot")
EMPTY_LOG = {"q": None, "cache": None, "model": None, "docs": [], "top": 0, "in_tok": 0,
             "out_tok": 0, "cost_usd": 0.0, "stop": None, "request_id": None, "err": None,
             "status": "ok", "ms": 0}

def handle(user: str, question: str) -> str:
    rec = {**EMPTY_LOG, "user": user}
    started = time.monotonic()
    try:
        if not allow(user):
            rec["status"] = "rate_limited"
            return "429: slow down"
        rec["q"] = redact(question)
        out = cached_answer(rec["q"])
        rec.update({k: v for k, v in out.items() if k in rec})
        rec["docs"] = [key for key, _ in out["hits"]]
        rec["top"] = max((s for _, s in out["hits"]), default=0)
        rec["status"] = "degraded" if out.get("err") else "ok"
        return out["text"]
    finally:
        rec["ms"] = round((time.monotonic() - started) * 1000)
        log.info(json.dumps(rec))

{"q":"Does Acme support SSO? My card [CARD] was charged, email [EMAIL]","cache":"miss","model":"claude-sonnet-5-5","docs":["sso","refunds"],"top":2,"in_tok":1040,"out_tok":325,"cost_usd":0.00533,"stop":"end_turn","request_id":"req_stub_0019","err":null,"status":"ok","ms":1,"user":"u2"}

The line is written in finally, so a failed call still logs its status and time. The outage in the same run logged OverloadedError. Buckets in memory are per server; production shares them, Rate Limiting & Cost.

What it costs, v1 to v6

Cost per question is tokens times price, per call, at the list prices read on 2026-10-08. Apart from the prices, these inputs are assumptions:

• 130 input and 325 output tokens (current tokenizer, about 30% more than older ones).
• The system prompt adds 130 input tokens; retrieval brings v3 to 1,040.
• 40% repeats (v4); 70% of misses to Haiku 5.5 (v5). Every miss calls the model: an upper bound.
• Thinking is off; 100 thinking tokens per Sonnet call add $1.00 per 1,000.

v3  (1,040 x $2 + 325 x $10) / 1,000,000 = $0.00533
Haiku call  (1,040 x $0.10 + 325 x $0.50) / 1,000,000 = $0.0002665
v5  0.6 x (0.7 x $0.0002665 + 0.3 x $0.00533) = $0.00107

Brackets give the monthly figure at 50,000 questions. Latency is not in the table; with assumed inputs (0.8 seconds to the first token, 65 tokens a second), a 325-token v1 answer takes 0.8 + 325 ÷ 65 = 5.8 seconds.

Next: The Paradigm Shift: Loops, Gates and State on Google Cloud — the same six parts, named as agent parts on Google Cloud.