Everyone draws roughly the same boxes, so the boxes are not what is assessed. The order is: restate, clarify the trust boundary, derive one number, then design and harden. In a support agent the number that picks the architecture is the escalation queue, not the token bill.
The split is the illustrative one from the 5-Phase Framework; confirm the round’s length with the recruiter. Pans sideways on a phone.
Restate it with one constraint added
- “A support agent for a mid-size SaaS that answers from our docs and drafts replies, and never issues a refund on its own.”
- “I’ll design the answer path first.”
- “So, a support bot.”
- Start drawing before the scope is named.
Rule of thumb: add the constraint the interviewer implied, while a wrong assumption is still cheap to fix.
“Design a support agent for a mid-size SaaS.” The ten minutes before the diagram and the twelve after it are where answers differ. The restatement shows which problem you are solving.
Ask the trust boundary first
| Bucket | You ask | It decides |
|---|---|---|
| Trust | Draft only, or send? Refund, and up to what amount? | Whether a send tool exists, and where the gate sits |
| Quality | What does a wrong reply cost, and who grades one? | What the eval set scores |
| Data | Where do docs and orders live, and how stale may they be? | The retrieval path and the reindex cadence |
| Cost and scale | Tickets a day, how peaky, and is a minute fine? | Queue or real time, and the model tiers |
What the agent may do without a person is the biggest fork: a read-only agent and one that can refund use the same model inside different systems. Cover the other buckets, then stop when the next answer would not change the design. Say your assumptions plainly: 20,000 tickets a day, about 60% answerable from documentation, drafts auto-sent only in low-risk categories, a minute is fine.
Derive the number before you draw
| Lever | Per ticket | A month |
|---|---|---|
| Every ticket on Sonnet 5.5, uncached | $0.030 | $18,000 |
| Cache the 2,000-token prefix (reads at 0.05×) | $0.0148 | $8,880 |
| Triage, then 60% on Haiku 5.5, if evals hold | $0.0065 | $3,887 |
Assumed per ticket: 4 model calls of 3,000 input tokens, of which 2,000 are the stable system prompt and tool definitions, and 600 output tokens in all; a triage call of 800 in and 20 out ($0.00009). List prices per million tokens, read 2026-10-08: Sonnet 5.5 $2 in, $10 out, $0.10 cache hit; Haiku 5.5 $0.10, $0.50, $0.01 (prompts up to 100K tokens). Cache writes are negligible at this rate; thinking is excluded, so each figure is a floor. 600,000 tickets a month.
The cache is the first lever, a free one: it cuts the bill by 51% without touching quality. Triage then saves $4,993 a month, which makes it an architectural argument rather than a preference. But halving the whole model bill saves about $1,943 a month, while cutting escalations by five points saves 100 agent-hours a day. That is the number the design should serve: what the agent can resolve safely, not what it costs per token. Tiered Model Routing prices the routed mix and its ceiling.
Then draw it
Blue: input. Amber: the model. Teal: the gate, the security layer. Green: output. Purple: memory. Numbers match the probes in step 05.
Untrusted text arrives in the ticket and in retrieved pages, so the tool gate, not a filter in front of the model, is the security layer. The bottom row is what makes the system operable: retrieval you can debug by stage, traces and evals you read on every change, and memory scoped to one customer. The support-agent design question has the full answer.
Where they push, and what strong sounds like
| Probe | Strong |
|---|---|
| 1 · “Why a separate triage step?” Weak: “It’s cleaner.” | “It sends 60% of tickets to a model priced at a twentieth of the main one, about $5.0K a month here, and rejects out-of-scope tickets before spending anything. But escalations are the bigger number.” |
| 2 · “What if it takes fifteen tool calls?” Weak: “Add a step limit.” | “A cap, yes, but first I read the trace: the same call repeated, or a null result it keeps retrying. Fix the descriptions, return errors it can act on, and let compaction shrink history instead of deleting turns.” |
| 3 · “Which actions need approval?” Weak: “Anything risky.” | “Gate by reversibility, in code between the tool call and its execution. Measured low-risk replies send; refunds above a ceiling and anything to a new contact wait for a person.” |
| 4 · “How do you know it works?” Weak: “We’d evaluate it.” | “20 to 50 tasks from real failures to start, graded on the end state and run several times each; escalation and reopen rates in production; traces I can read when one goes wrong.” |
Each strong answer names the failure before the fix and carries a number. The weak ones are not wrong; they are the first line of an answer that stopped. The step cap, the gate and the eval set are covered in The Agent Loop, Human in the Loop and Agent Evaluation.
Name the failure modes before they ask
- The knowledge goes stale, quietly
- Answers stay confident while pricing and policy move. A reindex pipeline with a freshness metric you alert on, not a quarterly reminder.
- Retrieval is the bug you misdiagnose
- Check whether the right passage was retrieved before touching the prompt; most “hallucination” here is a chunk boundary in the wrong place.
- Anything the agent reads can instruct it
- A ticket is text a stranger wrote. If the agent can act, enforcement lives in the gate, outside the model.
Naming a weakness in your own design reads as experience; waiting to be caught reads as the opposite. The failure-modes catalogue is worth an hour before an onsite, with chunking strategies and the injection checklist for the last two.
Eight minutes in without a number: you are describing a category of system, not designing one. A guardrail in front of the model and nothing after it. Semantic caching as the first cost lever, when prompt caching is free and larger. Batch for work with a one-minute target: batch results can take up to 24 hours.
“Your agent costs too much. What do you change first?” Split the bill, cache the stable prefix, then route only the share an eval holds on the small model. Then turn the question: at this volume the model spend is $3.9K a month and escalations are 500 hours a day, so the resolution rate is the lever that matters.
The interviewer says the agent may issue refunds up to $50. What changes on your diagram?
A refund tool appears behind the gate, with the ceiling checked in code from the tool’s arguments and a person approving anything above it. Each call carries an idempotency key so a retry cannot refund twice.
Why cache before routing, when routing saves more here?
Caching is a free win: same model, same quality, 51% off. Routing trades quality for cost, so it waits for an eval that shows the small model holds on that share of tickets.
Answers are wrong and the team wants to rewrite the prompt. What do you check first?
Whether the right passage was retrieved, and at which stage it was lost. A prompt cannot fix a chunk that never reached the model.
- Restate with one constraint, then clarify the trust boundary before anything else.
- Derive one number out loud before the first box: here $3.9K a month in tokens and 500 human hours a day.
- Cache first, route second, and let the escalation rate drive the design.
- The tool gate is the security layer; the bottom row makes it operable.
- Volunteer the failure modes: stale knowledge, retrieval, injection through tickets.