System Design 101

System Design 101 for AI Engineers

The seven classic primitives every architecture diagram in this guide assumes you already know — each one explained from zero, drawn once, and then re-explained for the LLM era, where the parts stay the same but the physics change.

System design sounds like a senior-engineer subject, but it is really seven ordinary ideas wearing a trench coat: clients and servers, APIs, latency, caches, load balancers, databases, and queues. If you have ever waited in line for coffee, you already understand the hard parts. This section teaches each idea in plain English, then shows the one way it changes when the “server” on the other end is a language model.

The rest of this guide — RAG pipelines, agent loops, guardrails — is assembled from these seven parts. Each gets three beats:

• (a) a plain-English definition with an everyday analogy
• (b) a small diagram
• (c) how it mutates when the workload is LLM calls instead of web requests

Terms in bold are defined where they first appear; nothing here assumes prior vocabulary.

Primitive	The one-line version	The LLM mutation
1 · Client–server	One side asks, the other answers.	Answers arrive slowly, token by token, never identical twice.
2 · APIs & JSON	The fixed menu, and the order slip.	Four knobs: model, messages, max_tokens, temperature.
3 · Latency & throughput	Your wait vs. the shop’s capacity.	One number becomes two: TTFT and tokens/sec.
4 · Caching	Keep expensive answers lying around.	Match by meaning, not by exact string.
5 · Load balancing	More lanes, plus a traffic director.	Your fleet scales; the provider’s rate limit doesn’t.
6 · Databases	Where the truth is written down.	A new species stores meaning (vectors).
7 · Message queues	Take a number; we’ll call you.	Agent jobs run for minutes — they can’t live in a web request.

1 · Client–Server & the Request/Response Cycle

(a) The idea. A client is whatever asks for something; a server is whatever does the work and answers. Their entire relationship is one loop: the client sends a request, the server thinks, then sends back a response. That loop — request, work, response — is the atom of every system you will ever design.

The everyday analogy is a restaurant. You (the client) never walk into the kitchen. You tell the waiter what you want; the waiter carries your order to the kitchen (the server); the kitchen cooks; the waiter carries the plate back. The waiter’s route is the network, and the agreed way of writing down orders is HTTP (HyperText Transfer Protocol — the house rules for formatting web requests and responses).

2 · APIs & JSON

(a) The idea. An API (Application Programming Interface) is the menu a server publishes: the fixed list of things you may ask for, and the exact format you must ask in. JSON (JavaScript Object Notation) is the order slip nearly everyone agreed on: plain text made of {"key": value} pairs, nested as deep as you like, that every language can read and write. “Call the API” means: send an HTTP POST with a JSON body to a specific URL (the endpoint), with your API key in a header. The restaurant analogy holds: the API is the printed menu, JSON is the waiter’s order pad, your API key is the membership card you flash before the kitchen cooks for you. One rule surprises beginners: the call is stateless — the kitchen has no memory of previous orders. Want the model to “remember” the conversation? Staple the entire conversation to every new order.

3 · Latency vs. Throughput, and Reading p50 / p95 / p99

(a) The idea. Latency is how long one request takes — your personal wait for your coffee. Throughput is how many requests the whole system completes per second — how many coffees per hour the shop sells. The two are independent: a shop can have a five-minute wait (bad latency) and still serve 500 people an hour (great throughput) with enough baristas in parallel. Adding capacity raises throughput; it does nothing for one customer’s wait. This matters because LLM systems are usually bottlenecked on throughput (rate limits, primitive 5) while users feel latency.

Now the percentile family, from zero. Write down the latency of every request and sort the list fastest to slowest. The p50 (the median) is the middle value: half of all requests were faster. The p95 is the value 95% of requests beat — only the slowest 1-in-20 was worse. The p99 is the slowest 1-in-100 request. Why not just the average? Because averages lie about pain: if 99 requests take 50 ms and one takes 5 seconds, the average is a pleasant 99 ms — and one real user had a miserable time. Dashboards track p95/p99 because the tail is where user complaints live.

4 · Caching

(a) The idea. A cache is a place you keep a copy of an expensive answer so you never compute or fetch it twice. The barista who starts making “the usual” when you walk in is running a cache. The first request is a miss (do the work, save a copy); every repeat is a hit (serve the copy, skip the work). Two knobs define every cache: the TTL (time-to-live — how long a copy counts as fresh) and the hit rate (the fraction of requests you actually saved). The trade-off never changes: freshness vs. speed. Longer TTL, more hits, staler answers.

5 · Load Balancing & Horizontal Scaling

(a) The idea. When one server can’t keep up, you have two moves. Vertical scaling: buy a bigger machine — simple, but there is always a biggest machine, and it is expensive. Horizontal scaling: buy more ordinary machines — the supermarket opening more checkout lanes. Horizontal only works if any lane can serve any customer, which is why systems prize stateless servers: one that keeps no memory between requests can be cloned, killed, and added freely. The load balancer is the employee at the front of the lanes, sending each arriving customer to an open, healthy server (it pings servers for health and stops sending work to ones that stop answering).

6 · Databases: SQL vs. NoSQL

(a) The idea, part one — SQL. A SQL database (PostgreSQL, MySQL) stores data in tables: rows and columns with a strict schema (a declared contract: every user row must have an id, an email, a plan). Think of a spreadsheet that refuses to let you improvise new columns — annoying until the day it saves you. Its superpower is transactions: multi-step changes that are all-or-nothing (move money from A to B and both halves happen or neither does — this guarantee is called ACID). When the data is your source of truth — users, orders, payments, tickets — SQL is the default answer.

(a) The idea, part two — NoSQL. A NoSQL database (MongoDB, Redis, DynamoDB) relaxes the contract to gain flexibility or raw speed. Document stores keep free-form JSON objects — a folder of notes where every note can have different fields. Key-value stores keep dumb, blazing-fast key → value pairs. The trade: you give up some guarantees (joins, strict types, sometimes immediate consistency) for a shape that fits your data and easy horizontal scaling. Most real systems run both: SQL for truth, NoSQL for speed and odd-shaped data.

7 · Message Queues & Async Jobs

(a) The idea. A message queue is a to-do list that sits between two services. Instead of doing slow work while the user waits, your app writes a job (a small message: “summarize ticket #4821”) onto the queue and replies instantly. A separate pool of workers — background processes — pulls jobs off the queue and does them at its own pace. This is the deli counter with a ticket number: you don’t stand at the slicer while they make your sandwich; you take number 47 and they call you. Redis queues, RabbitMQ, SQS, and Kafka all implement this one idea at different scales. Two vocabulary words come free: at-least-once delivery (the queue guarantees your job runs, but may run it twice) and idempotent (a worker written so running the same job twice is harmless — the standard defense against duplicates).

8 · Capstone: A Simple AI Support Bot, All Seven Pieces at Once

Here is the entire section on one whiteboard: a customer asks a question, and every primitive you just learned does exactly one job. Trace the numbers. When an interviewer says “design a customer-support chatbot,” this diagram — plus judgment about what breaks first — is the answer they are grading.

Walk it once, out loud, the way you would in an interview. 1 — The browser sends an HTTPS request with a JSON body (primitives 1, 2). 2 — The load balancer hands it to any app replica, because replicas are stateless (5). 3 — The app checks the semantic cache; on a hit the answer returns in milliseconds and the LLM is never called (4). On a miss, the app gathers context — the ticket and conversation from Postgres, the most relevant help-center articles from the vector database (6) — and 4 — sends prompt plus context to the Claude API (2). 5 — The answer streams back token by token (3), and the app saves it to the cache for the next similar question. 6–7 — Anything slow — summarizing the resolved ticket, emailing a transcript, escalating to a human — goes onto the queue for a worker, which writes results back to Postgres when done (7). Notice what is not in the diagram: nothing exotic. Seven ordinary parts, arranged with intent.

Real-World Numbers: Two Companies, Two Eras, Same Primitives

Code: Two Snippets Worth Typing Out Yourself

Both run on any Python 3 install (the second needs pip install anthropic and an API key). The comments are the lesson — read them line by line.

Snippet A — p50 / p95 / p99 from raw latencies

import statistics

# 20 fake request latencies (ms) — same shape as the bar chart above.
latencies_ms = [
    45, 52, 48, 61, 55, 50, 47, 59, 63, 51,
    49, 54, 58, 46, 53, 57, 60, 44, 210, 480,
]

def percentile(data, p):
    """Return the value below which p% of the requests fall."""
    ordered = sorted(data)            # percentiles need SORTED data
    # quantiles(..., n=100) cuts the data into 100 equal slices;
    # slice p-1 is the p-th percentile. method='inclusive' matches
    # how most monitoring tools (Datadog, Grafana) compute it.
    return statistics.quantiles(ordered, n=100, method="inclusive")[p - 1]

p50 = percentile(latencies_ms, 50)  # median: half of requests were faster
p95 = percentile(latencies_ms, 95)  # only 1 request in 20 was slower
p99 = percentile(latencies_ms, 99)  # only 1 request in 100 was slower

print(f"p50  = {p50:.0f} ms   # the typical user")
print(f"p95  = {p95:.0f} ms  # the tail begins")
print(f"p99  = {p99:.0f} ms  # the unlucky 1%")
print(f"mean = {statistics.mean(latencies_ms):.0f} ms   # watch the average lie")

# Output:
# p50  = 54 ms   # the typical user
# p95  = 224 ms  # the tail begins
# p99  = 429 ms  # the unlucky 1%
# mean = 82 ms   # watch the average lie
# The mean (82) is 8x lower than what the slowest user felt (480).
# That gap is why dashboards chart p95/p99, not averages.

Snippet B — Your first real Claude API call, every parameter explained

# Setup:  pip install anthropic
#         export ANTHROPIC_API_KEY="sk-ant-..."   # SDK reads it automatically
import anthropic

client = anthropic.Anthropic()  # picks up ANTHROPIC_API_KEY from the environment

message = client.messages.create(
    # WHICH MODEL you are renting. Haiku is the fast, cheap tier —
    # right for a first call ($1 / million input tokens as of July 2026).
    # Model IDs change as new versions ship; check the docs.
    model="claude-haiku-4-5",

    # THE CONVERSATION SO FAR: a list of {"role": ..., "content": ...} dicts.
    # Roles alternate "user" (the human / your app) and "assistant" (the model).
    # The API is STATELESS — it remembers nothing, so multi-turn chat means
    # re-sending the whole history on every call (and paying for it again).
    messages=[
        {"role": "user", "content": "Explain what an API is in two sentences."}
    ],

    # HARD CAP on reply length, in tokens (~0.75 words each). Required on
    # every call. The model stops when it hits this, even mid-thought —
    # size it to the longest answer you actually want to pay for.
    max_tokens=300,

    # RANDOMNESS DIAL, 0.0 to 1.0 (default 1.0). 0 = most predictable,
    # picks the most likely token nearly every time; 1 = full creativity.
    # Use ~0 for factual answers, classifications, and anything you parse.
    temperature=0.0,

    # STANDING INSTRUCTIONS that shape every reply (persona, rules, tone).
    # Not part of the user/assistant back-and-forth; the model treats it
    # as higher-priority context. This is where a support bot's policy lives.
    system="You are a patient teacher explaining concepts to a beginner.",
)

# The reply is a list of content blocks (text, tool calls, ...); take the text.
print(message.content[0].text)

# The usage object is your receipt — this is exactly what you are billed for.
print(message.usage)  # e.g. Usage(input_tokens=38, output_tokens=87)

Failure Modes & Trade-offs, in Numbers

Every primitive fails in a characteristic way. Seniors are the people who can quote the numbers without looking them up.

Primitive	Classic failure mode	The numbers that expose it	The LLM-era twist
Client–server	Slow downstream work hangs the request until a gateway gives up.	Proxy timeouts commonly default to 30–60 s; an LLM answer can legitimately need 5–120 s.	Stream tokens (TTFT ≈ 0.5–2 s) instead of waiting for the full answer; never hold a web request open for a multi-minute agent run.
APIs & JSON	Caller and server disagree on the contract.	One renamed field breaks every client; status codes are the error contract (400 your fault, 429 slow down, 500 their fault).	The model’s own output is the new contract — ask for JSON and it may still wrap it in prose. Parse defensively; temperature 0 narrows variance.
Latency & throughput	Dashboard shows a healthy average while the tail burns.	Snippet A: mean 82 ms vs. p99 429 ms on the same 20 requests.	Track TTFT and tokens/sec separately; a 20 s total with a 1 s TTFT feels fine, a 20 s spinner does not.
Caching	Stale copies served as fresh; hit rate too low to matter.	TTL too long → wrong answers; 5% hit rate → 95% of traffic still pays full price. Netflix: cache-replication p99 < 1 s across continents.	Exact-match caching hits ~never on free text; semantic matching is what gets support bots to 30–50% hit rates.
Load balancing	One sick replica keeps receiving traffic; thundering herd on recovery.	Health checks every few seconds; a failed check pulls a replica out of rotation.	The ceiling is the provider, not your fleet: Start tier = 1,000 RPM / 2M ITPM shared by all replicas (Anthropic, July 2026). More servers ≠ more LLM throughput.
Databases	N+1 queries; strict schema where you needed flexibility (or vice versa).	50 serial queries × 5 ms = 250 ms added to one page load.	Vector similarity search adds ~10–100 ms; approximate indexes trade a few % recall for 10–100× speed.
Message queues	Duplicate deliveries; one poison job retried forever.	At-least-once queues can deliver a job twice — non-idempotent workers email a customer twice.	Agent jobs run minutes, not ms: set per-job token/step budgets and a dead-letter queue, or one looping agent burns your daily budget before lunch.

The Interview Angle

Checkpoint: Five Questions Before You Move On

Your dashboard shows p50 = 300 ms and p99 = 12 s for the same endpoint. Is the system healthy, and what does each number tell you?

Both can be true at once — that is the point of percentiles. p50 = 300 ms means the typical user is fine. p99 = 12 s means 1 in 100 requests takes twelve seconds: for an LLM endpoint that is probably a long answer or a rate-limit retry, not a bug. Split the metric: chart TTFT (want p95 ≈ 0.5–2 s) separately from total time (scales with answer length). If TTFT’s p99 is 12 s, then you have a problem — queueing or throttling upstream.

Traffic doubled, so you doubled your app servers — and the 429 errors got worse. Why didn’t horizontal scaling work?

The bottleneck was never your fleet. Every replica shares one provider account with one rate limit (Anthropic Start tier: 1,000 RPM / 2M input tokens per minute, July 2026). More servers just means more clients competing for the same token bucket. The fixes live elsewhere: cache more, queue and pace the calls, batch, use a smaller model for easy queries, retry with backoff honoring retry-after — or move up a usage tier.

“How do I reset my password?” and “I forgot my password lol” are the same question. Which cache serves both, and why does the other one fail?

A semantic cache serves both: it stores questions as embeddings (meaning fingerprints) and hits when a new question is close enough in meaning. An exact-match cache keys on the literal string, so the two questions are different keys and both pay full price. On free-text user input, exact match is close to worthless — that is the whole motivation for semantic caching.

Your support bot works great in a 3-turn test but the bill explodes in week two when users have 40-turn conversations. Which primitive explains it?

APIs & JSON — specifically statelessness. The API remembers nothing, so your app re-sends the entire conversation history on every call and you are billed for those input tokens every time: turn 40 re-pays for 39 previous turns. This is why production bots add history truncation/summarization and prompt caching — both are direct consequences of primitive 2.

Sketch the support-bot diagram from memory. Where does the queue go, and what breaks first at 10× traffic?

Client → load balancer → stateless app replicas → semantic cache → (on miss) Postgres + vector DB for context → Claude API → streamed response. The queue hangs off the app servers for slow work — ticket summaries, escalation emails, re-indexing — processed by workers that write results back to Postgres. At 10× traffic the first wall is the provider rate limit (429s), the second is cache effectiveness (hit rate decides how much of the 10× even reaches the model), the third is database connections. Your app replicas are fine — statelessness was the whole point.