AI System Design Vocabulary
The working glossary as a routing hub: twenty terms defined with a number each, the rest routed to the course that owns them, arranged by where each term sits on a request path.
Last updated
After this section you can
- Place a term from a model call on the stop of the request path where it acts.
- Define the twenty core terms with a number or a trade-off attached.
- Find the section that owns any other term, so you link instead of re-explaining it.
AI Vocabulary
Find the stop on a request where each term acts, and the number behind the twenty terms that have no deeper section.
A request passes seven stops on its way to a model and back. Each term acts at one stop. Twenty are defined here with a number; the rest are routed to their owners.
Blue: request. Amber: model. Green: answer. Red: failures. Teal: measurement. Purple: compliance. Pans sideways on a phone.
Stops 1 to 4 are on the path. Five and six run beside it, and seven covers what leaves.
The edge: admission, then metering
- API gateway
- One proxy in front of every model call: keys, budgets, fallbacks, logs. It holds every provider key.
- Rate limit
- A cap on requests or tokens per window; past it the API answers HTTP 429. Tenants share the organization's limit.
- TPM / RPM
- Requests (RPM), input tokens (ITPM), output tokens (OTPM), per model class. Cache reads do not count toward ITPM on most models.
A buffering proxy breaks streamed answers. A spend-cap 429 has no retry-after and fails until the reset at 00:00 UTC on the 1st.
What the model may see
- Screenthe input for personal data and injection
- Checkthe cache, under a key scoped to the tenant
- Retrieveonly what this user may read
- Assemblethe prompt: instructions, retrieved text, history
Instructions and retrieved text share one token stream, so the system prompt cannot defend itself. Controls sit in code; the permission check runs inside the retrieval query.
Where the model's time goes
| Lane | First token | Done |
|---|---|---|
| Cold prompt | 1.4 s | 8.4 s |
| Cached prefix | 0.5 s | 7.5 s |
The same two lanes as a table, for a reader without scripts.
Latency is time to first token plus output. Cold: 1.4 s, then 420 tokens at 60 a second (7 s): 8.4 s. Cached: 7.5 s. The lanes assume thinking is off; with adaptive thinking on, the first visible token comes after it.
Three caches sit here. The KV cache is attention state inside the server; prompt caching bills a repeated prefix at a fraction; semantic caching returns a stored answer.
- TTFT
- Time to the first visible token. Queue, prefill and any thinking come before it.
- Batching
- The Batch API takes 50% off for work that can wait up to 24 hours; most batches finish within an hour.
Act: the model asks, your code runs
The model asks for issue_refund on order A-4021, $300: a tool_use block with an input.
Your code checks the $500 rule, then runs the refund with a key from the run id and tool_use id.
A tool_result block with the same tool_use id goes back in the next user message.
The $500 rule lives in the tool, not the prompt. The loop is in The Agent Loop; the block is in Inside a Tool Call.
- Guardrails
- Deterministic checks in code around the model: input filters, output validators, action allow-lists. Classifiers add a second layer, never the first.
Reliability: when a dependency misbehaves
Calls 1 to 5 each fail after their retries, with 529.
The breaker opens. Call 6 sends nothing and returns the degraded answer.
open for 30 sAfter 30 s, one probe: success closes the breaker, failure reopens it.
half-openA retry handles a blip; a breaker handles an outage. The SDK already retries twice, so a loop in your code gives (1 + 2) × (1 + 2) = 9 calls per tier.
Overload Control owns breakers and retry budgets; Model Routing owns failover. Server-side fallbacks retry only refusals, so outage failover belongs in your gateway.
- Retry
- Repeat a transient failure: 408, 409, 429 and 5xx. Never other 4xx, such as 400, 401 or 413.
- Exponential backoff
- random(0, min(cap, base × 2^attempt)): capped full jitter. Honor retry-after first.
- Timeout
- The longest you wait. The SDK default is 10 minutes; set a deadline per task.
- Idempotency
- Doing it twice has the effect of doing it once. The Messages API has no key, so your tool keeps one.
- Circuit breaker
- After N consecutive failures, stop calling for a cooldown, then send one probe. Illustrative: N is 5, cooldown 30 s.
- Graceful degradation
- A named, reduced answer while a dependency is down.
- Failover
- Switch tier or provider only when the failures are independent. Sonnet to Haiku on one provider shares its outage.
- Backpressure
- A full downstream stage slows or refuses the producer.
- Queue depth
- Jobs waiting now. Illustrative: a climb from 200 to 5,000 means consumers are behind.
Measurement: the rail that tells you
- SLO
- A target you measure, such as 99.9% of requests succeeding in a 30-day month. The gap to 100% is the error budget.
Prefer percentiles for latency targets. An illustrative p50 of 900 ms with a p99 of 12 s is a red system behind a green average.
Compliance: the words that decide whether you may ship
- Red teaming
- Attacking your own system before users do, on every change. No pass rate proves safety.
- SOC 2
- An auditor's report that controls operated over a period. Security is the mandatory criterion.
- GDPR
- EU data-protection law. Chapter V limits transfers outside the EEA; fines reach EUR 20M or 4% of worldwide annual turnover, whichever is higher.
- HIPAA
- The US health-data law. Claude's route is a BAA plus a HIPAA-ready organization; features not eligible return 400.
Procurement asks these four before a deal. Cite the article or criterion, and say as of when.
Every other term in this course, by the stop where it acts, with the section that owns it.
| Stop · word | In one line |
|---|---|
| 1 · Load balancer | Spreads requests over identical servers, so you can scale out. Scale up hits a ceiling. Owner: System Design 101 |
| 1 · Sharding | Splits data by a key, such as tenant ID, so each machine owns a slice. Owner: Scaling 10k to 1M |
| 1 · Quota | A longer budget, such as monthly spend. A spend-cap 429 has no retry-after, so retries fail until the reset. Owner: Rate Limiting & Cost |
| 2 · RAG | Fetches document pieces at question time into the prompt. Chunking, BM25 and reranking belong here too. Owner: RAG Architecture |
| 2 · Embedding | Text as a vector, so similar meanings land at nearby points. Owner: Embeddings |
| 2 · Cosine similarity | A closeness score from −1 to 1. Cut-offs such as 0.82 are model-specific: calibrate on labelled pairs. Owner: Embeddings |
| 2 · Vector database | A store built for nearest-neighbor search over many vectors, usually approximate (ANN). Owner: Vector Indexes & Stores |
| 2 · Agent memory | What an agent keeps across steps and sessions: this conversation, and facts stored about the user. Owner: Agent Memory |
| 2 · Orchestrator | A router classifies and dispatches. An orchestrator also decomposes, delegates and synthesizes. Owner: Hierarchical Delegation |
| 2 · PII | Personal data such as names, emails and IDs. Detect and redact it before the prompt leaves your boundary. Owner: PII Detection |
| 2 · Prompt injection | Instructions hidden in input or retrieved text. OWASP ranks it LLM01:2026; no reliable prevention exists. Owner: Prompt Injection Defense |
| 2 · Semantic caching | Reuses an answer for a similar question. Key it by tenant and permissions, and measure the wrong-hit rate first. Owner: Semantic Caching |
| 3 · Token | The unit a model reads and bills. The 1M context window is counted in tokens; 1M is about 555K words. Owner: Tokens, Context Windows & Sampling |
| 3 · Temperature | Not a dial on the newest models: a non-default value gets a 400 on Sonnet 5.5, Opus 5.5 and Fable 5.1. Steer with effort. Owner: Tokens, Context Windows & Sampling |
| 3 · System prompt | The standing instructions ahead of user messages. Assume it can be read. The prompt is the whole request. Owner: Prompt Engineering Foundations |
| 3 · Fine-tuning | Training on your examples to change behavior. Facts that change belong in retrieval. Owner: When to Fine-Tune |
| 3 · RLHF | Training that ranks candidate answers with people and prefers the winners. Owner: How LLMs Are Built |
| 3 · Hallucination | Fluent, confident and wrong. Its detection is covered in Hallucination Detection. Owner: Tokens, Context Windows & Sampling |
| 3 · Cache | A recent result kept so it need not be recomputed. The three LLM caches are compared in step 3. Owner: System Design 101 |
| 3 · Throughput | Work finished per unit of time: requests per second, or tokens per second for a server. Owner: System Design 101 |
| 3 · KV cache | Attention state kept on the server: about 0.33 MB per token for a 70B-class model, so 10K tokens is about 3.3 GB. Owner: How LLMs Are Built |
| 3 · Quantization | Storing weights at lower precision, such as 16-bit to 4-bit: less memory, a small quality cost. Owner: Inference Optimization |
| 3 · Prompt caching | The provider keeps a repeated prefix. A hit costs 0.05× input on Sonnet 5.5; prompts under 512 tokens are not cached. Owner: Prompt Caching |
| 4 · Agent | A model in a loop: reason, call a tool, read the result, repeat. ReAct is the reasoning pattern inside it. Owner: The Agent Loop |
| 4 · Tool call | A structured request naming a function and its input; your code runs it. Owner: How LLMs Actually Call Tools |
| 4 · Planner-executor | A strong model plans the steps and cheaper models carry them out. Re-planning is part of the pattern. Owner: Planner-Executor |
| 4 · MCP | An open protocol for exposing tools and data to models. Donated to the Agentic AI Foundation on 2025-12-09. Owner: MCP Overview |
| 4 · Audit log | A record of who did what, and when: every tool call, admin action and data access. Owner: Audit & Compliance |
| 5 · Message queue | A waiting line so a producer can hand off work and return at once. Owner: Event-Driven Async |
| 5 · Canary rollout | A small share of traffic to the change. Watch the metrics, and widen only if they hold. Owner: Deployment & Rollout |
| 6 · Latency percentile | p50, p95 and p99: the latency that 50%, 95% or 99% of requests beat. Averages hide the tail. Owner: System Design 101 |
| 7 · CMEK | Customer-managed keys in the customer's own KMS. Anthropic's CMEK covers one workspace, and only data written after the key takes effect. Owner: Multi-Tenant Isolation |
| 7 · RLS | The database filters each tenant's rows. Table owners bypass the policy unless it is forced. Owner: Multi-Tenant Isolation |
- Temperature as a knob: a non-default value is a 400 on the newest models.
- One 60-second timeout for every call: it cuts off long answers.
- Counting requests alone. At the Start tier's 2,000,000 ITPM, 4,000 uncached tokens a request cap you at 500 a minute, under the 1,000 RPM.
- A refund with no key: the Messages API has none, so the tool must bring one.
“A provider returns 429s and 529s for twenty minutes. What do your retries, breaker and SLO do?” A junior says retry with backoff. A senior retries only 408, 409, 429 and 5xx, honors retry-after, and opens a breaker after 5 failures for 30 s.
A staff engineer ties it to the error budget, 43.2 minutes a month at 99.9%. The trap is the spend-cap 429: no retry-after, and it fails until the monthly reset.
A client gets HTTP 429 with retry-after: 12. What does it do first, and what is wrong with sleep(2**n + random()) as the only policy?
It waits 12 seconds, because the server named the wait, then retries. The formula ignores retry-after and has no cap, its jitter is too small to spread clients, and it retries a spend-cap 429 that no wait fixes.
The model asks for a refund tool call and the tool times out. Where does the idempotency key live, what must hold across a replay, and why not on the model call?
In your tool, from the run id and tool_use id, saved before the tool runs; the key must match on replay. Not on the model call: the Messages API has no key, and a replayed call mints a new tool_use id. See Agent Durability.
You are at 30% of your requests-per-minute limit and still get 429s. Name two other meters that can cause it, and one lever that lowers the load on one of them.
ITPM and OTPM, per model class. Prompt caching lowers ITPM: cache reads do not count toward it on most models. Lowering max_tokens does not help, because OTPM counts only generated tokens.
- Retry only transient failures, with capped jitter and retry-after first. Never retry a spend-cap 429.
- The idempotency key lives in your side-effecting tool and stays the same on every replay.
- Write SLOs on percentiles; 99.9% leaves 43.2 minutes of error budget a month.
- Twenty terms are defined here; thirty-three route to their owners.