System Design 101

System Design 101

The seven primitives every architecture diagram uses, placed on one request path, and what changes when the server is a language model.

One request, one streamed answer

POST /v1/messages

{
  "model": "claude-sonnet-5-5",
  "max_tokens": 1024,
  "stream": true,
  "messages": [
    {
      "role": "user",
      "content": "Order status?"
    }
  ]
}

HTTP/1.1 200 OK

event: error
data: {"type": "error", "error": {"type": "overloaded_error", "message": "Overloaded"}}

The 200 only says the stream started. Read blocks by type, not content[0].text: on Sonnet 5.5 adaptive thinking is on by default, and the first block can be a thinking block. Branch on stop_reason, and leave max_tokens room for thinking. The call is Messages API's.

The reference question is 1,040 input and 325 output tokens: the older 800 and 250, plus about 30% for the Claude 4.7 tokenizers. Count your own with the token counting endpoint, not a words-per-token rule.

Per million tokens, 1,040 input at $2 and 325 output at $10 is $0.00533 on Sonnet 5.5, before thinking tokens. Haiku 5.5, the smallest current model, is $0.000267 for the same question: 20 times less.

The mean hides the slow answer

A percentile is the value that p% of requests are at or below. A task of n sequential calls meets a call from the slowest 1% with probability 1 − 0.99ⁿ, so agents live in the tail.

The script makes 1,020 synthetic latencies: 1,000 with a median of 60 ms, and 20 slow ones.

# A mean hides the tail (synthetic data, in ms).
import math
import random
random.seed(7)  # same numbers every run
fast = [random.lognormvariate(math.log(60), 0.3)
        for _ in range(1000)]
slow = [random.uniform(1500, 5000)
        for _ in range(20)]
latencies = fast + slow
def percentile(data, p):
    """The p-th percentile, nearest rank."""
    ranked = sorted(data)
    k = math.ceil(p / 100 * len(ranked))
    return ranked[k - 1]
mean = sum(latencies) / len(latencies)
print(f"n = {len(latencies)}")
print(f"mean = {mean:.0f} ms")
for p in (50, 95, 99):
    v = percentile(latencies, p)
    print(f"p{p} = {v:.0f} ms")

n = 1020
mean = 122 ms
p50 = 60 ms
p95 = 104 ms
p99 = 2950 ms

The mean, 122 ms, is slower than at least 95 requests in 100 (the p95 is 104 ms). The p99 is 2,950 ms, almost three seconds. A p99 needs at least 100 samples. The mean suits a bill, not a wait.

Latency sets how many requests are in flight

Little's law: requests in flight equal the arrival rate times the time each spends in the system. A slow model keeps more requests open at the same rate. Slot sizing is Scaling the Agent Runtime's.

Three caches, three different risks

A cache stores a value under a key for a time to live: a hit returns it, a miss computes and stores it. The LLM caches differ in their keys.

Prefix reads: 0.05× on Sonnet 5.5 and Opus 5.5, 0.025× on Fable 5.1, 0.1× on Haiku 5.5. Writes cost 1.25× for 5 minutes or 2× for 1 hour. Prompt Caching has the economics.

More servers, same provider ceiling

Stateless replicas add capacity in your code, not at the provider. Every replica draws on one organisation-level bucket per model class, with three meters: requests (RPM), input tokens (ITPM) and output tokens (OTPM).

At Start, Opus 5.5 and Sonnet 5.5 each allow 1,000 requests, 2M input and 400K output tokens a minute (Haiku 5.5 matches). For the reference question: 1,000 questions on RPM, 1,923 on ITPM, 1,231 on OTPM. The lowest binds first.

At the reference shape, checked 2026-10-08.

OTPM counts generated tokens, so a high max_tokens costs nothing against it. Cache reads are not in ITPM: 2M ITPM at an 80% hit rate serves 10M input tokens a minute.

A 429 is a meter over its line (with retry-after), an acceleration limit after a sharp ramp, or the spend cap: $500 a month at Start, with no retry-after, reset at 00:00 UTC on the first unless you request more.

The SDK retries twice and honours retry-after. With your own loop, three attempts of three make 9 calls per question on one meter.

Pick the store by the question it answers

A vector index is approximate, and a new article is findable only once embedded. Which store to run is Vector Indexes & Stores's question.

Move slow work off the request path

1. Enqueue. A handler writes a job with an id to the queue.
2. Return 202. The client gets 202 Accepted and the id at once.
3. Work. A worker runs the job and writes the result.
4. Collect. The client polls, or a webhook reports back.

At-least-once

A job can arrive twice.

Idempotent

A repeat leaves the same result.

Dead-letter queue

Where jobs go when their retries run out.

An agent run lasts minutes, so it cannot wait in a web request. The model API gives no idempotency key: keep one per side effect in your code (Agent Durability). Work nobody waits for can use the Batch API, at 50% off. Queue mechanics are Event-Driven Async's.

Sources, checked 2026-10-08: Streaming, Errors, Rate limits, Pricing and Models overview.

Next: AI Vocabulary: Every Term at Its Stop on the Request Path — each term from this page, defined in one line