Scale & Cost

"Cost reduction playbook:

• (1)Model tiering — use Flash/Haiku for 80% of tasks (classification, simple Q&A), Pro/Opus only for complex reasoning. That alone cuts 60-70%.
• (2)Prompt caching — system prompt + tool definitions are identical across calls. Cache them. Saves 90–95% on the cached input tokens for repeated calls.
• (3)Semantic caching — if someone asked a similar question in the last 24h, serve from cache. Expected 30-40% hit rate for support use cases.
• (4)Context trimming — summarize old tool observations instead of keeping full text.
• (5)Batch API — for non-urgent tasks, use the batch endpoint at 50% discount. Combined, these typically achieve 5-8x cost reduction."

"The architecture shouldn't change — the infrastructure scales.:

• (1)Stateless agents on Cloud Run/GKE — auto-scale horizontally.
• (2)Queue-based ingestion via Pub/Sub — decouples request rate from processing rate.
• (3)Provisioned throughput on Vertex AI for predictable latency under load.
• (4)Sharded vector indices — partition by tenant or region.
• (5)Regional deployment — deploy in 3 regions, route by user geography.
• (6)Cache layers become critical — semantic cache hit rate determines your cost scaling. The key insight: at 10K users you can afford to be synchronous. At 1M, everything must be async with graceful degradation."

“I’d start from the budget and work backwards, because at a tight enough number the budget picks the architecture. Say twenty cents a claim at fifty thousand claims a month.

At twenty cents I can’t put every claim through a frontier model with a large context. So: a triage step on a small model classifies the claim and retrieves only the policy sections that apply. Keeping retrieval tight is the biggest token lever, because retrieved text is what actually fills the prompt.

Then a decision tier. Clean claims that match a policy rule with high confidence get auto-approved by the small model. Ambiguous ones escalate to the frontier model with full context. Anything the frontier model isn’t confident about, or anything above a dollar threshold, goes to a human.

And then I’d point out that the economics live in that distribution, not in the prompts. Human review is the expensive line by an order of magnitude — if I want the cost down, I move the escalation rate, not the token count.”

“It is a prefix cache. The provider hashes the front of your prompt, and if it matches something it has already processed it reuses the computed attention state instead of recomputing it. Two things follow from the word prefix: it only works from the very start of the prompt, and it breaks the moment any byte before the cached point changes.

So the whole game is prompt layout. Stable things first — system prompt, tool schemas, few-shot examples, the long document. Volatile things last — the user’s message, the timestamp, anything personalised.

The classic mistake is rendering ‘today is 2026-08-03 14:32:11’ or the user’s name into the top of the system prompt. That is a miss on every single call, and nobody notices, because the system still works — it is just several times more expensive than it should be.

Caveats: there’s a minimum cacheable length, a cache write costs a little more than a normal token, and the TTL is minutes. So it pays on repeated traffic, not on a long tail of one-off requests.”

“The users are waiting synchronously, so this is a latency-under-throughput problem, not a throughput problem — and that rules out the obvious design.

Static batching is wrong here: you wait for a batch to fill, run it, and everyone pays for the longest sequence in it. What I want is continuous batching — the scheduler admits new requests at each decoding step and evicts finished ones, so a short request never waits behind a long one.

Concretely: an admission queue, a scheduler that runs every decoding step, and a paged KV cache so capacity is bounded by memory pages rather than by the longest sequence anyone might send. The batch isn’t 100 because someone wrote 100 — it is whatever fits in KV memory at the context length I actually serve.

The knob that matters is how much queueing delay I’ll accept. I’d set a maximum wait, something like ten milliseconds, and a maximum batch, and fire on whichever comes first. And I’d shed load rather than let the queue grow, because an unbounded queue makes everyone’s latency bad instead of failing a few requests quickly.”

“Three-x overnight with no deploy means volume, retries, or context growth — and I can tell which within about ten minutes if the traces carry the right attributes.

First question: did request count go up, or did cost per request go up? If it’s per-request, it isn’t a traffic story at all. Then I split tokens per request into prompt and completion. Prompt tokens climbing against a flat request count is almost always context growth — a retrieval change, a memory system that started including more history, or a prefix cache that stopped hitting.

The one that catches people is retries. A tool starts timing out, the agent retries, every retry replays the whole conversation, and spend triples while request count doesn’t move at all. That’s why I want tokens attributed per step and per trace, not a monthly total on a provider dashboard.

And the cheap safeguard is a per-trace token budget with an alert, so this is a page at two-x rather than a surprise on the invoice.”

“Before I build a router I spend the cheaper levers: cache the prefix, then sweep effort on one model. A router has to beat that whole curve on cost per completed task, not just one model’s default.

Then I route in two stages. Hard rules first, as yes or no: does the prompt fit, do the tools work on that tier, may this data go there. Then a cheap classifier — Haiku 5.5 at low effort with a structured output — that labels the request, and escalates when it’s unsure, because a wrong ‘easy’ costs a bad answer and a wrong ‘hard’ costs a few tenths of a cent.

The decision rule is two prices. At 800 tokens in and 250 out, a Haiku 5.5 call is about $0.0002 and a Sonnet 5.5 call about $0.004. If a wrong answer costs me 80 cents, the small tier pays only while its extra wrong rate stays under about half a percent. I measure that from a weekly judged sample, not from the router’s confidence.

On a 60/25/15 mix that’s about $2,400 per million requests against $4,100 all on Sonnet — roughly 40% off — and I keep each conversation on one tier, because caches and thinking blocks don’t move between models.”

“On current Claude models thinking is adaptive: the model decides how much to reason on each turn, and output_config.effort — low, medium, high, xhigh, max — bounds it. There’s no token budget for thinking any more.

Effort moves three things. Cost, because thinking is billed as output at five times the input price whether I display it or not. Latency, because thinking runs before the first text token. And behaviour: lower effort makes fewer, more consolidated tool calls; higher effort explores and checks more, which can sometimes mean fewer turns overall. And max_tokens has to fit thinking plus the answer.

I pick it per route from a sweep on my own evals, judged by cost per completed task, not cost per call: low for classification and subagents, medium or high for most work, xhigh for coding agents. A useful pattern is to run low and re-run only the failures at high, if I have a checker that catches them.

Two rules: set it explicitly after a model change — Opus 5.5 defaults to medium, Sonnet 5.5 to high — and pin it per route, because changing effort between requests invalidates the cache.”