Scale & Cost

"Cost reduction playbook:

• (1)Model tiering — use Flash/Haiku for 80% of tasks (classification, simple Q&A), Pro/Opus only for complex reasoning. That alone cuts 60-70%.
• (2)Prompt caching — system prompt + tool definitions are identical across calls. Cache them. Saves 80% on input tokens for repeated calls.
• (3)Semantic caching — if someone asked a similar question in the last 24h, serve from cache. Expected 30-40% hit rate for support use cases.
• (4)Context trimming — summarize old tool observations instead of keeping full text.
• (5)Batch API — for non-urgent tasks, use the batch endpoint at 50% discount. Combined, these typically achieve 5-8x cost reduction."

"The architecture shouldn't change — the infrastructure scales.:

• (1)Stateless agents on Cloud Run/GKE — auto-scale horizontally.
• (2)Queue-based ingestion via Pub/Sub — decouples request rate from processing rate.
• (3)Provisioned throughput on Vertex AI for predictable latency under load.
• (4)Sharded vector indices — partition by tenant or region.
• (5)Regional deployment — deploy in 3 regions, route by user geography.
• (6)Cache layers become critical — semantic cache hit rate determines your cost scaling. The key insight: at 10K users you can afford to be synchronous. At 1M, everything must be async with graceful degradation."

“I’d start from the budget and work backwards, because at a tight enough number the budget picks the architecture. Say twenty cents a claim at fifty thousand claims a month.

At twenty cents I can’t put every claim through a frontier model with a large context. So: a triage step on a small model classifies the claim and retrieves only the policy sections that apply. Keeping retrieval tight is the biggest token lever, because retrieved text is what actually fills the prompt.

Then a decision tier. Clean claims that match a policy rule with high confidence get auto-approved by the small model. Ambiguous ones escalate to the frontier model with full context. Anything the frontier model isn’t confident about, or anything above a dollar threshold, goes to a human.

And then I’d point out that the economics live in that distribution, not in the prompts. Human review is the expensive line by an order of magnitude — if I want the cost down, I move the escalation rate, not the token count.”

“It is a prefix cache. The provider hashes the front of your prompt, and if it matches something it has already processed it reuses the computed attention state instead of recomputing it. Two things follow from the word prefix: it only works from the very start of the prompt, and it breaks the moment any byte before the cached point changes.

So the whole game is prompt layout. Stable things first — system prompt, tool schemas, few-shot examples, the long document. Volatile things last — the user’s message, the timestamp, anything personalised.

The classic mistake is rendering ‘today is 2026-08-03 14:32:11’ or the user’s name into the top of the system prompt. That is a miss on every single call, and nobody notices, because the system still works — it is just several times more expensive than it should be.

Caveats: there’s a minimum cacheable length, a cache write costs a little more than a normal token, and the TTL is minutes. So it pays on repeated traffic, not on a long tail of one-off requests.”

“The users are waiting synchronously, so this is a latency-under-throughput problem, not a throughput problem — and that rules out the obvious design.

Static batching is wrong here: you wait for a batch to fill, run it, and everyone pays for the longest sequence in it. What I want is continuous batching — the scheduler admits new requests at each decoding step and evicts finished ones, so a short request never waits behind a long one.

Concretely: an admission queue, a scheduler that runs every decoding step, and a paged KV cache so capacity is bounded by memory pages rather than by the longest sequence anyone might send. The batch isn’t 100 because someone wrote 100 — it is whatever fits in KV memory at the context length I actually serve.

The knob that matters is how much queueing delay I’ll accept. I’d set a maximum wait, something like ten milliseconds, and a maximum batch, and fire on whichever comes first. And I’d shed load rather than let the queue grow, because an unbounded queue makes everyone’s latency bad instead of failing a few requests quickly.”

“Three-x overnight with no deploy means volume, retries, or context growth — and I can tell which within about ten minutes if the traces carry the right attributes.

First question: did request count go up, or did cost per request go up? If it’s per-request, it isn’t a traffic story at all. Then I split tokens per request into prompt and completion. Prompt tokens climbing against a flat request count is almost always context growth — a retrieval change, a memory system that started including more history, or a prefix cache that stopped hitting.

The one that catches people is retries. A tool starts timing out, the agent retries, every retry replays the whole conversation, and spend triples while request count doesn’t move at all. That’s why I want tokens attributed per step and per trace, not a monthly total on a provider dashboard.

And the cheap safeguard is a per-trace token budget with an alert, so this is a page at two-x rather than a surprise on the invoice.”