Tradeoffs & Scenarios

"Four levers:

• (1)Parallel tool calls — if tools are independent, call them simultaneously. 3 sequential x 500ms = 1.5s → 3 parallel = 500ms.
• (2)Semantic cache — if this query was asked before, serve from cache in <100ms.
• (3)Model tiering — use Flash/Haiku for the routing step, only escalate to Pro for the final synthesis. Flash is 5-10x faster.
• (4)Streaming — start sending tokens to the user while the last tool call is still running. Perceived latency drops dramatically."

"Multiple layers:

• (1)Schema exposure — don't give the agent all 500 tables. Give it a curated catalog of 15-20 relevant tables with descriptions.
• (2)Query validation — the MCP server validates every query before execution: no SELECT *, no full table scans, max rows limit, timeout after 30s.
• (3)Dry-run cost estimation — BigQuery can estimate bytes scanned before running. Reject queries that would scan > 10GB.
• (4)Per-user quotas — 100 queries/day, max 50GB scanned/day.
• (5)Row-level security — query runs as the user, not a super-account."

"Five concrete changes:

• (1)BAA — must have a signed Business Associate Agreement with every vendor in the chain (GCP, model provider). Vertex AI supports BAA.
• (2)PHI handling — all patient data redacted by Cloud DLP before any LLM call. The model never sees raw PHI.
• (3)Encryption — CMEK for data at rest, mTLS for transit, VPC-SC perimeter around the entire system.
• (4)Audit trail — every access to PHI logged with who, when, what, why. Retained 6 years.
• (5)Zero data retention — must confirm model provider doesn't retain prompts/responses for training. Vertex AI ZDR is on by default. I'd also add access reviews every 90 days and annual penetration testing."

"Common causes of quality drift:

• (1)Data drift — the knowledge base hasn't been updated. New products, pricing changes, policy updates aren't in the RAG corpus. Fix: automated re-indexing pipeline.
• (2)Usage pattern drift — users are asking questions the agent wasn't designed for. Fix: classify query types, track 'out-of-scope' rate.
• (3)Model version change — the provider silently updated the model. Fix: pin model versions, run golden set on every version change.
• (4)Prompt injection at scale — users found ways to jailbreak. Fix: review flagged outputs. Debug process: run the golden set from week 1 — if it still passes, the issue is data/usage drift, not model quality."

"Defense in depth:

• (1)Channel separation — system prompt is in the 'system' role, retrieved documents are wrapped in <document> tags in the 'user' role. The model is instructed to treat document content as data, never as instructions.
• (2)Input sanitization — scan retrieved docs for known injection patterns before including in prompt.
• (3)Output validation — a lightweight classifier checks if the response contains system prompt content, internal instructions, or out-of-scope tool calls.
• (4)Dual-LLM pattern — for high-stakes outputs, a second model reviews the first model's output for policy violations.
• (5)Behavioral testing — red-team the agent weekly with known injection attacks."

“I split it into two numbers, because they are not the same problem. There is total time, and there is time to first signal. People tolerate forty seconds far better than they tolerate eight seconds of a blank spinner. So the target isn’t ‘make it fast’, it’s ‘never be silent for more than about a second’.

Concretely: stream tokens as they come, and emit a progress event for every tool call — ‘searching four knowledge bases’, ‘reading the Q3 contract’. That is real work, and showing it is honest rather than decorative.

Then I ask whether they should be waiting at all. Past about thirty seconds, make it a job: return a handle immediately, do the work in the background, notify on completion. That is a product decision more than an engineering one.

And let them out. Cancel has to actually cancel — propagate the abort into the in-flight tool call. Hiding the spinner is not cancelling.”

“First I find out what ‘no data leaving’ actually means to them, because there are three versions and they differ by an order of magnitude in cost. One: nothing goes to a new third party — a managed model endpoint inside the cloud region and tenancy they already use is fine. Two: nothing leaves their VPC — now I’m self-hosting open weights on GPUs in that VPC. Three: air-gapped. That last one is a different product.

Assume version two. The real cost is that I lose the frontier model, and that shows up as a quality regression rather than an outage — which is exactly the kind of thing nobody notices for a month. So the first artefact I build is an eval set against the current system, before we migrate, so the delta is a number we agreed on instead of a complaint in month three.

Then the unglamorous half: who runs those GPUs, who patches them, who is awake at 3am. Self-hosting is an operating commitment, not a deployment step.”

“The first three steps already happened — that’s the entire problem. An agent that takes real actions is a distributed transaction with a language model as the coordinator, and there is no rollback for ‘the money moved’.

So I design for it three ways. Every write tool takes an idempotency key, so a retry after a timeout can’t charge anyone twice. Every step’s result is checkpointed, so a resume starts at step 4 instead of replaying step 1. And every irreversible step has a compensating action written before it ships — refund for charge, cancellation for booking — because undo has to be something you built, not something you hope for.

Then I order the workflow so the irreversible steps land last wherever I can. If steps one to three are reads and step four sends the email, a failure at four costs nothing at all.”