Hard / Curveball

"Four areas change:

• (1)Embeddings — switch to multilingual model (e.g., multilingual-e5-large). Test retrieval quality per language — it degrades for low-resource languages.
• (2)Chunking — sentence boundaries differ. Use language-aware tokenizers. Arabic is RTL — ensure your pipeline handles it.
• (3)Evaluation — build golden sets per language. LLM-as-judge must be multilingual or use per-language judges.
• (4)Tool schemas — keep in English (models handle English schemas best). But response generation should be in the user's language. Add 'Respond in {detected_language}' to the system prompt. Cost implication: multilingual embeddings are larger; retrieval quality is usually 10-15% lower for non-English."

"Prevention layers:

• (1)Action limits — no single agent action can exceed $X without human approval. For financial actions, implement 4-eyes principle.
• (2)Idempotency — every write action has an idempotency key. Retries don't double-execute.
• (3)Reversibility — prefer reversible actions. Don't delete; soft-delete. Don't send; draft.
• (4)Insurance via audit trail — full decision trace proves the agent followed its instructions. Liability typically sits with the company that deployed the agent, not the model provider — your terms of service should reflect this.
• (5)Graceful degradation — when confidence is below threshold, the agent must route to a human, not guess. The meta-answer: the agent should never be the sole decision-maker for high-value actions."

"This is why architecture matters:

• (1)MCP servers don't change — they're model-agnostic. All tool integrations survive.
• (2)Prompts need tuning — each model family responds differently to system prompts. Budget 1-2 weeks for prompt engineering.
• (3)Evaluation is the safety net — run the golden set on Gemini, compare scores vs Claude. Only migrate when quality parity is confirmed.
• (4)Agent loop is the same — tool_use/tool_result follows the same pattern across providers with minor field name changes. The lesson: decouple your intelligence layer from your integration layer. The model is a replaceable component."

"Workflow-per-agent pattern:

• (1)Router agent — a thin classifier that receives the user request, identifies the workflow type, and dispatches to the appropriate specialist agent.
• (2)Specialist agents — each workflow has its own agent with its own system prompt, tools, and eval criteria. Deployed as separate services.
• (3)Shared infrastructure — all agents share the same MCP servers, vector store, and observability pipeline.
• (4)Configuration-driven — agent behavior defined in YAML/JSON configs, not code. New workflows added by writing a config + prompt, not deploying new code. This avoids the monolith while keeping infrastructure costs shared."

"Agent A/B testing framework:

• (1)Split by user cohort, not by request — the same user should get the same variant for consistency.
• (2)Metrics to compare: task completion rate, CSAT, cost per task, latency, hallucination rate. Need all to be equal or better, not just one.
• (3)Shadow mode first — run variant B on all traffic but only show variant A's output. Compare offline. Only promote B to live when confident.
• (4)Statistical significance — agent outputs are high-variance. Need larger sample sizes than UI tests. Typically 1000+ tasks per variant.
• (5)Prompt version tracking — every prompt change is a versioned artifact in git. A/B test maps to prompt version A vs B."

"Never in the agent, never in the prompt.:

• (1)Secret Manager — all credentials stored in GCP Secret Manager, rotated automatically.
• (2)Per-MCP service accounts — each MCP server has its own service account with minimum required permissions.
• (3)Workload Identity Federation — for MCP servers running on GKE/Cloud Run, no static keys at all. Identity is asserted by the platform.
• (4)User-scoped tokens — when the MCP needs to act as the user (e.g., read their Gmail), use OAuth with the user's delegated token, stored per-session, never persisted.
• (5)The LLM never sees credentials — it emits 'I want to call tool X with args Y.' Your dispatcher adds the credentials. Separation of intent from execution."

"This is a tone/brand problem, not an accuracy problem.:

• (1)Brand voice system prompt — define the voice: 'Professional but approachable. No jargon. Lead with the business impact, not the technical detail.'
• (2)Audience-aware formatting — detect the recipient's role. VP gets a 3-bullet summary with metrics. Engineer gets the detailed analysis.
• (3)Tone classifier — post-generation filter that scores tone (formal/casual, confident/hedging, concise/verbose). Flag outputs that don't match the target profile.
• (4)Feedback loop — VP's edits are gold training data. Analyze what they change — it's usually not the facts, it's the framing."

"Structured debugging:

• (1)Segment failures — by task type, user segment, time of day, input length. Is it 20% across the board or 80% on one category?
• (2)Read the traces — pull 20 failed traces. Classify: tool error? Model hallucination? Timeout? Wrong tool selection? Context overflow?
• (3)Find the common pattern — usually 1-2 root causes explain 80% of failures.
• (4)Fix and verify — fix the root cause, replay the failed traces, confirm they now pass.
• (5)Add regression tests — add the failed cases to the golden set so this never regresses.
• (6)Monitor — set an alert on failure rate. The meta-insight: the observability you built before production is what makes this debugging possible in 1 hour instead of 1 week."

":

• (1)A golden evaluation set — 200 tasks with expected outputs. If I can't measure quality, I can't ship safely. This is the most under-invested thing in AI projects.
• (2)A kill switch — one button that routes all requests to humans. When things go wrong (and they will), I need to stop harm instantly.
• (3)An audit trail — every decision the agent makes, with the full trace of why. For compliance, for debugging, and for the inevitable 'why did the agent do X?' question from the customer's CISO. Everything else — caching, scaling, fancy UX — can come after launch. These three are non-negotiable for responsible deployment."

“That’s the metric telling me it has stopped measuring the thing I care about. So I’d treat the complaints as ground truth and the eval as the thing under suspicion.

It’s usually one of three things. The eval set no longer looks like production — it was built in March, the traffic mix moved, and I’m optimising against a distribution that doesn’t exist any more. Or the rubric measures something merely correlated with quality, most often length or confidence, and if you optimise hard enough against a proxy you get the proxy. Or the complaints are about something the eval never covered at all: latency, tone, being confidently wrong in a way the rubric scores as correct.

So the move is to go and read fifty complaints and label what actually went wrong. Then check whether the eval set contains a single case like that. It usually doesn’t, and now I know what to add.

Then I’d fix the process rather than the set: sample production traffic into the eval set continuously, so it drifts with reality instead of away from it.”

“I’d push back on ‘more autonomous’ as a goal, because it isn’t one. Autonomy is a cost you pay to get speed. The real question is which specific approvals are worth removing.

So I’d make it concrete: list every action the agent can take, and score two things per action — how reversible is it, and how accurate are we, measured rather than asserted. Cheap and reversible with high measured accuracy: automate it. Expensive and irreversible: keep the human permanently, and stop treating that as a failure. The middle is where the work is, and that’s where you buy confidence with evaluation before you spend it on autonomy.

The mechanism is graduated and per action type: log-only, then propose-and-approve, then auto-with-undo, then auto. Each promotion has an entry criterion — a measured agreement rate, on a real sample, over a real period.

And the thing worth saying out loud: ‘more autonomous’ usually means ‘the approval queue is a bottleneck’. Sometimes the right answer is a better queue, not less oversight.”

“I’d sort the system into two piles: things that get more valuable as models improve, and scaffolding that exists only because the current model is limited.

The durable pile: evals, because a more capable system needs them more, not less. Data and retrieval — proprietary context is the moat, and no model gets it for free. Tool surface and integrations, because a smarter model still cannot call an API you never built. Auth, permissions and audit, because those are about who is allowed, and no model improvement changes that. And the product decisions: what should happen, and what a good outcome even is.

The scaffolding pile: chaining a task into six steps because the model can’t hold it in one. Aggressive context compression. Routing between three model sizes. A fine-tune that exists to fix a formatting problem. Those are compensations, and every one of them is work I will eventually delete.

So I’d build the scaffolding when I need it, keep it thin, and keep it behind an interface so deleting it is a small change. What I would not do is architect the whole system around a limitation with a twelve-month half-life.”