How do you do A/B testing on an agent? It's not like testing a button color.

"Agent A/B testing framework:

• (1)Split by user cohort, not by request — the same user should get the same variant for consistency.
• (2)Metrics to compare: task completion rate, CSAT, cost per task, latency, hallucination rate. Need all to be equal or better, not just one.
• (3)Shadow mode first — run variant B on all traffic but only show variant A's output. Compare offline. Only promote B to live when confident.
• (4)Statistical significance — agent outputs are high-variance. Need larger sample sizes than UI tests. Typically 1000+ tasks per variant.
• (5)Prompt version tracking — every prompt change is a versioned artifact in git. A/B test maps to prompt version A vs B."