How to Answer
"Three evaluation strategies:
- (1)Outcome-based eval — I don't care about the path, I care about the result. Did the agent produce the correct answer? Was the customer satisfied? Use golden sets with expected outcomes and LLM-as-judge for quality.
- (2)Trajectory-based eval — for important tasks, check that the agent took reasonable steps. Did it call the right tools? Did it ask for the right data? Score the trajectory, not just the output.
- (3)A/B testing — run the new version on 10% of traffic, compare CSAT, accuracy, cost, latency vs the current version. Only promote if all metrics are equal or better. The key: separate what from how. The outcome must be deterministic (correct answer); the path can vary."