How to Answer
“Outcome eval asks whether it got the right answer. Trajectory eval asks whether it got there sensibly. You need both, and they fail in opposite directions.
Outcome-only leaves you blind to an agent that got the right answer by calling twelve tools, reading data it had no business reading, and getting lucky. That agent is slow, expensive and one prompt change from being wrong — and all of that risk is invisible in the score.
Trajectory-only over-constrains. There are usually several good paths, and if you assert a specific tool sequence you’ll fail correct runs and end up maintaining a change-detector instead of an eval suite.
So outcome is the gate — it decides pass or fail. Trajectory gets scored as properties rather than as a sequence: did it stay inside the step budget, did it call anything it wasn’t entitled to, did it repeat a call with identical arguments, did it act before checking. All checkable without asserting one right path.
And when the outcome is right but the trajectory is bad, I don’t call that a pass. That’s a latent failure, and it gets its own metric.”