Evaluation & Quality

"Three evaluation strategies:

• (1)Outcome-based eval — I don't care about the path, I care about the result. Did the agent produce the correct answer? Was the customer satisfied? Use golden sets with expected outcomes and LLM-as-judge for quality.
• (2)Trajectory-based eval — for important tasks, check that the agent took reasonable steps. Did it call the right tools? Did it ask for the right data? Score the trajectory, not just the output.
• (3)A/B testing — run the new version on 10% of traffic, compare CSAT, accuracy, cost, latency vs the current version. Only promote if all metrics are equal or better. The key: separate what from how. The outcome must be deterministic (correct answer); the path can vary."

"Multi-layer:

• (1)Citation enforcement — every factual claim must cite a retrieved chunk. Claims without citations are flagged.
• (2)Claim verification — extract individual claims from the answer, check each against the source material. Score = supported claims / total claims.
• (3)Self-consistency — generate the answer 3 times with temperature > 0. If answers diverge significantly, confidence is low — flag for human review.
• (4)Programmatic checks — for numbers, dates, prices: verify against the source data directly. No LLM needed.
• (5)'I don't know' calibration — train the model to say 'I don't have enough information' when context is insufficient. Measure the rate and verify it's appropriate."

“Three layers, answering three different questions.

Offline evals answer ‘is this change better?’ — a curated set with graded rubrics, run on every prompt or model change before it ships. Small, high quality, human-labelled.

Regression suites answer ‘did we break something we already fixed?’ Every production incident and every red-team finding becomes a case here. This set only grows, and it’s what makes shipping fast, because it’s what lets you change a prompt on a Friday.

Live signals answer ‘is it actually working?’ — containment rate, escalation rate, resolution time, repeat contacts. They’re the only ones that reflect reality, and they lag, so you can’t ship on them alone.

The organising idea is that a case moves left over its life. Something fails in production, shows up in a live signal, becomes a regression case, and if it represents a class it graduates into the offline set. That flow is the system — the three layers are just where cases sit.”

“A judge is a model, so it has every failure mode a model has, plus one that’s worse: you stop reading outputs, because now you have a number.

So I treat it as a classifier that must be validated. Take a few hundred cases, have humans grade them, have the judge grade them, and measure agreement — Cohen’s kappa, not raw accuracy, because on a skewed set ‘always say pass’ scores ninety percent. Below about 0.6 the judge isn’t ready, and usually neither is the rubric.

The biases worth naming: position bias in pairwise comparisons, so swap the order and re-run. Verbosity bias, where longer answers score higher, so control for length. Self-preference, where a model favours its own family’s outputs, so don’t judge with the model that generated.

And the rubric matters more than the model. A binary criterion with a clear definition gets high agreement; ‘rate helpfulness one to ten’ gets noise. Then re-validate whenever the judge’s model version moves, because a judge drifting quietly is what invalidates months of results.”

“Two weeks is plenty for a real eval set, as long as you stop trying to build a big one.

Day one: pull two hundred real requests from whatever already exists — logs, tickets, transcripts, the questions the sales team demoed. Stratify by intent and by difficulty, and deliberately over-sample the strange ones, because easy cases don’t discriminate between two versions.

Then expert time, spent precisely. The customer’s expert doesn’t have days, they have an afternoon. So I don’t ask them to grade two hundred outputs. I ask them to answer forty cases the way they’d want the agent to — that gives me references — and to tell me what ‘wrong’ means in their domain. That conversation is worth more than the labels.

From there the forty become the seed, the model generates variations a human accepts or rejects in bulk, and the remaining hundred and sixty are graded by a judge validated against those forty.

Then ship version one with the caveat that it’s version one. A perfect eval set delivered in month three is worth less than a rough one that catches a regression in week three.”

“Outcome eval asks whether it got the right answer. Trajectory eval asks whether it got there sensibly. You need both, and they fail in opposite directions.

Outcome-only leaves you blind to an agent that got the right answer by calling twelve tools, reading data it had no business reading, and getting lucky. That agent is slow, expensive and one prompt change from being wrong — and all of that risk is invisible in the score.

Trajectory-only over-constrains. There are usually several good paths, and if you assert a specific tool sequence you’ll fail correct runs and end up maintaining a change-detector instead of an eval suite.

So outcome is the gate — it decides pass or fail. Trajectory gets scored as properties rather than as a sequence: did it stay inside the step budget, did it call anything it wasn’t entitled to, did it repeat a call with identical arguments, did it act before checking. All checkable without asserting one right path.

And when the outcome is right but the trajectory is bad, I don’t call that a pass. That’s a latent failure, and it gets its own metric.”