The number went up and the complaints went up with it. That is not a paradox — it is the normal outcome of measuring one thing and shipping another.
Almost every team building on models eventually reports the same experience: eval scores improving while user satisfaction does not. It is rarely a broken metric. It is a metric that stopped describing production some weeks ago, and nobody noticed because it kept returning a number.
When the score and the complaints disagree, treat the complaints as ground truth and the eval as the suspect. Usually the eval set no longer looks like your traffic — it was built in March, the mix moved, and you have been optimising against a distribution that no longer exists.
Three questions, three different systems
“How do you evaluate it?” is three questions wearing one coat, and they need different machinery. Conflating them is why teams end up with one spreadsheet that answers none of them well.
That last line is the loop worth building deliberately: production tells you what real requests look like, which keeps the offline set honest, which is the only thing that makes the score mean anything. The data flywheel is that loop written down.
The metrics, and what each one can actually tell you
| Metric | Answers | Blind to |
|---|---|---|
| Task success rate | Did it do the thing? | How it got there — including reading data it should not have |
| Retrieval recall@k | Was the answer even available to the model? | Whether the model then used it |
| Groundedness / citation rate | Is each claim traceable to a source? | Whether the cited source is correct |
| Trajectory score | Were the steps sensible? | Whether the outcome was right |
| Escalation rate | How often it hands off | Whether escalating was the right call |
| Cost and latency per completed task | Whether it is viable | Quality entirely — always pair it |
| User feedback | Whether people are happy | Silent failures — most unhappy users just leave |
The pairing rule matters more than any single row. Success rate without cost hides an agent that burns four dollars a task; cost without success rate hides one that got cheap by giving up early. Agent evaluation and cost control are the two halves.
Measure retrieval separately, or you will blame the model
In any RAG system, the first question is not “was the answer good” but “was the answer reachable”. Those are different failures with different fixes, and mixing them wastes weeks.
# Two numbers, in this order. The second only means something if the first is high.
recall = retrieval_recall_at_k(EVAL, k=6) # was the passage retrieved at all?
quality = answer_quality(EVAL) # given it, did the model use it well?
print(f"recall@6 {recall:.0%} answer quality {quality:.0%}")
# recall 61%, quality 88% -> chunking and retrieval problem, not a model problem
# recall 94%, quality 62% -> now it is a prompt, context or model problem
The first row is the common one, and it is almost always a boundary in the wrong place — see chunking strategies, then RAG evaluation and RAGAS for the component-level breakdown.
An LLM judge is a classifier, so validate it like one
A judge has every failure mode a model has, plus one that is worse: you stop reading outputs, because now there is a number. It can still be the right tool — but only after you know how well it agrees with you.
- Have humans grade a few hundred cases, have the judge grade the same ones, and measure agreement with Cohen’s kappa rather than raw accuracy. On a skewed set, 90% accuracy can be worse than guessing.
- Give it a rubric with examples, not an adjective. “Rate helpfulness 1–5” produces noise; a rubric with a worked example per level produces a signal.
- Watch for position and verbosity bias. Judges favour the first option and the longer answer. Randomise order; check whether your winner is just wordier.
- Re-validate when you change the judge model. A judge upgrade silently moves every historical score.
The eval set is the artifact, not the harness
Teams over-invest in tooling and under-invest in the two hundred examples that decide whether the tooling tells the truth. Two weeks is plenty, provided you stop trying to build a big one.
Draw it from real traffic
Logs, tickets, transcripts, the questions sales demoed. Questions written while looking at a document retrieve that document far too easily and inflate every number downstream.
Stratify, and over-sample the strange
By intent and by difficulty. Easy cases teach you nothing after the first week; the awkward ten percent is where every regression shows up first.
Grow it from incidents
Every production failure becomes a permanent case. This is the cheapest quality mechanism available and the one most often skipped — it is also what turns the second layer of the diagram into something real.
Version it with the prompt
A score is meaningless without knowing which set produced it. When the set changes, old numbers are not comparable, and someone will compare them anyway unless the version makes it obvious.
Pick metrics in pairs so one cannot be gamed by ignoring the other, split retrieval from generation before you debug either, validate any judge against human labels before trusting its number, and rebuild the eval set from real traffic on a schedule — because the failure mode is not a wrong metric, it is a metric that quietly stopped describing your users.
The full treatment, with the dashboards, the rubric templates and the production wiring, is in the premium guide: evaluation and observability end to end, the metrics that belong on a dashboard, and hallucination and off-brand detection. For the interview version of this topic, the evaluation questions are their own group.