LLM Evaluation Metrics That Actually Matter

Almost every team reports the same thing sooner or later: eval scores rising while user complaints rise too. It is rarely a broken metric. It is a test set that stopped looking like production weeks ago and kept returning a number. When the score and the complaints disagree, treat the complaints as ground truth and the eval as the suspect.

Choose metrics in pairs

The pairing rule matters more than any row. Success without cost hides an agent that got expensive; cost without success hides one that got cheap by giving up. So count cost per resolved task:

Claude Opus 5.5 list prices, $4 in and $20 out per million tokens, no caching. The variant is 20% cheaper per task and 7% dearer per outcome, before the 200 extra tasks a person now handles (Metrics & Dashboards).

Score retrieval before the answer

In a RAG system the first question is not “was the answer good” but “was it reachable”. Recall@k needs no judge: label each golden question with the source span that answers it, and count how often that span is in the top k. Store spans, not chunk ids, because ids change with every re-chunk and a stale label reads as a retrieval failure. MRR and nDCG add rank order; precision@k says how much of what the model reads is useful (RAG Evaluation). The usual cause of a low first row is a boundary in the wrong place: RAG chunking strategies.

Use RAGAS for the claims a rule cannot check

import anthropic
from ragas.llms.base import llm_factory
from ragas.metrics.collections import ContextRecall, Faithfulness

client = anthropic.AsyncAnthropic()           # ascore needs an async client
judge = llm_factory("claude-sonnet-5-5", provider="anthropic", client=client)
for arg in ("temperature", "top_p"):          # ragas sends both; current models refuse them
    judge.model_args.pop(arg, None)
# ragas parses content[0].text: no thinking first (Sonnet 5.5 rejects "disabled")
judge.model_args["thinking"] = {"type": "between_tools"}

async def score(row):
    f = await Faithfulness(llm=judge).ascore(
        user_input=row["q"], response=row["answer"],
        retrieved_contexts=row["contexts"])
    r = await ContextRecall(llm=judge).ascore(
        user_input=row["q"], retrieved_contexts=row["contexts"],
        reference=row["reference"])          # a human-written answer
    return f.value, r.value                   # floats from 0 to 1, or NaN
# asyncio.run(score(row)) for one row; asyncio.gather for a batch

RAGAS 0.4.3 is still the newest release on PyPI (checked 2026-10-21); install it with pip install "ragas==0.4.3" "langchain-community==0.4.1", because langchain-community 0.4.2 removed a module ragas imports. Run end to end in a clean install with a scripted judge, not a live API call; the Sonnet 5.5 thinking settings are from Anthropic’s thinking docs. Make one live call and check the block types before a batch.

Each metric is a judge reading fields of one logged row, so a run has a price: about 11 judge calls per row for the four core metrics at k = 5 (RAGAS). Context precision and recall need a reference a person wrote, so they run on the golden set only; faithfulness and answer relevancy can run on sampled live traffic. A reply with no claims scores NaN, not 1: print the NaN count beside every mean, or an alert on the mean never fires.

Validate the judge like the classifier it is

from sklearn.metrics import accuracy_score, cohen_kappa_score

human = ["pass"] * 180 + ["fail"] * 20          # 200 human-labelled answers
lazy  = ["pass"] * 200                          # a judge that always says pass
judge = ["pass"] * 172 + ["fail"] * 8 + ["pass"] * 10 + ["fail"] * 10

for name, j in (("always-pass", lazy), ("judge", judge)):
    print(f"{name:>11}  accuracy {accuracy_score(human, j):.0%}"
          f"  kappa {cohen_kappa_score(human, j):.2f}")
# always-pass  accuracy 90%  kappa 0.00
#       judge  accuracy 91%  kappa 0.48

Same 200 answers, 20 of them failures by human label; run on scikit-learn 1.9.1.

On a skewed set, accuracy flatters a judge, so measure agreement with Cohen’s kappa, which subtracts the agreement chance alone would give. Before a judge gates a release, have people label a few dozen to a hundred cases; the MT-Bench authors found a strong judge agreed with human raters over 80% of the time, about as often as humans agreed with each other (The Eval Stack).

Give it one yes/no question per criterion with a worked example, not “rate helpfulness 1 to 10”. Pin the judge model and the rubric version, and re-measure agreement when either changes: a judge upgrade silently moves every historical score.

Grade agents on the end state, k times

An agent takes a different path each run and can land on the right result by luck. Grade the world it left behind in code (exactly one refund row, the right amount), then check the path against constraints such as “no issue_refund before verify_identity”, and run each task several times. For anything customer-facing, gate on pass^k, how often it succeeds every time (Grading Agents). Keep cost and steps per task in the same report: a prompt change can hold success at 92% and double the cost.

Size the set for the change you need to see

Margin on a pass rate near 0.85, 1.96 × √(p(1 − p) / n); the same table as RAG Evaluation.

A 3-point gain on 200 questions is inside the noise. Compare two systems question by question instead: count the questions only A gets right and only B gets right. Twelve disagreements split nine to three are not yet a result; thirty split twenty-five to five are. A launch gate should test the lower bound of the interval, not the point estimate (RAG Best Practices).

Build the golden set; it is the artifact

1. Draw from trafficLogs, tickets, transcripts. Questions written from a document flatter search.
2. Over-sample the strangeStratify by intent; keep about one case in five an edge case or an attack.
3. Grow from incidentsEvery confirmed failure becomes a permanent case.
4. Version itWith the prompt and the judge: a score means nothing without its set.

{"id": "refund-017",
 "source": "prod trace 2026-08-14, redacted",
 "input": "I was charged twice for order A-1042.",
 "checks": {"must_call": ["get_order"], "must_not_call": ["issue_refund"]},
 "rubric": ["explains the duplicate-charge policy"]}

Use the cheapest grader that can decide each check: a tool call happened or it did not, so code grades it; only tone and policy need a judge. Teams over-invest in harnesses and under-invest in these few hundred cases, which decide whether any harness tells the truth. Never write reference answers with the current model: they reward imitating it and punish a better successor.

Read production signals for how each one lies

Divide by tasks, not by votes, and size the judged sample before you set an alert: at 100 judged answers a day, a fixed 0.80 line on a true 0.85 fires by chance on about 7% of days (RAG Monitoring). Pin exact model ids, cluster live queries weekly for new topics, and send each confirmed failure back down the dashed arrow in the first figure.

Next: The Eval Stack: Offline to Online — CI gates, judge calibration, A/B tests and the traces every layer reads. For retrieval specifically, RAG Evaluation.