How to Answer
“Three layers, answering three different questions.
Offline evals answer ‘is this change better?’ — a curated set with graded rubrics, run on every prompt or model change before it ships. Small, high quality, human-labelled.
Regression suites answer ‘did we break something we already fixed?’ Every production incident and every red-team finding becomes a case here. This set only grows, and it’s what makes shipping fast, because it’s what lets you change a prompt on a Friday.
Live signals answer ‘is it actually working?’ — containment rate, escalation rate, resolution time, repeat contacts. They’re the only ones that reflect reality, and they lag, so you can’t ship on them alone.
The organising idea is that a case moves left over its life. Something fails in production, shows up in a live signal, becomes a regression case, and if it represents a class it graduates into the offline set. That flow is the system — the three layers are just where cases sit.”