1:1 mentoring with Big Tech AI engineers
Q44Premium

Design the evaluation infrastructure for production agents at enterprise scale.

Evaluation & Quality

EvaluationProductionMonitoringArchitecture

Asked at Sierra · Scale AI · Databricks

How to Answer

“Three layers, answering three different questions.

Offline evals answer ‘is this change better?’ — a curated set with graded rubrics, run on every prompt or model change before it ships. Small, high quality, human-labelled.

Regression suites answer ‘did we break something we already fixed?’ Every production incident and every red-team finding becomes a case here. This set only grows, and it’s what makes shipping fast, because it’s what lets you change a prompt on a Friday.

Live signals answer ‘is it actually working?’ — containment rate, escalation rate, resolution time, repeat contacts. They’re the only ones that reflect reality, and they lag, so you can’t ship on them alone.

The organising idea is that a case moves left over its life. Something fails in production, shows up in a live signal, becomes a regression case, and if it represents a class it graduates into the offline set. That flow is the system — the three layers are just where cases sit.”

The deep dive — diagrams, tradeoff tables, and the follow-up trap

Loading…