How to Answer
“That’s the metric telling me it has stopped measuring the thing I care about. So I’d treat the complaints as ground truth and the eval as the thing under suspicion.
It’s usually one of three things. The eval set no longer looks like production — it was built in March, the traffic mix moved, and I’m optimising against a distribution that doesn’t exist any more. Or the rubric measures something merely correlated with quality, most often length or confidence, and if you optimise hard enough against a proxy you get the proxy. Or the complaints are about something the eval never covered at all: latency, tone, being confidently wrong in a way the rubric scores as correct.
So the move is to go and read fifty complaints and label what actually went wrong. Then check whether the eval set contains a single case like that. It usually doesn’t, and now I know what to add.
Then I’d fix the process rather than the set: sample production traffic into the eval set continuously, so it drifts with reality instead of away from it.”