How to Answer
“A judge is a model, so it has every failure mode a model has, plus one that’s worse: you stop reading outputs, because now you have a number.
So I treat it as a classifier that must be validated. Take a few hundred cases, have humans grade them, have the judge grade them, and measure agreement — Cohen’s kappa, not raw accuracy, because on a skewed set ‘always say pass’ scores ninety percent. Below about 0.6 the judge isn’t ready, and usually neither is the rubric.
The biases worth naming: position bias in pairwise comparisons, so swap the order and re-run. Verbosity bias, where longer answers score higher, so control for length. Self-preference, where a model favours its own family’s outputs, so don’t judge with the model that generated.
And the rubric matters more than the model. A binary criterion with a clear definition gets high agreement; ‘rate helpfulness one to ten’ gets noise. Then re-validate whenever the judge’s model version moves, because a judge drifting quietly is what invalidates months of results.”