The “judge” model receives the question, the answer to evaluate and an explicit rubric (accuracy, tone, following instructions), and returns a score or a verdict.
The method makes it possible to test thousands of answers, where human review handles a few dozen. But the judge has its own biases: it prefers long answers, can be swayed by the order of presentation and sometimes gets it wrong.
Good practices
- Calibrate the judge on a set of hand-labelled cases, and measure how often it agrees with humans.
- Ask for reasoning before the verdict.
- Keep deterministic tests wherever possible.