LLM-as-a-judge: having one AI evaluate another AI's answers
When you can no longer compare word for word, you can ask an LLM to grade the answers. Powerful, but full of traps: here is how to use it without fooling yourself.
When you test an AI assistant, human review of every answer quickly becomes impossible. One technique has become the standard way to scale: LLM-as-a-judge, or LLM evaluator. A second model is asked to grade the first model’s answers against precise criteria.
The idea is simple. Applying it well is much less so.
How it works
For each test scenario, the LLM judge receives:
- the question asked;
- the answer produced by the agent under test;
- optionally a reference answer or expected elements;
- an explicit grading rubric.
It returns a score and a justification, ideally in a structured format (JSON) that the test can read.
A good rubric makes a good judge
The classic mistake is to ask “Is this answer good? Score from 1 to 10”. The result is vague and unstable. A good rubric breaks the judgement down into binary criteria, or criteria with only a few levels:
- Does the answer contain information X? (yes / no)
- Does it contain a claim that isn’t in the source document? (yes / no)
- Does it stay within the product’s scope? (yes / no)
- Is the tone appropriate for a customer? (yes / partly / no)
Closed questions give more reproducible verdicts, and failures that are easier to understand.
The traps to know about
An LLM judge has its own biases, documented in the literature on model evaluation:
- Length bias: it tends to prefer long answers.
- Position bias: when asked to compare two answers, the order in which they’re presented can sway the verdict.
- Self-preference: it may rate favourably answers produced by a model from its own family.
- Instability: the same evaluation run twice can give two different scores.
How to protect yourself
- Calibrate the judge: have humans grade a sample of answers, then compare with the LLM’s verdicts. Until the agreement is satisfactory, rework the rubric.
- Pin the settings: same model, same version, low temperature, and a prompt versioned in the repository like code.
- Swap the order in pairwise comparisons, and only keep consistent verdicts.
- Keep deterministic assertions for everything that allows it (format, tool calls, forbidden words). The judge is only for what truly requires judgement.
- Review the failures: every failure flagged by the judge deserves a human look, at least at the beginning.
Where it fits in your strategy
The LLM judge is a tool for measurement at scale, not an oracle. I use it to track how quality evolves from one version to the next across hundreds of scenarios, and to spot regressions. Release decisions, on the other hand, remain based on verifiable criteria and on human review of the critical cases.
In a future article, I’ll show how to plug this kind of evaluation into CI, alongside your usual Playwright tests.
