LLM-as-a-judge: setup, calibration, and failure modes

A team wired up a judge model to score 400 support replies overnight. The dashboard read 94 percent acceptable. Two weeks later a manual review of 60 of the same replies scored them at 68 percent. The judge had been rewarding long, polite answers that never solved the customer's problem. The number was not wrong by accident. Nobody had ever checked it against a person.

Short version

LLM-as-a-judge means using one language model to score the output of another.
It is fast and cheap. It is only trustworthy after you compare it to human scores.
A good judge prompt gives one narrow criterion, a fixed scale, and the reason before the score.
Measure agreement with humans on at least 50 cases before you trust any judge number.
Watch for four biases: position, self-preference, length, and tone.
Recheck the judge whenever you change its prompt or its model version.

What LLM-as-a-judge means

A large language model, or LLM, is a system that produces text from text. LLM-as-a-judge is the practice of giving one model an output produced by your feature and asking it to score that output against written criteria.

It solves a real problem. Human review is slow and expensive. Rules and string matching cannot tell you whether an answer was helpful. A judge model can score 500 outputs in ten minutes for a few dollars.

It also creates a real problem. A judge is a model, so it has the same weaknesses as the model it is scoring. An unchecked judge gives you confident numbers with no relationship to quality, which is worse than having no numbers at all.

Where a judge fits, and where it does not

Use a judge for qualities that need reading and understanding.

Is the answer grounded in the source document?
Does it answer the question the user asked?
Is the tone appropriate for a support reply?
Is answer A better than answer B?

Do not use a judge where a cheaper check works. Never ask a judge whether the JSON is valid, whether the total equals 1284.50, or whether the reply is under 80 words. A schema validator, an equality check, and a word count are faster, free, and always right. Reserve the judge for what only reading can decide.

Writing the judge prompt

Three rules cover most of it: one criterion at a time, a small fixed scale, and reasoning before the score.

Bad judge prompt "Rate this customer support reply from 1 to 10 on overall quality."

That asks for four judgements at once on a scale where nobody can define the gap between 6 and 7. You will get 7s and 8s forever.

Good judge prompt "You are scoring one property of a support reply: groundedness. Groundedness means every factual claim in the reply is supported by the ticket history provided below. Opinions and greetings do not count as claims.

Ticket history: {{context}} Reply to score: {{output}}

Work through each factual claim in the reply. For each one, quote the line in the ticket history that supports it, or write NOT SUPPORTED. Then output JSON only: {"reasoning": "<your check, under 120 words>", "unsupported_claims": ["..."], "score": "grounded" | "partly_grounded" | "not_grounded"}"

What that prompt gets right:

One criterion. Groundedness only. Run separate judges for helpfulness and tone.
The criterion is defined. Including what does not count.
A three-point scale with named levels. People and models both handle three or five named levels better than a 1 to 10 number line.
Reasoning first, score last. Making the model list evidence before deciding produces steadier scores than asking for the number alone.
Structured output. You can parse it, count it, and store it.

Two more settings. Run the judge at temperature 0 so repeated runs are close. And pin the judge model version, for example in a config field judge_model: "<provider>-<model>-2026-04". A judge that silently upgrades makes every historical score incomparable.

Calibrating against humans

Calibration means checking that the judge agrees with people, and fixing the prompt until it does. Do this once before you trust the judge, and again after every change to it.

Pick 50 to 100 outputs that cover the full quality range. Include clear passes, clear failures, and awkward middle cases. A set of only good outputs teaches you nothing.
Have two people score them using the same written criteria you gave the judge. Same scale, same definitions.
Settle the human disagreements first. If two people disagree on 30 percent of cases, your criteria are unclear and the judge cannot possibly do better. Rewrite and rescore.
Run the judge on the same set, blind to the human scores.
Compare. Build a small table of judge score against human score.
Read the disagreements one by one. This is the step that produces the fix. Usually you will find a missing definition, for example that the judge counts a greeting as a claim.
Edit the judge prompt and rerun. Repeat until agreement stops improving.

For the comparison, raw agreement percentage is a fine starting point, but it flatters you when most cases are the same class. Cohen's kappa corrects for agreement that would happen by chance. In plain terms, kappa of 0 means the judge is no better than guessing with the same overall rates, and 1 means perfect agreement. As a working bar: below 0.4 the judge is not usable, 0.4 to 0.6 is usable for spotting trends only, above 0.6 is good enough to gate releases on, and above 0.8 is rare and worth double-checking for a leak of the answer into the prompt.

Also record where the judge is wrong, not just how often. A judge that is slightly harsh across the board is easy to live with. A judge that misses one specific failure type, such as invented dates, is dangerous, because that failure now scores clean forever.

The four failure modes

Bias	What happens	How to detect it	How to reduce it
Position	In A/B comparisons, the judge favours whichever answer came first	Run every pair twice with the order swapped	Swap and average; treat flips as ties
Self-preference	The judge scores text from its own model family higher	Score the same outputs with a judge from a different provider	Use a different family for the judge than the feature
Length	Long, detailed answers score higher regardless of accuracy	Plot score against word count	Add "length is not a quality signal" and cap reasoning
Tone	Confident, polite phrasing scores higher than hedged but correct text	Compare hedged and confident versions of the same fact	Ask for claim-level evidence rather than a global impression

Position bias is the easiest to fix and the most often ignored. If you ask a judge to pick between answer A and answer B, run it a second time with the answers swapped. If the winner changes, the judge has no real preference. Count that as a tie rather than a win.

Self-preference bias matters when you use the same provider for both the feature and the judge. The safe default is to use a different model family for judging. If you cannot, calibrate more often and check a sample against a second provider each quarter.

Length bias is common and quietly corrupts everything. Plot judge score against output length on your calibration set. If the line slopes upward and your human scores do not, the judge is measuring effort rather than quality.

Tone bias is the hardest. A model that states a wrong fact plainly often scores above one that states a right fact carefully. Asking for claim-by-claim evidence, as in the good prompt above, is the strongest counter.

Running a judge in your pipeline

Keep the judge out of your fast pipeline. It costs money and takes time, and a per-commit judge run trains developers to skip the pipeline.

A workable setup:

Judge runs nightly over the full dataset, and on demand for prompt changes.
Every judge result is stored with the input, output, reasoning, score, judge model version, and judge prompt version.
The report shows score by tag, not just an average. "Groundedness fell 9 points on multi-document cases" is actionable; "score fell 2 points" is not.
A weekly sample of 20 judged cases goes to a human. This is your ongoing calibration check, and it catches drift before it becomes a surprise.

Treat the judge prompt as code. It lives in git, changes go through review, and every change triggers a fresh calibration run. A judge prompt edited quietly in a dashboard is a broken measuring stick nobody knows is broken.

Frequently asked questions

Can the judge be the same model as the feature? It can, but it will score its own style favourably. Prefer a different model family for judging, and calibrate against humans more often if you cannot.

How many cases do I need for calibration? At least 50, covering good, bad, and borderline outputs. Fewer than that and one disagreement moves your agreement number by several points.

Should the judge see the reference answer? Yes, when you have one. Giving the judge the expected answer turns a vague quality call into a comparison, which raises agreement with humans.

What score scale works best? Three or five named levels, such as grounded, partly grounded, not grounded. Numeric 1 to 10 scales pull almost everything into the middle and are hard to define.

How often should I recalibrate? Whenever the judge prompt changes, whenever the judge model version changes, and at least once a quarter. Also recalibrate if the judge score and your human sample start to drift apart.