Who Checks the AI Judge? Calibrate It Before It Gates

TL;DR An LLM judge that agrees with your reviewers 60% of the time produces numbers that look exactly like a good one, and a release review quotes them. LLM judge validation puts a gate in front of every score. Only here: the TPR, TNR and kappa in a judge's validation.md are recomputed by lint from its labels, so a judge cannot be typed into trust, and a gate that names an unvalidated judge fails lint.

Do this: Count the Pass and Fail labels in the test split of one judge you use; under 30 of either, keep it out of every gate.

The orders support assistant has a failure mode that code cannot decide, a refund timeline that contradicts the policy. Part 2 ended there: "in about a week" and 7 days look the same to an assertion. A model has to judge it, and that judge is now part of your measurement. This part asks who checks it.

This series, in four parts:

1. An AI Eval Pipeline That Can Say NOT-EVALUABLE
2. Read 100 Real Traces Before You Write One Test
3. Who Checks the AI Judge? Calibrate It Before It Gates (you are here)
4. Read the Scorecard: Intervals, Gates and a Frozen Baseline

Every number labelled "sample" comes from the illustrative scenario in the bundle's showcase, docs/showcase/pipeline.html, not from a measured product.

Try a code assertion before any judge

/eval-judge tries a code assertion for every failure mode first. FM-001 is decided in code: every id-shaped token in the answer must appear in a tool result. No labels, no model, no variance. The bundle ships seven such assertions, listed in assertions/registry.yaml.

Only what survives this table becomes a judge: one binary decision about one failure mode, JDG-<slug>-vN.

Score the judge on labels it never saw

The human labels are split once into train, dev and test, and never reshuffled. You tune the prompt against dev. The test split is scored once, at the end, and each of dev and test needs at least 30 Pass and 30 Fail labels.

Then /eval-validate runs calibrate.py, which computes the result from the judge's saved verdicts on the test split. Here it is on the bundle's broken test fixture, where the split is far too small:

## Rates
TPR 1.000 · TNR 0.000 · FPR 1.000 · FNR 0.000 · kappa 0.000
## Verdict
UNVALIDATED — test split has 1 Fail examples, need 30; test split has 0 Pass examples, need 30; TNR 0.000 < 0.8; kappa 0.000 < 0.6; TPR-FPR 0.000 < 0.5: bias correction would amplify noise

TPR is the share of real failures the judge catches, TNR the share of good answers it lets through. The thresholds come from EVAL-PROFILE.md: both at least 0.80, kappa at least 0.60, and TPR minus FPR at least 0.50. Below that last one, the bias correction only amplifies noise.

Version 1 fails, and stays on file

In the sample, JDG-refund-timeline-v1 scored TPR 0.698 on a test split of 43 Fail and 42 Pass labels. It missed 13 labelled failures, all hedged ranges such as "3 to 5 days" that do not contain the policy figure. Verdict: UNVALIDATED.

The fix was a sentence in the prompt naming that case as a failure condition. A new prompt is a new version, so v2 was scored on the same test split, again in the sample: TPR 0.860, TNR 0.905, kappa 0.765, VALIDATED. Version 1 stays in the history.

Every false positive and false negative is listed one by one in validation.md. The rates decide the verdict. The list is what you read when you rewrite the prompt.

The numbers are computed, not typed

Lint recomputes the rates from the labels and refuses a validation.md that states different ones:

ERROR   E023  evals/judges/tone/validation.md
         JDG-tone-v1 validation.md disagrees with what its test labels and predictions compute: TPR stated 0.308, computed 1.000; TNR stated 0.870, computed 0.000; kappa stated 0.201, computed 0.000. Paste calibrate.py's output instead (EVAL-CONTRACTS 5.3)
ERROR   E010  evals/suites/broken.yaml
         gates on JDG-tone-v1, whose verdict is UNVALIDATED. The honest state is NOT-EVALUABLE: remove the gate or fix the judge (EVAL-CONTRACTS 5.4)

The runner enforces it too: a suite with an unvalidated judge ends NOT-EVALUABLE, decided by judge not validated. And a new judge model, even a provider-side bump, invalidates the validation (E022): bump the version and score it again.

Try it

python3 .evals/scripts/shared/calibrate.py JDG-<slug>-vN
python3 .evals/scripts/shared/lint_evals.py

Before you trust any judge you already run, find its test split and count the Pass and Fail labels. If either is under 30, its TPR is a number, not evidence. In the sample scenario both judges are now validated, so the assistant can finally be run on 220 cases. What a run reports, and when it says NOT-EVALUABLE anyway, is the job of part 4, Read the Scorecard: Intervals, Gates and a Frozen Baseline.

Read this with your AI

Read the post "Who Checks the AI Judge? Calibrate It Before It Gates".
1. Summarise it in 5 bullets for an engineer who relies on an LLM judge.
2. Ask me 3 questions to check I understood; wait, then correct me.
3. List what I would need (labels, splits, thresholds) to validate one judge I use today.