---
title: "Who Checks the AI Judge? Calibrate It Before It Gates"
author: "Harry"
published: 2026-10-31T04:46:18.361Z
updated: 2026-10-31T05:06:37.518Z
category: "AI Evaluation"
tags: ["ai-eval"]
canonical: "https://fxblog.dev/posts/llm-judge-validation-trust-gate"
---

# Who Checks the AI Judge? Calibrate It Before It Gates

## Key takeaways

- Try a code assertion for every failure mode before writing a judge; a judge is only for what code cannot decide.
- Human labels are split once into train, dev and test, and the test split is scored once, with at least 30 Pass and 30 Fail labels.
- A judge is VALIDATED when TPR and TNR are at least 0.80, kappa at least 0.60 and TPR minus FPR at least 0.50.
- Lint recomputes the rates from the labels and fails a validation.md that states different ones, and fails a gate that names an unvalidated judge.
- A new prompt or a new judge model is a new version that must be validated again, and a failed version stays on file.

> **TL;DR** An LLM judge that agrees with your reviewers 60% of the time produces numbers that look exactly like a good one, and a release review quotes them. LLM judge validation puts a gate in front of every score. Only here: the TPR, TNR and kappa in a judge's `validation.md` are recomputed by lint from its labels, so a judge cannot be typed into trust, and a gate that names an unvalidated judge fails lint.
>
> **Do this:** Count the Pass and Fail labels in the test split of one judge you use; under 30 of either, keep it out of every gate.

The orders support assistant has a failure mode that code cannot decide, a refund timeline that contradicts the policy. [Part 2](02-llm-error-analysis.md) ended there: "in about a week" and 7 days look the same to an assertion. A model has to judge it, and that judge is now part of your measurement. This part asks who checks it.

**This series, in four parts:**

1. **An AI Eval Pipeline That Can Say NOT-EVALUABLE**
2. **Read 100 Real Traces Before You Write One Test**
3. **Who Checks the AI Judge? Calibrate It Before It Gates** (you are here)
4. **Read the Scorecard: Intervals, Gates and a Frozen Baseline**

Every number labelled "sample" comes from the illustrative scenario in the bundle's showcase, `docs/showcase/pipeline.html`, not from a measured product.

## Try a code assertion before any judge

`/eval-judge` tries a code assertion for every failure mode first. FM-001 is decided in code: every id-shaped token in the answer must appear in a tool result. No labels, no model, no variance. The bundle ships seven such assertions, listed in `assertions/registry.yaml`.

| Failure mode                         | Assertion?                                  |
| ------------------------------------ | ------------------------------------------- |
| invents an order number              | yes, ids must appear in a tool result       |
| cites a chunk that was not retrieved | yes, cited ids must be in the retrieved set |
| tone acceptable to a frustrated user | no, it needs judgement                      |

Only what survives this table becomes a judge: one binary decision about one failure mode, `JDG-<slug>-vN`.

## Score the judge on labels it never saw

The human labels are split once into train, dev and test, and never reshuffled. You tune the prompt against dev. The test split is scored once, at the end, and each of dev and test needs at least 30 Pass and 30 Fail labels.

Then `/eval-validate` runs `calibrate.py`, which computes the result from the judge's saved verdicts on the test split. Here it is on the bundle's broken test fixture, where the split is far too small:

```text
## Rates
TPR 1.000 · TNR 0.000 · FPR 1.000 · FNR 0.000 · kappa 0.000
## Verdict
UNVALIDATED — test split has 1 Fail examples, need 30; test split has 0 Pass examples, need 30; TNR 0.000 < 0.8; kappa 0.000 < 0.6; TPR-FPR 0.000 < 0.5: bias correction would amplify noise
```

TPR is the share of real failures the judge catches, TNR the share of good answers it lets through. The thresholds come from `EVAL-PROFILE.md`: both at least 0.80, kappa at least 0.60, and TPR minus FPR at least 0.50. Below that last one, the bias correction only amplifies noise.

## Version 1 fails, and stays on file

In the sample, `JDG-refund-timeline-v1` scored TPR 0.698 on a test split of 43 Fail and 42 Pass labels. It missed 13 labelled failures, all hedged ranges such as "3 to 5 days" that do not contain the policy figure. Verdict: `UNVALIDATED`.

![The pipeline showcase at the validate step: JDG-grounded-v1 is VALIDATED with TPR 0.837, and JDG-refund-timeline-v1 is UNVALIDATED with TPR 0.698 below the 0.80 threshold](https://fxblog.dev/media/01a19047-85f8-7177-ad1b-89a25ef88a70/1600)

_The showcase's sample: two judges, one trusted, one not yet_

The fix was a sentence in the prompt naming that case as a failure condition. A new prompt is a new version, so `v2` was scored on the same test split, again in the sample: TPR 0.860, TNR 0.905, kappa 0.765, `VALIDATED`. Version 1 stays in the history.

Every false positive and false negative is listed one by one in `validation.md`. The rates decide the verdict. The list is what you read when you rewrite the prompt.

## The numbers are computed, not typed

Lint recomputes the rates from the labels and refuses a `validation.md` that states different ones:

```text
ERROR   E023  evals/judges/tone/validation.md
         JDG-tone-v1 validation.md disagrees with what its test labels and predictions compute: TPR stated 0.308, computed 1.000; TNR stated 0.870, computed 0.000; kappa stated 0.201, computed 0.000. Paste calibrate.py's output instead (EVAL-CONTRACTS 5.3)
ERROR   E010  evals/suites/broken.yaml
         gates on JDG-tone-v1, whose verdict is UNVALIDATED. The honest state is NOT-EVALUABLE: remove the gate or fix the judge (EVAL-CONTRACTS 5.4)
```

The runner enforces it too: a suite with an unvalidated judge ends `NOT-EVALUABLE`, decided by `judge not validated`. And a new judge model, even a provider-side bump, invalidates the validation (E022): bump the version and score it again.

> **Warning**
>
> The judge must come from a different model family than the system it judges in any gated suite (E020). Same-family judging inflates scores.

## Try it

```sh
python3 .evals/scripts/shared/calibrate.py JDG-<slug>-vN
python3 .evals/scripts/shared/lint_evals.py
```

Before you trust any judge you already run, find its test split and count the Pass and Fail labels. If either is under 30, its TPR is a number, not evidence. In the sample scenario both judges are now validated, so the assistant can finally be run on 220 cases. What a run reports, and when it says NOT-EVALUABLE anyway, is the job of part 4, **Read the Scorecard: Intervals, Gates and a Frozen Baseline**.

## Read this with your AI

```text
Read the post "Who Checks the AI Judge? Calibrate It Before It Gates".
1. Summarise it in 5 bullets for an engineer who relies on an LLM judge.
2. Ask me 3 questions to check I understood; wait, then correct me.
3. List what I would need (labels, splits, thresholds) to validate one judge I use today.
```

> **Success**
>
> You can tell a judge you may quote from one you may not: its rates are recomputed from labels it never saw. Next: **Read the Scorecard: Intervals, Gates and a Frozen Baseline**.

## FAQ

### Why must the judge be a different model family from the system?

Same-family judging inflates scores, so lint error E020 fires when a gated suite uses a judge from the same family as the system under evaluation.

### What happens to a suite whose judge is not validated?

It can still run in exploratory mode, but every number carries the UNVALIDATED marker and the suite verdict is NOT-EVALUABLE, never PASS.