Read the Scorecard: Intervals, Gates and a Frozen Baseline

TL;DR A scorecard that says 87% hides that 60 cases cannot support it, and a flat total hides one failure mode collapsing. An LLM evaluation scorecard in ai-eval reports every rate with its 95% interval and gates on the interval's lower bound. Only here: the regression gate compares the total and every failure mode to a frozen baseline and fails on the largest drop.

Do this: Run /eval-audit on one suite you already trust, and check that every gate reads a lower bound.

Part 3, Who Checks the AI Judge? Calibrate It Before It Gates, ended with both judges validated and the orders support assistant ready to run on 220 cases (sample). This part reads what that run reports, and when it still says NOT-EVALUABLE.

This series, in four parts:

1. An AI Eval Pipeline That Can Say NOT-EVALUABLE
2. Read 100 Real Traces Before You Write One Test
3. Who Checks the AI Judge? Calibrate It Before It Gates
4. Read the Scorecard: Intervals, Gates and a Frozen Baseline (you are here)

Every number labelled "sample" comes from the illustrative scenario in the bundle's showcase, docs/showcase/pipeline.html, not from a measured product.

A rate is a range, so three runs get three verdicts

Take one gate, 0.80, and three sample runs. stats.py computes each interval with the Wilson method.

A dashboard showing only 87% would call run B a pass. Its lower bound is 0.758, so the run could not answer. More cases fix that, not a rerun.

Declare every gate, and each one counts

Gates live in the suite file, each on the metric its name says. The seed suite carries these lines commented out, ready to switch on:

gates:
  task_success_ci_lower: 0.80
  regression_margin: 0.05
baseline: BASE-2026-09-16-example

Every declared gate is evaluated. The suite is FAIL if any gate fails, PASS only if all pass, and NOT-EVALUABLE otherwise, including for a gate the runner cannot measure. The exit codes follow: 0 PASS, 1 FAIL or a preflight STOP, 3 NOT-EVALUABLE, 4 aborted on the budget. A gated suite must also name its system_version (the runner never fills it in), and below 30 cases it is refused, as part 1's STOP showed.

A frozen baseline catches what a total hides

/eval-report --freeze writes BASE-<id>, a record of the system, model, prompt, dataset and judges, plus the rate of each failure mode. A later change is compared to it:

paired · −1.4pp [−4.6, +1.8] · 8 regressions, 5 improvements · McNemar p 0.581
per FM · no failure mode moved more than 0.05

The comparison is paired, case against itself, so the answer is a difference with its interval: no change we can detect. The regression gate takes the drop in the total and in each failure mode, and fails on the largest one. A steady total can hide one failure mode collapsing while another improves.

Where it stops: layers, board and limits

The profile sets a risk tier, and the tier's table names the layers that are mandatory:

The README and the profile said a mandatory layer left off was an error. While checking this post against the code I found lint never looked, so I fixed the code rather than the sentence: from ai-eval 1.4.0, a mandatory layer left off in §Layers is lint error E024. The orders support assistant is tier medium, and the profile template ships with L3 and L4 off, so setting the tier is enough to see it:

ERROR   E024  .claude/EVAL-PROFILE.md
         L3 is mandatory for risk tier 'medium' but §Layers does not enable it: the pipeline would skip it and the system would look evaluated. Enable it, or change the tier (EVAL-CONTRACTS 8.1)

L4 gets the same error. The npm bundle is still 1.3.0 and does not check this yet.

Start with L0: it makes no model calls, runs in under ten seconds and is gated at 100%. The VS Code board reads .evals/state.json, never writes a file, and shows NOT-EVALUABLE as itself. L6, production monitoring, is not built: /eval-monitor stops with a phase note.

Try it

/eval-run task --env staging
/eval-report <slug> --freeze
/eval-drift <slug> --against BASE-<id> --change model
/eval-audit

/eval-audit asks one question: are these numbers safe to act on? Run it before you trust a suite someone else wrote, and once a quarter after that.

Read this with your AI

Read the post "Read the Scorecard: Intervals, Gates and a Frozen Baseline".
1. Summarise it in 5 bullets for an engineer who reports LLM quality to a release review.
2. Ask me 3 questions to check I understood; wait, then correct me.
3. Take one pass rate I quote today and tell me what interval it would carry.