--- title: "Read the Scorecard: Intervals, Gates and a Frozen Baseline" author: "Harry" published: 2026-10-31T04:46:18.740Z updated: 2026-10-31T05:06:38.166Z category: "AI Evaluation" tags: ["ai-eval"] canonical: "https://fxblog.dev/posts/llm-evaluation-scorecard-confidence-interval" --- # Read the Scorecard: Intervals, Gates and a Frozen Baseline ## Key takeaways - Every rate carries a 95% Wilson interval, and a gate compares the lower bound, so 52 of 60 against a 0.80 gate is NOT-EVALUABLE, not a pass. - A suite is FAIL if any gate fails, PASS only if every gate passes, and NOT-EVALUABLE otherwise, including for a gate the runner cannot measure. - The runner exits 0 for PASS, 1 for FAIL or a preflight STOP, 3 for NOT-EVALUABLE and 4 when aborted on the budget. - A frozen baseline records the system, model, prompt, dataset, judges and each failure mode's rate, and the regression gate fails on the largest drop in the total or any one mode. - From ai-eval 1.4.0, a layer the profile's risk tier makes mandatory but its layer table leaves off is lint error E024; production monitoring, L6, is specified but not built. > **TL;DR** A scorecard that says 87% hides that 60 cases cannot support it, and a flat total hides one failure mode collapsing. An LLM evaluation scorecard in ai-eval reports every rate with its 95% interval and gates on the interval's lower bound. Only here: the regression gate compares the total and every failure mode to a frozen baseline and fails on the largest drop. > > **Do this:** Run `/eval-audit` on one suite you already trust, and check that every gate reads a lower bound. [Part 3](03-llm-judge-validation.md), **Who Checks the AI Judge? Calibrate It Before It Gates**, ended with both judges validated and the orders support assistant ready to run on 220 cases (sample). This part reads what that run reports, and when it still says NOT-EVALUABLE. **This series, in four parts:** 1. **An AI Eval Pipeline That Can Say NOT-EVALUABLE** 2. **Read 100 Real Traces Before You Write One Test** 3. **Who Checks the AI Judge? Calibrate It Before It Gates** 4. **Read the Scorecard: Intervals, Gates and a Frozen Baseline** (you are here) Every number labelled "sample" comes from the illustrative scenario in the bundle's showcase, `docs/showcase/pipeline.html`, not from a measured product. ## A rate is a range, so three runs get three verdicts Take one gate, 0.80, and three sample runs. `stats.py` computes each interval with the Wilson method. | Run | Passed | Rate | 95% interval | Verdict | | --- | ---------- | ----- | --------------- | ------------- | | A | 205 of 220 | 0.932 | \[0.891, 0.958] | PASS | | B | 52 of 60 | 0.867 | \[0.758, 0.931] | NOT-EVALUABLE | | C | 150 of 220 | 0.682 | \[0.618, 0.740] | FAIL | A dashboard showing only 87% would call run B a pass. Its lower bound is 0.758, so the run could not answer. More cases fix that, not a rerun. ![The three-verdict panel from the pipeline showcase: the same 0.80 gate with PASS at 205 of 220, NOT-EVALUABLE at 52 of 60 and FAIL at 150 of 220](https://fxblog.dev/media/01a19047-962b-7341-af8c-8a6f83911146/1600) _Same gate, three sample runs: only the middle one is not an answer_ ## Declare every gate, and each one counts Gates live in the suite file, each on the metric its name says. The seed suite carries these lines commented out, ready to switch on: ```yaml gates: task_success_ci_lower: 0.80 regression_margin: 0.05 baseline: BASE-2026-09-16-example ``` Every declared gate is evaluated. The suite is `FAIL` if any gate fails, `PASS` only if all pass, and `NOT-EVALUABLE` otherwise, including for a gate the runner cannot measure. The exit codes follow: 0 PASS, 1 FAIL or a preflight STOP, 3 NOT-EVALUABLE, 4 aborted on the budget. A gated suite must also name its `system_version` (the runner never fills it in), and below 30 cases it is refused, as part 1's `STOP` showed. ## A frozen baseline catches what a total hides `/eval-report --freeze` writes `BASE-`, a record of the system, model, prompt, dataset and judges, plus the rate of each failure mode. A later change is compared to it: ```text paired · −1.4pp [−4.6, +1.8] · 8 regressions, 5 improvements · McNemar p 0.581 per FM · no failure mode moved more than 0.05 ``` The comparison is paired, case against itself, so the answer is a difference with its interval: no change we can detect. The regression gate takes the drop in the total and in each failure mode, and fails on the largest one. A steady total can hide one failure mode collapsing while another improves. > **Note** > > With no readable baseline, the regression gate is NOT-EVALUABLE ("no readable baseline to regress against"), not a pass. ## Where it stops: layers, board and limits The profile sets a risk tier, and the tier's table names the layers that are mandatory: | Tier | The system | Mandatory layers | | ------ | ----------------------------------------------------- | ---------------- | | low | drafts text a human always reviews | L0 L1 L2 | | medium | answers users directly, or calls read-only tools | L0 to L4 | | high | takes actions with side effects, handles money or PII | L0 to L6 | The README and the profile said a mandatory layer left off was an error. While checking this post against the code I found lint never looked, so I fixed the code rather than the sentence: from ai-eval 1.4.0, a mandatory layer left off in `§Layers` is lint error E024. The orders support assistant is tier `medium`, and the profile template ships with L3 and L4 off, so setting the tier is enough to see it: ```text ERROR E024 .claude/EVAL-PROFILE.md L3 is mandatory for risk tier 'medium' but §Layers does not enable it: the pipeline would skip it and the system would look evaluated. Enable it, or change the tier (EVAL-CONTRACTS 8.1) ``` L4 gets the same error. The npm bundle is still 1.3.0 and does not check this yet. Start with L0: it makes no model calls, runs in under ten seconds and is gated at 100%. The VS Code board reads `.evals/state.json`, never writes a file, and shows NOT-EVALUABLE as itself. L6, production monitoring, is not built: `/eval-monitor` stops with a phase note. ## Try it ```text /eval-run task --env staging /eval-report --freeze /eval-drift --against BASE- --change model /eval-audit ``` `/eval-audit` asks one question: are these numbers safe to act on? Run it before you trust a suite someone else wrote, and once a quarter after that. ## Read this with your AI ```text Read the post "Read the Scorecard: Intervals, Gates and a Frozen Baseline". 1. Summarise it in 5 bullets for an engineer who reports LLM quality to a release review. 2. Ask me 3 questions to check I understood; wait, then correct me. 3. Take one pass rate I quote today and tell me what interval it would carry. ``` > **Success** > > You can read a scorecard for what it can and cannot say: every rate has an interval, and a change is judged on the largest drop in the total or any failure mode. Start with `/eval-scope` on your own system. ## FAQ ### Why not just report the pass rate? A pass rate without an interval hides how many cases it rests on. 52 of 60 is 0.867, but its lower bound is 0.758, so against a 0.80 gate the run cannot answer. ### What does /eval-audit check? Whether the numbers are safe to act on: an unvalidated judge in a gated suite, a gate on a point estimate, too few cases, leakage from the dataset into a prompt, an open failure mode with no case, and a suite that has stopped discriminating.