--- title: "An AI Eval Pipeline That Can Say NOT-EVALUABLE" author: "Harry" published: 2026-10-31T04:46:17.600Z updated: 2026-10-31T05:06:36.229Z category: "AI Evaluation" tags: ["ai-eval"] canonical: "https://fxblog.dev/posts/ai-evaluation-pipeline-ai-eval" --- # An AI Eval Pipeline That Can Say NOT-EVALUABLE ## Key takeaways - A quality number from an AI feature is a guess unless you know who checked the judge, how many cases it rests on and how sure the number is. - Ask five questions of any AI-evaluation setup: were real traces read, was the judge calibrated, does the gate read an interval, is change judged per failure mode, can it say it cannot tell. - ai-eval takes your traces and one profile file through seven steps, scope to report, and outputs a scorecard with one verdict per suite. - A suite has three verdicts, PASS, FAIL and NOT-EVALUABLE, and the seed suite returns NOT-EVALUABLE with exit code 3 even though 9 of 9 attempts passed. - ai-eval measures only: defects and sign-off stay in the QA bundle, and production monitoring is specified but not built. > **TL;DR** Your AI feature has a quality number, and nobody can say who checked the judge behind it, how many cases it rests on or how sure it is. ai-eval is an AI evaluation pipeline for Claude Code. Only here: a run can end NOT-EVALUABLE, exit code 3, even when 9 of 9 attempts passed. > > **Do this:** Install with `npx agentic-fx@latest install --bundle eval`, run the seed suite, and read why 9 of 9 still ends NOT-EVALUABLE. Your orders support assistant tells a customer that order 881234 is out for delivery. The order tool returned nothing, and the assistant made the id up. You add test prompts, an LLM grades them, the dashboard turns green, and a review quotes the number. **This series, in four parts:** 1. **An AI Eval Pipeline That Can Say NOT-EVALUABLE** (you are here) 2. **Read 100 Real Traces Before You Write One Test** 3. **Who Checks the AI Judge? Calibrate It Before It Gates** 4. **Read the Scorecard: Intervals, Gates and a Frozen Baseline** ## Five questions to ask of any AI-evaluation setup The usual setup is imagined prompts, a model that grades the answers, and one percentage. Ask these five of it, and of this pipeline: 1. **Did anyone read your real traces first?** Otherwise the tests cover failures you imagined. 2. **Was the judge scored against human labels it never saw?** An uncalibrated judge produces numbers that look real. 3. **Does every rate carry an interval, and does the gate read its lower bound?** Otherwise a lucky run passes. 4. **Is a change judged per failure mode?** A flat total can hide one failure collapsing. 5. **Can it answer "I can't tell"?** If the only outputs are green and red, it is guessing. ## How ai-eval works: input, process, output Two things go in: your production traces and one config file, `.claude/EVAL-PROFILE.md`. Then seven slash commands run in Claude Code. Each writes a file the next one reads, and each refuses one shortcut. | Step | Command | Writes | Refuses | | ---------- | ---------------- | -------------------------------------------------- | --------------------------------- | | 1 Scope | `/eval-scope` | `system-card.md`, the rules that must never break | a rule nobody can test | | 2 Explore | `/eval-explore` | failure modes coded from about 100 real traces | tests for imagined failures | | 3 Data | `/eval-data` | an immutable dataset, at least 30% real traces | a dataset made of guesses | | 4 Judge | `/eval-judge` | a code assertion first, a judge only if needed | a judge where code would decide | | 5 Validate | `/eval-validate` | `validation.md`, the judge scored on unseen labels | an unproven judge in a gate | | 6 Run | `/eval-run` | repeated cases, every rate with a 95% interval | one lucky run | | 7 Report | `/eval-report` | the scorecard; `--freeze` fixes a baseline | a regression hidden in an average | What comes out is one verdict per suite, PASS, FAIL or NOT-EVALUABLE, with the gate that decided it. The board in VS Code shows the same thing. ## One real run: nine passes, no PASS The bundle's seed suite, `task-example`, has three cases repeated three times on a stub adapter that calls no model. It shows the verdicts, not a model's quality. ```text NOT-EVALUABLE — task-example 2026-10-30-0537 (decided by no gate declared) pass^1 1.000 [0.701, 1.000] (n=9) pass^3 1.000 [0.439, 1.000] (n=3) flaky 0.000 [0.000, 0.561] (n=3) ``` Every attempt passed. Exit code 3. Nine attempts cannot rule out a true rate near 70%, and nobody declared what the suite must reach. (`pass^1` is the rate per attempt, `pass^3` the rate per case.) I shipped the seed suite with no gates on purpose. A suite earns a gate only once the evaluators that decide it can be trusted, so until then the seed reports numbers and never a PASS. Add a gate to this three-case suite anyway and the runner refuses: ```text STOP: gated suite runs on 3 cases. Below 30, a gate cannot distinguish a real regression from sampling noise (EVAL-CONTRACTS 8.3). ``` ![The ai-eval board on the seed project after a real run: task-example is NOT-EVALUABLE and ungated, pass^3 100% with a 95% interval of 43.9% to 100% over 3 cases](https://fxblog.dev/media/01a19047-6b70-76bf-8e47-d9ac7346ce77/1600) _Nine of nine attempts passed, and the board still shows NOT-EVALUABLE_ ## Install it, and what it does not do ```sh cd your-project npx agentic-fx@latest install --bundle eval ``` You need Node 20+, Python 3 and Claude Code. The installer adds the `.evals/` workspace with the seed above, and the VS Code board. > **Warning** > > ai-eval measures; it does not fix. Defects and release sign-off stay in the QA bundle. Production monitoring, `/eval-monitor`, is specified but not built. The repo has no results from a real pilot yet, and every LLM judge needs human labels before it can gate anything. ## Try it ```sh python3 .evals/scripts/shared/lint_evals.py python3 .evals/scripts/shared/runner.py --suite task-example ``` Then fill `.claude/EVAL-PROFILE.md`. The assistant invented order 881234 in one trace, but you do not know how often. Part 2, **Read 100 Real Traces Before You Write One Test**, finds out before a test is written. ## Read this with your AI ```text Read the post "An AI Eval Pipeline That Can Say NOT-EVALUABLE". 1. Summarise it in 5 bullets for an engineer who ships an LLM feature. 2. Ask me 3 questions to check I understood; wait, then correct me. 3. Answer the five questions in the post for my current evaluation setup. ``` > **Success** > > You can ask five questions of any AI-evaluation setup; in ai-eval the answer can be NOT-EVALUABLE, even at 9 of 9. Next: **Read 100 Real Traces Before You Write One Test**. ## FAQ ### Does ai-eval fix the failures it finds? No. It measures. Defect lifecycle, quarantine and release sign-off stay in the QA bundle, and the evidence is handed over with /eval-signoff. ### What do I need to run it? Node 20 or later, Python 3 and Claude Code. The command npx agentic-fx@latest install --bundle eval adds the commands, the .evals workspace with a seed that lints clean and the VS Code board.