An AI Eval Pipeline That Can Say NOT-EVALUABLE

TL;DR Your AI feature has a quality number, and nobody can say who checked the judge behind it, how many cases it rests on or how sure it is. ai-eval is an AI evaluation pipeline for Claude Code. Only here: a run can end NOT-EVALUABLE, exit code 3, even when 9 of 9 attempts passed.

Do this: Install with npx agentic-fx@latest install --bundle eval, run the seed suite, and read why 9 of 9 still ends NOT-EVALUABLE.

Your orders support assistant tells a customer that order 881234 is out for delivery. The order tool returned nothing, and the assistant made the id up. You add test prompts, an LLM grades them, the dashboard turns green, and a review quotes the number.

This series, in four parts:

1. An AI Eval Pipeline That Can Say NOT-EVALUABLE (you are here)
2. Read 100 Real Traces Before You Write One Test
3. Who Checks the AI Judge? Calibrate It Before It Gates
4. Read the Scorecard: Intervals, Gates and a Frozen Baseline

Five questions to ask of any AI-evaluation setup

The usual setup is imagined prompts, a model that grades the answers, and one percentage. Ask these five of it, and of this pipeline:

1. Did anyone read your real traces first? Otherwise the tests cover failures you imagined.
2. Was the judge scored against human labels it never saw? An uncalibrated judge produces numbers that look real.
3. Does every rate carry an interval, and does the gate read its lower bound? Otherwise a lucky run passes.
4. Is a change judged per failure mode? A flat total can hide one failure collapsing.
5. Can it answer "I can't tell"? If the only outputs are green and red, it is guessing.

How ai-eval works: input, process, output

Two things go in: your production traces and one config file, .claude/EVAL-PROFILE.md. Then seven slash commands run in Claude Code. Each writes a file the next one reads, and each refuses one shortcut.

What comes out is one verdict per suite, PASS, FAIL or NOT-EVALUABLE, with the gate that decided it. The board in VS Code shows the same thing.

One real run: nine passes, no PASS

The bundle's seed suite, task-example, has three cases repeated three times on a stub adapter that calls no model. It shows the verdicts, not a model's quality.

NOT-EVALUABLE — task-example 2026-10-30-0537 (decided by no gate declared)
pass^1 1.000 [0.701, 1.000] (n=9)
pass^3 1.000 [0.439, 1.000] (n=3)
flaky  0.000 [0.000, 0.561] (n=3)

Every attempt passed. Exit code 3. Nine attempts cannot rule out a true rate near 70%, and nobody declared what the suite must reach. (pass^1 is the rate per attempt, pass^3 the rate per case.)

I shipped the seed suite with no gates on purpose. A suite earns a gate only once the evaluators that decide it can be trusted, so until then the seed reports numbers and never a PASS. Add a gate to this three-case suite anyway and the runner refuses:

STOP: gated suite runs on 3 cases. Below 30, a gate cannot distinguish a real regression from sampling noise (EVAL-CONTRACTS 8.3).

Install it, and what it does not do

cd your-project
npx agentic-fx@latest install --bundle eval

You need Node 20+, Python 3 and Claude Code. The installer adds the .evals/ workspace with the seed above, and the VS Code board.

Try it

python3 .evals/scripts/shared/lint_evals.py
python3 .evals/scripts/shared/runner.py --suite task-example

Then fill .claude/EVAL-PROFILE.md. The assistant invented order 881234 in one trace, but you do not know how often. Part 2, Read 100 Real Traces Before You Write One Test, finds out before a test is written.

Read this with your AI

Read the post "An AI Eval Pipeline That Can Say NOT-EVALUABLE".
1. Summarise it in 5 bullets for an engineer who ships an LLM feature.
2. Ask me 3 questions to check I understood; wait, then correct me.
3. Answer the five questions in the post for my current evaluation setup.