Read 100 Real Traces Before You Write One Test

TL;DR Most LLM test suites measure the failures their author imagined, so the number says nothing about how your system really fails. LLM error analysis fixes that: you read real traces and name the failure modes before any test exists. Only here: lint rejects a failure mode backed by fewer than three real trace references, and rejects an open failure mode that no dataset case probes.

Do this: Read 30 of your own traces before you write a test, and keep only the failure modes with three trace references.

Part 1, An AI Eval Pipeline That Can Say NOT-EVALUABLE, left the orders support assistant inventing order 881234 in one trace. How often it does that, and in which kinds of message, is unknown. This part answers it from real traces, before a single test is written.

This series, in four parts:

1. An AI Eval Pipeline That Can Say NOT-EVALUABLE
2. Read 100 Real Traces Before You Write One Test (you are here)
3. Who Checks the AI Judge? Calibrate It Before It Gates
4. Read the Scorecard: Intervals, Gates and a Frozen Baseline

Every number labelled "sample" comes from the illustrative scenario in the bundle's showcase, docs/showcase/pipeline.html, not from a measured product.

Write the rule first, so two people cannot disagree

/eval-scope writes .evals/system-card.md. Its ## Hard policies section holds one rule per line, and each must be testable: could two people reading the same output disagree about whether it was broken?

- Never state a delivery date that is not returned by `get_shipment_status`.
- Never state an order id that did not come from a tool result.

"Be accurate and helpful" fails that test. A rule you cannot make testable is written as SCOPE-GAP, never softened until it sounds testable.

Read the first 30 traces yourself

/eval-explore --from-profile --count 100 pulls a batch of traces, each the record of one request: the input, the tool calls and the answer. A failure mode is one named way the system goes wrong. The first 30 are yours to read, because after 30 you can judge a proposed failure mode in seconds, and before 30 you cannot judge one at all.

1. Open coding: write free notes on what is wrong with each trace. Pick no categories yet.
2. Axial coding: group the notes into failure modes.
3. Stop at saturation: when new traces add no new mode. In the sample scenario that was trace 84 of 100.

A suggestion the AI wrote is not a label: rows tagged ANN-AI count only after a human confirms them.

A failure mode needs three real traces

The seed ships FM-001, .evals/taxonomy/FM-001-invented-order-id.md: 4 of 100 coded traces, severity blocking, and three trace references as evidence. One had a six-digit id the tool could not find; one had no id at all and the answer invented one; one belonged to another customer. Its detector is assert:no-invented-ids, decided in code with no judge.

Lint holds every failure mode to that bar:

ERROR   E003  evals/taxonomy/FM-001-thin-evidence.md
         1 trace ref(s) under ## Evidence, need 3. Under 3 observations it is a hypothesis, not a failure mode.

In the sample, "curt to frustrated users" had two observations. It stayed in the notes.

Build the dataset from what you found

/eval-data writes .evals/datasets/DS-<slug>-v1.jsonl, one case per combination of dimensions, taking the input from a real trace wherever one exists. The rules:

• At least 30% of cases have origin: trace. The orchestrator refuses a dataset below that unless you state in writing that no traces exist.
• Every open failure mode has a case. Lint reports the rest as an error.
• Once a run references it, the dataset never changes; fixing one row is v2.

ERROR   E008  evals/taxonomy/FM-003-uncovered.md
         FM-003 is open but no dataset case probes it: the suite will report this failure mode as fine, permanently

The seed dataset DS-example-v1 has 3 cases, 2 of them from traces. Its .meta.md records the weights: the happy path is 96% of traffic and 33% of the dataset, so rates must be read against those weights or they overstate the failure rate.

Try it

/eval-scope
/eval-explore --from-profile --count 100

Read the first 30 traces before you accept any suggestion, then count how many references each failure mode has. FM-001 has a rule, evidence and a code detector. The second failure mode in the sample, a wrong refund timeline, does not: an assertion cannot tell "in about a week" from 7 days. Part 3, Who Checks the AI Judge? Calibrate It Before It Gates, asks what happens when a model has to judge that.

Read this with your AI

Read the post "Read 100 Real Traces Before You Write One Test".
1. Summarise it in 5 bullets for an engineer who ships an LLM feature.
2. Ask me 3 questions to check I understood; wait, then correct me.
3. Take 10 of my own traces and list the failure modes you would propose, with a trace reference for each.