--- title: "Read 100 Real Traces Before You Write One Test" author: "Harry" published: 2026-10-31T04:46:17.981Z updated: 2026-10-31T05:06:36.889Z category: "AI Evaluation" tags: ["ai-eval"] canonical: "https://fxblog.dev/posts/llm-error-analysis-failure-modes" --- # Read 100 Real Traces Before You Write One Test ## Key takeaways - Write hard policies two people cannot disagree on, and mark any rule you cannot test as SCOPE-GAP instead of softening it. - Read the first 30 traces yourself, with open coding first and axial coding second, and stop at saturation. - A failure mode needs at least three real trace references, and lint error E003 rejects one with fewer. - A dataset needs at least 30% of its cases from real traces, a case for every open failure mode, and never changes after a run references it. - Rates from a dataset built failure-rich must be read against its sampling weights, or they overstate the failure rate. > **TL;DR** Most LLM test suites measure the failures their author imagined, so the number says nothing about how your system really fails. LLM error analysis fixes that: you read real traces and name the failure modes before any test exists. Only here: lint rejects a failure mode backed by fewer than three real trace references, and rejects an open failure mode that no dataset case probes. > > **Do this:** Read 30 of your own traces before you write a test, and keep only the failure modes with three trace references. [Part 1](01-ai-evaluation-pipeline.md), **An AI Eval Pipeline That Can Say NOT-EVALUABLE**, left the orders support assistant inventing order 881234 in one trace. How often it does that, and in which kinds of message, is unknown. This part answers it from real traces, before a single test is written. **This series, in four parts:** 1. **An AI Eval Pipeline That Can Say NOT-EVALUABLE** 2. **Read 100 Real Traces Before You Write One Test** (you are here) 3. **Who Checks the AI Judge? Calibrate It Before It Gates** 4. **Read the Scorecard: Intervals, Gates and a Frozen Baseline** Every number labelled "sample" comes from the illustrative scenario in the bundle's showcase, `docs/showcase/pipeline.html`, not from a measured product. ## Write the rule first, so two people cannot disagree `/eval-scope` writes `.evals/system-card.md`. Its `## Hard policies` section holds one rule per line, and each must be testable: could two people reading the same output disagree about whether it was broken? ```text - Never state a delivery date that is not returned by `get_shipment_status`. - Never state an order id that did not come from a tool result. ``` "Be accurate and helpful" fails that test. A rule you cannot make testable is written as `SCOPE-GAP`, never softened until it sounds testable. ## Read the first 30 traces yourself `/eval-explore --from-profile --count 100` pulls a batch of traces, each the record of one request: the input, the tool calls and the answer. A failure mode is one named way the system goes wrong. The first 30 are yours to read, because after 30 you can judge a proposed failure mode in seconds, and before 30 you cannot judge one at all. 1. **Open coding:** write free notes on what is wrong with each trace. Pick no categories yet. 2. **Axial coding:** group the notes into failure modes. 3. **Stop at saturation:** when new traces add no new mode. In the sample scenario that was trace 84 of 100. A suggestion the AI wrote is not a label: rows tagged `ANN-AI` count only after a human confirms them. ![The ai-eval board on the seed project: FM-001, an order id that did not come from a tool result, is blocking and open; DS-example-v1 has 3 cases, 66.7% from real traces](https://fxblog.dev/media/01a19047-7c2c-7546-a273-bd58b3fd5d8a/1600) _FM-001 on the board: blocking, and still open_ ## A failure mode needs three real traces The seed ships FM-001, `.evals/taxonomy/FM-001-invented-order-id.md`: 4 of 100 coded traces, severity `blocking`, and three trace references as evidence. One had a six-digit id the tool could not find; one had no id at all and the answer invented one; one belonged to another customer. Its detector is `assert:no-invented-ids`, decided in code with no judge. Lint holds every failure mode to that bar: ```text ERROR E003 evals/taxonomy/FM-001-thin-evidence.md 1 trace ref(s) under ## Evidence, need 3. Under 3 observations it is a hypothesis, not a failure mode. ``` In the sample, "curt to frustrated users" had two observations. It stayed in the notes. > **Tip** > > Before you build anything for a failure mode, check whether one prompt line fixes it. In the sample, "replies in English to Vietnamese messages" was fixed that way, the traces were re-pulled, and it was gone. ## Build the dataset from what you found `/eval-data` writes `.evals/datasets/DS--v1.jsonl`, one case per combination of dimensions, taking the input from a real trace wherever one exists. The rules: - At least 30% of cases have `origin: trace`. The orchestrator refuses a dataset below that unless you state in writing that no traces exist. - Every open failure mode has a case. Lint reports the rest as an error. - Once a run references it, the dataset never changes; fixing one row is `v2`. ```text ERROR E008 evals/taxonomy/FM-003-uncovered.md FM-003 is open but no dataset case probes it: the suite will report this failure mode as fine, permanently ``` The seed dataset `DS-example-v1` has 3 cases, 2 of them from traces. Its `.meta.md` records the weights: the happy path is 96% of traffic and 33% of the dataset, so rates must be read against those weights or they overstate the failure rate. ## Try it ```text /eval-scope /eval-explore --from-profile --count 100 ``` Read the first 30 traces before you accept any suggestion, then count how many references each failure mode has. FM-001 has a rule, evidence and a code detector. The second failure mode in the sample, a wrong refund timeline, does not: an assertion cannot tell "in about a week" from 7 days. Part 3, **Who Checks the AI Judge? Calibrate It Before It Gates**, asks what happens when a model has to judge that. ## Read this with your AI ```text Read the post "Read 100 Real Traces Before You Write One Test". 1. Summarise it in 5 bullets for an engineer who ships an LLM feature. 2. Ask me 3 questions to check I understood; wait, then correct me. 3. Take 10 of my own traces and list the failure modes you would propose, with a trace reference for each. ``` > **Success** > > You can name how your system fails from real traces, and lint will not let a failure mode stand on fewer than three of them, or stay open without a case. Next: **Who Checks the AI Judge? Calibrate It Before It Gates**. ## FAQ ### How many traces do I need to read? The profile asks for at least 100 traces to code, until saturation, and a human codes the first 30 before any AI suggestion. Rows labelled ANN-AI are suggestions until a human confirms them. ### What if I have no production traces yet? The orchestrator refuses a dataset below 30% real cases unless you state in writing that no traces exist and accept the consequence.