--- title: "When a Test Goes Red: Triage, the Heal Guard and Quarantine" author: "Harry" published: 2026-10-31T04:14:27.017Z updated: 2026-10-31T05:06:01.587Z category: "AI Agents: SDLC & STLC" tags: ["stlc"] canonical: "https://fxblog.dev/posts/ai-test-failure-triage-heal-guard" --- # When a Test Goes Red: Triage, the Heal Guard and Quarantine ## Key takeaways - A run never fixes anything: each failure gets one of four labels, FUNCTIONAL, SCRIPT, SETUP/ENV or FLAKY, with its evidence. - FLAKY needs an observed passing re-run; a hunch does not count. - npm run qa:heal-guard blocks a fix that removes, weakens or suppresses an assertion; a changed expected value passes only if the case says so or a named human approves. - A quarantined test still runs and still reports its real status, for at most 30 days. - A defect closes as fixed only after a newer verification run in which its case passed, and a recurrence re-opens the same file. > **TL;DR** When an AI agent's test goes red, the cheapest fix is to hide it, and a green run made by editing the test tells everyone the feature works. AI test failure triage in stlc removes that option. Only here: a fix must pass `npm run qa:heal-guard`, which diffs it against the base and blocks a removed or weakened assertion, a skip or a retry, and writes its verdict to `reports/heal-guard/`. > > **Do this:** Before you fix your next red test, label it FUNCTIONAL, SCRIPT, SETUP/ENV or FLAKY, and write down the evidence for that label. Part 2 ended with ten scripts for orders checkout. The first uat run came back red: `TC-INT-ORDERS-001` expected a `tax_total` of 4.20 and got null. At 2 a.m. the agent has three cheap options: loosen the assertion so `null` passes, add a retry, or skip the test. Any of them turns the run green and hides what is wrong. **This series, in four parts:** 1. **A QA Pipeline Your AI Agent Can't Fake** 2. **Your First stlc Run: Checkout From Docs to Scripts** 3. **When a Test Goes Red: Triage, the Heal Guard and Quarantine** (you are here) 4. **The QA Board and the Release Gate** ## A red test gets a label, not a fix `/qa-run` never edits a test or the product. It writes each failure to `failures.md` with the evidence and exactly one label: | Label | Means | | ------------ | ------------------------------------ | | `FUNCTIONAL` | the product is wrong | | `SCRIPT` | the test is wrong | | `SETUP/ENV` | data, fixture or environment | | `FLAKY` | an identical immediate re-run passed | Checkout's failure was `SETUP/ENV`: the uat seed predates the tax migration. The product was fine, and loosening the assertion would have buried that. The other red test, `TC-E2E-ORDERS-001`, is `FUNCTIONAL`. The new order never appears in the list, so it goes to `/qa-defect` as `DEF-orders-001`. `FLAKY` needs a re-run someone saw pass. I hold that line even for CI: a retry there is recorded as `retried-in-ci`, never `FLAKY`, because nobody ran the one immediate, unchanged re-run the label means. ![The latest run on the stlc board: a sample orders run at 67 % with one failure classified SETUP-ENV, the expected and actual values, and the command that class leads to](https://fxblog.dev/media/01a18fc9-ca4c-7036-b52a-7844628c1160/1600) _The red test stays red, with its label and next command beside it_ ## The heal guard: what a fix may not do For a `SCRIPT` failure, `/qa-heal` repairs the test, and `npm run qa:heal-guard` checks the diff against the base. `/qa-heal` and `/qa-flaky` are not done until it exits 0. You can run it yourself on any diff. ![The heal guard on a SCRIPT fix: a removed or weakened assertion, or a skip, retry or wait, is blocked; a changed expected value passes only if the case says so](https://fxblog.dev/media/01a18fc9-e11c-737e-ab2d-738db3172767/1600) - **removed**, **weakened** and **suppressed** (`skip`, `only`, `retries:`, `toPass()`) changes are blocked, always; - a changed **expected** value passes only if every new value is already in the case, or a named person approves it with `--human ""`. So changing `4.20` to `null`, the move an agent reaches for first, is blocked, and the verdict lands in `reports/heal-guard/` either way. The guard's own tests try 20 ways to fake a green heal, and each one must be blocked. ## Flaky tests are quarantined in the open `TC-E2E-ORDERS-002`, the order list filter, fails on CI when the filter chips render late. A re-run passes, so the case is quarantined in the ledger (shortened here: the real one also has `Evidence` and `Released` columns): ```text | Case ID | Since | Until | Reason | Ticket | | TC-E2E-ORDERS-002 | 2026-10-05 | 2026-10-30 | filter chips render late on CI (timing) | TASK-201 | ``` The grant lasts at most 30 days, lint fails once it expires unreleased, and the case still runs and reports its true status. Checkout's second quarantine, `TC-API-ORDERS-002`, ran out on 2026-10-10 without a release. Lint flags it. > **Note** > > Unit tests and performance scenarios are never quarantined. A flaky unit test is a defect in the test, and a flaky SLO is a finding about the system. ## A defect closes on evidence The rounding bug from part 1 has a file now: `/qa-escape` opened `DEF-orders-002` with `Origin: ESCAPED`. Its escape analysis names the phase that missed it and the type that should have caught it (in this example, `DESIGN` and unit). ![DEF-orders-002, the escaped rounding bug: opened by /qa-escape with backfill cases, closed as fixed only after a newer passing verification run, re-opened on recurrence](https://fxblog.dev/media/01a18fc9-ef1d-7221-977b-d48af9b2a169/1600) 1. **Backfill:** design adds a unit case for a three-line order. 2. **Verify:** `/qa-defect orders --verify` looks for a run newer than every recurrence in which that case passed. 3. **Close:** only then does the file get `Resolution: fixed (verified )`. Nobody gets to call it fixed on their word. A repeat failure re-opens the same file. ## Try it ```text Here is a failing Playwright test and its error. Before changing anything, label the failure FUNCTIONAL, SCRIPT, SETUP/ENV or FLAKY, and show the evidence for the label. If it is SCRIPT, propose a fix that keeps every assertion at least as strict as before, and say which line a heal guard would block if you loosened it. ``` Checkout now has two red tests with known causes, two quarantines (one expired) and two open defects, all in files and none easy to see at a glance. May checkout ship? Part 4, **The QA Board and the Release Gate**, opens the board on this exact state. ## Read this with your AI ```text Read the post "When a Test Goes Red: Triage, the Heal Guard and Quarantine". 1. Summarise it in 5 bullets for a QA engineer. 2. Ask me 3 questions to check I understood; wait, then correct me. 3. Look at my last three test fixes and say which ones a heal guard would have blocked. ``` > **Success** > > The run stays red until the cause is fixed, and every heal leaves a guard verdict on file. Next: **The QA Board and the Release Gate**. ## FAQ ### Can the agent just add a retry to make a flaky test pass? No. Retries, timeouts, waits, skips and toPass() are suppressed changes the heal guard never allows. A flaky case goes to quarantine in the open, with an expiry date of at most 30 days, and keeps running. ### What happens when a fixed bug comes back? The same defect is re-opened with a recurrence line, never duplicated. It can close again only after a newer verification run in which its case passed.