A QA Pipeline Your AI Agent Can't Fake

TL;DR AI-written tests go green and still ship bugs, and nobody can say what a green run proves. stlc is an AI test automation pipeline for Claude Code. Only here: the state of a feature, such as 6 of 12 cases passed and 2 failing, is computed from the files each step wrote, never from the agent's summary.

Do this: Run npx agentic-fx@latest install --bundle qa, fill .claude/PROJECT-PROFILE.md, then run /qa-analyze on one small feature's spec.

You ask an AI agent to "write tests for checkout". A minute later you have forty green tests. A week later a customer reports that orders with three lines round the total wrong. None of the forty builds a three-line order, and nobody can say which requirement any of them was meant to prove.

The agent was fast. Nobody asked it for proof.

This series, in four parts:

1. A QA Pipeline Your AI Agent Can't Fake (you are here)
2. Your First stlc Run: Checkout From Docs to Scripts
3. When a Test Goes Red: Triage, the Heal Guard and Quarantine
4. The QA Board and the Release Gate

Five questions to ask any AI test pipeline

I built stlc around these five. Put them to any tool, stlc too.

1. Can you trace each test to a requirement? Without it, "coverage" is the agent's word.
2. Are the run's files kept? A green run is a claim until its evidence is on disk.
3. Can the agent make red green by editing the test? Loosening, retrying and skipping all do it.
4. Does anything judge the cases before they are automated? A vague case scripted is a vague test.
5. Does a file decide whether it ships? Or does someone say "looks fine"?

How stlc works, step by step

Inside Claude Code, each step is one slash command that writes a file the next one reads.

1. Analyze. /qa-analyze orders docs/orders-spec.md writes REQ-001 to REQ-005. Stops: tests written from the code, not the spec.
2. Plan. /qa-plan orders writes a traceability table with zero rows. Stops: a pre-filled coverage number.
3. Design and review. 12 cases, each citing a REQ, then an optional read-only review. Stops: vague cases reaching automation.
4. Script. One spec per case. Stops: scripts with no case behind them.
5. Run. /qa-env uat, then /qa-run orders --env uat. Stops: guessing whether uat is ready, and a quiet fix of a red test.
6. Report. Gate rows and a verdict. Stops: a verdict nobody can check.

See where checkout stands

The numbers in this series come from Sample Orders App, the fixture of stlc's VS Code board (the installer adds it unless you pass --no-addons). Checkout ends the series at 6 of 12 cases passed, 2 failing, 2 blocked, 2 not run.

The board only reads files, so every number traces to a case, a run or a report. Getting that derivation right took me more than one try. My first draft read the estimate file as the test plan and guessed quarantine from the case id. Each of those mistakes is now a test that fails if it comes back.

Try it

cd your-project
npx agentic-fx@latest install --bundle qa

Then fill .claude/PROJECT-PROFILE.md and start with one small feature. Part 2, Your First stlc Run: Checkout From Docs to Scripts, fills the traceability table column by column.

Read this with your AI

Read the post "A QA Pipeline Your AI Agent Can't Fake".
1. Summarise it in 5 bullets for a test lead.
2. Ask me 3 questions to check I understood; wait, then correct me.
3. Answer the five questions in the post for my current test setup.