---
title: "A QA Pipeline Your AI Agent Can't Fake"
author: "Harry"
published: 2026-10-31T04:14:26.263Z
updated: 2026-10-31T05:06:00.331Z
category: "AI Agents: SDLC & STLC"
tags: ["stlc"]
canonical: "https://fxblog.dev/posts/ai-test-automation-pipeline-stlc"
---

# A QA Pipeline Your AI Agent Can't Fake

## Key takeaways

- An AI agent writes tests fast, but without traceability, evidence and rules for fixes, a green run proves nothing.
- Ask five questions of any AI test pipeline: can tests be traced to requirements, is the evidence kept, can red be turned green by editing, is anything judging the cases, does a file decide shipping.
- stlc takes one feature from raw docs to a report with a verdict, one command per step, each step writing a file the next one reads.
- Two checks can stop the route: a REVISE FIRST case review blocks scripting once a review is on file, and the environment check marks each test type READY, NOT READY or BLOCKED.
- Every project-specific value lives in one file, PROJECT-PROFILE.md, and the state of a feature, such as 6 of 12 cases passed, is read from files, never from the agent's summary.

> **TL;DR** AI-written tests go green and still ship bugs, and nobody can say what a green run proves. stlc is an AI test automation pipeline for Claude Code. Only here: the state of a feature, such as 6 of 12 cases passed and 2 failing, is computed from the files each step wrote, never from the agent's summary.
>
> **Do this:** Run `npx agentic-fx@latest install --bundle qa`, fill `.claude/PROJECT-PROFILE.md`, then run `/qa-analyze` on one small feature's spec.

You ask an AI agent to "write tests for checkout". A minute later you have forty green tests. A week later a customer reports that orders with three lines round the total wrong. None of the forty builds a three-line order, and nobody can say which requirement any of them was meant to prove.

The agent was fast. Nobody asked it for proof.

**This series, in four parts:**

1. **A QA Pipeline Your AI Agent Can't Fake** (you are here)
2. **Your First stlc Run: Checkout From Docs to Scripts**
3. **When a Test Goes Red: Triage, the Heal Guard and Quarantine**
4. **The QA Board and the Release Gate**

## Five questions to ask any AI test pipeline

I built stlc around these five. Put them to any tool, stlc too.

1. **Can you trace each test to a requirement?** Without it, "coverage" is the agent's word.
2. **Are the run's files kept?** A green run is a claim until its evidence is on disk.
3. **Can the agent make red green by editing the test?** Loosening, retrying and skipping all do it.
4. **Does anything judge the cases before they are automated?** A vague case scripted is a vague test.
5. **Does a file decide whether it ships?** Or does someone say "looks fine"?

![Forty green tests without stlc versus with it: without, nobody knows which requirement each covers and the bug ships; with it, every case traces to a REQ id and a gap shows](https://fxblog.dev/media/01a18fc9-7678-7431-84a4-8a98637ba202/1600)

_Same agent, same checkout: the difference is what the files can prove_

## How stlc works, step by step

Inside Claude Code, each step is one slash command that writes a file the next one reads.

1. **Analyze.** `/qa-analyze orders docs/orders-spec.md` writes REQ-001 to REQ-005. Stops: tests written from the code, not the spec.
2. **Plan.** `/qa-plan orders` writes a traceability table with zero rows. Stops: a pre-filled coverage number.
3. **Design and review.** 12 cases, each citing a REQ, then an optional read-only review. Stops: vague cases reaching automation.
4. **Script.** One spec per case. Stops: scripts with no case behind them.
5. **Run.** `/qa-env uat`, then `/qa-run orders --env uat`. Stops: guessing whether uat is ready, and a quiet fix of a red test.
6. **Report.** Gate rows and a verdict. Stops: a verdict nobody can check.

> **Note**
>
> Two checks can stop the route. A REVISE FIRST review blocks `/qa-script`. `/qa-env` marks each test type READY, NOT READY or BLOCKED, and security and performance runs are refused on environments your profile does not authorise. stlc never edits your product code; every project value lives in `.claude/PROJECT-PROFILE.md`.

## See where checkout stands

The numbers in this series come from Sample Orders App, the fixture of stlc's VS Code board (the installer adds it unless you pass `--no-addons`). Checkout ends the series at 6 of 12 cases passed, 2 failing, 2 blocked, 2 not run.

![The stlc board for orders checkout at the end of the series: step 6 of 10, 6 of 12 cases passed, 2 failing, 2 blocked, and tiles for defects, quarantine and pass rate](https://fxblog.dev/media/01a18fc9-7d11-73b4-adf2-0182c8427af1/1600)

_Two failing tests, in plain sight, not tuned away_

The board only reads files, so every number traces to a case, a run or a report. Getting that derivation right took me more than one try. My first draft read the estimate file as the test plan and guessed quarantine from the case id. Each of those mistakes is now a test that fails if it comes back.

## Try it

```sh
cd your-project
npx agentic-fx@latest install --bundle qa
```

Then fill `.claude/PROJECT-PROFILE.md` and start with one small feature. Part 2, **Your First stlc Run: Checkout From Docs to Scripts**, fills the traceability table column by column.

## Read this with your AI

```text
Read the post "A QA Pipeline Your AI Agent Can't Fake".
1. Summarise it in 5 bullets for a test lead.
2. Ask me 3 questions to check I understood; wait, then correct me.
3. Answer the five questions in the post for my current test setup.
```

> **Success**
>
> 6 of 12 passed is counted from the files the steps wrote, not taken from the agent's summary. Next: **Your First stlc Run: Checkout From Docs to Scripts**.

## FAQ

### Does stlc change my application code?

No. It reads product code to ground its cases and never edits it. Unit and mobile tests are staged in its own folder and handed to developers through a HANDOFF.md, landing in your repo only when your profile says apply.

### Which test types does stlc cover?

Nine: unit, API, integration, end-to-end, security, performance, accessibility, visual and layout UI, and mobile. Mobile is off by default, and security and performance run only on environments your profile authorizes.