---
title: "A Green Pipeline That Ran Zero Tests"
author: "Harry"
published: 2026-11-02T01:59:44.986Z
category: "DevOps & CI/CD"
tags: ["ci-cd"]
canonical: "https://fxblog.dev/posts/ci-quality-gate-not-evaluable"
---

# A Green Pipeline That Ran Zero Tests

## Key takeaways

- A test command that collects zero tests exits 0, so a pipeline that only reads exit codes turns green with nothing tested.
- Running assert-tests-ran.sh after every test command fails the job when the report is missing or holds zero tests.
- In the ci-cd templates, test jobs only run tests and publish reports, and one gate job with when: always decides the outcome.
- gate.py returns 0 for pass, 1 for fail and 3 for not evaluable, so a run with no evidence never reads as a pass.
- What blocks a merge lives in one versioned quality-gate.yml, and every new gate runs as advisory for two weeks before it blocks.

> **TL;DR** A test command that collects nothing exits 0, and an exit code is all most pipelines look at. A CI quality gate closes that hole only if it has three outcomes: pass, fail, and not evaluable when no evidence arrived. Here the test jobs only report, and one gate job reads every report against a policy file kept in git.
>
> **Do this:** Add `./scripts/assert-tests-ran.sh reports/junit.xml` after every test command, then run `gate.py` on your reports and see which of the three outcomes you get.

A typo in the test path. A marker expression that matches nothing. A framework that quietly found no files. Each one exits 0, the job turns green, and the merge goes in with nothing tested.

**This series, in two parts:**

1. **A Green Pipeline That Ran Zero Tests** (you are here)
2. **The Retry Button Is Hiding Your Flaky Tests**

Both parts follow a merge request through the GitLab templates in [agentic-fx/ci-cd](https://gitlab.com/agentic-fx/ci-cd): a `unit` job writes `reports/junit.xml`, and a `gate` job decides. The reports below are the repo's own self-test fixtures.

## Why does a green pipeline prove so little?

Because a pipeline reads exit codes, and an exit code says the command finished, not that anything was tested. The usual routes:

- a test command that collected zero tests exits 0;
- `|| true` after a test command turns every failure into success;
- `allow_failure: true` makes the test optional, and it "will decay within a month".

I made the second mistake in this repo. The nightly DAST job ended in `|| true`. ZAP exits 0 when clean, 2 on warnings, 1 on failing alerts and 3 when the scan itself broke, and `|| true` hid all four. A scan that never reached staging was green. The job now reads the code:

```yaml
      [ "$rc" -eq 0 ] || [ "$rc" -eq 2 ] || { echo "ZAP exit $rc (1 = FAIL alerts, 3 = scan error)"; exit "$rc"; }
```

## Count the tests, not the exit code

`scripts/assert-tests-ran.sh` opens the report the test command wrote and fails when the file is missing or holds zero tests. It reads JUnit XML and CTRF. In the templates it runs straight after the tests:

```yaml
  script:
    - npm ci --prefer-offline --cache .npm
    - npm run test:unit -- --coverage --reporter=junit --outputFile=reports/junit.xml
    - ./scripts/assert-tests-ran.sh reports/junit.xml
```

Here it is on the self-test's report with `tests="0"`. Exit code 1:

```text
assert-tests-ran: 0 test(s) across 1 report file(s)
assert-tests-ran: expected at least 1 test(s), found 0
  A suite that collects nothing exits 0. Check the test path, the marker
  expression, and that the framework found your files.
```

A second argument raises the floor. The self-test checks that a 12-test report fails `assert-tests-ran.sh reports/junit.xml 50`.

## Tests report, and only the gate decides

The obvious design lets every test job fail the pipeline. I decided against it in ADR 0002. A failed test job skips its own reporting, so the run that most needs evidence has none, and the policy ends up spread over many jobs.

So the test jobs publish evidence, and one gate job at the end reads it all:

Tests report; the gate decides

```mermaid
flowchart LR
  U[unit] -->|reports/junit.xml| G{gate.py}
  V[integration, api] -->|JUnit| G
  S[sast, secret-scan] -->|SARIF| G
  P[(quality-gate.yml)] --> G
  G -->|exit 0| A[PASS]
  G -->|exit 1| B[FAIL]
  G -->|exit 3| C[NOT EVALUABLE]
```

```yaml
gate:
  stage: gate
  when: always                    # must run even when an upstream job failed
  # …
  script:
    - pip install --quiet lxml
    - python scripts/gate.py --policy quality-gate.yml --reports reports/ --junit-out reports/gate.xml
        --summary reports/gate-summary.md --record reports/gate-history.jsonl
```

> **Note**
>
> The cost I accepted: a reader who sees `continue-on-error: true` on a test step assumes the tests are optional. It sits on the test step only, so the upload always runs.

## Three outcomes, not two

`gate.py` returns 0 for pass, 1 for fail and 3 for not evaluable. I ran it with the repo's `quality-gate.yml` on four of the self-test's report sets:

| Evidence                         | Verdict                                                       | Exit |
| -------------------------------- | ------------------------------------------------------------- | ---- |
| 12 tests, 1 skipped, none failed | PASS                                                          | 0    |
| 10 tests, 2 failed               | FAIL: `pass rate 80.0% below the required 100.0% (2 failing)` | 1    |
| a report with 0 tests            | `NOT EVALUABLE -- 0 tests found, expected >= 1`               | 3    |
| no report directory              | `NOT EVALUABLE -- no test reports under reports/`             | 3    |

The gate also prints what it does not check, such as `not enforced here: blocking.contract (needs its own job)`, so a rule written in the policy is never mistaken for a rule the script enforces.

> **Tip**
>
> Exit 3 still fails the job, like any non-zero code. What changes is the log line: "no evidence arrived" instead of "a test failed". On an image without PyYAML, the gate once crashed reading its own policy and exited 1, which read as failed tests. An unreadable policy now returns 3.

## Put the policy in one file

What blocks a merge lives in `quality-gate.yml`, reviewed like code. Changing it is a merge request, not a click in a settings page.

```yaml
gates:
  blocking:
    pass_rate: 1.0
    min_tests: 1
    coverage_floor: 60.0
    coverage_delta_min: -0.5      # percentage points allowed to drop vs the base branch
```

Every new gate starts under `advisory` for two weeks, so you see what it would have blocked first.

> **Warning**
>
> Lowering a threshold because it failed is a ratchet pointed the wrong way. Do it once, with a reason, in a commit.

## Try it

```sh
curl -fsSL https://gitlab.com/agentic-fx/ci-cd/-/raw/main/scripts/assert-tests-ran.sh \
  -o scripts/assert-tests-ran.sh && chmod +x scripts/assert-tests-ran.sh
./scripts/assert-tests-ran.sh reports/junit.xml
```

Then vendor `scripts/gate.py` and `quality-gate.yml` the same way and run the gate on your own reports.

On the merge request through the GitLab templates, the passing gate prints `tests 12 | failed 0 | skipped 1 | flaky 0 | pass rate 100.0%`. Now suppose `t.wobbly` fails on its first try, the framework retries it, and it passes. The pass rate still says 100.0%. Part 2, **The Retry Button Is Hiding Your Flaky Tests**, runs that report through the same gate.

## Read this with your AI

```text
Read the post "A Green Pipeline That Ran Zero Tests".
1. Summarise it in 5 bullets for an engineer who owns a CI pipeline.
2. Ask me 3 questions to check I understood; wait, then correct me.
3. Read my pipeline config and list every place where a test step could exit 0 with nothing tested.
```

> **Success**
>
> Your pipeline can now tell "no test ran" apart from "every test passed", and one file says what blocks a merge. Next: **The Retry Button Is Hiding Your Flaky Tests**.

## FAQ

### Why not let each test job fail the pipeline?

A failed test job skips its own reporting, so the run that most needs evidence has none, and the policy ends up spread over many jobs. ADR 0002 in the repo makes the test jobs report and one gate job decide.

### Does exit code 3 still fail the pipeline?

Yes. Any non-zero exit fails the job. Exit 3 tells you that no evidence arrived, such as zero tests or no report, rather than that a test failed.

### Do I need PyYAML to run gate.py?

No. gate.py reads the subset of YAML that quality-gate.yml uses with the standard library, and a JSON policy file also works.
