Skip to content
fx-blog

A Green Pipeline That Ran Zero Tests

A test command that collects nothing exits 0, and the pipeline goes green. A CI quality gate with three outcomes fixes it: test jobs only report, one gate reads every report against quality-gate.yml, and no evidence means NOT EVALUABLE, exit 3.

Harry5 min readIntermediate
  • Judgment
  • Verification
MarkdownSave

Key takeaways

  • A test command that collects zero tests exits 0, so a pipeline that only reads exit codes turns green with nothing tested.
  • Running assert-tests-ran.sh after every test command fails the job when the report is missing or holds zero tests.
  • In the ci-cd templates, test jobs only run tests and publish reports, and one gate job with when: always decides the outcome.
  • gate.py returns 0 for pass, 1 for fail and 3 for not evaluable, so a run with no evidence never reads as a pass.
  • What blocks a merge lives in one versioned quality-gate.yml, and every new gate runs as advisory for two weeks before it blocks.
Contents

TL;DR A test command that collects nothing exits 0, and an exit code is all most pipelines look at. A CI quality gate closes that hole only if it has three outcomes: pass, fail, and not evaluable when no evidence arrived. Here the test jobs only report, and one gate job reads every report against a policy file kept in git.

Do this: Add ./scripts/assert-tests-ran.sh reports/junit.xml after every test command, then run gate.py on your reports and see which of the three outcomes you get.

A typo in the test path. A marker expression that matches nothing. A framework that quietly found no files. Each one exits 0, the job turns green, and the merge goes in with nothing tested.

This series, in two parts:

  1. A Green Pipeline That Ran Zero Tests (you are here)
  2. The Retry Button Is Hiding Your Flaky Tests

Both parts follow a merge request through the GitLab templates in agentic-fx/ci-cd: a unit job writes reports/junit.xml, and a gate job decides. The reports below are the repo's own self-test fixtures.

Why does a green pipeline prove so little?

Because a pipeline reads exit codes, and an exit code says the command finished, not that anything was tested. The usual routes:

  • a test command that collected zero tests exits 0;
  • || true after a test command turns every failure into success;
  • allow_failure: true makes the test optional, and it "will decay within a month".

I made the second mistake in this repo. The nightly DAST job ended in || true. ZAP exits 0 when clean, 2 on warnings, 1 on failing alerts and 3 when the scan itself broke, and || true hid all four. A scan that never reached staging was green. The job now reads the code:

yamlplatforms/gitlab/templates/40-gate-release.yml (dast job)
      [ "$rc" -eq 0 ] || [ "$rc" -eq 2 ] || { echo "ZAP exit $rc (1 = FAIL alerts, 3 = scan error)"; exit "$rc"; }

Count the tests, not the exit code

scripts/assert-tests-ran.sh opens the report the test command wrote and fails when the file is missing or holds zero tests. It reads JUnit XML and CTRF. In the templates it runs straight after the tests:

yamlplatforms/gitlab/templates/20-test.yml (unit job)
  script:
    - npm ci --prefer-offline --cache .npm
    - npm run test:unit -- --coverage --reporter=junit --outputFile=reports/junit.xml
    - ./scripts/assert-tests-ran.sh reports/junit.xml

Here it is on the self-test's report with tests="0". Exit code 1:

text./scripts/assert-tests-ran.sh reports/junit.xml
assert-tests-ran: 0 test(s) across 1 report file(s)
assert-tests-ran: expected at least 1 test(s), found 0
  A suite that collects nothing exits 0. Check the test path, the marker
  expression, and that the framework found your files.

A second argument raises the floor. The self-test checks that a 12-test report fails assert-tests-ran.sh reports/junit.xml 50.

Tests report, and only the gate decides

The obvious design lets every test job fail the pipeline. I decided against it in ADR 0002. A failed test job skips its own reporting, so the run that most needs evidence has none, and the policy ends up spread over many jobs.

So the test jobs publish evidence, and one gate job at the end reads it all:

mermaid
flowchart LR
  U[unit] -->|reports/junit.xml| G{gate.py}
  V[integration, api] -->|JUnit| G
  S[sast, secret-scan] -->|SARIF| G
  P[(quality-gate.yml)] --> G
  G -->|exit 0| A[PASS]
  G -->|exit 1| B[FAIL]
  G -->|exit 3| C[NOT EVALUABLE]
yamlplatforms/gitlab/templates/40-gate-release.yml
gate:
  stage: gate
  when: always                    # must run even when an upstream job failed
  # …
  script:
    - pip install --quiet lxml
    - python scripts/gate.py --policy quality-gate.yml --reports reports/ --junit-out reports/gate.xml
        --summary reports/gate-summary.md --record reports/gate-history.jsonl

Three outcomes, not two

gate.py returns 0 for pass, 1 for fail and 3 for not evaluable. I ran it with the repo's quality-gate.yml on four of the self-test's report sets:

Evidence Verdict Exit
12 tests, 1 skipped, none failed PASS 0
10 tests, 2 failed FAIL: pass rate 80.0% below the required 100.0% (2 failing) 1
a report with 0 tests NOT EVALUABLE -- 0 tests found, expected >= 1 3
no report directory NOT EVALUABLE -- no test reports under reports/ 3

The gate also prints what it does not check, such as not enforced here: blocking.contract (needs its own job), so a rule written in the policy is never mistaken for a rule the script enforces.

Put the policy in one file

What blocks a merge lives in quality-gate.yml, reviewed like code. Changing it is a merge request, not a click in a settings page.

yamlquality-gate.yml
gates:
  blocking:
    pass_rate: 1.0
    min_tests: 1
    coverage_floor: 60.0
    coverage_delta_min: -0.5      # percentage points allowed to drop vs the base branch

Every new gate starts under advisory for two weeks, so you see what it would have blocked first.

Try it

Try it
curl -fsSL https://gitlab.com/agentic-fx/ci-cd/-/raw/main/scripts/assert-tests-ran.sh \
  -o scripts/assert-tests-ran.sh && chmod +x scripts/assert-tests-ran.sh
./scripts/assert-tests-ran.sh reports/junit.xml

Then vendor scripts/gate.py and quality-gate.yml the same way and run the gate on your own reports.

On the merge request through the GitLab templates, the passing gate prints tests 12 | failed 0 | skipped 1 | flaky 0 | pass rate 100.0%. Now suppose t.wobbly fails on its first try, the framework retries it, and it passes. The pass rate still says 100.0%. Part 2, The Retry Button Is Hiding Your Flaky Tests, runs that report through the same gate.

Read this with your AI

textPaste into your AI
Read the post "A Green Pipeline That Ran Zero Tests".
1. Summarise it in 5 bullets for an engineer who owns a CI pipeline.
2. Ask me 3 questions to check I understood; wait, then correct me.
3. Read my pipeline config and list every place where a test step could exit 0 with nothing tested.

FAQ

Why not let each test job fail the pipeline?

A failed test job skips its own reporting, so the run that most needs evidence has none, and the policy ends up spread over many jobs. ADR 0002 in the repo makes the test jobs report and one gate job decide.

Does exit code 3 still fail the pipeline?

Yes. Any non-zero exit fails the job. Exit 3 tells you that no evidence arrived, such as zero tests or no report, rather than that a test failed.

Do I need PyYAML to run gate.py?

No. gate.py reads the subset of YAML that quality-gate.yml uses with the standard library, and a JSON policy file also works.

Sources

  1. ci-cd on GitLab
  2. ci-cd docs/04-quality-gates.md
  3. ci-cd ADR 0002: tests report, only the gate decides

Was this useful?

Only you and the owner see your answer and rating. Send private feedback

Enjoyed it? Support fx-blog with a coffee.

The Retry Button Is Hiding Your Flaky Tests

A test that fails, is retried and passes counts as green. Flaky tests in CI need rules: retry only runner faults, let the gate block a pass on retry, run one commit five times to find the rest, and quarantine each with an owner and a 30-day expiry.

4 min readIntermediate

Comments

Oldest first

No comments yet. Start the conversation.

You sign in first; what you typed stays here. · 0 / 2,000