A Green Pipeline That Ran Zero Tests
A test command that collects nothing exits 0, and the pipeline goes green. A CI quality gate with three outcomes fixes it: test jobs only report, one gate reads every report against quality-gate.yml, and no evidence means NOT EVALUABLE, exit 3.
Key takeaways
- A test command that collects zero tests exits 0, so a pipeline that only reads exit codes turns green with nothing tested.
- Running assert-tests-ran.sh after every test command fails the job when the report is missing or holds zero tests.
- In the ci-cd templates, test jobs only run tests and publish reports, and one gate job with when: always decides the outcome.
- gate.py returns 0 for pass, 1 for fail and 3 for not evaluable, so a run with no evidence never reads as a pass.
- What blocks a merge lives in one versioned quality-gate.yml, and every new gate runs as advisory for two weeks before it blocks.
Contents
FAQ
Why not let each test job fail the pipeline?
A failed test job skips its own reporting, so the run that most needs evidence has none, and the policy ends up spread over many jobs. ADR 0002 in the repo makes the test jobs report and one gate job decide.
Does exit code 3 still fail the pipeline?
Yes. Any non-zero exit fails the job. Exit 3 tells you that no evidence arrived, such as zero tests or no report, rather than that a test failed.
Do I need PyYAML to run gate.py?
No. gate.py reads the subset of YAML that quality-gate.yml uses with the standard library, and a JSON policy file also works.
Sources
Only you and the owner see your answer and rating. Send private feedback
Enjoyed it? Support fx-blog with a coffee.
Related posts
The Retry Button Is Hiding Your Flaky Tests
A test that fails, is retried and passes counts as green. Flaky tests in CI need rules: retry only runner faults, let the gate block a pass on retry, run one commit five times to find the rest, and quarantine each with an owner and a 30-day expiry.
Comments
Oldest firstNo comments yet. Start the conversation.
A typo in the test path. A marker expression that matches nothing. A framework that quietly found no files. Each one exits 0, the job turns green, and the merge goes in with nothing tested.
This series, in two parts:
Both parts follow a merge request through the GitLab templates in agentic-fx/ci-cd: a
unitjob writesreports/junit.xml, and agatejob decides. The reports below are the repo's own self-test fixtures.Why does a green pipeline prove so little?
Because a pipeline reads exit codes, and an exit code says the command finished, not that anything was tested. The usual routes:
|| trueafter a test command turns every failure into success;allow_failure: truemakes the test optional, and it "will decay within a month".I made the second mistake in this repo. The nightly DAST job ended in
|| true. ZAP exits 0 when clean, 2 on warnings, 1 on failing alerts and 3 when the scan itself broke, and|| truehid all four. A scan that never reached staging was green. The job now reads the code:Count the tests, not the exit code
scripts/assert-tests-ran.shopens the report the test command wrote and fails when the file is missing or holds zero tests. It reads JUnit XML and CTRF. In the templates it runs straight after the tests:Here it is on the self-test's report with
tests="0". Exit code 1:A second argument raises the floor. The self-test checks that a 12-test report fails
assert-tests-ran.sh reports/junit.xml 50.Tests report, and only the gate decides
The obvious design lets every test job fail the pipeline. I decided against it in ADR 0002. A failed test job skips its own reporting, so the run that most needs evidence has none, and the policy ends up spread over many jobs.
So the test jobs publish evidence, and one gate job at the end reads it all:
Three outcomes, not two
gate.pyreturns 0 for pass, 1 for fail and 3 for not evaluable. I ran it with the repo'squality-gate.ymlon four of the self-test's report sets:pass rate 80.0% below the required 100.0% (2 failing)NOT EVALUABLE -- 0 tests found, expected >= 1NOT EVALUABLE -- no test reports under reports/The gate also prints what it does not check, such as
not enforced here: blocking.contract (needs its own job), so a rule written in the policy is never mistaken for a rule the script enforces.Put the policy in one file
What blocks a merge lives in
quality-gate.yml, reviewed like code. Changing it is a merge request, not a click in a settings page.Every new gate starts under
advisoryfor two weeks, so you see what it would have blocked first.Try it
Then vendor
scripts/gate.pyandquality-gate.ymlthe same way and run the gate on your own reports.On the merge request through the GitLab templates, the passing gate prints
tests 12 | failed 0 | skipped 1 | flaky 0 | pass rate 100.0%. Now supposet.wobblyfails on its first try, the framework retries it, and it passes. The pass rate still says 100.0%. Part 2, The Retry Button Is Hiding Your Flaky Tests, runs that report through the same gate.Read this with your AI