The Retry Button Is Hiding Your Flaky Tests
A test that fails, is retried and passes counts as green. Flaky tests in CI need rules: retry only runner faults, let the gate block a pass on retry, run one commit five times to find the rest, and quarantine each with an owner and a 30-day expiry.
Key takeaways
- A retry is legitimate when the runner or network failed, and hides a defect when a test failed.
- The ci-cd GitLab templates retry jobs only on runner_system_failure, stuck_or_timeout_failure, api_failure and scheduler_failure, never on script_failure.
- gate.py counts each test that passed only on retry, and with flaky.new_allowed set to 0 it fails a run whose pass rate is 100.0%.
- A nightly job that runs one commit five times and diffs the outcomes with flake_report.py finds flakes that a single retry hides.
- A quarantined test keeps running outside the blocking gate with an issue, an owner and a 30-day expiry, and an expired one is deleted.
Contents
FAQ
Is any retry of a test acceptable?
One retry inside the test framework is acceptable when it is recorded and reported, because it works as a flake detector. Retries as normal practice, such as --reruns 3 by default, are not.
Does gate.py enforce the quarantine cap of 15 tests or 30 days?
No. gate.py checks only flaky.new_allowed and prints that quarantine_max is not enforced there, so the cap needs a job of its own.
What flake rate should I aim for?
The repo's flaky-test doc sets a target below 1%, and says that past roughly 2% engineers stop reading failures and start pressing retry.
Sources
Only you and the owner see your answer and rating. Send private feedback
Enjoyed it? Support fx-blog with a coffee.
Related posts
A Green Pipeline That Ran Zero Tests
A test command that collects nothing exits 0, and the pipeline goes green. A CI quality gate with three outcomes fixes it: test jobs only report, one gate reads every report against quality-gate.yml, and no evidence means NOT EVALUABLE, exit 3.
Comments
Oldest firstNo comments yet. Start the conversation.
Part 1, A Green Pipeline That Ran Zero Tests, left a merge request through the GitLab templates with a gate that prints
flaky 0. Its question: ift.wobblyfails, the framework retries it and it passes, should the merge go in? This is what the same gate says about that report.Pass rate 100.0%. Verdict FAIL, exit 1. That is on purpose.
This series, in two parts:
Retry the runner, never the test
A retry is fine when the infrastructure failed. When a test failed, a retry hides a defect. The repo's flaky-test doc draws the line like this:
retry: { when: runner_system_failure }(GitLab)retry: 2unconditionally--reruns 3as the permanent defaultEvery job in the GitLab templates inherits this default. I left
script_failureout of the list on purpose:Why does the gate block a test that passed?
A test that passed on retry failed once on the same commit. That makes it flaky by definition, with no analysis needed. Playwright and pytest-rerunfailures mark such a case with a
<flakyFailure>element in the JUnit report, andgate.pycounts them against one policy line:So one retry inside the test framework is allowed, as a detector. Its result becomes evidence for the gate instead of a quiet pass.
Find the flakes one retry hides
A single retry only catches the flakes that happen to fail first. The nightly
flake-sweepjob runs the same commit five times and diffs the outcomes:The
|| truehere is allowed: the report decides, not the test command. On the self-test's three runs, wheret.wobblyfails once,flake_report.pyprints this and exits 1:A test that fails in every run lands in its own list. That is a real failure, not a flake.
Quarantine with an owner and an expiry
Quarantine means the test keeps running but leaves the blocking gate, with an issue, an owner and a 30-day expiry. The doc's rule for each case:
Mark it in code, so the marker travels with the test:
The cap in
quality-gate.ymlis 15 tests or 30 days. One honest gap:gate.pydoes not check that cap itself. It printsnot enforced here: blocking.flaky.quarantine_max (needs its own job), so today the cap needs a job of its own.Fix the cause, starting with sleep
The doc lists seven causes in the order you meet them. The first four:
sleep(2)is the number one cause. Wait for a condition instead.Then publish the flake rate weekly next to the pass rate, with a target under 1%. A rising flake rate means the suite is failing, even while every pipeline is green.
Try it
Then search your CI config for
script_failure,--rerunsand unconditionalretry:. The merge request through the GitLab templates now has a gate that says "no evidence" when nothing ran, and blockst.wobblyuntil someone owns it.Read this with your AI