Skip to content
fx-blog

The Retry Button Is Hiding Your Flaky Tests

A test that fails, is retried and passes counts as green. Flaky tests in CI need rules: retry only runner faults, let the gate block a pass on retry, run one commit five times to find the rest, and quarantine each with an owner and a 30-day expiry.

Harry4 min readIntermediate
  • Verification
MarkdownSave

Key takeaways

  • A retry is legitimate when the runner or network failed, and hides a defect when a test failed.
  • The ci-cd GitLab templates retry jobs only on runner_system_failure, stuck_or_timeout_failure, api_failure and scheduler_failure, never on script_failure.
  • gate.py counts each test that passed only on retry, and with flaky.new_allowed set to 0 it fails a run whose pass rate is 100.0%.
  • A nightly job that runs one commit five times and diffs the outcomes with flake_report.py finds flakes that a single retry hides.
  • A quarantined test keeps running outside the blocking gate with an issue, an owner and a 30-day expiry, and an expired one is deleted.
Contents

TL;DR A test that fails, is retried and then passes counts as green in most pipelines, and past roughly a 2% flake rate people stop reading failures and press retry. Flaky tests in CI are a rule problem: retry only infrastructure faults, let the gate block any test that passed only on retry, and find the rest by running one commit several times. Here the gate fails a run whose pass rate is 100.0%.

Do this: Limit your CI retries to runner faults, then run your suite five times on one commit and feed the reports to flake_report.py.

Part 1, A Green Pipeline That Ran Zero Tests, left a merge request through the GitLab templates with a gate that prints flaky 0. Its question: if t.wobbly fails, the framework retries it and it passes, should the merge go in? This is what the same gate says about that report.

textpython scripts/gate.py --policy quality-gate.yml --reports reports/
gate: FAIL
  tests 2 | failed 0 | skipped 0 | flaky 1 | pass rate 100.0%
  …
  BLOCKED: 1 test(s) passed only on retry; policy allows 0. Quarantine them with an owner and an expiry (docs/05-flaky-tests.md).

Pass rate 100.0%. Verdict FAIL, exit 1. That is on purpose.

This series, in two parts:

  1. A Green Pipeline That Ran Zero Tests
  2. The Retry Button Is Hiding Your Flaky Tests (you are here)

Retry the runner, never the test

A retry is fine when the infrastructure failed. When a test failed, a retry hides a defect. The repo's flaky-test doc draws the line like this:

Legitimate Illegitimate
Runner died, network partition, image pull failure Test failed, so run it again
retry: { when: runner_system_failure } (GitLab) retry: 2 unconditionally
Rerunning a whole job after a spot-instance eviction --reruns 3 as the permanent default

Every job in the GitLab templates inherits this default. I left script_failure out of the list on purpose:

yamlplatforms/gitlab/templates/00-common.yml
default:
  retry:
    max: 2
    # Infrastructure faults only. Never script_failure: a retried test failure
    # is a hidden defect and a growing flaky suite. See docs/05-flaky-tests.md.
    when: [runner_system_failure, stuck_or_timeout_failure, api_failure, scheduler_failure]

Why does the gate block a test that passed?

A test that passed on retry failed once on the same commit. That makes it flaky by definition, with no analysis needed. Playwright and pytest-rerunfailures mark such a case with a <flakyFailure> element in the JUnit report, and gate.py counts them against one policy line:

yamlquality-gate.yml
    flaky:
      new_allowed: 0              # no new test may pass only on retry

So one retry inside the test framework is allowed, as a detector. Its result becomes evidence for the gate instead of a quiet pass.

Find the flakes one retry hides

A single retry only catches the flakes that happen to fail first. The nightly flake-sweep job runs the same commit five times and diffs the outcomes:

yamlplatforms/gitlab/templates/40-gate-release.yml
flake-sweep:
  stage: scheduled
  # …
  script:
    # Same commit, five runs. Tests with mixed outcomes are flaky by definition.
    - npm ci --prefer-offline --cache .npm
    - for i in 1 2 3 4 5; do npm run test:unit -- --reporter=junit --outputFile="reports/run-$i.xml" || true; done
    - python scripts/flake_report.py reports/run-*.xml --out reports/flaky.md

The || true here is allowed: the report decides, not the test command. On the self-test's three runs, where t.wobbly fails once, flake_report.py prints this and exits 1:

textpython scripts/flake_report.py reports/run-*.xml
# Flake report (3 runs of the same commit)

- tests observed: **2**
- flaky: **1** (50.00%)
- consistently failing (a real failure, not a flake): **0**

| Test | Failed | Runs | Rate |
|---|---|---|---|
| `t.wobbly` | 1 | 3 | 33% |

A test that fails in every run lands in its own list. That is a real failure, not a flake.

Quarantine with an owner and an expiry

Quarantine means the test keeps running but leaves the blocking gate, with an issue, an owner and a 30-day expiry. The doc's rule for each case:

mermaid
flowchart LR
  F[flaky test found] --> K{critical path?}
  K -->|yes| X[fix it now]
  K -->|no, cause unknown| Q[quarantine: owner + 30 days]
  Q -->|fixed| B[back in the gate]
  Q -->|expired| D[delete it]

Mark it in code, so the marker travels with the test:

pythondocs/05-flaky-tests.md
@pytest.mark.flaky_quarantine(issue="QA-1423", expires="2026-10-15")
def test_checkout_under_load(): ...

The cap in quality-gate.yml is 15 tests or 30 days. One honest gap: gate.py does not check that cap itself. It prints not enforced here: blocking.flaky.quarantine_max (needs its own job), so today the cap needs a job of its own.

Fix the cause, starting with sleep

The doc lists seven causes in the order you meet them. The first four:

  1. Waiting on time instead of state. sleep(2) is the number one cause. Wait for a condition instead.
  2. Shared mutable state. Two tests on the same user or order id. Generate unique data per test.
  3. Order dependence. Test B passes only after test A. Randomise the order in the nightly run.
  4. Real time and timezones. Tests that fail at month end or midnight UTC. Freeze the clock.

Then publish the flake rate weekly next to the pass rate, with a target under 1%. A rising flake rate means the suite is failing, even while every pipeline is green.

Try it

Try it
for i in 1 2 3 4 5; do
  pytest --junitxml="reports/run-$i.xml" || true
done
python scripts/flake_report.py reports/run-*.xml

Then search your CI config for script_failure, --reruns and unconditional retry:. The merge request through the GitLab templates now has a gate that says "no evidence" when nothing ran, and blocks t.wobbly until someone owns it.

Read this with your AI

textPaste into your AI
Read the post "The Retry Button Is Hiding Your Flaky Tests".
1. Summarise it in 5 bullets for an engineer who owns a CI pipeline.
2. Ask me 3 questions to check I understood; wait, then correct me.
3. Read my CI config and list every retry that would rerun a failed test rather than a dead runner.

FAQ

Is any retry of a test acceptable?

One retry inside the test framework is acceptable when it is recorded and reported, because it works as a flake detector. Retries as normal practice, such as --reruns 3 by default, are not.

Does gate.py enforce the quarantine cap of 15 tests or 30 days?

No. gate.py checks only flaky.new_allowed and prints that quarantine_max is not enforced there, so the cap needs a job of its own.

What flake rate should I aim for?

The repo's flaky-test doc sets a target below 1%, and says that past roughly 2% engineers stop reading failures and start pressing retry.

Sources

  1. ci-cd docs/05-flaky-tests.md
  2. ci-cd scripts/flake_report.py
  3. ci-cd GitLab template 00-common.yml

Was this useful?

Only you and the owner see your answer and rating. Send private feedback

Enjoyed it? Support fx-blog with a coffee.

A Green Pipeline That Ran Zero Tests

A test command that collects nothing exits 0, and the pipeline goes green. A CI quality gate with three outcomes fixes it: test jobs only report, one gate reads every report against quality-gate.yml, and no evidence means NOT EVALUABLE, exit 3.

5 min readIntermediate

Comments

Oldest first

No comments yet. Start the conversation.

You sign in first; what you typed stays here. · 0 / 2,000