---
title: "The Retry Button Is Hiding Your Flaky Tests"
author: "Harry"
published: 2026-11-02T01:59:45.802Z
category: "DevOps & CI/CD"
tags: ["ci-cd"]
canonical: "https://fxblog.dev/posts/flaky-tests-ci-retry-quarantine"
---

# The Retry Button Is Hiding Your Flaky Tests

## Key takeaways

- A retry is legitimate when the runner or network failed, and hides a defect when a test failed.
- The ci-cd GitLab templates retry jobs only on runner\_system\_failure, stuck\_or\_timeout\_failure, api\_failure and scheduler\_failure, never on script\_failure.
- gate.py counts each test that passed only on retry, and with flaky.new\_allowed set to 0 it fails a run whose pass rate is 100.0%.
- A nightly job that runs one commit five times and diffs the outcomes with flake\_report.py finds flakes that a single retry hides.
- A quarantined test keeps running outside the blocking gate with an issue, an owner and a 30-day expiry, and an expired one is deleted.

> **TL;DR** A test that fails, is retried and then passes counts as green in most pipelines, and past roughly a 2% flake rate people stop reading failures and press retry. Flaky tests in CI are a rule problem: retry only infrastructure faults, let the gate block any test that passed only on retry, and find the rest by running one commit several times. Here the gate fails a run whose pass rate is 100.0%.
>
> **Do this:** Limit your CI retries to runner faults, then run your suite five times on one commit and feed the reports to `flake_report.py`.

[Part 1](01-ci-quality-gate-not-evaluable.md), **A Green Pipeline That Ran Zero Tests**, left a merge request through the GitLab templates with a gate that prints `flaky 0`. Its question: if `t.wobbly` fails, the framework retries it and it passes, should the merge go in? This is what the same gate says about that report.

```text
gate: FAIL
  tests 2 | failed 0 | skipped 0 | flaky 1 | pass rate 100.0%
  …
  BLOCKED: 1 test(s) passed only on retry; policy allows 0. Quarantine them with an owner and an expiry (docs/05-flaky-tests.md).
```

Pass rate 100.0%. Verdict FAIL, exit 1. That is on purpose.

**This series, in two parts:**

1. **A Green Pipeline That Ran Zero Tests**
2. **The Retry Button Is Hiding Your Flaky Tests** (you are here)

## Retry the runner, never the test

A retry is fine when the infrastructure failed. When a test failed, a retry hides a defect. The repo's flaky-test doc draws the line like this:

| Legitimate                                           | Illegitimate                          |
| ---------------------------------------------------- | ------------------------------------- |
| Runner died, network partition, image pull failure   | Test failed, so run it again          |
| `retry: { when: runner_system_failure }` (GitLab)    | `retry: 2` unconditionally            |
| Rerunning a whole job after a spot-instance eviction | `--reruns 3` as the permanent default |

Every job in the GitLab templates inherits this default. I left `script_failure` out of the list on purpose:

```yaml
default:
  retry:
    max: 2
    # Infrastructure faults only. Never script_failure: a retried test failure
    # is a hidden defect and a growing flaky suite. See docs/05-flaky-tests.md.
    when: [runner_system_failure, stuck_or_timeout_failure, api_failure, scheduler_failure]
```

## Why does the gate block a test that passed?

A test that passed on retry failed once on the same commit. That makes it flaky by definition, with no analysis needed. Playwright and pytest-rerunfailures mark such a case with a `<flakyFailure>` element in the JUnit report, and `gate.py` counts them against one policy line:

```yaml
    flaky:
      new_allowed: 0              # no new test may pass only on retry
```

So one retry inside the test framework is allowed, as a detector. Its result becomes evidence for the gate instead of a quiet pass.

> **Warning**
>
> Three retries as normal practice turn a reliability problem into a slow pipeline and a false sense of safety. `scripts/guard_weakening.py` fails a merge request that adds `retry:` with `script_failure` or `--retries` of 3 or more.

## Find the flakes one retry hides

A single retry only catches the flakes that happen to fail first. The nightly `flake-sweep` job runs the same commit five times and diffs the outcomes:

```yaml
flake-sweep:
  stage: scheduled
  # …
  script:
    # Same commit, five runs. Tests with mixed outcomes are flaky by definition.
    - npm ci --prefer-offline --cache .npm
    - for i in 1 2 3 4 5; do npm run test:unit -- --reporter=junit --outputFile="reports/run-$i.xml" || true; done
    - python scripts/flake_report.py reports/run-*.xml --out reports/flaky.md
```

The `|| true` here is allowed: the report decides, not the test command. On the self-test's three runs, where `t.wobbly` fails once, `flake_report.py` prints this and exits 1:

```text
# Flake report (3 runs of the same commit)

- tests observed: **2**
- flaky: **1** (50.00%)
- consistently failing (a real failure, not a flake): **0**

| Test | Failed | Runs | Rate |
|---|---|---|---|
| `t.wobbly` | 1 | 3 | 33% |
```

A test that fails in every run lands in its own list. That is a real failure, not a flake.

## Quarantine with an owner and an expiry

Quarantine means the test keeps running but leaves the blocking gate, with an issue, an owner and a 30-day expiry. The doc's rule for each case:

What happens to a flaky test

```mermaid
flowchart LR
  F[flaky test found] --> K{critical path?}
  K -->|yes| X[fix it now]
  K -->|no, cause unknown| Q[quarantine: owner + 30 days]
  Q -->|fixed| B[back in the gate]
  Q -->|expired| D[delete it]
```

Mark it in code, so the marker travels with the test:

```python
@pytest.mark.flaky_quarantine(issue="QA-1423", expires="2026-10-15")
def test_checkout_under_load(): ...
```

The cap in `quality-gate.yml` is 15 tests or 30 days. One honest gap: `gate.py` does not check that cap itself. It prints `not enforced here: blocking.flaky.quarantine_max (needs its own job)`, so today the cap needs a job of its own.

> **Danger**
>
> The cheap way out is to skip the test. `guard_weakening.py` fails that diff: `tests/test_orders.py: a Python test skipped: @pytest.mark.skip(reason="flaky")`, unless the line carries a `ci-allow:` reason for a reviewer to read.

## Fix the cause, starting with sleep

The doc lists seven causes in the order you meet them. The first four:

1. **Waiting on time instead of state.** `sleep(2)` is the number one cause. Wait for a condition instead.
2. **Shared mutable state.** Two tests on the same user or order id. Generate unique data per test.
3. **Order dependence.** Test B passes only after test A. Randomise the order in the nightly run.
4. **Real time and timezones.** Tests that fail at month end or midnight UTC. Freeze the clock.

Then publish the flake rate weekly next to the pass rate, with a target under 1%. A rising flake rate means the suite is failing, even while every pipeline is green.

## Try it

```sh
for i in 1 2 3 4 5; do
  pytest --junitxml="reports/run-$i.xml" || true
done
python scripts/flake_report.py reports/run-*.xml
```

Then search your CI config for `script_failure`, `--reruns` and unconditional `retry:`. The merge request through the GitLab templates now has a gate that says "no evidence" when nothing ran, and blocks `t.wobbly` until someone owns it.

## Read this with your AI

```text
Read the post "The Retry Button Is Hiding Your Flaky Tests".
1. Summarise it in 5 bullets for an engineer who owns a CI pipeline.
2. Ask me 3 questions to check I understood; wait, then correct me.
3. Read my CI config and list every retry that would rerun a failed test rather than a dead runner.
```

> **Success**
>
> A test that passes only on retry can no longer reach your default branch unnoticed, and every flaky one has an owner and an end date.

## FAQ

### Is any retry of a test acceptable?

One retry inside the test framework is acceptable when it is recorded and reported, because it works as a flake detector. Retries as normal practice, such as --reruns 3 by default, are not.

### Does gate.py enforce the quarantine cap of 15 tests or 30 days?

No. gate.py checks only flaky.new\_allowed and prints that quarantine\_max is not enforced there, so the cap needs a job of its own.

### What flake rate should I aim for?

The repo's flaky-test doc sets a target below 1%, and says that past roughly 2% engineers stop reading failures and start pressing retry.
