---
title: "A Test That Passed on Retry Is Not Green"
author: "Harry"
published: 2026-11-02T02:00:02.816Z
category: "Test Automation"
tags: ["test-automation"]
canonical: "https://fxblog.dev/posts/playwright-flaky-tests-release-gate"
---

# A Test That Passed on Retry Is Not Green

## Key takeaways

- A test that passed only on retry is recorded with the status flaky, and a pipeline that ignores that field reports it as green.
- Retrying in CI only, with the isolated strategy that reruns failures in one worker after the parallel run, stops a test from passing on retry because the load moved.
- A release gate after the suite reads the JSON report, because it carries tags and flaky status as fields and JUnit does not.
- The gate fails the build on any failing or flaky @smoke or @critical test, on @regression below 95%, and on any failing test with no suite tag.
- A flaky test gets one fix attempt and is then quarantined with a ticket; raising the retry count to make a suite green is never the fix.

> **TL;DR** CI retries a failing test, the retry passes, and the pipeline goes green with a flaky test inside it. Playwright flaky tests stay visible only if something downstream refuses them: here a release gate reads the JSON report and fails the build on any flaky `@smoke` or `@critical` pass.
>
> **Do this:** Add a step after your test job that reads `report.json` and fails when a smoke or critical test has the status `flaky`.

Your login smoke test fails on the first try. CI runs it again, it passes, and the pipeline goes green. Nobody opens the report, because the badge is green. The flake is still there, and the next retry may not be so kind.

**This series, in two parts:**

1. **A Test That Passed on Retry Is Not Green** (you are here)
2. **API Tests That Fail When the Contract Moves**

The running example is the demo suite in the [playwright template](https://gitlab.com/agentic-fx/playwright): it runs out of the box against public practice systems, a shop in the browser and a booking API over HTTP.

## Why does a retry turn a flaky test green?

Because the retry runs under the same conditions that made the test flaky, and sometimes it gets lucky. A test that only fails when many workers hit the page at once can pass on its second try, simply because the load has moved.

So the template retries in CI only, and it retries late:

```ts
  retries: IS_CI ? 2 : 0,

  // 'isolated' (1.62+) runs retries sequentially in a single worker at the end
  // of the run. A test that only fails under parallel load stops passing on
  // retry by luck, so a green retry actually means something.
  retryStrategy: IS_CI ? 'isolated' : 'immediate',
```

That makes a retry-pass rarer. It does not make it a pass. The runner still records it, with the status `flaky`, and most pipelines never read that field.

## Make the gate read flaky as a failure

I put the release gate after the suite, as its own step, and had it read the JSON report rather than JUnit. JUnit has no field for tags or for flaky status; the JSON report carries both. The gate is `scripts/quality-gate.mjs`, and this is the branch that matters:

```js
  if (s.flaky > 0 && !ALLOW_FLAKY) {
    failures.push(
      `@${tag} has ${s.flaky} flaky test(s) — a retry-pass is not a release signal. ` +
        'Fix them, or set GATE_ALLOW_FLAKY=1 deliberately.',
    );
  }
```

It runs that for `@smoke` and `@critical`. `@regression` gets a pass-rate threshold instead, 95% by default, because a large suite always has some noise. A failing test with no suite tag fails the gate by name, so nothing slips past it untagged. A clean run prints this (the README's example):

```text
Quality gate (test-results/report.json)
  @smoke      4/4 passed, 0 flaky, 0 failed
  @critical   2/2 passed, 0 flaky, 0 failed
  @regression  5/5 passed (100.0%, threshold 95%), 0 flaky
  @api         2/2 passed, 0 failed
  @db          2 skipped — no DB_* configuration for this ENV
  overall      10/10 passed, 0 flaky, 0 failed, 2 skipped

QUALITY GATE PASSED.
```

> **Warning**
>
> `GATE_ALLOW_FLAKY=1` turns the flaky check off. It exists for the day you decide to ship anyway, as a team. Don't fix a red pipeline by loosening the gate.

## Let the tests finish, then let the gate fail the build

If the pipeline stops at the first red test job, the report and traces never get published, and you lose the evidence for the flake. All four pipelines in the template publish the reports whatever the tests did, merge the shards, and only then run the gate.

Where the release gate sits

```mermaid
flowchart LR
  A[pre-test gate] --> B[sharded tests]
  B --> C[merge reports]
  C --> D[publish]
  D --> E[release gate]
  E -- fails --> F[notify]
```

In `.gitlab-ci.yml` the report job runs `when: always`: `npx playwright merge-reports --reporter html,junit,json,allure-playwright blob-report`, then `npm run gate`.

## Find the cause from the trace, not from a rerun

A flaky test you cannot explain comes back. The template records a trace on the first retry, with three kinds of snapshot:

```ts
    trace: {
      mode: 'on-first-retry',
      snapshots: { dom: true, aria: true, screen: true },
    },
```

The DOM snapshot replays the page, the ARIA snapshot shows what a screen reader saw, and the screen is the rendered frame. Together they answer "why did this locator miss?" from the CI artifact, with `npx playwright show-trace <trace.zip>`.

> **Tip**
>
> Read the error type first. A product bug fails an assertion, expected against actual. An automation failure times out on a locator or a fixture.

The demo suite's rules block two habits that make tests flaky. Locators use the `data-test` attribute, never a styling class or DOM position. Hard waits are a lint error (`'playwright/no-wait-for-timeout': 'error'`), so a test waits on an assertion that retries until the page catches up:

```ts
      await loginPage.login('locked_out_user', envConfig.users.standard.password);
      await expect(loginPage.errorMessage).toContainText('locked out');
```

## Fix it once, or quarantine it

The flaky policy in `CONTRIBUTING.md` is short:

| When        | What happens                                                                   |
| ----------- | ------------------------------------------------------------------------------ |
| Weekly      | Flaky tests are triaged from the HTML report's Flaky filter                    |
| First time  | One fix attempt                                                                |
| Still flaky | Tag it `@quarantine` with a ticket; gates run with `--grep-invert @quarantine` |
| Release     | `npm run test:release-gate` runs `@critical` with `--fail-on-flaky-tests`      |
| Never       | Raise the retry count to make a suite green                                    |

## Try it

```sh
npm run test:release-gate
npm run gate
```

Then look at the gate's `@api` line. When a booking API test fails, the gate does not blame the test. It says "check for a contract change before blaming the tests", because a broken API test usually means the backend moved. In the demo suite, nothing yet says what the booking response is allowed to look like, so nobody can tell. Part 2, **API Tests That Fail When the Contract Moves**, writes that contract down for the booking API and fails the suite when it moves.

## Read this with your AI

```text
Read the post "A Test That Passed on Retry Is Not Green".
1. Summarise it in 5 bullets for an engineer who owns a CI test suite.
2. Ask me 3 questions to check I understood; wait, then correct me.
3. Look at my CI config and tell me where a retry-pass would turn my pipeline green.
```

> **Success**
>
> You can make a pass-on-retry fail the release instead of hiding in a green pipeline. Next: **API Tests That Fail When the Contract Moves**.

## FAQ

### Should I turn retries off to see flaky tests?

No. Keep retries in CI so a run finishes and records a trace on the first retry, but make a release gate treat a flaky smoke or critical test as a failure.

### Can the gate ever let a flaky test through?

Only when someone sets GATE\_ALLOW\_FLAKY=1 on purpose. The template treats that as a team decision, not a way to turn a red pipeline green.

### How do I tell a product bug from an automation failure?

Read the error type. A product bug fails an assertion with expected and actual values; an automation failure times out on a locator or a fixture.
