Skip to content
fx-blog

A Test That Passed on Retry Is Not Green

CI retries a failing test, the retry passes, and the pipeline goes green with a flaky test inside it. How the playwright template stops that: retries run isolated at the end, and a release gate reads the JSON report and fails on any flaky smoke or critical pass.

Harry4 min readIntermediate
  • Judgment
  • Verification
MarkdownSave

Key takeaways

  • A test that passed only on retry is recorded with the status flaky, and a pipeline that ignores that field reports it as green.
  • Retrying in CI only, with the isolated strategy that reruns failures in one worker after the parallel run, stops a test from passing on retry because the load moved.
  • A release gate after the suite reads the JSON report, because it carries tags and flaky status as fields and JUnit does not.
  • The gate fails the build on any failing or flaky @smoke or @critical test, on @regression below 95%, and on any failing test with no suite tag.
  • A flaky test gets one fix attempt and is then quarantined with a ticket; raising the retry count to make a suite green is never the fix.
Contents

TL;DR CI retries a failing test, the retry passes, and the pipeline goes green with a flaky test inside it. Playwright flaky tests stay visible only if something downstream refuses them: here a release gate reads the JSON report and fails the build on any flaky @smoke or @critical pass.

Do this: Add a step after your test job that reads report.json and fails when a smoke or critical test has the status flaky.

Your login smoke test fails on the first try. CI runs it again, it passes, and the pipeline goes green. Nobody opens the report, because the badge is green. The flake is still there, and the next retry may not be so kind.

This series, in two parts:

  1. A Test That Passed on Retry Is Not Green (you are here)
  2. API Tests That Fail When the Contract Moves

The running example is the demo suite in the playwright template: it runs out of the box against public practice systems, a shop in the browser and a booking API over HTTP.

Why does a retry turn a flaky test green?

Because the retry runs under the same conditions that made the test flaky, and sometimes it gets lucky. A test that only fails when many workers hit the page at once can pass on its second try, simply because the load has moved.

So the template retries in CI only, and it retries late:

tsplaywright.config.ts
  retries: IS_CI ? 2 : 0,

  // 'isolated' (1.62+) runs retries sequentially in a single worker at the end
  // of the run. A test that only fails under parallel load stops passing on
  // retry by luck, so a green retry actually means something.
  retryStrategy: IS_CI ? 'isolated' : 'immediate',

That makes a retry-pass rarer. It does not make it a pass. The runner still records it, with the status flaky, and most pipelines never read that field.

Make the gate read flaky as a failure

I put the release gate after the suite, as its own step, and had it read the JSON report rather than JUnit. JUnit has no field for tags or for flaky status; the JSON report carries both. The gate is scripts/quality-gate.mjs, and this is the branch that matters:

jsscripts/quality-gate.mjs
  if (s.flaky > 0 && !ALLOW_FLAKY) {
    failures.push(
      `@${tag} has ${s.flaky} flaky test(s) — a retry-pass is not a release signal. ` +
        'Fix them, or set GATE_ALLOW_FLAKY=1 deliberately.',
    );
  }

It runs that for @smoke and @critical. @regression gets a pass-rate threshold instead, 95% by default, because a large suite always has some noise. A failing test with no suite tag fails the gate by name, so nothing slips past it untagged. A clean run prints this (the README's example):

textnpm run gate
Quality gate (test-results/report.json)
  @smoke      4/4 passed, 0 flaky, 0 failed
  @critical   2/2 passed, 0 flaky, 0 failed
  @regression  5/5 passed (100.0%, threshold 95%), 0 flaky
  @api         2/2 passed, 0 failed
  @db          2 skipped — no DB_* configuration for this ENV
  overall      10/10 passed, 0 flaky, 0 failed, 2 skipped

QUALITY GATE PASSED.

Let the tests finish, then let the gate fail the build

If the pipeline stops at the first red test job, the report and traces never get published, and you lose the evidence for the flake. All four pipelines in the template publish the reports whatever the tests did, merge the shards, and only then run the gate.

mermaid
flowchart LR
  A[pre-test gate] --> B[sharded tests]
  B --> C[merge reports]
  C --> D[publish]
  D --> E[release gate]
  E -- fails --> F[notify]

In .gitlab-ci.yml the report job runs when: always: npx playwright merge-reports --reporter html,junit,json,allure-playwright blob-report, then npm run gate.

Find the cause from the trace, not from a rerun

A flaky test you cannot explain comes back. The template records a trace on the first retry, with three kinds of snapshot:

tsplaywright.config.ts
    trace: {
      mode: 'on-first-retry',
      snapshots: { dom: true, aria: true, screen: true },
    },

The DOM snapshot replays the page, the ARIA snapshot shows what a screen reader saw, and the screen is the rendered frame. Together they answer "why did this locator miss?" from the CI artifact, with npx playwright show-trace <trace.zip>.

The demo suite's rules block two habits that make tests flaky. Locators use the data-test attribute, never a styling class or DOM position. Hard waits are a lint error ('playwright/no-wait-for-timeout': 'error'), so a test waits on an assertion that retries until the page catches up:

tstests/ui/login.spec.ts
      await loginPage.login('locked_out_user', envConfig.users.standard.password);
      await expect(loginPage.errorMessage).toContainText('locked out');

Fix it once, or quarantine it

The flaky policy in CONTRIBUTING.md is short:

When What happens
Weekly Flaky tests are triaged from the HTML report's Flaky filter
First time One fix attempt
Still flaky Tag it @quarantine with a ticket; gates run with --grep-invert @quarantine
Release npm run test:release-gate runs @critical with --fail-on-flaky-tests
Never Raise the retry count to make a suite green

Try it

Try it
npm run test:release-gate
npm run gate

Then look at the gate's @api line. When a booking API test fails, the gate does not blame the test. It says "check for a contract change before blaming the tests", because a broken API test usually means the backend moved. In the demo suite, nothing yet says what the booking response is allowed to look like, so nobody can tell. Part 2, API Tests That Fail When the Contract Moves, writes that contract down for the booking API and fails the suite when it moves.

Read this with your AI

textPaste into your AI
Read the post "A Test That Passed on Retry Is Not Green".
1. Summarise it in 5 bullets for an engineer who owns a CI test suite.
2. Ask me 3 questions to check I understood; wait, then correct me.
3. Look at my CI config and tell me where a retry-pass would turn my pipeline green.

FAQ

Should I turn retries off to see flaky tests?

No. Keep retries in CI so a run finishes and records a trace on the first retry, but make a release gate treat a flaky smoke or critical test as a failure.

Can the gate ever let a flaky test through?

Only when someone sets GATE_ALLOW_FLAKY=1 on purpose. The template treats that as a team decision, not a way to turn a red pipeline green.

How do I tell a product bug from an automation failure?

Read the error type. A product bug fails an assertion with expected and actual values; an automation failure times out on a locator or a fixture.

Sources

  1. playwright template on GitLab
  2. scripts/quality-gate.mjs
  3. CONTRIBUTING.md: flaky test policy

Was this useful?

Only you and the owner see your answer and rating. Send private feedback

Enjoyed it? Support fx-blog with a coffee.

Comments

Oldest first

No comments yet. Start the conversation.

You sign in first; what you typed stays here. · 0 / 2,000