A Test That Passed on Retry Is Not Green
CI retries a failing test, the retry passes, and the pipeline goes green with a flaky test inside it. How the playwright template stops that: retries run isolated at the end, and a release gate reads the JSON report and fails on any flaky smoke or critical pass.
Key takeaways
- A test that passed only on retry is recorded with the status flaky, and a pipeline that ignores that field reports it as green.
- Retrying in CI only, with the isolated strategy that reruns failures in one worker after the parallel run, stops a test from passing on retry because the load moved.
- A release gate after the suite reads the JSON report, because it carries tags and flaky status as fields and JUnit does not.
- The gate fails the build on any failing or flaky @smoke or @critical test, on @regression below 95%, and on any failing test with no suite tag.
- A flaky test gets one fix attempt and is then quarantined with a ticket; raising the retry count to make a suite green is never the fix.
Contents
FAQ
Should I turn retries off to see flaky tests?
No. Keep retries in CI so a run finishes and records a trace on the first retry, but make a release gate treat a flaky smoke or critical test as a failure.
Can the gate ever let a flaky test through?
Only when someone sets GATE_ALLOW_FLAKY=1 on purpose. The template treats that as a team decision, not a way to turn a red pipeline green.
How do I tell a product bug from an automation failure?
Read the error type. A product bug fails an assertion with expected and actual values; an automation failure times out on a locator or a fixture.
Sources
Only you and the owner see your answer and rating. Send private feedback
Enjoyed it? Support fx-blog with a coffee.
Related posts
API Tests That Fail When the Contract Moves
An API test that checks one field stays green while the backend renames or retypes another. API contract testing in Python: approved JSON Schema files, checked against every live response and against your own model, with the JSON path in every failure.
Comments
Oldest firstNo comments yet. Start the conversation.
Your login smoke test fails on the first try. CI runs it again, it passes, and the pipeline goes green. Nobody opens the report, because the badge is green. The flake is still there, and the next retry may not be so kind.
This series, in two parts:
The running example is the demo suite in the playwright template: it runs out of the box against public practice systems, a shop in the browser and a booking API over HTTP.
Why does a retry turn a flaky test green?
Because the retry runs under the same conditions that made the test flaky, and sometimes it gets lucky. A test that only fails when many workers hit the page at once can pass on its second try, simply because the load has moved.
So the template retries in CI only, and it retries late:
That makes a retry-pass rarer. It does not make it a pass. The runner still records it, with the status
flaky, and most pipelines never read that field.Make the gate read flaky as a failure
I put the release gate after the suite, as its own step, and had it read the JSON report rather than JUnit. JUnit has no field for tags or for flaky status; the JSON report carries both. The gate is
scripts/quality-gate.mjs, and this is the branch that matters:It runs that for
@smokeand@critical.@regressiongets a pass-rate threshold instead, 95% by default, because a large suite always has some noise. A failing test with no suite tag fails the gate by name, so nothing slips past it untagged. A clean run prints this (the README's example):Let the tests finish, then let the gate fail the build
If the pipeline stops at the first red test job, the report and traces never get published, and you lose the evidence for the flake. All four pipelines in the template publish the reports whatever the tests did, merge the shards, and only then run the gate.
In
.gitlab-ci.ymlthe report job runswhen: always:npx playwright merge-reports --reporter html,junit,json,allure-playwright blob-report, thennpm run gate.Find the cause from the trace, not from a rerun
A flaky test you cannot explain comes back. The template records a trace on the first retry, with three kinds of snapshot:
The DOM snapshot replays the page, the ARIA snapshot shows what a screen reader saw, and the screen is the rendered frame. Together they answer "why did this locator miss?" from the CI artifact, with
npx playwright show-trace <trace.zip>.The demo suite's rules block two habits that make tests flaky. Locators use the
data-testattribute, never a styling class or DOM position. Hard waits are a lint error ('playwright/no-wait-for-timeout': 'error'), so a test waits on an assertion that retries until the page catches up:Fix it once, or quarantine it
The flaky policy in
CONTRIBUTING.mdis short:@quarantinewith a ticket; gates run with--grep-invert @quarantinenpm run test:release-gateruns@criticalwith--fail-on-flaky-testsTry it
Then look at the gate's
@apiline. When a booking API test fails, the gate does not blame the test. It says "check for a contract change before blaming the tests", because a broken API test usually means the backend moved. In the demo suite, nothing yet says what the booking response is allowed to look like, so nobody can tell. Part 2, API Tests That Fail When the Contract Moves, writes that contract down for the booking API and fails the suite when it moves.Read this with your AI