Quality Gates in CI: Make Your Tests Block a Bad Merge

TL;DR Tests that run only when someone remembers put the quality gate in a person, and the bad merge goes through on the day that person is busy. Quality gates in CI run the suites in tiers, each one blocking the next, with a commit gate under ten minutes, fresh data on every run and a quarantine lane for flaky tests.

Do this: Time the checks on your last pull request. If they took more than ten minutes, list which tests could move to a nightly run.

Part 5 left the ShopLite checkout with five kinds of tests (designed cases, API, UI, contracts and scans), and every one of them runs when someone remembers to run it, on one laptop, against whatever data is lying around. This part moves them into the pipeline so a failing check stops the merge.

The repo's CI guide opens with the line this part rests on: quality gates live in the pipeline, not in a person. A person forgets, or is on holiday the week of the release.

The QA Skills Path, in seven parts:

1. The QA Skills Path for 2026: What to Learn, in What Order
2. Test Design Techniques: Pick the Few Tests That Find Bugs
3. Test Below the UI: Your First API Tests in Python
4. Why Your UI Tests Flake, and the Fix for Web and Mobile
5. Green Tests, Broken Checkout: The Checks Functional Tests Miss
6. Quality Gates in CI: Make Your Tests Block a Bad Merge (you are here)
7. AI in Testing: Use It Without Trusting It, and Test AI Features

Four tiers, each blocking the next

Match the cost of a test to the stage that runs it, so fast feedback comes first. Here are the repo's four tiers, with the ShopLite suites from parts 2 to 5 placed in them:

Two rules make this a gate. A failing tier blocks the next, because a warning-only suite trains everyone to ignore it. And the commit gate stays under about ten minutes, or engineers bypass it.

The pipeline, in one file

The repo ships the same tiers for GitLab CI and GitHub Actions; the ideas are identical and only the names differ (parallel: 4 in GitLab is strategy.matrix in Actions). The UI stage is the one to read closely:

e2e:
  stage: e2e
  image: mcr.microsoft.com/playwright/python:v1.64.1-noble
  parallel: 4                       # shard across 4 runners
  script:
    - pip install -r requirements.txt
    - pytest tests/e2e --shard-id=$CI_NODE_INDEX --num-shards=$CI_NODE_TOTAL
        --junitxml=report-e2e.xml --alluredir=allure-results
  artifacts:
    when: always                    # keep failure artifacts!
    paths: [allure-results/, test-results/]
    reports: { junit: report-e2e.xml }

Four shards cut the wall-clock time by about four. when: always keeps part 4's screenshots and traces when the job fails, which is when you need them, and the JUnit report shows failures in the merge request. QA_PASSWORD comes from the CI secret store, never from a file in the repo.

Fresh data on every run

Ask why a suite is flaky and the answer is often data: two tests share one user, and one empties the cart while the other is paying. So every test creates what it needs, with a unique tag, and cleans up even when it fails:

@pytest.fixture
def discount_code():
    code = f"QA{uuid.uuid4().hex[:10].upper()}"   # unique, 12 chars, tagged "QA"
    requests.post(f"{BASE}/test-support/discount-codes",
                  json={"code": code, "percent": 10, "minSubtotal": 50}, timeout=10).raise_for_status()
    yield code
    requests.delete(f"{BASE}/test-support/discount-codes/{code}", timeout=10)  # runs even if the test fails

A killed job skips teardown, so a scheduled job deletes tagged data older than a day. Cleaner still is a new environment: the repo's docker-compose.test.yml runs Postgres on tmpfs, so the database is gone when the pipeline stops.

When the build goes red

The first question on a red build is whether the test failed or the environment did. They look the same and have different owners. The repo's Linux recipe greps the log for each kind:

grep -iE "(connection refused|timed out|no space left|killed|OOM)" build.log
grep -E "(FAILED|AssertionError|assert)" build.log | head -20

Its sample report finds 49 new failures in the checkout suite, all one error: ConnectionRefusedError: payment-service:8080. The suite hadn't changed; the payment service wasn't running. When it is the test, and only sometimes, it goes to quarantine:

flowchart LR
  R[Red build] --> Q{Test or environment?}
  Q -->|environment| P[Hand to the platform team]
  Q -->|test| F{Passes on retry?}
  F -->|no| B[Real failure: report it]
  F -->|yes, flaky| Z[Quarantine today, owner + deadline]
  Z --> X[Fix the cause, restore to the gate]

A quarantined test leaves the gate but keeps running nightly, so you still collect data on it.

Try it

pip install pytest-xdist
pytest tests/unit -n auto --junitxml=report-unit.xml

Run your fastest suite with -n auto and time it: that number is your commit gate.

The ShopLite checkout now has a gate that blocks a bad merge without anyone watching. Two questions are left. An AI assistant could have drafted half of these tests, so how would you know which ones can't fail? And ShopLite has just added a shopping assistant that answers customers in its own words. Part 7, AI in Testing: Use It Without Trusting It, and Test AI Features, answers both.

Read this with your AI

Read the post "Quality Gates in CI: Make Your Tests Block a Bad Merge".
1. I'll paste my CI config. Sort its jobs into the four tiers and flag anything in the wrong one.
2. Tell me which of my tests share data, from the test names and fixtures I paste.
3. Draft a one-paragraph flaky-test policy my team could agree on this week.