Green Tests, Broken Checkout: The Checks Functional Tests Miss

TL;DR A green functional suite says the checkout works for one honest user with a mouse. It can't see a renamed field that breaks every order, a checkout that slows down at peak, an open security hole, or a payment form a keyboard can't finish. Three fixes are non-functional testing (load against an agreed target, a security scan, an accessibility check); the fourth is a contract test between services.

Do this: Ask your product owner for the latency target of your busiest page. If nobody has one, that is your first finding.

Part 4 left the ShopLite checkout green on three browsers and two phones, for one user with a mouse, against services that never change. This part checks what happens with 2,000 users, a keyboard, and a pricing team that renames one field.

The repo tells the rename as a story. The checkout reads discountAmount from the pricing API. The pricing team renames it discount_amount, deploys on Tuesday, and checkout shows "€NaN" to every customer. Both teams had green tests, each against its own idea of the other.

The QA Skills Path, in seven parts:

1. The QA Skills Path for 2026: What to Learn, in What Order
2. Test Design Techniques: Pick the Few Tests That Find Bugs
3. Test Below the UI: Your First API Tests in Python
4. Why Your UI Tests Flake, and the Fix for Web and Mobile
5. Green Tests, Broken Checkout: The Checks Functional Tests Miss (you are here)
6. Quality Gates in CI: Make Your Tests Block a Bad Merge
7. AI in Testing: Use It Without Trusting It, and Test AI Features

A contract test catches the rename

A consumer-driven contract records what the checkout reads from the pricing API, and the pricing API's pipeline proves it still returns that. With Pact:

flowchart LR
  C[checkout-web test] -->|pact file| B[Pact Broker]
  B --> P[pricing-api replays it]
  P -->|verified or not| B
  B --> D{can-i-deploy?}
  D -->|no| X[deploy stops]

The consumer test uses matchers such as like and decimal, so the contract fixes the shape of discountAmount, not its value. Before either side deploys, one command asks the broker whether the versions agree:

pact-broker can-i-deploy --pacticipant checkout-web --version "$GIT_COMMIT" --to-environment production

The rename now fails in a pipeline, not in front of customers.

Load-test against a number someone agreed

Every threshold in the repo's performance guide is marked as an example. The real target is a service level objective, such as "p95 checkout latency under 800 ms over 28 days" (95% of requests faster than that). With no SLO, run a baseline and agree targets with the product owner first. Then k6 can fail the run for you:

thresholds: {
  'http_req_duration': ['p(95)<500'],
  'http_req_failed': ['rate<0.001'],
},

Measure percentiles, since a fast average can hide a slow p99. Use an open model (a fixed arrival rate, k6's constant-arrival-rate), because a fixed pool of virtual users slows down with the system and hides the problem. Little's Law sizes it: 50 requests a second at 0.5 s response and 2.5 s think time need about 150 concurrent users.

In the browser, Core Web Vitals measure what the customer feels: LCP ≤ 2.5 s, INP ≤ 200 ms, CLS ≤ 0.1. INP replaced FID in March 2024.

Scan for the cheap security wins

A ZAP baseline scan spiders the site and reports what it sees without attacking, so it is safe to run on every merge request against a test environment:

docker run --rm -v "$(pwd)":/zap/wrk:rw -t ghcr.io/zaproxy/zaproxy:stable \
  zap-baseline.py -t https://shop.staging.example.test -r zap-report.html -c zap-rules.conf

It exits 0 when clean, 1 on a FAIL, 2 on a WARN, 3 on a scanner error; -I ignores warnings while you triage the first report. Add npm audit or pip-audit for vulnerable dependencies and gitleaks for committed secrets.

Check that everyone can pay

The European Accessibility Act has applied to e-commerce in the EU since 28 June 2025, and the standards behind it are built on WCAG; level AA of WCAG 2.2 is the usual target. The repo's axe test scans states, not only pages; here, ShopLite's discount error:

// … goto('/checkout') first
await page.getByLabel('Discount code').fill('NOPE');
await page.getByRole('button', { name: 'Apply' }).click();
await expect(page.getByText('This code is not valid')).toBeVisible();

// Scan only the part of the page that changed
const results = await new AxeBuilder({ page })
  .include('#discount-section')
  .withTags(WCAG_AA)
  .analyze();

expect(results.violations).toEqual([]);

In Deque's study of real audits, automated checks covered about 57% of issues by volume. The rest needs a person, so put the mouse away and buy something: Tab moves in reading order, focus is always visible, payment radios work with arrow keys, and a failed "Apply" never throws focus to the top of the page.

Try it

npm install -D @axe-core/playwright
npx playwright test checkout.a11y.spec.ts

The ShopLite checkout now has five kinds of tests: designed cases, API, UI, contracts and these scans. Every one of them runs when someone remembers to run it, on one laptop, against whatever data is lying around. Part 6, Quality Gates in CI: Make Your Tests Block a Bad Merge, moves them all into the pipeline so a failing check stops the merge.

Read this with your AI

Read the post "Green Tests, Broken Checkout: The Checks Functional Tests Miss".
1. Ask me 5 questions about my product, then tell me which of the four checks I'm missing most.
2. Draft the questions I should ask my product owner to agree a latency SLO.
3. Give me a 10-minute keyboard-only checklist for my own checkout or sign-up page.