---
title: "Green Tests, Broken Checkout: The Checks Functional Tests Miss"
author: "Harry"
published: 2026-10-31T04:45:51.936Z
updated: 2026-10-31T05:06:18.752Z
category: "QA & Testing"
tags: ["qa-skills-path"]
canonical: "https://fxblog.dev/posts/non-functional-testing-for-testers"
---

# Green Tests, Broken Checkout: The Checks Functional Tests Miss

## Key takeaways

- Two teams can both have green tests while their services disagree on a field name; a consumer-driven contract test catches it before the deploy.
- Load-test targets come from a service level objective agreed with the business, not from example thresholds.
- Measure latency percentiles, and use an open workload model so the load does not slow down with the system.
- A ZAP baseline scan is passive and safe to run on every merge request against a test environment, but it cannot find business-logic flaws.
- Automated accessibility checks found about 57% of issues by volume in Deque's study; a keyboard and screen reader pass finds the rest.

> **TL;DR** A green functional suite says the checkout works for one honest user with a mouse. It can't see a renamed field that breaks every order, a checkout that slows down at peak, an open security hole, or a payment form a keyboard can't finish. Three fixes are non-functional testing (load against an agreed target, a security scan, an accessibility check); the fourth is a contract test between services.
>
> **Do this:** Ask your product owner for the latency target of your busiest page. If nobody has one, that is your first finding.

Part 4 left the ShopLite checkout green on three browsers and two phones, for one user with a mouse, against services that never change. This part checks what happens with 2,000 users, a keyboard, and a pricing team that renames one field.

The repo tells the rename as a story. The checkout reads `discountAmount` from the pricing API. The pricing team renames it `discount_amount`, deploys on Tuesday, and checkout shows "€NaN" to every customer. Both teams had green tests, each against its own idea of the other.

**The QA Skills Path, in seven parts:**

1. **The QA Skills Path for 2026: What to Learn, in What Order**
2. **Test Design Techniques: Pick the Few Tests That Find Bugs**
3. **Test Below the UI: Your First API Tests in Python**
4. **Why Your UI Tests Flake, and the Fix for Web and Mobile**
5. **Green Tests, Broken Checkout: The Checks Functional Tests Miss** (you are here)
6. **Quality Gates in CI: Make Your Tests Block a Bad Merge**
7. **AI in Testing: Use It Without Trusting It, and Test AI Features**

## A contract test catches the rename

A consumer-driven contract records what the checkout reads from the pricing API, and the pricing API's pipeline proves it still returns that. With Pact:

Pact, from consumer test to deploy

```mermaid
flowchart LR
  C[checkout-web test] -->|pact file| B[Pact Broker]
  B --> P[pricing-api replays it]
  P -->|verified or not| B
  B --> D{can-i-deploy?}
  D -->|no| X[deploy stops]
```

The consumer test uses matchers such as `like` and `decimal`, so the contract fixes the shape of `discountAmount`, not its value. Before either side deploys, one command asks the broker whether the versions agree:

```sh
pact-broker can-i-deploy --pacticipant checkout-web --version "$GIT_COMMIT" --to-environment production
```

The rename now fails in a pipeline, not in front of customers.

## Load-test against a number someone agreed

Every threshold in the repo's performance guide is marked as an example. The real target is a service level objective, such as "p95 checkout latency under 800 ms over 28 days" (95% of requests faster than that). With no SLO, run a baseline and agree targets with the product owner first. Then k6 can fail the run for you:

```javascript
thresholds: {
  'http_req_duration': ['p(95)<500'],
  'http_req_failed': ['rate<0.001'],
},
```

Measure percentiles, since a fast average can hide a slow p99. Use an open model (a fixed arrival rate, k6's `constant-arrival-rate`), because a fixed pool of virtual users slows down with the system and hides the problem. Little's Law sizes it: 50 requests a second at 0.5 s response and 2.5 s think time need about 150 concurrent users.

In the browser, Core Web Vitals measure what the customer feels: LCP ≤ 2.5 s, INP ≤ 200 ms, CLS ≤ 0.1. INP replaced FID in March 2024.

## Scan for the cheap security wins

A ZAP baseline scan spiders the site and reports what it sees without attacking, so it is safe to run on every merge request against a test environment:

```sh
docker run --rm -v "$(pwd)":/zap/wrk:rw -t ghcr.io/zaproxy/zaproxy:stable \
  zap-baseline.py -t https://shop.staging.example.test -r zap-report.html -c zap-rules.conf
```

It exits 0 when clean, 1 on a FAIL, 2 on a WARN, 3 on a scanner error; `-I` ignores warnings while you triage the first report. Add `npm audit` or `pip-audit` for vulnerable dependencies and `gitleaks` for committed secrets.

> **Warning**
>
> A clean scan does not mean a safe checkout. No scanner knows that `SAVE10` should not apply twice, and that kind of logic flaw is where most serious findings are. Part 3's two-user test stays in the suite.

## Check that everyone can pay

The European Accessibility Act has applied to e-commerce in the EU since 28 June 2025, and the standards behind it are built on WCAG; level AA of WCAG 2.2 is the usual target. The repo's axe test scans states, not only pages; here, ShopLite's discount error:

```ts
// … goto('/checkout') first
await page.getByLabel('Discount code').fill('NOPE');
await page.getByRole('button', { name: 'Apply' }).click();
await expect(page.getByText('This code is not valid')).toBeVisible();

// Scan only the part of the page that changed
const results = await new AxeBuilder({ page })
  .include('#discount-section')
  .withTags(WCAG_AA)
  .analyze();

expect(results.violations).toEqual([]);
```

In Deque's study of real audits, automated checks covered about 57% of issues by volume. The rest needs a person, so put the mouse away and buy something: `Tab` moves in reading order, focus is always visible, payment radios work with arrow keys, and a failed "Apply" never throws focus to the top of the page.

> **Tip**
>
> WCAG 2.2 asks for click targets of at least 24×24 CSS pixels, or enough spacing. Check the "+" and "−" quantity buttons first.

## Try it

```sh
npm install -D @axe-core/playwright
npx playwright test checkout.a11y.spec.ts
```

The ShopLite checkout now has five kinds of tests: designed cases, API, UI, contracts and these scans. Every one of them runs when someone remembers to run it, on one laptop, against whatever data is lying around. Part 6, **Quality Gates in CI: Make Your Tests Block a Bad Merge**, moves them all into the pipeline so a failing check stops the merge.

## Read this with your AI

```text
Read the post "Green Tests, Broken Checkout: The Checks Functional Tests Miss".
1. Ask me 5 questions about my product, then tell me which of the four checks I'm missing most.
2. Draft the questions I should ask my product owner to agree a latency SLO.
3. Give me a 10-minute keyboard-only checklist for my own checkout or sign-up page.
```

> **Success**
>
> You can see what functional tests can't, and you have a small check for each blind spot. Next: **Quality Gates in CI: Make Your Tests Block a Bad Merge**.

## FAQ

### Is contract testing non-functional testing?

No. A contract test checks that two services agree on the shape of their requests and responses, which is functional. It belongs with load, security and accessibility checks because a green functional suite misses all four.

### Where do performance test thresholds come from?

From your service level objectives, such as p95 checkout latency under 800 ms over 28 days. If there is no SLO, run a baseline, agree targets with the product owner and write them down before judging any result.

### Is a clean Lighthouse or axe report enough for accessibility?

No. Automated tools catch only part of the problems. Complete the main flow with a keyboard only and with a screen reader, and check dynamic states such as error messages and modals.