Skip to content
fx-blog

API Tests That Fail When the Contract Moves

An API test that checks one field stays green while the backend renames or retypes another. API contract testing in Python: approved JSON Schema files, checked against every live response and against your own model, with the JSON path in every failure.

Harry4 min readIntermediate
  • Verification
MarkdownSave

Key takeaways

  • A test that asserts one field passes while the backend renames or retypes another, so the break surfaces later in the UI suite or in production.
  • An approved contract is a JSON Schema file per response with required fields, types and additionalProperties set to false, and editing it is reviewed as an API change.
  • Contract regression validates every live response against the approved file and reports each violation by its JSON path.
  • A second check needs no network: it compares the model with the approved contract and fails on a removed property, a changed type or a field no longer required.
  • Strict models with extra set to forbid make even an ordinary happy-path test fail when the API returns a field nobody declared.
Contents

TL;DR An API test that checks one field stays green while the backend renames or retypes another, and the break turns up in the UI suite or in production. API contract testing in Python fixes that with one approved JSON Schema file per response, checked twice: against every live response, and against your own model before it ships.

Do this: Commit a JSON Schema for one endpoint you test, with required and additionalProperties: false, and validate every live response against it.

Part 1, A Test That Passed on Retry Is Not Green, ended on the gate's @api line: "check for a contract change before blaming the tests". In the demo suite, nothing said what the booking response is allowed to look like, so a failing booking test could not point at the backend. This part writes that contract down, in the python-api-automation template.

A test asserts that firstname is "Bob". The backend renames another field, or turns totalprice into a string. The test stays green, because it never looked.

This series, in two parts:

  1. A Test That Passed on Retry Is Not Green
  2. API Tests That Fail When the Contract Moves (you are here)

What does a contract look like as a file?

A contract is a JSON Schema in contracts/, one per response, that says which fields must be there, their types, and that nothing else may appear. The demo suite tests restful-booker's booking API, and this is the approved shape of GET /booking/{id}:

jsoncontracts/booking.schema.json
  "description": "APPROVED response contract for GET /booking/{id}. Changing this file = changing the API contract; it must go through review (see CODEOWNERS).",
  "type": "object",
  "additionalProperties": false,
  "required": ["firstname", "lastname", "totalprice", "depositpaid", "bookingdates"],
  "properties": {
    "firstname": { "type": "string" },
    "lastname": { "type": "string" },
    "totalprice": { "type": "integer" },
    "depositpaid": { "type": "boolean" },

I keep the contract in its own folder instead of only in the test code, because test code is what a test author edits to make a test pass. CODEOWNERS puts /contracts/ under its own section, so editing that file is reviewed as an API change.

Check every live response against it

The first check is contract regression: the real response, parsed against the approved file. assert_matches_contract in src/api_framework/utils/contract.py collects every violation and reports each one by its JSON path:

pythonsrc/api_framework/utils/contract.py
    validator = Draft202012Validator(load_contract(contract_name))
    errors = sorted(validator.iter_errors(response.json()), key=lambda e: list(e.path))
    if errors:
        details = "\n".join(
            f"  - {'/'.join(map(str, error.path)) or '<root>'}: {error.message}" for error in errors
        )
        raise AssertionError(f"Response violates contract {contract_name}:\n{details}")

The test that uses it is two lines of intent:

pythontests/api/booking/test_booking_contract.py
    response = booking_service.get(created_booking.bookingid)
    assert_status(response, 200)

    assert_matches_contract(response, "booking.schema.json")

Catch the break in your own model before it ships

The second check needs no network. It compares the Booking model the framework uses against the same approved file, from a consumer's point of view, and lists three kinds of breaking change:

Change in the model What breaking_changes reports
A property is gone <path>: property removed
A type changed <path>: type changed <approved> -> <current>
A required field became optional <path>: no longer guaranteed (was required)

Additions are not breaking for a response, so they pass. The test fails with the whole list:

pythontests/api/booking/test_booking_contract.py
    approved = load_contract("booking.schema.json")
    current = inline_refs(Booking.model_json_schema())

    problems = breaking_changes(approved, current)

    assert not problems, "Breaking API contract changes detected:\n" + "\n".join(problems)
mermaid
flowchart LR
  L[live response] --> A[assert_matches_contract]
  M[Booking model] --> B[breaking_changes]
  C[(contracts/booking.schema.json)] --> A
  C --> B
  A -- violation --> F[fails with the JSON path]
  B -- removed or retyped --> F

Make the model strict as well

The models in src/api_framework/models/booking.py refuse fields they do not declare:

pythonsrc/api_framework/models/booking.py
class Booking(BaseModel):
    model_config = ConfigDict(extra="forbid")

    firstname: str
    lastname: str
    totalprice: int
    depositpaid: bool
    bookingdates: BookingDates
    additionalneeds: str | None = None

Every endpoint in BookingService has a raw method and a *_validated twin, such as get_validated, that checks the status and parses the body into the model. Happy-path tests use the twin. So even a test that never mentions contracts fails loudly when the API starts returning a field nobody declared.

Try it

Try it
make install
pytest -m "regression or smoke" -n auto

The contract tests carry the regression marker, so they run in the full suite. Then pick the endpoint your UI leans on most, write its schema into contracts/, and add one assert_matches_contract call to a test you already have. The demo suite's booking checks show the whole pattern in two files.

Read this with your AI

textPaste into your AI
Read the post "API Tests That Fail When the Contract Moves".
1. Summarise it in 5 bullets for an engineer who writes API tests in Python.
2. Ask me 3 questions to check I understood; wait, then correct me.
3. Take one JSON response from my API and draft its JSON Schema with required fields and additionalProperties set to false.

FAQ

Is adding a new field to a response a breaking change?

Not for consumers, so breaking_changes lets additions pass. The live check still fails on it while the contract sets additionalProperties to false, which forces someone to approve the new field.

Why keep JSON Schema files when the Pydantic models already validate?

The models are code a test author edits. The files in contracts/ are the approved contract, owned in CODEOWNERS, so changing them is reviewed as an API change, and both checks compare against them.

Sources

  1. python-api-automation on GitLab
  2. src/api_framework/utils/contract.py
  3. contracts/booking.schema.json

Was this useful?

Only you and the owner see your answer and rating. Send private feedback

Enjoyed it? Support fx-blog with a coffee.

A Test That Passed on Retry Is Not Green

CI retries a failing test, the retry passes, and the pipeline goes green with a flaky test inside it. How the playwright template stops that: retries run isolated at the end, and a release gate reads the JSON report and fails on any flaky smoke or critical pass.

4 min readIntermediate

Comments (1)

Oldest first
Nguyễn Hoàng Thân

It's great!

You sign in first; what you typed stays here. · 0 / 2,000