---
title: "API Tests That Fail When the Contract Moves"
author: "Harry"
published: 2026-11-02T02:00:03.354Z
category: "Test Automation"
tags: ["test-automation"]
canonical: "https://fxblog.dev/posts/api-contract-testing-python-json-schema"
---

# API Tests That Fail When the Contract Moves

## Key takeaways

- A test that asserts one field passes while the backend renames or retypes another, so the break surfaces later in the UI suite or in production.
- An approved contract is a JSON Schema file per response with required fields, types and additionalProperties set to false, and editing it is reviewed as an API change.
- Contract regression validates every live response against the approved file and reports each violation by its JSON path.
- A second check needs no network: it compares the model with the approved contract and fails on a removed property, a changed type or a field no longer required.
- Strict models with extra set to forbid make even an ordinary happy-path test fail when the API returns a field nobody declared.

> **TL;DR** An API test that checks one field stays green while the backend renames or retypes another, and the break turns up in the UI suite or in production. API contract testing in Python fixes that with one approved JSON Schema file per response, checked twice: against every live response, and against your own model before it ships.
>
> **Do this:** Commit a JSON Schema for one endpoint you test, with `required` and `additionalProperties: false`, and validate every live response against it.

[Part 1](01-playwright-flaky-tests-release-gate.md), **A Test That Passed on Retry Is Not Green**, ended on the gate's `@api` line: "check for a contract change before blaming the tests". In the demo suite, nothing said what the booking response is allowed to look like, so a failing booking test could not point at the backend. This part writes that contract down, in the [python-api-automation](https://gitlab.com/agentic-fx/python-api-automation) template.

A test asserts that `firstname` is "Bob". The backend renames another field, or turns `totalprice` into a string. The test stays green, because it never looked.

**This series, in two parts:**

1. **A Test That Passed on Retry Is Not Green**
2. **API Tests That Fail When the Contract Moves** (you are here)

## What does a contract look like as a file?

A contract is a JSON Schema in `contracts/`, one per response, that says which fields must be there, their types, and that nothing else may appear. The demo suite tests restful-booker's booking API, and this is the approved shape of `GET /booking/{id}`:

```json
  "description": "APPROVED response contract for GET /booking/{id}. Changing this file = changing the API contract; it must go through review (see CODEOWNERS).",
  "type": "object",
  "additionalProperties": false,
  "required": ["firstname", "lastname", "totalprice", "depositpaid", "bookingdates"],
  "properties": {
    "firstname": { "type": "string" },
    "lastname": { "type": "string" },
    "totalprice": { "type": "integer" },
    "depositpaid": { "type": "boolean" },
```

I keep the contract in its own folder instead of only in the test code, because test code is what a test author edits to make a test pass. `CODEOWNERS` puts `/contracts/` under its own section, so editing that file is reviewed as an API change.

## Check every live response against it

The first check is contract regression: the real response, parsed against the approved file. `assert_matches_contract` in `src/api_framework/utils/contract.py` collects every violation and reports each one by its JSON path:

```python
    validator = Draft202012Validator(load_contract(contract_name))
    errors = sorted(validator.iter_errors(response.json()), key=lambda e: list(e.path))
    if errors:
        details = "\n".join(
            f"  - {'/'.join(map(str, error.path)) or '<root>'}: {error.message}" for error in errors
        )
        raise AssertionError(f"Response violates contract {contract_name}:\n{details}")
```

The test that uses it is two lines of intent:

```python
    response = booking_service.get(created_booking.bookingid)
    assert_status(response, 200)

    assert_matches_contract(response, "booking.schema.json")
```

> **Note**
>
> `additionalProperties: false` is what catches a rename. The old name is missing and it was required; the new name is a property the contract never declared.

## Catch the break in your own model before it ships

The second check needs no network. It compares the `Booking` model the framework uses against the same approved file, from a consumer's point of view, and lists three kinds of breaking change:

| Change in the model              | What `breaking_changes` reports                |
| -------------------------------- | ---------------------------------------------- |
| A property is gone               | `<path>: property removed`                     |
| A type changed                   | `<path>: type changed <approved> -> <current>` |
| A required field became optional | `<path>: no longer guaranteed (was required)`  |

Additions are not breaking for a response, so they pass. The test fails with the whole list:

```python
    approved = load_contract("booking.schema.json")
    current = inline_refs(Booking.model_json_schema())

    problems = breaking_changes(approved, current)

    assert not problems, "Breaking API contract changes detected:\n" + "\n".join(problems)
```

One contract, two checks

```mermaid
flowchart LR
  L[live response] --> A[assert_matches_contract]
  M[Booking model] --> B[breaking_changes]
  C[(contracts/booking.schema.json)] --> A
  C --> B
  A -- violation --> F[fails with the JSON path]
  B -- removed or retyped --> F
```

## Make the model strict as well

The models in `src/api_framework/models/booking.py` refuse fields they do not declare:

```python
class Booking(BaseModel):
    model_config = ConfigDict(extra="forbid")

    firstname: str
    lastname: str
    totalprice: int
    depositpaid: bool
    bookingdates: BookingDates
    additionalneeds: str | None = None
```

Every endpoint in `BookingService` has a raw method and a `*_validated` twin, such as `get_validated`, that checks the status and parses the body into the model. Happy-path tests use the twin. So even a test that never mentions contracts fails loudly when the API starts returning a field nobody declared.

> **Warning**
>
> Don't retry your way past a contract either. The client retries transport errors only; a 5xx or a 429 is a result, and retrying it would hide a real defect.

## Try it

```sh
make install
pytest -m "regression or smoke" -n auto
```

The contract tests carry the `regression` marker, so they run in the full suite. Then pick the endpoint your UI leans on most, write its schema into `contracts/`, and add one `assert_matches_contract` call to a test you already have. The demo suite's booking checks show the whole pattern in two files.

## Read this with your AI

```text
Read the post "API Tests That Fail When the Contract Moves".
1. Summarise it in 5 bullets for an engineer who writes API tests in Python.
2. Ask me 3 questions to check I understood; wait, then correct me.
3. Take one JSON response from my API and draft its JSON Schema with required fields and additionalProperties set to false.
```

> **Success**
>
> A green API run now means the response still has the approved shape, and a renamed or retyped field fails with its JSON path. Start with the one endpoint your UI depends on most.

## FAQ

### Is adding a new field to a response a breaking change?

Not for consumers, so breaking\_changes lets additions pass. The live check still fails on it while the contract sets additionalProperties to false, which forces someone to approve the new field.

### Why keep JSON Schema files when the Pydantic models already validate?

The models are code a test author edits. The files in contracts/ are the approved contract, owned in CODEOWNERS, so changing them is reviewed as an API change, and both checks compare against them.
