API Tests That Fail When the Contract Moves
An API test that checks one field stays green while the backend renames or retypes another. API contract testing in Python: approved JSON Schema files, checked against every live response and against your own model, with the JSON path in every failure.
Key takeaways
- A test that asserts one field passes while the backend renames or retypes another, so the break surfaces later in the UI suite or in production.
- An approved contract is a JSON Schema file per response with required fields, types and additionalProperties set to false, and editing it is reviewed as an API change.
- Contract regression validates every live response against the approved file and reports each violation by its JSON path.
- A second check needs no network: it compares the model with the approved contract and fails on a removed property, a changed type or a field no longer required.
- Strict models with extra set to forbid make even an ordinary happy-path test fail when the API returns a field nobody declared.
Contents
FAQ
Is adding a new field to a response a breaking change?
Not for consumers, so breaking_changes lets additions pass. The live check still fails on it while the contract sets additionalProperties to false, which forces someone to approve the new field.
Why keep JSON Schema files when the Pydantic models already validate?
The models are code a test author edits. The files in contracts/ are the approved contract, owned in CODEOWNERS, so changing them is reviewed as an API change, and both checks compare against them.
Sources
Only you and the owner see your answer and rating. Send private feedback
Enjoyed it? Support fx-blog with a coffee.
Related posts
A Test That Passed on Retry Is Not Green
CI retries a failing test, the retry passes, and the pipeline goes green with a flaky test inside it. How the playwright template stops that: retries run isolated at the end, and a release gate reads the JSON report and fails on any flaky smoke or critical pass.
Comments (1)
Oldest firstIt's great!
Part 1, A Test That Passed on Retry Is Not Green, ended on the gate's
@apiline: "check for a contract change before blaming the tests". In the demo suite, nothing said what the booking response is allowed to look like, so a failing booking test could not point at the backend. This part writes that contract down, in the python-api-automation template.A test asserts that
firstnameis "Bob". The backend renames another field, or turnstotalpriceinto a string. The test stays green, because it never looked.This series, in two parts:
What does a contract look like as a file?
A contract is a JSON Schema in
contracts/, one per response, that says which fields must be there, their types, and that nothing else may appear. The demo suite tests restful-booker's booking API, and this is the approved shape ofGET /booking/{id}:I keep the contract in its own folder instead of only in the test code, because test code is what a test author edits to make a test pass.
CODEOWNERSputs/contracts/under its own section, so editing that file is reviewed as an API change.Check every live response against it
The first check is contract regression: the real response, parsed against the approved file.
assert_matches_contractinsrc/api_framework/utils/contract.pycollects every violation and reports each one by its JSON path:The test that uses it is two lines of intent:
Catch the break in your own model before it ships
The second check needs no network. It compares the
Bookingmodel the framework uses against the same approved file, from a consumer's point of view, and lists three kinds of breaking change:breaking_changesreports<path>: property removed<path>: type changed <approved> -> <current><path>: no longer guaranteed (was required)Additions are not breaking for a response, so they pass. The test fails with the whole list:
Make the model strict as well
The models in
src/api_framework/models/booking.pyrefuse fields they do not declare:Every endpoint in
BookingServicehas a raw method and a*_validatedtwin, such asget_validated, that checks the status and parses the body into the model. Happy-path tests use the twin. So even a test that never mentions contracts fails loudly when the API starts returning a field nobody declared.Try it
The contract tests carry the
regressionmarker, so they run in the full suite. Then pick the endpoint your UI leans on most, write its schema intocontracts/, and add oneassert_matches_contractcall to a test you already have. The demo suite's booking checks show the whole pattern in two files.Read this with your AI