API schema changes: contract tests for safer agents

An agent can turn a minor API change into a real action error faster than a human operated client. A renamed response field may make it select every resource instead of one. A newly introduced default may widen a query. A changed status object may look like permission to retry, and the retry may repeat a charge, a deployment, or a deletion request.

The dangerous part is that these failures often look healthy in conventional monitoring. The provider returns HTTP 200. The client does not crash. The agent produces a plausible explanation. Contract tests need to check the meaning an agent assigns to API data, not merely whether JSON parses.

Agent decisions make compatibility a safety property

An API client used by a person often fails visibly when a response changes. A button shows an empty table, a form error appears, and someone investigates before taking the next action. Agents commonly convert a response directly into another request. They can make that conversion several times before anyone sees a transcript.

Consider a cleanup agent that calls GET /projects?state=inactive, reads each item's owner field, and asks for approval before archiving projects outside an allowlist. The provider later changes owner to owner_id, while retaining the old endpoint and status code. A loose parser maps a missing owner to an empty string. If the agent's rule says an empty owner is unassigned, it prepares a far broader archive request.

That is not an authentication failure or a prompt failure. It is an interpretation failure at the API boundary. The corrective control belongs there too.

Treat any field that affects one of these decisions as part of a safety contract:

• which object the agent may act on
• whether an object is eligible for the action
• the scope, amount, or destination of the action
• whether a prior action completed, failed, or needs a retry
• whether the agent should ask a person before proceeding

This distinction matters because transport compatibility is much weaker than behavioral compatibility. A service can preserve its endpoint, method, authentication scheme, and JSON syntax while breaking the decision that follows the response. Teams routinely call that a nonbreaking change because existing SDKs still deserialize it. For an autonomous caller, that label can be dangerously incomplete.

Build a small map of every response value that reaches an action selector, an authorization decision, or a retry branch. Do not begin with every property in a giant API specification. Begin with the values whose wrong interpretation changes what the agent can do.

A schema diff cannot prove behavioral compatibility

Schema diff tools catch useful changes, but they cannot tell you whether a change is safe for a particular agent. They compare declarations. An agent depends on semantics that often sit outside those declarations.

Suppose a provider changes limit from an optional parameter with an implicit default of 100 to an optional parameter with an implicit default of 1000. A standard OpenAPI diff may show no required property change. Yet an agent that omits limit can now inspect ten times as many objects and may send a batch action over all of them.

The OpenAPI Specification describes a Schema Object as a superset of JSON Schema vocabulary and says its properties provide information for request and response payloads. That is useful documentation and validation material. It does not state that state: "pending" permits a retry, or that an omitted limit remains capped at 100. Those are workflow assertions, and your tests must state them in plain terms.

Likewise, JSON Schema's default annotation does not force a validator or client to insert a value. Many developers assume it does. The JSON Schema documentation treats default as annotation data, not a command that changes an instance. If your safety depends on a value, make the client send it explicitly and test the exact outbound request. Do not expect a schema annotation to rescue an omission.

Use a diff tool as an alarm bell. Then classify each reported change by the action path it can affect:

• A renamed identifier can change the selected target.
• An enum addition can send the parser into an untested branch.
• A default change can alter scope without any request code change.
• A representation change can reverse the meaning of completion or failure.

The reverse also matters. A diff tool may report an added descriptive field that no agent reads. That deserves review but not a production freeze. Compatibility review gets better when it follows data into a decision rather than treating every schema line as equally risky.

Contract tests must pin requests and decisions

A useful contract test has two halves: it verifies the request the agent actually sends, then verifies the action the agent proposes after reading a provider response. Testing only one half leaves a large blind spot.

For defaults, capture a real HTTP request at a local test server or provider sandbox. The following Python example uses httpx.MockTransport to inspect an outgoing request. It prevents the common failure where a client silently relies on a provider default for a destructive operation.

import httpx

seen = []

def handler(request: httpx.Request) -> httpx.Response:
    seen.append({
        "method": request.method,
        "path": request.url.path,
        "query": dict(request.url.params),
    })
    return httpx.Response(200, json={"items": []})

client = httpx.Client(transport=httpx.MockTransport(handler))

response = client.get(
    "https://api.example.test/projects",
    params={"state": "inactive", "limit": "100"},
)

assert response.status_code == 200
assert seen == [{
    "method": "GET",
    "path": "/projects",
    "query": {"state": "inactive", "limit": "100"},
}]

The important assertion is not the 200 response. It is the explicit limit. If a refactor drops that parameter, the test fails before a provider's new default can widen the selection.

Then test the decision. Keep the agent's planning function separate from the code that performs the HTTP call, so the test can inspect a proposed action without making one.

from dataclasses import dataclass

@dataclass
class ArchivePlan:
    project_ids: list[str]
    requires_approval: bool

def plan_archives(items: list[dict], allowed_owners: set[str]) -> ArchivePlan:
    targets = []
    for item in items:
        owner = item.get("owner")
        if owner is None:
            raise ValueError("provider response lacks owner")
        if item["state"] == "inactive" and owner in allowed_owners:
            targets.append(item["id"])
    return ArchivePlan(targets, requires_approval=bool(targets))

items = [
    {"id": "p17", "state": "inactive", "owner": "team-a"},
    {"id": "p18", "state": "inactive", "owner": "team-b"},
]

plan = plan_archives(items, {"team-a"})
assert plan.project_ids == ["p17"]
assert plan.requires_approval is True

This test makes a choice that loose code often avoids: missing owner raises an error. For an action selection field, fail closed. Returning an empty string, None, or a guessed fallback might keep the process moving, but it replaces a detectable integration failure with a possibly unsafe plan.

Keep fixtures small enough that a reviewer can see why each item appears. A thousand object fixture may resemble production, but it hides the condition you intended to protect.

Renamed fields need explicit failure behavior

A renamed field should stop an action path unless you deliberately support both names during a defined migration. Silent fallback handling feels resilient in a demo and creates unreviewed semantics in production.

The worst version looks like this:

owner = item.get("owner", "")
if owner not in blocked_owners:
    archive(item["id"])

When owner disappears, every item appears to have an owner that is not blocked. The parser did exactly what the code requested. The engineer who wrote it likely wanted to avoid a KeyError. That small convenience converted absence of data into permission to act.

Write tests for all three situations: the expected field, the old field if you have a temporary compatibility promise, and neither field. The third test should state whether the agent stops, skips the object, or requests clarification. For target identity, authorization state, and action scope, stopping is usually the right answer.

If you support an alias, make its precedence obvious and temporary:

def read_owner(item: dict) -> str:
    if "owner" in item:
        return item["owner"]
    if "owner_id" in item:
        return item["owner_id"]
    raise ValueError("owner identity missing")

This code still needs a test for contradictory data. If both fields arrive and disagree, do not quietly prefer one. Raise an error and make the provider resolve the ambiguity. A compatibility layer should preserve a known old meaning, not invent a tie breaker for inconsistent records.

Teams sometimes argue that permissive parsing protects against provider evolution. It protects against harmless additions when you ignore unknown fields. It does not protect against missing fields that govern an action. Those cases require opposite behavior: accept extra information by default, but reject absent required meaning.

Defaults and omission require separate tests

An omitted property, an explicit null, and an explicit value are three different requests. Agents often blur them because general purpose serializers blur them too.

A request builder may omit dry_run when its internal value is None. The provider might interpret omission as false. A later provider release might change omission to mean "use account setting," where the account setting is false for one tenant and true for another. The agent code did not change, but the action changed.

Put decision bearing options into one of two categories. For options where the safe behavior is known, send the value every time. For options that need operator choice, require the choice before building a request. Avoid a third category called "let the server decide" for destructive or externally visible actions.

Test serialization with a table of exact cases. The point is to inspect the wire representation, not merely the language object before serialization.

Be equally strict with pagination. A response that adds next_cursor can tempt an agent to fetch every page automatically. That may be reasonable for a report and reckless for an action planner. Test both the maximum objects the planner may consider and the condition that permits a second page. A cursor is a continuation mechanism, not consent for unlimited scope.

Provider defaults also matter in responses. If an API begins omitting archivable when it is false, code such as if item.get("archivable", True) changes its behavior in the unsafe direction. For a field that grants permission, use an explicit comparison such as item.get("archivable") is True. It is less elegant and much easier to audit.

Response validation must preserve meaning, not just shape

Response validation should distinguish malformed data from unfamiliar but harmless data. Blanket strictness breaks clients when providers add fields. Blanket permissiveness turns missing evidence into a guess.

Define a narrow response model around the values that feed the next action. For each value, specify type, allowed states, and whether absence stops the workflow. A project identifier needs more than string; the agent needs a nonempty stable identifier that matches the identifier later sent in the archive request. A status needs more than string; the agent needs an enumerated state with a documented action for each member.

For example, this parser handles a changed status representation without granting itself permission to retry:

ALLOWED_STATES = {"queued", "running", "succeeded", "failed"}

def retryable(job: dict) -> bool:
    status = job.get("status")
    if status not in ALLOWED_STATES:
        raise ValueError(f"unknown job status: {status!r}")
    return status == "failed" and job.get("retry_allowed") is True

If the provider changes status from a string to an object such as {"phase":"failed"}, this code stops. That interruption is correct until someone decides how the new representation maps to the old workflow. If the provider adds cancelled, stopping is also correct until the team decides whether cancellation is terminal, retryable, or requires a person.

Do not make every unknown enum value an emergency for a read only display. The consequence should match the action. A reporting agent can label an unfamiliar state and continue. An agent that will retry a billing job or delete a resource must stop before it acts on a state it does not understand.

Test relationships between fields too. A response may validate structurally while containing a self contradictory combination such as status: "succeeded" and retry_allowed: true. Schema validation does not normally express every business invariant. A contract test should assert that a successful job creates no retry plan, regardless of a stray boolean.

Test the entire action route, not a convenient mock

Unit tests for parsers are necessary, but they do not prove that the deployed agent sends the intended request through its actual credential and execution route. Serialization libraries, wrappers, tool adapters, and retry middleware all change behavior in ways a direct function call cannot reveal.

Run a local contract server in CI that records requests and returns versioned fixtures. Point the agent's tool configuration at that server. The test should drive a realistic instruction, wait for the proposed plan or action record, and assert the recorded sequence: method, path, query, headers that are safe to inspect, body, and action count.

Do not put live secrets in this environment. Use a test credential that carries no authority, and verify that the agent never receives it in its prompt, tool result, exception text, or trace. A test that logs request headers carelessly can recreate the credential exposure it was meant to prevent.

For agents that make external HTTP or SSH calls through Sallyport, the action route can keep credentials out of the agent while preserving an inspectable result. That boundary does not repair a bad interpretation of a response, so run schema and decision contracts before allowing the external action.

Include failure fixtures, not just golden responses. Feed the exact conditions providers produce during changeovers: a missing selection field, a new enum member, an empty result with a continuation cursor, a content type change, and a 200 response containing an error object. A 200 error body is especially common in older APIs. If your parser assumes every 200 response contains the success shape, it may manufacture an empty plan or retry a request that already succeeded.

When testing retries, assert idempotency behavior. Give the contract server a first response that times out after recording the request, then a second response to the retry. The test must prove that the client sends an idempotency token where the API supports one, or that it stops and asks for confirmation when it cannot know whether the first action completed. Retrying a read is usually harmless. Retrying a transfer, email, deployment, or delete request is not.

Release gates should block semantic breakage

Run schema diffs, provider contract tests, and agent decision tests in the change path for both API producers and agent consumers. A release process that runs them only after deployment turns the tests into incident documentation.

For a provider change, require a review record that answers four concrete questions: which consumer assumptions change, what the old behavior was, how long both behaviors remain supported, and what fixture demonstrates the new behavior. This is less paperwork than arguing through a production regression with incomplete logs.

For an agent change, run the existing provider fixture suite before merging. If the agent now uses a newly added field, add fixtures for its absence and for values outside the happy path. Changes in prompt wording can alter tool arguments too, so test the tool call emitted by the whole agent rather than assuming the planner will keep choosing the same parameters.

Versioned fixtures make review practical. Store an identifier such as projects-list-v1 with the expected request and response pair. When a provider deliberately introduces projects-list-v2, keep the old fixture until the migration policy ends. Do not overwrite the old JSON and call the tests current. You lose the proof of what compatibility you dropped.

A useful gate reports failures in operational language. "Expected field owner missing, archive planning stopped" tells a reviewer what happened. "ValidationError at path items.0" is better than nothing, but it makes the reviewer reconstruct the risk during a release.

Approval screens cannot correct a misleading plan

Human approval remains a good control for external actions, but it comes too late if the agent built the wrong plan from a changed contract. A person who sees "Archive 847 inactive projects" may reject it. A person who sees "Archive project p17" cannot tell whether p17 came from an absent owner field, a widened default, or a parser that confused cancelled with failed.

Make approval records include the decision inputs that deserve human review: target identifiers, count, requested mode, and the response fields that made the target eligible. Keep the record compact. Dumping raw JSON on an approver moves the parsing task from code to a tired person.

Separately preserve a trace that lets an engineer reconstruct the action. Capture the provider response or a protected digest of it, the parser version, the contract fixture version, the generated request, and the resulting response. A tamper evident audit trail helps after the fact, but it should point to the decision boundary instead of merely recording that an HTTP call happened.

The next time an API team says a response change is cosmetic, ask them to run the agent contract suite against it. If the suite fails, the change has a behavior attached to it. Fix that before a polite 200 response becomes an unsafe action.