# Stale API documentation: test agent actions safely

Stale API documentation is an agent safety problem, not an editorial nuisance. A person may read an outdated example, hesitate, and ask a colleague. An autonomous coding agent will often turn that same example into a request, then use the response as evidence that it did the right thing.

That difference changes the standard. If your documentation teaches an agent how to invoke an API, rotate a token, delete a record, or reach a production host, treat the text as executable input. Test it alongside the tool itself. A page that was accurate when published but no longer matches the live service can steer an agent into an unsafe action even when the API is functioning exactly as its current maintainers intended.

The most dangerous drift is rarely a loud failure. A 404 gets attention. A request that still returns 200 while selecting more resources, using a changed default, or bypassing an expected confirmation does not. That is the kind of mismatch that produces a clean activity log and a bad afternoon.

## Documentation becomes part of the agent's control plane

An agent uses documentation to choose operations, fill parameters, interpret responses, and decide whether to retry. That makes examples, reference tables, authentication guides, and migration notes part of its control plane. The service code can be correct while this control plane tells the agent to use it incorrectly.

Teams often draw an artificial line between a tool definition and a guide. A tool definition says `deleteProject(project_id)`. The guide says which project identifier to obtain, whether a dry run exists, whether deletion cascades, and what to do after an authorization failure. The agent needs both. If either one lies, the resulting action can be wrong.

This is why a stale example is different from a misspelled paragraph. Consider an old instruction that says a missing `scope` parameter means "current project." A backend change later makes the same omission mean "all projects available to this credential." The endpoint still works. The example still parses. An agent that follows the old guide can now apply a supposedly local change across an account.

Documentation also sets the agent's confidence. Concrete snippets carry more weight than a vague warning in nearby prose. If one page says "use least privilege" and another gives a bearer token example with account wide access, the snippet wins in practice. Agents optimize for a path that produces a result.

Treat the following as action-bearing documentation:

- Request and command examples
- Parameter tables that describe defaults and allowed values
- Authentication, credential, and environment setup instructions
- Retry, pagination, idempotency, and error handling guidance
- Migration and deprecation instructions that tell users which operation replaces another

A useful distinction is between **syntax drift** and **meaning drift**. Syntax drift makes an example fail because a field or path changed. Meaning drift leaves the example valid but changes what it affects. Syntax drift embarrasses the writer. Meaning drift can damage data, spend money, expose records, or broaden access. Your test suite must detect both.

## A passing request can still prove the example wrong

A documentation test that checks only status codes catches the easy failures and misses the unsafe ones. HTTP success says the server accepted the request. It says nothing about whether the request targeted the intended object, produced the described side effect, or respected the stated boundary.

Suppose a reference page publishes this request:

```bash
curl -sS -X POST "$API_URL/v1/exports" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"project":"demo","include_archived":false}'
```

A basic test checks for `202 Accepted` and declares victory. That check misses several changes that matter:

- The service silently renames `project` to `project_id` and treats the old field as absent.
- `include_archived` changes from an opt out to an ignored compatibility field.
- The credential gains account wide visibility, so `demo` resolves to a different tenant's project.
- The endpoint still queues work but now exports attachments that the guide says it excludes.

The test needs to inspect the result and the service state, not merely the status. In a disposable account, create one active record and one archived record. Send the documented request. Poll the resulting job. Assert that the artifact contains the active record, omits the archived record, and records the expected project identifier. If the service cannot expose enough evidence to make that assertion, the documentation cannot safely promise the behavior either.

RFC 9110 defines HTTP status codes as the result of a request's handling. It does not claim that a successful status proves a caller's business intent. That sounds obvious, yet teams keep building documentation checks that reduce the protocol to `curl` plus `grep 200`. Use HTTP semantics for protocol assertions, then add assertions for the real outcome.

A good test names the claim it verifies. `export_excludes_archived_records` is useful. `docs_example_returns_success` mostly tells you that somebody ran a request.

## Examples need contract tests, not screenshot reviews

Copying an example from a docs page into a terminal during release review is better than nothing. It does not scale, it leaves no dependable evidence, and it favors the happy path. People also tend to repair the command locally and forget to repair the page.

Put executable examples in a structured source file, generate the rendered snippet from that source, and run the same source in CI. You can use OpenAPI examples, Markdown code extraction, or a separate fixture directory. The mechanism matters less than one property: the command a reader sees must be the command the test runs.

Do not quietly maintain a "test version" that has safer URLs, narrower scopes, or more complete headers than the published example. That split creates a comforting green build while the public instructions rot. Parameterize only values that must differ by environment, such as base URL, test credentials, and fixture identifiers. Keep method, path, body shape, and security relevant flags identical.

A small shell test can establish the pattern:

```bash
set -euo pipefail

project_id="docs-check-$RANDOM"
response=$(curl -sS -X POST "$API_URL/v1/projects" \
  -H "Authorization: Bearer $DOCS_TEST_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{\"id\":\"$project_id\",\"name\":\"Documentation check\"}")

printf '%s' "$response" | jq -e \
  --arg id "$project_id" \
  '.id == $id and .name == "Documentation check" and .archived == false'
```

The expected output from `jq -e` is the JSON value `true`, and a mismatch exits nonzero. The important part is not the shell syntax. The assertion encodes what the prose claims: the API creates a project with the supplied identifier, preserves the supplied name, and does not archive it by default.

Build a separate check for the rendered documentation. If a Markdown extractor pulls a block marked `bash` from the page, the test should run that block after substituting the approved environment variables. If you generate docs from an OpenAPI description, test the generated example rather than a hand copied cousin.

Screenshot review still has a place for readability. It cannot prove behavior. A reviewer will miss an omitted header surprisingly often, especially when the page contains several similar examples. Machines do not get tired of comparing a field to a fixture.

## Live service checks must cover defaults and failure paths

Most dangerous API changes land in defaults, authorization boundaries, and failure handling. Happy path tests avoid all three because they are easy to write and easy to keep green.

Test the omission cases that an agent might produce. Agents often build payloads conditionally, so an optional field may disappear when an earlier lookup returns no value. For each documented optional parameter, decide whether omission is safe, rejected, or meaningfully different. Then test the documented behavior directly.

For an operation with a `dry_run` field, run at least these cases in an isolated environment:

1. `dry_run: true` returns a plan and leaves the fixture unchanged.
2. `dry_run: false` performs the stated change on only the named fixture.
3. Omitting `dry_run` either rejects the request or produces the documented default.
4. A token with insufficient scope fails before any change occurs.
5. Repeating the documented request behaves as the idempotency guidance says it will.

The third case catches a routine source of accidental damage. A service team changes a default to favor interactive users, while the docs still assume the old default. A human interface may show a confirmation screen. An API client has no such screen.

Failure examples need tests too. Documentation often says "retry on 429" without saying whether the response includes `Retry-After`, whether the operation is safe to repeat, or whether the request needs an idempotency token. That advice can turn a brief rate limit into duplicate invoices, duplicate deployments, or repeated revocations.

Test the exact failure guidance. Force the throttled condition in a test service or controlled environment. Confirm that the documented client reads the stated header, waits as directed, and resends the same idempotency identifier when the API supports one. If the service cannot produce the error predictably, document the uncertainty instead of publishing a confident recipe.

OpenAPI's `default` keyword creates a related trap. In JSON Schema and OpenAPI descriptions, a declared default commonly communicates what tools may assume or display. It does not automatically make every server implementation apply that value. Check the deployed service with an omitted field. A schema default and a server default are separate claims until a test binds them together.

## Destructive workflows require disposable proof

Do not test destructive examples against a shared staging account and call it safe. Shared environments gather old fixtures, manual experiments, and credentials with unclear reach. Eventually a documentation test will match the wrong object, or a cleanup command will exceed its intended boundary.

Use a dedicated test tenant or account with credentials that can reach only the test resources. Create each fixture with a unique run marker. Retrieve it by that marker before mutation. After the test, verify the resulting state rather than assuming the API did what the response claimed.

A delete example should demonstrate a full lifecycle:

```text
create fixture: docs-delete-<run-id>
read fixture: confirm owner=test-suite and run_id=<run-id>
delete fixture: send the rendered documentation request
read fixture: expect the documented absence or tombstone state
list nearby fixtures: confirm unrelated fixtures remain
```

That last check matters. A deletion test that only confirms the chosen object disappeared cannot detect an overly broad selector. I have seen teams accept a bulk endpoint because their one fixture vanished as expected, while the endpoint also removed every resource with a similar prefix. The test needed a deliberately similar neighbor that was supposed to survive.

Do not let docs tell agents to use convenience selectors such as `latest`, `all`, a blank filter, or a human readable name when an immutable identifier exists. Those selectors feel friendly in a tutorial and become unsafe when an agent runs the recipe in a busy account. If an operation truly requires a broad selector, put the scope in the request body or command arguments where a reviewer can see it. Do not hide it in a server default.

For irreversible operations, publish a preflight read and make the example consume its output. First fetch the object, verify its immutable ID and relevant state, then issue the mutation. This adds friction. That friction is cheaper than explaining why an agent deleted the object that happened to share a display name.

## Tool tests must compare meaning, not just schemas

Schema validation is necessary, but schemas usually describe shape more faithfully than consequences. A request body can satisfy every type constraint while directing an action at the wrong environment or with the wrong privilege.

Build assertions around four questions: Who did the action affect? What state changed? What state did not change? Which identity authorized it? Those questions work for HTTP APIs, SSH commands, and internal tools.

For HTTP, capture a request identifier when the service provides one, then query the resulting resource or audit record in the test environment. Match the request identifier, actor, target, and mutation. For SSH, run commands against a disposable host, capture exit status and output, then inspect the host state with a separate verification command. Do not make the action command grade its own homework.

A useful fixture has contrast. If you test a command that should restart one service, create another service that must remain running. If you test a query limited to one repository, include a second repository that the same credential can see but the request must not touch. Without contrast, a broad action can look correct.

This is where many teams misuse contract testing. Consumer driven contract tools can confirm that a provider accepts a request shape and returns expected fields. They cannot determine whether the request selected the right production account, whether a `force` option acquired a new meaning, or whether a deletion cascaded beyond the documented object. Keep the contract test. Add an outcome test with fixtures designed to reveal overreach.

A tool description also needs testing. If a tool exposes `environment`, do not describe `production` as an acceptable value unless a test verifies that it routes to the documented host and uses the stated authorization path. Agents use descriptions to fill arguments. A stale description is just a prose version of a stale API example.

## An agent needs freshness evidence and a safe refusal path

An agent should not infer that documentation is current because it appears in a repository or an internal portal. Give it machine readable verification evidence tied to the operation it plans to call.

A simple manifest can be enough:

```json
{
  "operation": "POST /v1/exports",
  "documentation_source": "docs/api/exports.md#creating-an-export",
  "verified_in": "isolated-test-tenant",
  "verification_commit": "<commit-id>",
  "assertions": [
    "returns an export job",
    "omits archived fixtures when include_archived is false",
    "rejects a token without export scope"
  ],
  "review_required_when": ["production", "include_archived=true"]
}
```

The commit identifier is not a trust badge by itself. It lets a reviewer trace the documentation and test source that produced the evidence. Store a verification time in your own build records if your release process needs an age limit, but do not pretend a timestamp makes old behavior safe. A service deployment can invalidate yesterday's test.

The agent's decision rule should be plain. If the requested operation lacks a passing verification record for the deployed interface, it must either run a nonmutating preflight in an approved test context or ask a person to approve the exact action. It should not improvise based on a neighboring endpoint.

Separate "unknown" from "safe." Agents tend to fill gaps because completing a task receives positive feedback. Tool design must make abstention a successful outcome when evidence is missing. Return a reason such as: `documentation example has no verified outcome test for this operation`. That message gives the developer a repair target instead of a vague refusal.

Do not solve this with a long policy file that tries to enumerate every risky phrase in every document. The wording will drift faster than the rules. Bind a concrete operation to a concrete test and give the agent the result.

## Release gates work only when they block the misleading page

A documentation verification program fails when it produces reports that nobody must act on. The check must block publication, or at least block agent access to the affected example, when the service contract changes.

Connect the checks to changes in the API specification, route handlers, authentication middleware, SDK request builders, and documentation sources. A change to one of those areas should run the relevant example tests. If a test fails, the team has three honest choices: restore the old behavior, update the docs and tests for the new behavior, or mark the operation unavailable to agents until verification succeeds.

Manual review remains useful for judgment, but manual review alone is the wrong recommendation. It is popular because it feels cheap and preserves a fast publishing flow. It also asks a reviewer to mentally simulate a service, credentials, defaults, and state transitions from text. People cannot reliably do that across routine releases.

Make failures legible. A useful report names the page, code block, operation, test fixture, observed response, and violated assertion. "Documentation integration failed" creates a scavenger hunt. "exports.md line 42 says archived records are excluded; export artifact included fixture archived-run-817" gives the owner a direct repair.

Do not loosen a test just because a service change made it inconvenient. First decide whether the old promise was useful. If it was, restore it or state the new limitation prominently. If it was unsafe, remove the example rather than preserving it with a softer sentence. An agent will usually follow the remaining command.

Versioned documentation needs the same discipline. A page for an older API version may accurately describe an older deployment but still mislead an agent pointed at the current base URL. Put the version in the endpoint path, server URL, or tool metadata where the agent can bind it to the request. A heading that says "v1" somewhere above the fold is weak evidence.

## Authorization limits the blast radius but cannot repair bad instructions

Approval and credential isolation still matter because documentation checks will miss defects. They reduce the damage when an agent chooses the wrong operation. They do not convert a stale instruction into a correct one.

Keep the two jobs separate. Documentation verification asks, "Does this example describe the live service and its consequences?" Action authorization asks, "Should this agent be allowed to make this call now?" Combining them produces confusion. A user may approve a call because the agent says it will export one project, while the stale example actually exports every project visible to the credential.

For agent driven HTTP and SSH actions, Sallyport keeps credentials out of the agent and can require a person to authorize a session or an individual credential use. That is a useful last boundary when a documentation check finds no trustworthy evidence or when an action has consequences that deserve human review.

The approval screen should show the operation, target, and scope in terms a person can judge. "POST /v1/exports" is insufficient when the request body carries `include_archived=true` or an account wide selector. If your authorization layer cannot reveal the meaningful scope, narrow the tool interface until it can.

Audit records then give you the material to improve the tests. When a person revokes a run or questions an action, inspect the exact request, the documentation source the agent cited, and the verification evidence it had. Do not treat that review as blame assignment. Use it to add the missing fixture, assertion, or refusal condition.

## Start by testing the example most likely to cause regret

Do not begin with the cleanest GET request in the reference. Begin with the example that can delete, publish, rotate, grant, charge, or reach a production host. Give it an isolated fixture, run the exact rendered command, and assert both the intended change and the nearby change that must not happen.

Then wire that test to the documentation source and make a failure visible before publication or agent use. The work is less glamorous than writing a new tool description, but it removes a dangerous assumption: that a page is safe because it once passed review.

A live service changes. Its docs will change more slowly unless you force them to meet in a test. Make that meeting part of the release, before an agent turns an old sentence into an action.