Agent tool failure testing: catch unsafe retries early

Agent tool failure testing should focus on what the agent believes after an operation fails, not merely whether the tool emitted an error. A tool that reports "request failed" after a remote write may have completed creates a worse problem than a tool that stops with an incomplete answer. Agents will plan around whatever result you give them.

The happy path hides the decisions that decide whether an autonomous run is safe: retry, request approval, rotate access, repair data, or stop. I have seen tool suites with hundreds of green tests that had never forced a network drop between a remote command starting and its output returning. Those suites did not test the dangerous part.

A failure response is an input to the agent's planner

An agent treats a tool result as evidence. If the result implies that no change occurred, the agent may retry. If it implies that a credential expired, the agent may seek an authorized recovery path. If it implies success when the remote outcome is unknown, the agent may build several later actions on fiction.

Separate failures by what the caller can know. This distinction gets blurred constantly:

• A confirmed refusal means the remote service received the request and rejected it.
• A confirmed failure means the remote service returned a result that says it did not perform the requested work.
• An uncertain outcome means the caller cannot establish whether the remote side performed the work.
• A local failure means the tool failed before it could make a meaningful remote attempt.

A connection refused before a TCP session opens is usually local failure. An HTTP 403 is a confirmed refusal. A read timeout after sending a POST is uncertain unless the remote service gives you a way to look up the operation. These labels should appear in your test cases and your tool's result schema. Do not bury them in a sentence that an agent must interpret.

A compact result shape makes the contract testable:

{
  "ok": false,
  "category": "outcome_unknown",
  "operation": "create_deployment",
  "retry": "reconcile_first",
  "correlation_id": "case-ssh-017",
  "message": "Connection closed after the remote command started; remote completion is unknown."
}

The names do not matter much. The separation does. retry: "never" for a permission refusal and retry: "reconcile_first" for a timed-out write tell the agent different things without turning the entire error message into a prompt.

Do not return raw provider errors as the only interface. They change, often include irrelevant prose, and sometimes contain request material you should not hand to an agent. Preserve the original status, body, and headers in protected diagnostics. Return a stable, deliberately small result to the caller.

Build the matrix around operations and evidence

A useful matrix crosses each operation with the failure modes that can change its meaning. Start by listing tools that read, create, update, delete, trigger, or execute. A read that times out has a different recovery rule than a command that changes a production host.

Use this matrix as a starting artifact. Replace the operation names and expected records with your own, but do not delete the "remote effect known" column. That column forces the uncomfortable cases into the open.

Add rows for the operations that spend money, alter permissions, rotate credentials, or affect shared state. Those operations need more than one timeout row. Test a timeout before bytes leave the process, after request headers leave, after the service accepts the request, and while a response body arrives. The exact injection points depend on your protocol, but collapsing them into one generic "timeout" case loses the behavior you need to verify.

Every row needs four assertions:

1. Assert the tool result category and retry instruction.
2. Assert the agent's next action, including that it does not improvise a destructive retry.
3. Assert the remote state or the documented reason it remains unknown.
4. Assert the event trail contains the correlation ID and the observed outcome.

This is more work than asserting ok == false. It also catches the failures that matter after an agent has already taken several actions.

Expired credentials and denied actions require different recovery

An expired or revoked credential proves that authentication failed. A denied action proves that the caller authenticated but lacks permission for that operation, unless a provider deliberately obscures the distinction. Treating both as "access failed" produces bad agent behavior.

RFC 9110 defines 401 as an unauthenticated request and requires the server to send a WWW-Authenticate challenge. It defines 403 as a refusal to fulfill the request, even when the server does not disclose why. Providers do not always follow that distinction cleanly, so test the provider's actual response. Still, your tool should map its observed evidence into separate categories where it can do so honestly.

Make an expired credential test use a credential that was accepted during setup and is rejected during the actual call. A fake string that the service never recognized only tests an invalid credential branch. You want to catch caches, refresh code, and error mappers that behave differently once an access token has aged out.

A simple fixture can express both cases without exposing a secret:

cases:
  - id: expired-token
    request:
      method: POST
      path: /v1/releases
    fixture_response:
      status: 401
      headers:
        www-authenticate: Bearer error="invalid_token"
      body: {"error":"token_expired"}
    expect:
      category: authentication_failed
      retry: never
      secret_in_result: false

  - id: denied-release
    request:
      method: POST
      path: /v1/releases
    fixture_response:
      status: 403
      body: {"error":"insufficient_scope"}
    expect:
      category: authorization_denied
      retry: never
      secret_in_result: false

The secret_in_result assertion catches a mistake that shows up during frantic debugging: code attaches the outgoing authorization header or configuration object to an exception. Test serialized tool output, trace output, and whatever transcript reaches the agent. Redaction in one logger does not protect another logger.

Do not make the agent "try another credential" unless your system explicitly gives it a distinct, authorized identity to use. Blind credential selection can cross a privilege boundary while appearing to solve an availability problem. A test should prove that an auth failure stops the run or routes it to the approved human recovery path.

Malformed data needs a contract test, not a JSON parser test

Malformed data includes valid JSON that your code cannot safely use. Invalid syntax is the easy case. Production failures more often arrive as a field changing type, a required identifier disappearing, an error envelope replacing a success envelope, or a response truncated after a proxy closes a connection.

The JSON-RPC 2.0 specification separates parse errors (-32700) from invalid requests (-32600). That split is useful because it distinguishes unreadable bytes from a readable message that violates the protocol. Apply the same discipline to your domain responses: a parser success does not prove the response satisfies the tool contract.

For every provider response you consume, write fixtures that violate one assumption at a time:

• Replace a string ID with null, a number, and an object.
• Omit a field that later tool calls require for reconciliation.
• Return a success status with an error-shaped body.
• Return an error status with an HTML body or a truncated JSON document.
• Duplicate an item or change ordering when your code selects the first item.

Then assert the exact behavior. The tool should name the failing field or contract condition in protected diagnostics, return malformed_response to the agent, and perform no follow-on mutation based on guessed values.

A common bad pattern looks harmless: response.id || request.id. It keeps a workflow moving when a provider omits id, but it may cause an update or delete against an unrelated object if request identity and response identity differ. Test that missing response identity stops the operation. A failed workflow is cheaper than an incorrect write.

MCP tool clients need the same care. The Model Context Protocol tool result format supports an isError signal for a tool-level failure. Use it when the tool itself cannot complete its promised work, but keep the result content specific enough for an agent to choose a safe branch. Do not disguise a malformed upstream response as a normal text result that begins with "Error:". Many clients will treat that as successful tool execution and leave the agent to infer the rest.

Timeouts are ambiguous after a write begins

A timeout tells you that your deadline expired. It does not identify the state of the remote operation. This sounds obvious until a retry loop quietly turns a lost response into a duplicate invoice, two deployments, or a second restart.

Test timeout behavior at the boundary where certainty changes. Your fault injector or fake service should record each stage:

case=C05 request_id=case-http-005 received=true
case=C05 request_id=case-http-005 mutation_committed=true
case=C05 response_write=delayed
client case=C05 deadline_exceeded=true

The expected assertion is not "client got timeout." The assertion is that the client returns outcome_unknown, does not issue a second POST, and uses a status lookup or idempotency mechanism before proceeding.

Idempotency tokens help only when the remote API documents and honors them for the operation in question. Test them as a full sequence: submit a request with a unique token, delay the first response until the caller gives up, submit the same token through the recovery path, then verify the service reports one logical operation. Do not claim idempotency because you added a header that a provider ignores.

For operations without a reconciliation endpoint or idempotency support, say so in the tool result. The safe action may be to stop and ask a person to inspect the remote system. That is not an engineering failure. Pretending certainty because a workflow wants to continue is one.

Set timeouts by phase where your client supports it: connection, request write, first response byte, and total operation duration. A single large deadline hides whether a peer never accepted a connection or accepted a write then stalled. Your tests do not need to expose every phase to the agent, but your diagnostics need enough detail for an operator to reproduce the event.

Interrupted remote commands must preserve uncertainty

An SSH command has a failure window that HTTP developers often underestimate. The client may send the command, the remote shell may start it, and then the connection may close before the caller receives an exit status. A local process crash or a lost network route does not undo work that the remote host already started.

OpenSSH documents that its client returns the remote command's exit status when it can obtain one. When the transport breaks first, the caller does not have that status. Test that case deliberately rather than treating a nonzero local process exit as proof that the remote command failed.

Create a remote test command that records a start marker, waits, records a completion marker, and writes a recognizable result. Then terminate the local transport while the wait runs. Keep this confined to an isolated host or container you own.

# remote command used only in an isolated test environment
id="case-ssh-017"
printf '%s start\n' "$id" >> /tmp/agent-tool-test.log
sleep 20
printf '%s complete\n' "$id" >> /tmp/agent-tool-test.log

Run the command through the same SSH path that the tool uses, wait until the start marker appears, then terminate the local helper. After the remote wait has elapsed, inspect the log. Run the test twice: once where the remote process completes, and once where the remote side kills it after the start marker. Both produce a local interruption, yet they demand different recovery.

For commands that change state, design a reconciliation command before you design a retry. A service restart might query process uptime or a deployment revision. A package installation might query the installed version. A command that cannot be reconciled should require explicit human handling after interruption.

Avoid shell snippets that hide partial completion behind && chains and vague output. Emit a durable operation ID before the mutating part begins, and use that ID in later inspection. If the remote environment cannot retain any marker, the tool has no basis to tell an agent that retrying is safe.

Human denials are a normal result, not a broken test

A human declining an action should produce a distinct result that ends that action branch cleanly. Teams often test that an approval screen appears and forget to test the decline path, which leaves agents retrying, rephrasing the same request, or misreporting an approval failure as a network issue.

Test denial at each authorization boundary you expose. Verify that the tool does not connect to the remote service after a denial. Verify that it does not retain an approval for a later process or a later action that needs a fresh decision. Verify that the agent gets language it can use without treating the refusal as an invitation to find a workaround.

Sallyport's vault gate denies every action while locked, and its session and per-call approvals make those decisions testable without placing credentials in the agent process. That division is useful because a locked vault, a declined session, and a denied per-call use can all stop an operation for different reasons.

Approval fatigue is a test failure in its own right. If a harmless read generates repeated prompts during a normal run, people will approve without reading. If a destructive call inherits a broad approval accidentally, people never get the decision point they expected. Test the number, timing, and scope of prompts along with their presence.

Use a test agent that tries one approved action, one denied action, and one action after the process exits. The final call catches approval state that leaks past the intended session. Do not simulate this only by toggling an in-memory Boolean; launch a fresh process so the test shares the lifecycle your users actually run.

Logs must answer what happened without exposing access

A useful failure record lets you reconstruct causality: which agent run attempted which operation, what request identifier it used, what the remote system observed, what the tool returned, and what the agent did next. It should not need to contain the credential that authorized the call.

Log an event at every point where the answer can change. For a timed write, capture request construction, dispatch start, remote acceptance if your fixture can report it, deadline expiry, reconciliation attempt, and final classification. Include a correlation ID generated before the first network action. Do not derive it from a secret or reuse it across operations.

This record format works for a local test harness:

{"time":"2025-04-12T10:18:03Z","case":"C05","id":"case-http-005","event":"dispatch_started"}
{"time":"2025-04-12T10:18:03Z","case":"C05","id":"case-http-005","event":"remote_committed"}
{"time":"2025-04-12T10:18:08Z","case":"C05","id":"case-http-005","event":"client_timeout"}
{"time":"2025-04-12T10:18:08Z","case":"C05","id":"case-http-005","event":"result","category":"outcome_unknown"}

Your assertion can then compare client and fixture records by id. If the fixture says remote_committed and the tool says confirmed_failure, fail the test. That disagreement exposes an unsafe claim, even if all code paths returned a neat error object.

For a tamper-evident trail, test verification too. Sallyport projects its session and activity records from an encrypted hash-chained audit log, and sp audit verify checks the chain offline without needing a vault key. A failure test should add a known event sequence, verify it, alter a copied record, and assert that verification fails on the altered copy.

Do not put full request bodies into ordinary logs by default. Request data often carries personal data, source code, or tokens embedded by a careless caller. Log the operation name, target classification, correlation ID, result category, and a protected diagnostic reference. Expand collection only in a controlled test environment where you know what the fixtures contain.

Test the agent's recovery behavior, not only the adapter

Unit tests prove that an adapter maps 403 to authorization_denied. They do not prove that an agent stops after receiving it. Run a small end-to-end suite with a deterministic agent instruction and a fake remote service that exposes the injected event log.

Give each run a narrow task and an explicit boundary. For example: create one record, read it back, then attach a note. Delay the create response after committing the record. The correct agent behavior is to inspect by correlation ID or idempotency token before it attempts a second create. A test should fail if it creates another record, even if it eventually completes the task.

Keep the agent prompt stable for this suite. If you change the prompt, tool contract, fixture behavior, and model version at once, a failure tells you little. Record the tool transcript and the agent's next call, then compare it against permitted transitions:

create -> outcome_unknown -> lookup_by_request_id -> found -> attach_note
create -> outcome_unknown -> create

The first transition is permitted only if the lookup confirms the original create. The second is a failure. This state-machine check is more useful than judging whether the final prose response sounded sensible.

Run deterministic matrix cases on every change to tool code, result schemas, authorization handling, or retry logic. Run interruption cases repeatedly in isolated infrastructure because scheduling affects them. When a new incident appears, add the smallest reproduction to the matrix before you fix it. Otherwise the same attractive, wrong recovery path will return during the next refactor.

The standard for a tool is not that it keeps moving after every fault. The standard is that it tells the truth about what it knows, leaves evidence behind, and refuses to turn uncertainty into a second destructive action.