Agent tool timeouts: recovery plans that prevent repeats

A timeout is an observation about the client, not a verdict about the work. The caller gave up waiting. That is all it knows. When an agent turns that observation into "failed" and retries a state-changing call, it can create a second payment, a second deployment, a second support ticket, or a remote command that runs twice against a machine already under stress.

Agent tool timeouts need a recovery plan because agents act faster than the people supervising them and tend to treat tool output as ground truth. A missing response is not ground truth. The recovery path must decide whether to retry, wait, query for evidence, or stop and ask for a human decision. Build that path before giving an agent permission to make consequential calls.

A timeout leaves three plausible histories

After a client timeout, the request belongs to one of three broad histories: the service never received it, the service received it and has not finished, or the service finished but the client never got the result. Network failures can occur before a connection opens, while a request body travels, while the service works, or while the response returns. The same exception type may cover all four.

This distinction changes the next action. If a DNS lookup failed before any connection, a retry may be sensible. If the service accepted a request to delete a resource and the response disappeared, retrying may repeat a destructive operation. If the service queued an asynchronous job, a second submission may create a competing job while the first one is still running.

HTTP does not provide a magic bit that tells a client which history occurred. RFC 9110 describes request methods and the meaning of responses, including the distinction between safe and idempotent methods. It does not promise that a client can infer server execution from a lost response. That limitation is physical, not a missing SDK option.

Teams often blur two separate questions:

• Can this request be sent again without changing the intended final state?
• Did the original attempt actually reach and affect the service?

Idempotency addresses the first question. Reconciliation addresses the second. A system needs both. An idempotent PUT may be safe to repeat, yet a timeout still leaves you unable to tell whether downstream work triggered by that request has finished. A status lookup can establish outcome, yet it cannot protect you from duplicate work if the service accepts two indistinguishable creates.

Agents need this distinction in their tool contracts. A plain text error such as request timed out invites an improvisation. A structured result that says outcome: unknown tells the agent that it must leave the retry branch and enter the evidence branch.

Map the request path before choosing a retry

A useful recovery plan names the boundaries where evidence can exist. Start with the agent process, then the tool wrapper, connection pool, gateway or proxy if one exists, the service ingress, the application, the durable store, and any worker that handles asynchronous work. A timeout at one boundary says nothing reliable about the next boundary.

Consider a call to create a deployment. The agent sends a request through a tool. The client writes the full request body, the server commits a deployment record, then the connection breaks before the response reaches the client. The tool emits a timeout. The agent retries with a fresh request. The server now has two deployment records, each valid from its own point of view.

Now change one detail: the client times out while uploading the body, and the server rejects the partial body before application code runs. The same tool might still return timeout. In this case a retry may create exactly one deployment. The caller cannot tell these cases apart from the timeout alone.

Write down the evidence each component can produce. For a typical HTTP action, that includes:

• Client timestamps, selected target, request body digest, and a caller-generated operation ID.
• Service access logs that show whether ingress accepted the request.
• An application record that stores the operation ID with a committed result.
• Worker or queue records for actions that continue after the synchronous request.
• A read endpoint that returns the current state or operation status.

Do not make network logs your only source of truth. A load balancer log can show that bytes arrived, but it cannot prove that the database transaction committed. A database record can prove a commit, but it may not prove that an external provider received a later side effect. The authoritative record must match the action you are trying to prove.

For an email send, the provider's accepted-message identifier is stronger evidence than your application log saying "about to send." For a database migration, a migration table or transaction record is stronger than a shell process exit code that the caller never received. For a cloud resource create, an operation URL or resource tag with a caller-generated ID is better than a repeated create request.

Idempotency must belong to the operation, not the attempt

A retry scheme works only when every attempt for one intended action carries the same durable identifier. Generate the identifier before the first network call. Persist it alongside the action description. Reuse it after a process restart, a tool restart, or a handoff to a human operator.

Do not generate a new UUID inside a retry loop. That pattern looks careful in code review and defeats the entire purpose. The server sees every retry as a new request, which is exactly how duplicate operations appear.

A request can carry an idempotency value in a header or a body field, depending on the API. The transport detail matters less than the server rule. The server must atomically associate that value with the operation and its result. If two identical requests arrive at the same time, the server must serialize them or make one observe the other. A cache that expires before delayed retries finish does not provide reliable duplicate protection.

A practical HTTP request might look like this:

POST /deployments HTTP/1.1
Content-Type: application/json
Idempotency-Key: op_7d5d4d8e4e5a
X-Correlation-ID: run_42_task_9

{"repository":"api","revision":"a1b2c3d4","environment":"staging"}

The server should store the idempotency value with a fingerprint of the meaningful request fields and the resulting deployment or operation ID. If the same value arrives with a different revision or environment, reject it. Returning the first result for a different payload silently applies the wrong intent.

For an asynchronous operation, return a durable operation reference as soon as the server accepts the work:

{
  "operation_id": "dep_1842",
  "state": "accepted",
  "status_url": "/operations/dep_1842"
}

After a timeout, the agent queries for op_7d5d4d8e4e5a or dep_1842 before it considers another submission. If the API supports neither an idempotency value nor a lookup by external reference, classify the write as ambiguous by design. That may be acceptable for a disposable test resource. It is a poor fit for an autonomous action that creates cost or changes production state.

Do not call a method safe merely because it uses POST with a retry library. HTTP method names are hints about intended semantics, not protection against a server implementation that duplicates work. Read the specific API documentation, then test the duplicate behavior yourself.

Give agents an explicit unknown-outcome state

An agent should not receive only success or error from a tool that can touch the outside world. It needs a third result: unknown. That state prevents the most damaging model behavior, which is treating an incomplete transcript as permission to try a slightly different version of the same command.

Use a result contract that records the phase that failed, while admitting that the phase may be uncertain. For example:

{
  "outcome": "unknown",
  "operation_id": "op_7d5d4d8e4e5a",
  "correlation_id": "run_42_task_9",
  "transport_observation": "response deadline exceeded after request write",
  "retry_allowed": false,
  "reconcile": {
    "method": "GET",
    "path": "/operations/by-id/op_7d5d4d8e4e5a"
  }
}

The retry_allowed field must come from the tool or action definition, not from an agent's guess about English verbs. An agent cannot safely infer that create_release is harmless because the target happens to be a staging environment. A staging deployment can still send notifications, consume a shared quota, or mutate a release channel.

Make the agent follow a bounded recovery sequence:

1. Preserve the intended action, operation ID, target, and timeout observation in its run record.
2. Query the authoritative status source using the same operation ID or a service-issued reference.
3. Continue only on a confirmed terminal result. Retry only if the action definition permits it and the status source shows no accepted operation.
4. Stop and present the evidence when the service cannot establish the outcome within the action's recovery deadline.

The stop condition matters. An agent that polls forever consumes attention and can keep a task alive long after its original purpose has vanished. An agent that tries five variations of a write request can create a cleanup project for someone else. Give each operation a deadline for recovery, separate from the request deadline.

A human approval does not fix an unknown outcome by itself. Approval answers "may this caller attempt this action?" It does not answer "did the previous attempt succeed?" Keep authorization evidence and execution evidence separate in both your UI and your logs.

Read timeouts deserve different treatment from writes

A timed-out read usually has less severe duplicate risk, but it can still produce bad agent decisions. An agent may time out while listing resources, receive an incomplete or stale cached response elsewhere, and conclude that a resource does not exist. It then attempts a create that collides with reality.

Classify reads by the decision they support. A harmless dashboard refresh can retry with bounded backoff. A read used to decide whether to issue a write needs an explicit consistency rule. If the service offers an ETag, version, generation number, or read-after-write status endpoint, use it. If it offers eventual consistency only, make the waiting period and failure condition visible to the agent.

Avoid a universal retry count. A short metadata lookup may tolerate two quick retries. A report query that loads a warehouse may need one long deadline and no immediate repeat. A call that returns 429 Too Many Requests or an explicit service retry value needs different handling than a socket timeout. Treating every error as a transient network issue is how an agent turns a partial outage into avoidable load.

Backoff helps protect services, but it does not solve ambiguity. It spaces duplicate requests farther apart. Pair backoff with an idempotency value or a status check for every write that matters.

Use conditional writes where the API supports them. An If-Match request with a known ETag can prevent an agent from overwriting a resource that changed after its read. A create that accepts a client-selected resource ID can make retries converge on one object. These mechanisms protect state transitions; they do not replace a record of whether side effects outside that resource occurred.

SSH hides remote execution behind one broken stream

SSH timeout recovery needs more suspicion than HTTP recovery. A lost SSH session can occur after the remote host starts a command, during output transfer, after the command exits, or while a child process continues after its parent disconnects. A local shell message cannot tell you which one happened.

The dangerous pattern is a long compound command:

ssh deploy@host 'download-release && migrate-db && restart-service'

If the connection drops after migrate-db, rerunning the full command may run migrations twice or restart a service whose new release never finished downloading. The terminal transcript has collapsed several state transitions into one opaque result.

Split remote work into operations with durable, inspectable markers. A deployment can record a release ID before it begins, store migration versions in the database, and expose the active revision through a local status command. A recovery tool reconnects and asks for those markers before it does anything else.

For example, an agent can use a remote status command whose output is designed for machines rather than humans:

ssh deploy@host '/usr/local/bin/release-status --json'

{
  "release_id": "rel_202",
  "phase": "migrated",
  "active_revision": "9f24c1",
  "migration_version": "20250308_02"
}

The recovery action now has a basis for a decision. If phase is migrated, do not rerun migrations. If the host reports no release_id, the agent may start the operation only if the remote command guarantees that absence means no prior execution. If SSH cannot reconnect, the correct result remains unknown. Do not replace it with a hopeful rerun when the action changes a production host.

Use remote locks with care. A lock can prevent simultaneous runs, but a stale lock after a host failure can block recovery. Put the operation ID and an expiry policy in the lock record, and make inspection possible without deleting the lock blindly. A cleanup command that removes every old lock is another state-changing operation and needs its own evidence.

Sallyport can keep SSH credentials out of an agent process while its bundled sp-ssh helper performs the connection, but credential isolation does not make a dropped session safe to replay. The remote command still needs an operation ID, durable markers, and a reconciliation route.

Audit records help investigate, but they do not prove completion

An action log should preserve enough detail to reconstruct intent and recovery without storing secrets. Record the caller session, the time, target identity, operation ID, request digest or command template, authorization event, transport result, and final reconciled outcome. Do not record bearer tokens, private keys, or raw request bodies that may contain credentials or personal data.

Separate an attempted action from a completed action. A log line that says POST /deployments timeout is an attempt record. A later query that returns an operation in succeeded state is completion evidence. Keep both. Replacing the first record with a final success erases the most useful part of the incident: the period when the caller did not know.

Tamper evidence matters when an agent run triggers a later dispute. You need to answer which process made the call, what it was authorized to do, whether it received a result, and how the team established the final state. A mutable activity table is easy to search but weak evidence if a compromised process can rewrite history.

Sallyport records agent sessions and individual calls from a write-blind encrypted, hash-chained audit log, and sp audit verify can verify the chain offline over ciphertext. That record can show the gateway action and caller history, while the remote service or host remains the authority on whether the intended work completed.

Do not confuse a gateway audit event with an application transaction. If the gateway logged an outbound request, the request may still have failed before the service committed it. If the service committed it, the gateway may never have seen the response. Investigation works when records from both sides share an operation or correlation ID.

Test ambiguity deliberately before an outage does it for you

A timeout plan that has never faced a lost response is an assumption. Test the exact failure where the service completes the operation but the client loses the result. This is the case most teams skip because ordinary happy-path test doubles cannot express it.

Build a test endpoint or proxy fixture that accepts a request, commits its durable record, then delays or drops the response. Send the same operation ID twice. Verify that the service returns one logical operation, that the agent queries status after the timeout, and that the audit trail preserves both attempts and the reconciliation result.

Then test the opposite case: interrupt the request before the service accepts it. Confirm that recovery can retry only after it finds no operation record. The two tests may produce the same client exception. Your tool contract must lead to different actions because service evidence differs.

Exercise these cases as well:

• The service accepts the operation but its worker remains pending beyond the agent's recovery deadline.
• Two agent processes submit the same operation ID at nearly the same time.
• The status endpoint is unavailable while the primary write endpoint is healthy.
• An SSH command starts a child process, then the connection ends before final output.
• A human resumes a paused run after another operator has already reconciled the action.

The last case catches a problem that appears in real operations: recovery state must live outside the agent's conversational memory. Store the operation ID and current finding in a durable run record. A restarted agent should read that record and continue reconciliation, rather than invent a new action because it cannot see its previous transcript.

Set alerts on unknown outcomes that exceed their recovery deadlines. Do not alert on every first timeout if ordinary retries resolve safe reads. Alert when a consequential write lacks a confirmed terminal state, when duplicate operation IDs carry different payloads, or when remote markers contradict the expected progression. Those are the cases that need a person before an agent keeps moving.

A recovery plan should make refusal ordinary

The strongest timeout behavior is often refusal: "I cannot confirm whether the deployment request completed, so I will not submit another one." That is not a tool failure. It is the correct response to missing evidence around an irreversible or expensive action.

Make that response useful. Show the operation ID, target, last confirmed state, timestamps, and the exact status query or remote inspection that would settle the matter. If no authoritative query exists, say so plainly and route the decision to someone who understands the consequence of a duplicate.

Teams resist this because a retry feels productive and a pause feels slow. After enough duplicate writes and half-finished deployments, the trade becomes clear. A minute spent reconciling is cheaper than discovering that two systems each believe they performed the one action your agent was asked to perform.

Start with the writes that create money movement, external messages, releases, access changes, and data deletion. For each one, demand three answers from the API owner: what identifier binds retries to one operation, where can a caller query the result, and what proof exists when the connection dies. If any answer is missing, make the tool return unknown and require a deliberate human decision instead of teaching the agent to guess.