AI agent retry logging that exposes duplicate effects

An agent retry is not a second copy of an action. It is a second attempt to complete one declared action, and the audit trail must preserve that distinction. If your records show only a run of similar HTTP calls, you have made recovery and repetition look identical.

That gap matters most when the action changes the world: creating a ticket, revoking access, posting a payment, pushing a deployment, or running a command over SSH. The agent may receive a timeout after the destination already acted. It may also retry after a local failure before any bytes left the machine. Those cases demand different responses, yet too many systems write both as request failed, retrying.

I have seen teams investigate alleged duplicate actions by comparing timestamps and payloads manually. That is a poor substitute for an event model. Build the relationship into each record when the action occurs. An investigator should be able to select an operation and see its intent, every attempt, the reason each later attempt happened, and the outcome that eventually settled the question.

A retry belongs to an operation, not to a log line

Every side-effecting agent action needs two identities: an operation ID for the intended result and an attempt ID for one execution try. The operation ID persists from the moment the agent decides what it intends to do until you close or reconcile that intent. Each network call or SSH execution gets a fresh attempt ID.

Suppose an agent intends to disable an account. It creates operation op_7f2c. The first request, att_01, reaches the identity API but the connection closes before a response arrives. A second request, att_02, might be a justified recovery attempt. Both records must point to op_7f2c; att_02 must point directly to att_01 as the attempt that triggered it.

Do not use the conversation ID as the operation ID. One agent conversation can contain many actions, and one action may outlive a conversation when a supervisor resumes work. Do not use a request hash alone either. A hash describes bytes, while an operation describes the intended effect. Two requests can differ in harmless transport details and still belong to one operation. Conversely, identical bytes can create two distinct intended actions when the agent deliberately sends them twice.

Use an operation identity when the agent forms a commitment such as: "disable account A," "create one incident for alert B," or "run this migration once on host C." Capture that intent in structured fields. A plain natural-language summary is useful for people, but it cannot be the sole identity because wording drifts between agent runs.

A clean hierarchy looks like this:

• A session identifies one agent process run.
• An operation identifies one intended external effect.
• An attempt identifies one actual execution try.
• An observation identifies evidence received later, such as a callback, a read-after-write check, or an operator decision.

This hierarchy solves an awkward case that shows up often: an agent sends a request, times out, then asks a different endpoint whether the action happened. The lookup is not a retry. It is an observation attached to the original operation. Treating it as another attempt buries the most useful evidence in the wrong bucket.

Unknown is an outcome, not an error message

A timeout produces uncertainty, not proof of failure. Your log needs a state for that uncertainty, and it must keep it until later evidence resolves it.

Many client libraries collapse several events into one exception: connection refused, DNS failure, a response body that arrived too late, and a connection reset after the server committed a write. That convenience is fine for application control flow. It is unacceptable as the final audit record. The records must state what the caller observed and avoid claims about the destination that the caller cannot support.

For an attempt, separate the local observation from the resolved operation state. An attempt can be not_sent, sent_no_response, response_received, or execution_error. The operation can be open, succeeded, failed, unknown, or cancelled. Names vary, but the separation does not.

not_sent means the client stopped before dispatch. A local credential lookup failure might fit here. Retrying it cannot duplicate a remote side effect because no remote request went out.

sent_no_response means the caller dispatched the action but lacks a usable response. This is the dangerous state. An automatic retry may be safe only if the destination has a reliable deduplication mechanism, or if the action itself cannot create a duplicate effect.

response_received does not automatically mean success. A server can return a validation error, a conflict response, or a success response that describes asynchronous work. Store the status, relevant response fingerprint, and any destination-issued operation reference. Then set the operation state based on the contract of that particular API.

Avoid writing failed when you mean unknown. The word makes dashboards tidy, but it tells the next agent or operator to repeat an action that may already have happened. During an incident, that one dishonest field can turn a single mistaken action into a sequence of them.

HTTP semantics do not make a business action safe to repeat

HTTP methods describe protocol semantics, not your business guarantees. RFC 9110 says that a method is idempotent when the intended effect of multiple identical requests is the same as the effect of one request. It names PUT, DELETE, and the safe methods as idempotent, while POST is not idempotent by default.

That guidance is useful, but engineers overextend it. A DELETE request may be protocol-idempotent because deleting a missing resource leaves it missing. Your audit question may still be different: did the agent delete the correct account, did it trigger downstream cleanup twice, and did the second call use a different authority? The protocol label does not answer those questions.

PUT also causes trouble. A PUT that sets a resource to a fixed representation often tolerates retries. A PUT endpoint that triggers a notification, allocates a record, or runs an integration on every receipt does not offer the safety people assume. Read the destination's documented contract and test the behavior under a forced response loss. Method names are not evidence.

POST endpoints often support an idempotency token. Send a stable token derived from the operation ID, not from the attempt ID. If att_01 and att_02 have different tokens, you have defeated the feature that protects you from the retry.

A request envelope might look like this:

{
  "operation_id": "op_7f2c9c",
  "attempt_id": "att_01",
  "idempotency_key": "op_7f2c9c",
  "intent": {
    "kind": "disable_account",
    "subject_ref": "user:1842"
  },
  "destination": {
    "method": "POST",
    "route_template": "/v1/accounts/{id}/disable",
    "authority_ref": "vault:identity-prod"
  },
  "request_fingerprint": "sha256:...",
  "dispatch_state": "sent_no_response"
}

Do not log the authorization header, session cookie, private SSH material, or a body that contains secrets. Record a credential reference and a fingerprint of the canonical request representation. A fingerprint helps investigators compare attempts without turning the audit trail into another secret store.

The destination must honor the idempotency token for it to prevent repeat effects. When it does, record the destination's returned reference and whether the response came from a stored prior result. When it does not, the token is just an inert header and your retry policy must act accordingly.

An SSH retry can repeat more than one command

SSH makes retry accounting harder because one connection can carry shell syntax, pipelines, redirects, and commands that partially complete. A failed SSH session does not identify which parts of the remote command ran.

Consider this command:

create-user deployer && install-key deployer /tmp/new.pub && restart-service api

If the client loses the connection after dispatch, a retry can fail because the user already exists, install a key twice if the helper appends it, or restart a service for a second time. The shell's && only controls behavior within one execution. It does nothing to protect a fresh connection that reruns the full string.

Log the exact command only when it contains no secret material. Otherwise store a redacted display form and a canonical fingerprint. Record the host alias or host key reference, remote account reference, working directory if relevant, exit status if received, and the execution boundary. The boundary should say whether the helper started the command and whether it received an exit status, not merely whether the local caller reported an error.

The safer pattern is a remote script with an operation marker that the script checks before it acts. The marker must live where the target system can read it atomically. A database transaction, a deployment record, or a file created with exclusive creation can work, depending on the environment. A local agent cache cannot prove anything after a process crash or a second agent run.

For example, a deployment script can accept OPERATION_ID, write it to a release record before activation, and return the existing result if that record already exists. The audit event then captures both the local operation ID and the remote record ID. That gives an investigator a bridge between the agent record and evidence on the host.

Do not call arbitrary shell commands retryable because they are "mostly safe." Categorize command families. Read-only collection can retry freely. State-setting commands need an explicit convergence condition. Append-only, financial, destructive, or notification commands need a remote deduplication record or a human decision after an unknown result.

Every later attempt needs a stated reason and a parent

A retry event must name the attempt that caused it and the condition that justified another try. retry_count: 2 is too weak. It tells you there were previous calls, but it does not identify which call failed, whether the agent changed anything, or whether a human approved the continuation.

Use a controlled set of reasons, then attach supporting details separately. Useful reasons include connection_not_established, rate_limited, destination_5xx, response_lost_after_dispatch, credential_refreshed, and operator_requested. Do not let an agent invent a prose reason that looks reassuring but cannot be grouped or reviewed.

For each retry, preserve these links:

{
  "operation_id": "op_7f2c9c",
  "attempt_id": "att_02",
  "retry_of_attempt_id": "att_01",
  "retry_reason": "response_lost_after_dispatch",
  "retry_decision": "destination_idempotency_confirmed",
  "attempt_budget_remaining": 1,
  "request_fingerprint": "sha256:...",
  "prior_request_fingerprint": "sha256:..."
}

The two fingerprints should normally match. If they differ, log why. A changed timestamp header may be expected. A changed account identifier, amount, hostname, route, or authority is not a retry in the ordinary sense. It is a new operation or a manually amended intent, and the audit trail must say so.

This is where agent systems often mislead themselves. The model reads an error, changes a parameter to "fix" it, and calls the next request a retry. That is a new decision with a new possible effect. Linking it as a retry disguises a change of plan and makes review nearly impossible.

Bound the attempt budget per operation and record the budget decision. Retrying a rate-limit response after the destination's stated delay differs from retrying an unknown write after a timeout. The former often has an unambiguous remote response. The latter needs idempotency evidence or reconciliation before you repeat it.

Idempotency tokens and audit identities solve different jobs

An idempotency token tells a destination to treat repeated submissions as one operation. An audit operation ID tells your own investigators which attempts belong to one intent. Use both when you can, but never pretend one replaces the other.

The token may be scoped to a route, a merchant, a time window, or a particular account. Some APIs retain tokens only for a limited period. Some return the original response for a duplicate; others return a conflict. Some reject a reused token when the request body changes. Those details belong in the connector contract and in your test suite.

Your operation ID has a wider role. It connects the agent's session, approval evidence, request construction, transport attempts, remote response, and later reconciliation. It should remain valid even when a vendor API does not offer idempotency, when the action uses SSH, or when an operator performs the final recovery manually.

Do not generate a new operation ID after a restart just because process memory disappeared. Persist pending operations before dispatch. On recovery, inspect each unresolved operation and choose one of three paths: reconcile using remote evidence, retry under a documented idempotency guarantee, or escalate to a person. A restart is an engineering event, not permission to forget uncertainty.

A popular but wrong recommendation is to retry every failed write with exponential backoff. Backoff reduces pressure on a struggling service. It does not convert an unknown write into a safe write. The action's repeatability and the destination's deduplication behavior decide whether another attempt is acceptable.

Reconciliation closes uncertainty without rewriting history

Reconciliation means gathering later evidence about an unresolved operation. It does not mean editing the first attempt until it looks successful.

Say an agent creates an incident with a client-provided reference in the request body. The initial POST ends with sent_no_response. Before retrying, the agent queries incidents by that reference. If it finds one matching record, append an observation event that cites the original operation ID, the query fingerprint, the returned remote identifier, and the match criteria. Then close the operation as succeeded through reconciliation.

If the query finds nothing, be careful. Absence proves little when the API has replication delay, search indexing lag, or weak filtering. Record the negative observation with the time and endpoint used. Retry only if the destination's contract says the idempotency token remains effective, or wait and ask for a decision.

If the query finds two matching records, do not call the operation succeeded and move on. Close it as duplicate_effect_confirmed, preserve both remote identifiers, and create a separate remediation operation. The remediation must not share the original operation ID because it has a different intended effect.

Keep append-only events rather than mutable status rows as your source of truth. You can project a convenient current status for a user interface, but the evidence must preserve the transitions: intent created, attempt dispatched, response lost, lookup performed, remote record found, operation resolved. An investigator needs the sequence, including the mistaken retry if one occurred.

A tamper-evident log adds another property: it lets you verify that a later process did not quietly remove the first unknown attempt. Sallyport records agent sessions and individual actions from a write-blind encrypted, hash-chained audit log, and sp audit verify checks the chain offline over ciphertext. That helps preserve the timeline, but the event schema still needs the operation and attempt links described here.

Agent delegation needs one operation owner

Subagents make duplicate effects easier because each process may believe it owns the task. Give one process ownership of the operation ID, and require every delegated worker to carry that ID in its action context.

A planner can request that a worker gather information and another worker execute an action. The information-gathering calls should receive their own operations because they are separate intents. The execution worker should receive the original operation ID only when it acts on the same declared effect. Its individual calls use fresh attempt IDs and identify the worker process that made them.

Do not let workers independently retry an unknown write while the parent also retries it. The parent must receive the worker's dispatch state before it decides what to do. If the worker exits unexpectedly, mark the operation unresolved and reconcile. A missing child result is not an invitation for a supervisor to replay the command.

Per-session approval records also belong in the evidence chain. When an agent process gains authority to make a group of calls, log the process identity, approval time, and revocation time separately from the action outcome. Approval explains who was allowed to try. It does not establish whether the destination performed the action.

For sensitive operations, require a per-call approval for the retry itself when the first attempt had an unknown outcome. People make a better decision when the approval card states the original intent, the previous dispatch state, and the planned recovery method. A generic prompt saying "allow API call" hides the one fact that should slow them down.

The investigation view must show a timeline, not a pile of requests

An investigator needs a single operation page or query result that starts with intent and ends with the best supported outcome. Request logs sorted by time force the reviewer to reconstruct parentage under pressure, often across several systems with slightly different clocks.

Show the operation state at the top, but make the evidence available underneath. Each attempt should display its number, dispatch state, retry parent, reason, authority reference, destination, request fingerprint, response summary, and duration. Each observation should show what it checked and why that evidence changed or failed to change the state.

Do not conceal duplicate requests because the final effect was harmless. Harmless duplication today can become an expensive side effect after an API changes or an integration adds a webhook. The record should let reviewers distinguish "the agent retried correctly and the destination deduplicated" from "the agent sent the request twice and got lucky."

Treat a request body change during recovery as an explicit fork. The original operation stays open or receives its resolved outcome. The amended action gets a new operation ID and a link such as supersedes_operation_id. That record tells the truth: the agent did not merely retry, it changed what it intended to do.

Build one forced-failure test before trusting the design. Arrange for the destination to commit a known idempotent test action, then drop the response to the caller. Confirm that the next agent run retains the original operation ID, uses the same destination token, logs the unknown first attempt, performs the documented reconciliation or retry, and closes the operation with evidence. If your test cannot answer those points, an incident will not either.

The first field I look for in an agent action log is not an HTTP status. It is the stable operation ID. Without it, every retry investigation starts as guesswork. With it, you can ask the questions that matter: what did the agent intend, what left the machine, what changed between attempts, and what evidence supports the final outcome.