How to record canceled agent actions without lying

An audit log that labels every canceled agent action as "failed" is lying. The agent may have stopped waiting, but the database row might already exist, the deployment may already be rolling, or the remote command may still be running after the connection vanished.

The fix is not a longer list of failure codes. It is an outcome model that separates what the gateway observed from what happened at the destination. Investigators need to tell the difference between an action that never left the machine, one that the destination rejected, one that completed with a receipt, and one whose effect remains unknown.

The caller's result is not the action's result

A caller sees a narrow slice of an action: it submits work and waits for a response. The action itself crosses several systems that do not share a clock, a process lifetime, or a reliable return path. When the caller cancels, disconnects, or reaches its deadline, it has learned something about its own wait. It has not necessarily learned anything final about the remote effect.

This distinction matters most for state changing work. Creating a ticket, issuing a refund, applying infrastructure changes, deleting an object, rotating an access token, and running a remote command all have consequences that can survive a missing response. If your journal writes failed because the agent process exited first, a later retry can create a second ticket, issue a second refund, or run the destructive command twice.

Read operations need the same honesty, although the risk differs. A canceled fetch can return an incomplete view, and an agent might make a bad follow-up decision. It usually does not change the world by itself. A POST, PATCH, DELETE, or remote shell command can.

Keep three things separate in every record:

• Caller disposition: completed, canceled, disconnected, or timed out.
• Dispatch evidence: never started, started locally, bytes handed to transport, or receipt confirmed by the remote side.
• Effect outcome: no effect, succeeded, rejected, partially completed, or unknown effect.

Teams often merge the first and third fields because a single status column is convenient. That convenience lasts until an incident review. Then someone must explain why "request canceled" appears beside an object that plainly exists in production.

The practical rule is blunt: only write no effect when evidence rules out execution. Write unknown effect when execution remains plausible and you lack a trusted result. Unknown is not an embarrassing gap in the log. It is the accurate result of a distributed action with an interrupted observation path.

Record the evidence boundary, not a guessed story

Every action needs an explicit boundary after which the gateway can no longer honestly promise that nothing happened. Call it the dispatch boundary. For HTTP, that boundary may be when the request is committed to a connection and handed to the operating system, or when an upstream service acknowledges acceptance. For SSH, it may be when the helper has sent the command request over an authenticated channel.

Do not pretend that a single sent=true Boolean settles the matter. Local writes can be buffered. A transport library can report a write before a peer application reads the data. A peer can receive a request, apply the change, and lose the response on the way back. The log should describe the strongest evidence available, not turn an implementation detail into proof.

A useful action record has immutable identifiers and a sequence of observations. This is a compact shape that works for both an API call and a command execution:

{
  "action_id": "act_01JQ7M4V6K",
  "session_id": "ses_01JQ7M2Y8A",
  "channel": "http",
  "intent": {
    "method": "POST",
    "target": "api.example.internal/v1/releases",
    "request_fingerprint": "sha256:...",
    "idempotency_token": "release_01JQ7M4V6K"
  },
  "observations": [
    {"at": "2026-07-22T16:40:01Z", "kind": "authorized"},
    {"at": "2026-07-22T16:40:02Z", "kind": "dispatch_started"},
    {"at": "2026-07-22T16:40:03Z", "kind": "transport_write_completed"},
    {"at": "2026-07-22T16:40:33Z", "kind": "caller_deadline_exceeded"}
  ],
  "caller_disposition": "timed_out",
  "effect_outcome": "unknown_effect",
  "outcome_basis": "response_not_observed_after_dispatch"
}

The request fingerprint identifies what was attempted without placing a bearer credential, raw request body, or secret command argument in the journal. The identifier must remain stable across retries and later reconciliation. If an operator cannot connect the original request, its retry, and the eventual remote object, the audit trail does not answer the question that matters.

There is an important difference between dispatch_started and transport_write_completed. The first says the gateway began the operation. The second says its local transport accepted the outbound data. Neither says that the remote application executed it. If your implementation cannot distinguish them, record the weaker fact and say so in the outcome basis.

Cancellation before dispatch can mean no effect

A cancellation can support a no-effect conclusion, but only before the gateway commits the action to an outside channel. This is the clean case: the agent revokes its request while it is still waiting for local authorization, before the vault unlocks, before an HTTP request begins, or before an SSH command is handed to the transport helper.

The audit entry should show why the conclusion is safe. "Canceled" alone does not tell an investigator where cancellation landed. Write a stage and a local proof point.

{
  "action_id": "act_01JQ7P1N2R",
  "caller_disposition": "canceled",
  "effect_outcome": "no_effect",
  "outcome_basis": "cancellation_observed_before_dispatch",
  "last_observed_stage": "awaiting_authorization"
}

This result is also appropriate when the local gate denies the action before any outside request can begin. A missing approval, a locked vault, a revoked session, or an invalid local request may all produce no effect, provided the gateway never dispatched work. The journal should distinguish a denied request from a canceled request because they tell different stories about human control and agent behavior, but both can safely say the target saw nothing.

Do not apply that label after you have created a connection and started writing merely because the transport call returns a cancellation error. Many libraries use one error value for several paths: canceled while queued, canceled during a write, canceled while waiting for response headers, or canceled while reading a body. Those are not interchangeable.

Cancellation propagation has a useful purpose, but it is not a time machine. The gRPC cancellation guidance says a client cancellation signals that it no longer needs the RPC result and recommends that servers stop work and propagate cancellation to downstream work. That is good resource hygiene. It does not prove that an earlier side effect was undone, and it cannot undo an independently committed write.

If the receiving service offers an explicit cancellation endpoint tied to an operation identifier, record that as a second action. Its result can change the original effect outcome only if the service gives a trustworthy statement about the original operation. A best-effort cancel request that returns after a network break creates another unknown effect. It does not clean up the first one by magic.

A timeout after dispatch has an unknown effect

A deadline is a local limit on waiting. It is not a verdict on remote execution. Once an action crosses the dispatch boundary, a timeout must default to unknown effect unless a protocol receipt or a later check proves more.

HTTP makes this easy to get wrong because status labels sound final. RFC 9110 says a 408 means the server did not receive a complete request message within the time it was willing to wait. A 504 means a gateway did not receive a timely response from an upstream server. Both statements describe a particular observer and a particular exchange. Neither statement proves that a separate system did not act on data it already received.

Consider an agent that sends POST /v1/releases with a 30 second deadline. The API validates the request, inserts a release row, asks a deployment controller to start, and then stalls while formatting its response. At 30 seconds, the agent sees a timeout. The release exists. A retry without an idempotency token may make another release, even though the first call is marked "failed" in the agent transcript.

The truthful record looks like this:

{
  "caller_disposition": "timed_out",
  "effect_outcome": "unknown_effect",
  "outcome_basis": "deadline_after_transport_write_no_remote_receipt",
  "recovery_required": "lookup_by_idempotency_token"
}

Do not use failed as a shortcut for unknown. Reserve failure outcomes for facts you can establish: a remote service returned a validation error, a command returned a nonzero exit status, a connection could not be established before any request left the gateway, or a local authorization decision denied execution. A timeout does not meet that standard once outbound dispatch may have happened.

Partial completion deserves its own outcome when the remote side gives evidence. Batch APIs and scripts frequently do some work before they fail. If a service returns a list of completed object IDs followed by an error, record partial_effect, preserve the object IDs where policy permits, and capture the service's stated reason. Calling that simply "failed" hides the exact cleanup work an operator must do.

Lost acknowledgments need a separate record

A lost acknowledgment happens after the receiver may have acted but before the gateway receives a final receipt. It is common enough to deserve a named observation rather than a generic network error.

The sequence usually looks ordinary until the last moment:

1. The gateway authorizes and dispatches an action.
2. The remote service accepts it and performs, or queues, the requested work.
3. The response is delayed, the connection drops, or the local process exits.
4. The gateway has no durable proof of completion, even though the remote system may have durable proof.

The first mistake is to overwrite the original entry when a later lookup succeeds. That makes the journal read as if the gateway knew the outcome at the time. An investigator needs both facts: the initial call ended without acknowledgment, and a later reconciliation found a matching remote result.

Append an observation instead:

{
  "action_id": "act_01JQ7M4V6K",
  "reconciliation": {
    "at": "2026-07-22T16:43:10Z",
    "method": "GET /v1/operations/release_01JQ7M4V6K",
    "remote_reference": "op_8f2c",
    "result": "succeeded"
  },
  "effect_outcome": "succeeded",
  "outcome_basis": "remote_operation_lookup"
}

The original caller disposition remains timed_out. Do not rewrite it to completed. The caller did time out. The system later learned that the remote action succeeded. Those facts coexist without contradiction.

A remote receipt is only as trustworthy as its correlation. Matching on an object name, current timestamp, or agent-supplied natural language is weak. Prefer an idempotency token accepted by the target, an operation ID returned before long running work begins, or a provider request ID that the target guarantees is unique for the request. If the destination offers none, use a narrowly scoped read-back query and record why it is sufficient or why it remains ambiguous.

For example, discovering a new user named build-bot does not prove which create request made it. Discovering an object with a stored request token equal to the original action token is much better. The difference determines whether you can safely retry.

Idempotency turns recovery into a check instead of a gamble

Idempotency is not a retry permission slip attached after the fact. It is a contract established before dispatch. The client supplies a stable token, and the service guarantees that repeated requests with that token identify the same logical operation rather than creating new effects.

For every state changing HTTP integration, ask the service owner four direct questions:

• Does it accept a caller-supplied idempotency token?
• What request fields must remain identical when that token repeats?
• How long does it retain the token-to-result mapping?
• Can a caller retrieve the original result after a lost response?

If the answers are vague, do not advertise automatic retry as safe. "We usually deduplicate" is not a contract. A cache eviction, a regional failover, or a request parser change can turn that assumption into duplicated work.

Where the target has no idempotency support, split a risky operation where you can. Create a durable draft with a unique external reference, verify that draft, then issue the irreversible command against its returned identifier. This does not make every operation safe, but it creates a reconciliation point before the expensive or destructive part.

A deployment workflow illustrates the difference. A one-shot endpoint that both creates and starts a release leaves little room to recover from a lost acknowledgment. A two-call workflow can create a release using the action ID as an external reference, query that reference after a timeout, and start it only once the caller has a known release ID. The extra call is usually cheaper than explaining an unexpected production change.

Never generate a fresh idempotency token for an automatic retry of the same action. A new token declares that the retry is a new logical request. That may be correct when a human intentionally repeats work, but it defeats deduplication during recovery. Keep the original token with the action record and make the retry cite its parent action ID.

An idempotency contract also helps incident response. Investigators can ask one concrete question: "What did the target decide for token X?" Without it, they must infer intent from timestamps, logs, and names. That is slow, error prone, and often impossible after retention windows pass.

SSH closure and command completion are different facts

SSH invites a similar mistake because a closed session feels like a stopped command. It is not. A network break can cut off the client while the remote process continues under its parent shell, supervisor, or service manager. Conversely, a process can finish while the client misses the exit report.

RFC 4254 treats channel closure, end of file, and exit status as distinct protocol events. It recommends returning an exit status when a remote command terminates, but it does not make a channel close proof that you received one. The specification also says peers exchange close messages before each side considers the channel closed. That tells you about the SSH channel state, not whether the remote command made a filesystem or service change before the channel died.

Record SSH evidence as separate observations:

{
  "channel": "ssh",
  "observations": [
    {"kind": "command_request_sent"},
    {"kind": "stdout_received", "bytes": 1840},
    {"kind": "connection_lost"}
  ],
  "caller_disposition": "disconnected",
  "effect_outcome": "unknown_effect",
  "outcome_basis": "no_exit_status_or_remote_process_identity"
}

If you receive a valid exit status and a complete channel close after the command's output, you have strong evidence about the command process, though not an absolute proof of every external side effect it triggered. A script may successfully submit asynchronous work and exit zero before that work completes. Record the command outcome as succeeded, then model the external job as its own operation if the remote system gives you an ID.

Commands need a recovery plan before an agent runs them. Prefer commands that print or write a durable operation identifier. For a service restart, query the service manager using a known unit name and a before-and-after state. For a database migration, inspect the migration ledger rather than trusting terminal output. For a file operation, check a content hash and a version or generation marker, not only whether a path exists.

Avoid treating a signal sent to a local helper as proof that the remote process stopped. The signal may arrive before remote dispatch, after the remote command completed, or after the connection broke. Log the signal as a caller or gateway event. Change the effect outcome only when the remote endpoint gives evidence.

Investigation starts with chronology, not a final label

An investigator should be able to reconstruct an indeterminate action without guessing which log line came first. That requires a stable action ID, a session ID, monotonic event ordering within the gateway, and timestamps for observed events. Wall-clock time helps correlation, but it can drift across systems. Do not build your whole conclusion on two clocks agreeing to the millisecond.

A good investigation asks these questions in order:

1. Who authorized the action, and which agent process made the request?
2. Did the gateway cross its dispatch boundary?
3. Which transport observations occurred after dispatch?
4. Did any trusted remote receipt arrive?
5. If not, what reconciliation query can identify the original logical operation?

This sequence prevents a common bad habit: searching remote logs first, finding a similar event, and declaring it the answer. Start with the intended action and its correlation material. Then evaluate whether the remote evidence matches that exact operation.

The audit record must resist quiet alteration. If an operator can change unknown_effect to succeeded without preserving the prior state and the basis for the update, the record becomes an assertion rather than evidence. Append-only records, hash chaining, and offline verification make post-incident editing harder to hide. They do not prove that every remote system told the truth, but they preserve what the gateway observed and when it learned more.

Sallyport keeps agent runs and individual actions in separate journals projected from one encrypted, hash-chained audit log, and sp audit verify can verify the chain offline over ciphertext. That design is useful here because a later reconciliation can be recorded as a new fact without erasing the original timeout or disconnect.

Do not put secret-bearing request bodies into an audit trail just to improve incident response. Store sanitized targets, request fingerprints, approved operation identifiers, response classifications, and the minimum remote references needed for reconciliation. A log that solves one investigation by leaking credentials creates the next incident itself.

Outcome names should drive safe agent behavior

An outcome taxonomy only earns its place when the agent runtime responds differently to each result. If every non-success leads to an immediate retry, detailed audit labels are decoration.

Use these operational rules:

• Retry automatically after no_effect only when the original intent is still authorized and current.
• Retry after rejected only after the agent changes the invalid input or a human resolves the stated conflict.
• Reconcile unknown_effect before retrying any state changing action.
• Treat partial_effect as a cleanup or continuation task, not a blank slate.
• Escalate when reconciliation cannot identify a single matching remote operation.

The final rule is where teams get impatient. They want the agent to keep moving. That instinct is reasonable for a read request, but it is reckless for actions that spend money, alter access, change production, or delete data. An unresolved action should remain visible until a person or a reliable remote query closes the evidence gap.

Your approval flow should not hide this distinction either. A human who approved an action approved an attempt, not unlimited retries after an unknown result. If the next attempt can create a second effect, show that it is a retry of an indeterminate action and require a fresh decision when the risk calls for one.

Build this model before you add more channels or more autonomous behavior. The first timeout after an agent changes something important is a poor time to discover that your audit log has only two outcomes: success and whatever the client happened to see.