# Power loss during an audited agent action leaves unknown outcomes

A power loss during an audited agent action does not create a single failure. It creates an evidence problem. The agent, the local gateway, the operating system, the network, and the remote service can each stop at a different point. If you collapse that mess into "failed" because the agent did not get a response, you will eventually retry an action that already happened.

That mistake is easy to make because logs read like a story. An investigator sees an approved action, an outbound request, then a gap. The gap feels like a conclusion. It is not. It is an interval where several materially different outcomes remain possible, and the right response depends on which system owns the fact you need.

This is where an audit trail earns its keep. It should preserve what the local system knows, prove that retained records have not been quietly edited, and make uncertainty visible rather than papering it over. It cannot turn a lost response into proof that a database write, a payment, a deployment, or a remote command did not occur.

## A timeout is an evidence gap, not a confirmed failure

A timeout tells you that one participant did not observe a usable response before its deadline. It does not tell you whether the destination received the request, whether it began work, whether it committed work, or whether its response vanished on the return path.

Keep these outcomes separate in incident notes and in any user interface that reports agent actions:

- **Confirmed failure:** the destination or local executor returned durable evidence that it rejected or rolled back the requested operation.
- **Confirmed success:** the system that owns the changed state returned a receipt, and a follow-up read verifies the expected state.
- **Unknown outcome:** the action may have happened, but the available evidence cannot establish it either way.
- **Not attempted:** the gateway denied the action before it handed work to an executor or transport.

"Unknown" is not a timid spelling of "probably failed." It is an operational state with a different playbook. You pause automatic retries, preserve evidence, query the authoritative system, and only then decide whether a compensating action or a retry is safe.

People often add a fourth label, "partially successful," too early. Use it only when you can name the completed sub-operation and the incomplete sub-operation. A deployment pipeline that created an artifact but never promoted it has a partial result if both facts are verified. A request that disappeared after the client wrote bytes to a socket has an unknown result, even if the request looked simple.

RFC 9110 makes the same distinction in protocol terms. It permits automatic replay of idempotent methods after a communication failure because repeating the intended operation has the same intended effect. It warns against automatically retrying a non-idempotent request unless the client can establish that the original request was not applied or knows the application makes repeats safe. That is a semantic rule, not a transport trick.

An agent action gateway should report the local facts precisely. "Request sent; completion not observed" is useful. "Action failed" is a claim the local process may not be entitled to make.

## Power loss splits one action into durability boundaries

One action can cross several boundaries before anyone sees a final result. Write them down before you test, because a test that only crashes one process will otherwise produce misleading confidence.

A typical HTTP action has at least these boundaries:

1. The gateway accepts an authorized intent and records it locally.
2. The gateway constructs and starts transmitting the authenticated request.
3. The remote service receives enough of the request to begin processing it.
4. The remote service commits its side effect and creates a receipt.
5. The gateway receives the response and records the observed outcome.

An SSH action follows a related path, except the remote host can start a command before the local client learns its exit status. The host may also fork work that survives the session. A successful TCP or SSH connection says very little about where a command stopped.

Local writes have their own boundaries. A process can append an audit event, the operating system can accept that append into cache, the filesystem can order metadata and data, and storage can make the bytes survive loss of power. Those are not interchangeable events.

The POSIX `fsync()` specification says the call requests transfer of an open file's queued data to its storage device and does not return until the operation completes or reports an error. Its rationale also warns that the actual assurance depends on the implementation and storage configuration. That caveat matters: an application cannot infer power-loss safety merely because a write call returned successfully.

For investigation, assign every record an evidence class instead of treating every timestamp as equally durable:

| Evidence class | What it supports | What it cannot support |
| --- | --- | --- |
| Intent accepted | The gateway agreed to attempt a specific action | That any request left the machine |
| Dispatch started | The gateway began local execution or transport work | That the destination received complete input |
| Remote receipt | The destination claims it accepted or committed an operation | That the local system stored the receipt before crashing |
| Local completion record | The gateway observed and recorded a result | That the remote state remains unchanged after later work |
| Reconciliation read | A later query observed the destination's state | The exact moment that state changed, unless the destination records it |

This table is deliberately strict. A process log line that says "sending request" is dispatch evidence. It is not delivery evidence. A response body in memory is observation evidence. It is not durable local evidence until your persistence path survives the failure model you claim to test.

## Give every side effect an operation identity before testing failures

An investigator cannot reconcile an interrupted action if the remote system has no stable way to identify it. Add an operation identity before you write a chaos test, not after the first duplicate appears in production.

Use two identifiers when you can:

- A locally generated action ID that names the gateway's single attempt.
- A destination-recognized operation ID, idempotency key, request token, deployment ID, or transaction reference.

They may contain the same random value, but do not assume they mean the same thing. Your action ID identifies a local audit record. The remote identifier only becomes useful when the destination stores it with the side effect and offers a way to retrieve the resulting state.

For an API that supports idempotency keys, make the identity explicit in the request and record a digest of the meaningful payload. The digest lets you detect an operator accidentally attempting to reuse an old key with a changed request.

```http
POST /v1/releases HTTP/1.1
Host: deploy.example.internal
Idempotency-Key: 8b4dbdb6-61e1-49ea-a5b5-9ae225eb2af1
Content-Type: application/json

{
  "operation_id": "8b4dbdb6-61e1-49ea-a5b5-9ae225eb2af1",
  "service": "catalog",
  "artifact": "sha256:3c1f...",
  "environment": "production"
}
```

Do not log the authorization header, bearer token, or a full body that may contain secrets. Record the method, destination identity, safe request metadata, action ID, operation ID, and payload digest. You need enough evidence to compare attempts without creating a second credential leak in your audit system.

A useful intent record might look like this:

```json
{
  "event": "intent_accepted",
  "action_id": "act_01JX8F3Z6Z",
  "operation_id": "8b4dbdb6-61e1-49ea-a5b5-9ae225eb2af1",
  "channel": "http",
  "destination": "deploy.example.internal",
  "method": "POST",
  "path": "/v1/releases",
  "payload_sha256": "3c1f...",
  "authorization": "approved_for_session"
}
```

That record prevents a common investigation failure: someone compares a later retry to the first action by timestamp and endpoint alone, overlooks a changed artifact or account, and declares two different operations equivalent.

For SSH, put the operation ID where the remote host can retain it. A shell command can include it in its structured logs, a deployment script can write it into a release record, or a remote wrapper can reject a duplicate ID. Do not depend on the local SSH client transcript as your only proof.

```sh
ssh deploy@host.example.internal \
  '/usr/local/bin/release --operation-id 8b4dbdb6-61e1-49ea-a5b5-9ae225eb2af1 --artifact sha256:3c1f...'
```

If that command starts a background worker, have the worker persist the operation ID before it changes anything. Otherwise a disconnect leaves you with a host that may still be working and no reliable way to find the work.

## Simulate interruption at the boundaries that change your decision

A useful fault test kills the process at points that produce different investigator decisions. Randomly terminating a client in a loop finds bugs, but it does not teach anyone what the log means.

Build a test destination that can pause at controlled phases and can answer a reconciliation query by operation ID. It does not need to be elaborate. It needs to expose the difference between request received, side effect committed, and response sent.

Start with five cases:

1. **Crash before intent persistence.** The action should be absent from the durable local trail and absent from the destination. If the destination changed, your order of operations is wrong or another component issued the action.
2. **Crash after intent persistence but before dispatch.** The audit trail should show an accepted action with no dispatch record. This should classify as not attempted only if the gateway can prove it never handed the action to a transport or executor.
3. **Crash during request transmission.** The destination may see nothing, a partial request, or a complete request. Classify this as unknown unless the destination provides a definite rejection or lookup result.
4. **Crash after remote commit but before local completion persistence.** The destination should show the operation as completed while the local trail lacks a completion event. This is the test that exposes dangerous retry logic.
5. **Crash after local completion persistence but before the agent receives the response.** The local audit trail has the answer even though the agent thinks it timed out. A new agent session must query the action record instead of blindly issuing a second operation.

A local harness can coordinate the gateway and a test service with named pause files. The exact implementation will differ, but the test contract should not. The harness must tell you which boundary it reached before you interrupt it, then preserve the destination state for reconciliation.

```sh
# Terminal 1: start the test destination with controlled pauses.
./test-api --pause-after=commit --state-file ./tmp/remote-state.json

# Terminal 2: run one authorized action with a known operation ID.
./gateway-test invoke \
  --operation-id 8b4dbdb6-61e1-49ea-a5b5-9ae225eb2af1 \
  --pause-file ./tmp/client-dispatch.pause

# When the destination reports "committed", terminate the local process.
kill -9 "$(pgrep -f 'gateway-test invoke')"

# Terminal 3: inspect the destination without issuing another mutation.
./test-api lookup 8b4dbdb6-61e1-49ea-a5b5-9ae225eb2af1
```

`kill -9` tests a process crash. It does not test storage behavior under a hard power cut, and it does not cancel an action the remote service already accepted. That limitation is a feature when you are learning outcome classification. It forces the team to stop treating the death of the caller as proof about the callee.

For actual power-loss testing, use a disposable machine or virtualized environment where you can reproduce abrupt shutdowns safely. Do not test by yanking power from a workstation that contains the only copy of a production vault, audit material, or working tree. Save the exact build, storage mode, test inputs, and time source for each run. A test result without that context is an anecdote.

## An audit trail should state what it knows and preserve what it does not

A good audit record reads like a set of claims with clear scope. It does not pretend to contain omniscient truth about systems it cannot observe.

For an interrupted action, record separate events rather than overwriting one mutable status field. An append-only sequence can represent the truth without inventing a final answer:

```text
intent_accepted     action=act_01JX8F3Z6Z op=8b4d... principal=agent-process
transport_started   action=act_01JX8F3Z6Z channel=http
outcome_unobserved  action=act_01JX8F3Z6Z reason=local-process-terminated
reconciled_success  action=act_01JX8F3Z6Z receipt=rel_4921 source=remote-api
```

The third line should not say `remote_failed`. It only says the gateway failed to observe a terminal outcome. The fourth line adds a later claim from the source that owns the release state.

This distinction also makes tamper evidence more useful. A hash chain can expose changes to retained audit material, but it cannot recreate an event that never reached durable storage. If a machine loses power between an outbound call and a log append, an intact chain may end cleanly before the action's result. That is not necessarily a broken chain. It is a gap that needs reconciliation.

Sallyport projects its Sessions and Activity journals from one write-blind encrypted, hash-chained audit log, and `sp audit verify` checks that chain offline over ciphertext. That gives an investigator a strong way to test whether the retained local history has been altered, while leaving the remote outcome to remote evidence where necessary.

Keep authorization evidence separate from outcome evidence. An approval shows that a human or configured control allowed an action to proceed. It does not show the action completed. Mixing those claims is how an approved but interrupted production change becomes falsely reported as a completed change.

The same warning applies to timestamps. Wall-clock timestamps help investigators correlate systems, but they do not establish a global order when machines disagree or buffers delay writes. If order matters, record sequence numbers within each journal, retain remote receipt IDs, and capture the service's own timestamps during reconciliation.

## Network calls need remote reconciliation, not optimism

When an HTTP call loses its response, the destination is usually the authority on whether the requested state changed. Query it before you retry, and make the query specific enough to distinguish this operation from similar work.

The safest reconciliation order is:

1. Look up the operation ID or idempotency key at the destination.
2. If the destination returns a completed operation, compare its resource IDs and payload digest with the original intent.
3. If it returns a recorded rejection, retain that response as failure evidence.
4. If it has no record, check whether the API documents delayed processing, asynchronous queues, or eventual record creation before you issue a retry.
5. Retry only when the endpoint's semantics and the evidence make repetition safe.

Do not mistake a `GET` request for a harmless verification merely because it uses a safe method. Some APIs hide work behind a read endpoint, and some response caches lag behind the write path. Verify the provider's contract and, when possible, retrieve the specific created resource or operation record rather than searching a broad list by time.

HTTP method labels help but they do not settle application behavior. A `PUT` request can be idempotent at the HTTP layer while a server emits duplicate emails, charges usage, or triggers a deployment hook on every receipt. RFC 9110 explicitly notes that idempotence applies to the requested effect, while a server can still retain separate logs or produce other side effects. That is why the API owner's documented operation identity matters more than a verb in a client library.

A popular but bad recommendation is "retry every timeout twice." It sounds practical because many timeouts are transient. It converts a transient transport issue into duplicate money movement, duplicate account creation, or two production releases when the endpoint is not safely repeatable. A retry policy must name the operation type, the idempotency mechanism, the maximum delay, and the evidence that permits the retry.

For a remote service that does not offer a lookup or idempotency support, the honest answer may be that you cannot automatically resolve the outcome. Build a compensating process around that limitation, such as a human review queue with the exact request fingerprint and a read-only account that can inspect the affected state.

## SSH commands require proof from the remote host

An interrupted SSH session leaves more ambiguity than many API calls because the command may execute on the far side after the client has disappeared. A lost exit status is not a failed command. It is a missing observation.

Avoid one large shell line that does several mutations with no checkpoints. Break remote work into operations that have their own IDs and durable state. A deployment wrapper, for example, can record `received`, `validated`, `applied`, and `completed` for one operation ID, then expose a read-only status command.

```sh
/usr/local/bin/release-status \
  --operation-id 8b4dbdb6-61e1-49ea-a5b5-9ae225eb2af1
```

The wrapper should write its status before it starts a non-repeatable action, not after. If it allocates a cloud resource, publishes a package, or switches traffic, it should retain the provider receipt with the same operation ID. If it cannot do that, put the operation behind a remote queue that can.

Be suspicious of shell cleanup traps as recovery evidence. A local trap does not run after abrupt power loss. A remote trap may not run after a forced kill, and it may run while a child process continues. Cleanup logic can reduce mess, but it does not prove the final state.

Use a remote marker only when the marker and the side effect have a meaningful relationship. A line appended to `/tmp/action-done` does not prove a database migration committed. A migration table entry written in the same transaction is better evidence. A deployment record created by the deployment controller is better still.

Sallyport routes SSH through its bundled stateless `sp-ssh` helper, but the same rule applies: the local action record can establish what the gateway attempted and observed. Only the remote host, or the system changed by the command, can settle an unobserved remote outcome.

## Walk an investigator through one interrupted release

Suppose an agent requests a production release through an HTTP endpoint. The gateway records an authorized intent with action ID `act_01JX8F3Z6Z`, operation ID `8b4d...`, artifact digest `3c1f...`, and a session approval. It begins the request. The release service commits the release and assigns receipt `rel_4921`. Before the response reaches the gateway, the laptop loses power.

After restart, the agent's transcript says the call timed out. The local audit trail ends at `transport_started`. Someone who treats that line as proof of failure submits the same release again, this time with a fresh operation ID. The service creates a second release. If the release endpoint activates immediately, the second request may be merely noisy. If it triggers an irreversible migration, it may be expensive.

The correct investigation starts by freezing automatic retry for `act_01JX8F3Z6Z`. Verify the local audit chain. Record the last retained event, the local process termination evidence, the destination, payload digest, and operation ID. Then query the release service for `8b4d...`.

There are three useful results:

- The service returns `rel_4921` with artifact digest `3c1f...`. Classify the original action as confirmed success after reconciliation. The missing local completion remains an audit gap, not a reason to replay the release.
- The service returns a durable rejection tied to `8b4d...`. Classify it as confirmed failure. Preserve the rejection reason before deciding whether a corrected request is appropriate.
- The service returns no record. Check whether it queues requests before creating operation records and whether its query path has lag. If it cannot rule out delayed work, retain unknown outcome and escalate rather than issuing a blind retry.

Notice what does not decide the case: the agent's timeout, the gateway's process exit, or the fact that a human approved the action. Those facts matter, but none of them owns release state.

This walkthrough also exposes a design requirement. If the destination cannot search by operation ID, the gateway must either avoid unattended non-repeatable actions against that destination or require a human reconciliation path. Audit quality cannot compensate for an API that offers no durable way to identify its own side effects.

## Retry rules must be narrow enough to survive a bad day

A retry policy should say more than "retry on network errors." Write it as a decision table that an implementer and an incident responder can both follow.

| Action type | Evidence after interruption | Automatic retry? | Required safeguard |
| --- | --- | --- | --- |
| Read-only query | No response | Usually yes | Bounded retries and destination timeout limits |
| Idempotent update | No response | Yes, if the destination honors stable identity | Same resource identity and same request content |
| Create or trigger action | No response | Only after reconciliation or with documented idempotency | Destination lookup by operation ID |
| Money movement or destructive change | No response | No | Human review and authoritative state check |
| SSH command with side effects | Lost session | No | Remote operation status and duplicate rejection |

Do not let agents invent a new operation ID during recovery. That is a subtle duplicate generator. If a retry is valid, reuse the same destination-recognized identity and prove that the retry payload matches the original intent. If the desired change has changed, it is a new action that deserves a new authorization decision.

Set a terminal state for unresolved actions. "Unknown, awaiting reconciliation" is better than a job that retries forever because no one wrote the state machine to admit uncertainty. Give that state an owner, a deadline, and an escalation path. Otherwise an old ambiguous action becomes a surprise when someone revisits it weeks later with incomplete remote logs.

## Preserve evidence before systems age out of the answer

The first hour after an interruption is when remote services still have request traces, queue entries, and fresh operator context. Capture the facts before log retention, cache eviction, or another deployment removes them.

Preserve the local audit material in its original form and verify it before you copy excerpts into a ticket. Record the exact action ID, operation ID, destination, payload digest, authorization record, last local event, and local restart time. Then collect the remote operation record, resource revision, provider request ID, and the service time source used for the answer.

Do not repair the history by adding a fake completion event. Add a later reconciliation event that names its source and evidence. Investigators need to see the original boundary, because the boundary tells them what the system knew at the time.

The durable lesson is uncomfortable but simple: an audited action can be well authorized, carefully logged, and still have an unknown remote result after power loss. Build action identities, remote receipts, and reconciliation paths before you let an autonomous agent perform work that cannot safely happen twice. When the lights go out at the wrong moment, those choices decide whether your team investigates a gap or creates a second incident.