Partial SSH command failures: stop agents repeating work

An AI agent must treat a failed SSH command as an unknown state transition, not as permission to run the same text again. A command can create a user, reload a service, and then fail while writing a final file. A blind retry may produce a duplicate account, overwrite a hand-edited setting, or turn a recoverable deployment into an outage.

The usual advice, "check the exit code," is necessary but incomplete. Exit status describes how a process ended. Recovery requires a record of what the process completed, what state the host now reports, and which operation the agent intended to perform. Put those facts in a durable command receipt, then make the agent reconcile before it acts again.

An SSH failure leaves three different unknowns

A failed SSH action can mean the remote shell failed, the connection failed, or the controller stopped waiting. Those cases need different handling, yet agents often flatten all three into "command failed."

Consider a remote deployment script that performs these operations in order:

1. It writes a new application archive to a release directory.
2. It changes a current symlink to that release.
3. It restarts the service.
4. It runs a health check and exits nonzero because the check sees a temporary dependency error.

The release is live even though the command returned failure. Repeating the script may be harmless if every operation tolerates repetition. More often, the script creates a new release directory, truncates a log, rotates a credential, or performs a migration. The exit code alone cannot settle what occurred.

The second case is worse: an SSH transport interruption. The local process may receive 255 after a network break while the remote shell continues. A controller timeout has the same problem. The controller knows only that it lacks a final answer. It does not know whether the target received the request, whether the shell started, or whether the process is still making changes.

That distinction changes the agent's next action:

• A confirmed remote exit with a receipt calls for recovery based on the failed checkpoint.
• A transport failure calls for observation before any mutation.
• A controller timeout calls for observation and, if needed, an explicit cancellation procedure rather than a duplicate request.

Do not label all three as retries. A retry is an operation with a known safe repetition rule. An unknown remote state needs reconciliation.

Exit status reports a process, not a transaction

An SSH exit code gives useful evidence, but it is not a database commit record. The OpenSSH ssh manual says that ssh exits with the exit status of the remote command, or with 255 if an error occurred. That wording draws a boundary many automation systems ignore: a remote exit status describes the command when SSH receives it, while 255 covers SSH's own error path.

A zero exit code also needs interpretation. In POSIX shell, the status of a simple sequential list normally comes from the last command. This script can report success after a meaningful failure:

install -m 0644 app.conf /etc/myapp/app.conf
systemctl restart myapp
logger -t deploy "deployment finished"

If install fails but systemctl restart and logger return zero, the script's final status is zero. The agent sees success and may falsely claim the configuration changed. A final echo done creates the same lie.

Pipelines add another failure path. In Bash, the Bash Reference Manual documents that a pipeline's status is the last command's status unless pipefail is enabled. This command can produce a zero result if extraction exits cleanly after receiving no useful data:

curl --fail --silent https://example.invalid/build.tar.gz | tar -xz -C /srv/myapp

Use an explicit interpreter and declare the behavior you need:

#!/usr/bin/env bash
set -Eeuo pipefail

curl --fail --silent --show-error "$archive_url" | tar -xz -C "$release_dir"

-e tells Bash to stop on many unhandled failures, -u rejects unset variables, and pipefail preserves failure from earlier pipeline members. The E option lets an ERR trap apply inside functions and command substitutions. These settings improve failure reporting. They do not make a sequence atomic.

That last point matters. set -e runs after an operation has returned an error. It cannot reverse a directory that a previous command created or restore a service that a previous command restarted. It also has exceptions that surprise people: commands tested by if, commands on the left side of && or ||, and several compound contexts do not always cause an exit. Write explicit checks around actions whose failure changes recovery.

Define the operation boundary before writing the command

An agent cannot recover a vague instruction such as "deploy the service". The remote command needs a named operation with a postcondition that an observer can test.

For a release change, the operation might be: "Set /srv/myapp/current to release 2025-04-18.3, then confirm that the active service reports that release." For a database change, it might be: "Apply migration add_invoice_index exactly once and confirm that its migration record exists." The command text is an implementation detail. The operation and its postcondition determine whether recovery can proceed.

Split an operation at irreversible or externally visible boundaries. A useful checkpoint is not every shell line. Record a checkpoint after a state change that would alter the next decision. In a deployment, that may include archive verified, release directory populated, symlink switched, service restarted, and health state observed.

Avoid the popular but wrong recommendation to make every remote command "idempotent" and then retry forever. Idempotence applies to a particular operation and a stated desired state. mkdir -p /srv/app may be repeatable. useradd deploy is only repeatable if the agent checks whether the existing account has the expected UID, group, home directory, and shell. ALTER TABLE may fail on a second execution, or worse, a loosely written migration may apply a related change twice.

A command can be safe to repeat while its surrounding workflow is unsafe. Restarting a service might be repeatable, but restarting it halfway through a configuration copy may expose an incomplete file. Put the state validation beside the action. Do not ask an agent to infer it from a generic rule.

For each operation, define four fields before granting access:

• An operation ID that remains the same during recovery.
• A desired postcondition that a read-only command can inspect.
• Checkpoints that describe completed state changes.
• A recovery action for each incomplete checkpoint.

The operation ID is not cosmetic. If the controller creates a fresh identifier on every attempt, the target cannot distinguish a continuation from a new request. That is how repeated migrations and duplicate provisioning happen.

Write a receipt before and after each state change

A durable receipt turns a partial failure into an inspectable event. Write it on the target host before the first mutation, update it after each meaningful checkpoint, and make updates atomic.

The following Bash script is deliberately plain. It deploys an already staged release by switching a symlink and restarting a system service. It does not claim to solve every deployment method. It shows the receipt mechanics an agent needs.

#!/usr/bin/env bash
set -Eeuo pipefail

operation_id=${1:?operation ID required}
release=${2:?release path required}
service=${3:?service name required}
state_dir=/var/lib/agent-ops
receipt="$state_dir/$operation_id.receipt"
tmp="$receipt.$$"

mkdir -p "$state_dir"
chmod 0700 "$state_dir"

write_receipt() {
  cat >"$tmp" <<EOF
operation_id=$operation_id
release=$release
service=$service
checkpoint=$1
updated_at=$(date -u +%Y-%m-%dT%H:%M:%SZ)
EOF
  chmod 0600 "$tmp"
  mv -f "$tmp" "$receipt"
}

fail() {
  status=$?
  write_receipt "failed:$status"
  exit "$status"
}
trap fail ERR

if [[ -f "$receipt" ]]; then
  . "$receipt"
  case "$checkpoint" in
    complete)
      printf 'operation already complete: %s\n' "$operation_id"
      exit 0
      ;;
    switched|restarted)
      printf 'operation requires reconciliation: %s\n' "$checkpoint" >&2
      exit 75
      ;;
  esac
fi

[[ -d "$release" ]]
write_receipt "release_verified"

ln -sfn "$release" /srv/myapp/current
write_receipt "switched"

systemctl restart "$service"
write_receipt "restarted"

active_target=$(readlink -f /srv/myapp/current)
[[ "$active_target" == "$release" ]]
systemctl is-active --quiet "$service"
write_receipt "complete"
printf 'operation complete: %s\n' "$operation_id"

The temporary file and mv matter. On a single filesystem, rename replaces the receipt in one operation, so a reader gets either the earlier complete receipt or the new complete receipt rather than half a file. Keep the state directory writable only by the account that owns the operation. If an untrusted user can edit receipts, the agent's recovery logic accepts fiction as evidence.

The ERR trap records the exit status when Bash handles a failure. It cannot run when the machine loses power, receives an uncatchable signal, or dies abruptly. That is why the script records progress after each completed state change instead of relying on a final trap alone.

Do not source arbitrary receipt formats as this small example does unless the directory has strict ownership and permissions. In production, prefer JSON parsed by a known parser, or a fixed line format that rejects unexpected fields. The example sources only a file it just created under a protected directory to keep the shell code readable.

The receipt should state observed facts, not optimistic intent. checkpoint=switched means the symlink command returned successfully. It does not mean the service loaded the new release. complete follows the explicit postcondition checks. That distinction prevents an agent from treating a written command as a completed operation.

Make the agent ask for reconciliation, not a new command

After a nonzero result, the agent should preserve the original operation ID and issue read-only checks first. It should not regenerate the deployment command with minor wording changes. New wording does not create a new state transition.

For the deployment receipt above, a reconciliation command can inspect both the durable record and the live postcondition:

operation_id='release-7f3b'
cat "/var/lib/agent-ops/$operation_id.receipt"
printf 'current='
readlink -f /srv/myapp/current
systemctl is-active myapp
systemctl show myapp --property=ActiveState --property=SubState --no-pager

The output has a shape the agent can parse without pretending that prose is evidence:

operation_id=release-7f3b
release=/srv/myapp/releases/2025-04-18.3
service=myapp
checkpoint=restarted
updated_at=2025-04-18T14:05:12Z
current=/srv/myapp/releases/2025-04-18.3
active
ActiveState=active
SubState=running

Here, the symlink points at the requested release and the service is active, but the receipt stopped at restarted. The remote process may have died after restarting the service and before it could write complete. The recovery procedure can run the postcondition checks again and, if they pass, write a completion receipt through a narrowly scoped recovery command. It should not restart the deployment from the top.

A useful controller protocol keeps request, result, and recovery separate. For example:

{
  "operation_id": "release-7f3b",
  "action": "deploy_release",
  "target": "app-01",
  "arguments": {
    "release": "/srv/myapp/releases/2025-04-18.3",
    "service": "myapp"
  },
  "mode": "reconcile"
}

The target should accept mode: reconcile only for read-only inspection or for a prewritten completion path that verifies the postcondition. Do not let the agent send an arbitrary shell string marked reconcile. That label has no security meaning when the command can mutate anything.

Exit code 75 in the sample is a deliberate temporary-failure signal. The exact number matters less than a documented contract: the agent sees that it must observe state, not submit an automatic retry. Reserve distinct controller outcomes for completed, reconcile_required, rejected_before_start, and transport_unknown. A single Boolean success field destroys the information recovery needs.

Timeouts and disconnects demand proof of remote state

A timeout is a local observation. The local client gave up waiting; it did not necessarily stop the remote command. Treating a timeout as cancellation is one of the fastest ways to duplicate a remote action.

A controller can reduce ambiguity by making operations single-flight. Before a mutation, the remote script creates an exclusive lock associated with its operation ID. A later attempt sees the lock and chooses between waiting, inspecting the process, or reporting that recovery needs a human decision.

For a simple host-level lock, flock is often enough:

exec 9>/var/lib/agent-ops/deploy.lock
if ! flock -n 9; then
  printf 'another deployment operation is active\n' >&2
  exit 75
fi

This protects only processes that honor the same lock. It does not protect against an administrator running a separate deployment procedure, nor does it survive every design error in your scripts. For a database migration, use the database's own advisory lock or migration lock where available. For an API action, use an idempotency token supported by that API. The lock must live beside the state it protects.

When the controller reconnects after a timeout, inspect in this order:

1. Read the receipt for the original operation ID.
2. Check whether the original process still runs, if the operation has a reliable process marker.
3. Test the operation's postcondition with read-only commands.
4. Choose an explicit recovery action or escalate when observations disagree.

Do not use process presence as your only signal. A process can exist while blocked on an external dependency, and an absent process says little about what it changed before exiting. The receipt and postcondition provide stronger evidence.

SSH multiplexing needs the same care. A master connection can hide an individual command's failure behind a shared transport, and a controller may confuse a closed channel with a failed operation. Capture the remote command's stdout, stderr, raw SSH exit status, start time, and end time as one action record. Preserve stderr even if the agent summarizes it. The raw data often shows whether Bash rejected an unset variable, a remote command exited 75, or SSH itself returned 255.

Some changes need compensation, not retries

Many operations cannot be made safely repeatable after the fact. Credential rotation, destructive cleanup, payment-like API calls, and schema migrations need a compensating procedure or an operator decision.

Take a credential rotation workflow. The command may create a new credential, update a service, verify access, and then revoke the old credential. If it fails after creation but before the service update, retrying can create yet another credential and leave several live secrets. The receipt should record the identifier of the newly created credential immediately. Recovery can then test which credential the service uses and decide whether to update, revoke, or retain the new one.

Do not put secret material in the receipt, standard output, or agent context. Store only a nonsecret identifier or fingerprint where that identifier itself does not grant access. The recovery process needs to know which object exists, not its private value.

Database migrations bring a different trap. A migration framework's history table can tell you that a named migration completed, but it may not describe an interrupted data backfill that ran outside the framework's transaction. Write migrations so the schema change, backfill progress, and completion marker have separate checks. If the database supports transactional DDL for your operation, use it, but do not assume every DDL statement or external side effect rolls back with the transaction.

For externally visible actions, prefer an API idempotency token to SSH-based guessing. If the remote host calls an API that accepts an idempotency key, persist that token in the receipt before the request. On recovery, query the API by the same token or resubmit with the same token according to the API's documented semantics. Generating a new token for every agent attempt defeats the feature.

The rule is blunt: if you cannot state how to determine whether an action happened, do not give an autonomous agent permission to retry it. Require a human to inspect the target or redesign the operation around a durable state record.

Shell scripts need a contract the agent can enforce

A script intended for agent use should expose a narrow, machine-readable contract. Agents are poor at reconstructing state from chatty logs, colored terminal output, and a mixture of warnings and success messages.

Use stable exit categories, one result object, and explicit operation IDs. For example, write a final JSON line only after the command knows its outcome:

{"operation_id":"release-7f3b","outcome":"reconcile_required","checkpoint":"switched","exit_code":75}

Keep ordinary diagnostic output on stderr and reserve stdout for the result record if your controller can enforce that convention. A shell command that prints banners, progress bars, and JSON all to stdout invites parser failures. If the command streams useful progress, write the durable receipt first and let the controller treat the final structured record as a convenience rather than the sole record.

Do not let the agent choose arbitrary checkpoint names, receipt paths, service names, or interpreters. Expose a reviewed command with constrained arguments. A command wrapper that accepts free-form shell after -- has merely moved the problem behind a cleaner label.

The command contract should also say which failures are safe to retry. For example, a package download can return a retryable network error before it changes the host. A symlink switch followed by a lost connection is not retryable until reconciliation verifies the link target. This classification belongs to the operation author, who understands the effects, not to a model guessing from stderr.

Use a test host and interrupt the script at every checkpoint. Send a termination signal during archive extraction, after the symlink switch, during restart, and after the final health check. Then run the reconciliation path and inspect whether it reaches the right decision. If you have never interrupted an operation on purpose, you do not know whether its retry behavior is safe.

Authorization and audit records must preserve the recovery story

An agent needs authority to perform a recovery read, but a read that leads to a new mutation should still follow the normal authorization boundary. Do not hide a second deployment inside a command called status.

Sallyport can keep the SSH credential outside the agent and records both the agent run and each action, which helps preserve who authorized a recovery attempt and what command it sent. Its audit trail does not replace the target-side receipt: the audit record can prove that an action request occurred, while the receipt and postcondition explain the target's resulting state.

Keep these records separate in your incident notes. Session identity answers which agent process held permission. The individual action record answers which command ran against which host and what it returned. The target receipt answers where that operation stopped. When a deployment goes wrong, collapsing those facts into one chat transcript wastes the evidence you need.

Grant per-call confirmation to operations with irreversible effects, especially when reconciliation may lead to credential deletion, schema repair, or cleanup. The extra approval is useful when the agent's evidence conflicts: for example, the receipt says a switch completed but the active service still reports an old release. That is a decision point, not a routine retry.

A good recovery design makes the conservative action cheap. Give the agent a read-only reconciliation command, a durable operation ID, and a defined escalation result. Then a disconnect produces an inspectable record instead of a second attempt that changes the system again.

Start by fixing the command that people already retry

Find the SSH command your team reruns after a timeout or a red deployment message. Add an operation ID, a protected receipt, and one read-only postcondition check before changing anything else.

Then interrupt it on purpose. If the recovery path cannot tell whether the first attempt switched the state, it is not ready for an autonomous agent. Rewrite the operation until the answer comes from the target host, not from confidence in an exit code.