SSH command success checks: prove remote work completed

An SSH command that exits with status 0 has completed one narrow job: the remote program told its shell that it succeeded. That is useful evidence, but it is not proof that a deployment reached the right release, a service stayed healthy, or a configuration change took effect.

I have seen plenty of automation declare victory because ssh host command returned zero, then discover that the command wrote to the wrong directory, queued work that later failed, or restarted a service that immediately crashed. The fix is not more hopeful logging. Treat a remote mutation as complete only after three separate facts agree: SSH reached the remote program, the program returned the expected result, and an independent read confirms the state you meant to create.

An SSH exit code reports one layer of the operation

An exit status tells you about process completion, not about the full operational outcome. A remote command sits inside a stack of assumptions: DNS and network connectivity, host identity, authentication, shell behavior, command parsing, dependencies, remote permissions, and the state you hoped to change.

The OpenSSH ssh(1) manual says that ssh exits with the status of the remote command, or with 255 if an error occurs. That distinction matters. A status of 255 usually means the SSH client could not establish or maintain the session as required. A status of 1, 2, or another nonzero value normally came from the remote command. A zero came from that command too.

That wording does not say that zero means "the production change is correct." It cannot. SSH has no idea whether /srv/app/current points at the release you intended, whether a daemon accepts requests after systemctl restart, or whether a database migration committed the rows your application needs.

POSIX defines a zero exit status as successful completion for a command. The useful part of that definition is its narrowness. It describes the command's contract. If your contract only says "run this shell line," then zero proves very little. Give the command a tighter contract, then test the state outside that command.

A practical way to think about failure is to separate it into three layers:

• SSH failure: the client could not connect, authenticate, verify the host, or complete the session.
• Command failure: the remote process detected an error and returned a nonzero status.
• Outcome failure: the process returned zero, but the intended remote state is absent, wrong, incomplete, or later reversed.

Teams often collapse the last two. That makes incident reports vague and retries dangerous. If you know which layer failed, you know whether to inspect credentials and connectivity, fix the command, or repair the remote state.

Success text is evidence only when you define it precisely

Expected output checks catch mistakes that exit status cannot see, but only if the output has an explicit contract. Searching a human-oriented transcript for words like success, complete, or deployed is weak evidence. Many tools print those words before a later command fails, and wrappers often print them after they merely submit asynchronous work.

Make the remote command emit a single record that carries the identity and state you expect. JSON is often convenient, but a fixed delimited line works if you control the values. The important part is that the script prints the record only after it has done the work it claims to report.

For example, suppose a release script switches a symlink to a release directory. This remote script emits the final target, rather than a vague progress message:

#!/bin/sh
set -eu

release="$1"
base=/srv/example/releases
link=/srv/example/current

[ -d "$base/$release" ]
ln -sfn "$base/$release" "$link"

actual=$(readlink "$link")
[ "$actual" = "$base/$release" ]
printf 'RELEASE_TARGET=%s\n' "$actual"

A caller can require an exact output line:

expected="RELEASE_TARGET=/srv/example/releases/2025.06.14"
output=$(ssh deploy@web-01 '/usr/local/sbin/activate-release 2025.06.14' 2>&1)
status=$?

if [ "$status" -ne 0 ]; then
  printf 'remote command failed, status=%s\n%s\n' "$status" "$output" >&2
  exit "$status"
fi

if ! printf '%s\n' "$output" | grep -Fxq "$expected"; then
  printf 'remote command returned unexpected output:\n%s\n' "$output" >&2
  exit 1
fi

grep -Fxq matters here. It checks one complete, fixed line. A loose expression such as grep deployed accepts junk, partial matches, and misleading progress logs. If your output includes dynamic fields, parse a structured document with a real parser rather than trying to make a regular expression understand nested data.

Do not turn output matching into a second copy of the whole command. It should answer one narrow question: did the remote program say it reached the named state? The follow-up read answers whether that statement remains true where it matters.

Remote writes need a read from the owner of the state

A follow-up read gives the strongest confirmation because it asks the component that owns the changed state. The right read depends on the change, and it often differs from the command that made the change.

For a symlink switch, readlink checks the filesystem. For a package rollout, query the installed package version. For a service restart, ask the service manager for active state and then perform a request against the service. For a database mutation, read the row or use the application's supported status endpoint. For a queued task, query the task record until it reaches a terminal state.

The common bad pattern looks like this:

ssh deploy@web-01 'deploy-release 2025.06.14 && systemctl restart example'

It can return zero while the service is "active" but points at an old release because the deployment script wrote to a different path. It can also return zero when the service manager accepts the restart but the process dies moments later. The shell saw two programs return zero. Users still get a broken service.

Use a read that makes the desired condition observable. A deploy check might look like this:

ssh deploy@web-01 '
  test "$(readlink /srv/example/current)" = /srv/example/releases/2025.06.14 &&
  systemctl is-active --quiet example &&
  curl --fail --silent --show-error http://127.0.0.1:8080/healthz
'

This is better than the first command because it checks three claims that the original command only assumed. It still does not prove that every external user can reach the service. If the change affects a public endpoint, run a suitable check from the network position that users actually occupy. Local loopback health checks catch a process failure, not a firewall or load balancer mistake.

Read-after-write checks should use an independent path where possible. Running a second function inside the same deployment script is better than nothing, but it can inherit the same mistaken variable, wrong host, or mocked dependency. A separate command that queries the filesystem, service manager, API, or database reduces that shared failure mode.

A command must state whether it is synchronous

Many remote commands return zero because they accepted work, not because the work finished. That is a correct result for a queue submission, a background job, a service reload request, or an orchestration API. It becomes a bug when the caller treats acceptance as completion.

Make the two contracts distinct in names and output. submit-backup can report a job ID and zero when the server accepted the request. wait-backup can return zero only after that specific job reports completion. Do not call both actions backup and hope the operator remembers which version runs on which host.

A remote script that starts work in the background needs particular care. This shell line returns success after it starts a process, even if that process fails immediately:

long-task >/var/log/long-task.log 2>&1 &
printf 'started\n'

The process's later exit code is unavailable to the SSH caller. Capture it somewhere durable and query it later, or keep the session open until the task reaches a meaningful state. If you detach the task, write an operation ID to a file or database and return that ID. The caller then polls or waits for the operation record.

A useful completion record has enough information to reconcile the work:

operation=4f2c1a status=accepted release=2025.06.14

The caller should not accept that record as a completed deployment. It should query operation=4f2c1a and require a terminal status such as completed, together with a check of the produced state. This may feel fussy for a simple script. It is much less fussy than guessing whether it is safe to run the script again after a timeout.

Timeouts deserve the same distinction. A local timeout tells you that the caller stopped waiting. It does not tell you that the remote command stopped. The network might have dropped after the remote host committed a change. Before retrying, inspect the remote state or query the operation ID. Retrying a release activation may be harmless if it is idempotent. Retrying a payment, a secret rotation, or an email send can compound the damage.

Shell composition hides failures unless you write for them

Remote shell syntax can convert a real failure into a successful final exit status. This is one of the oldest ways an SSH run looks clean while leaving a damaged machine behind.

Consider this command:

ssh ops@db-01 'backup-db; upload-backup; prune-old-backups'

The remote shell returns the status of prune-old-backups, its final command. If the backup fails but pruning succeeds, the overall SSH command returns zero. The transcript may contain an error near the top, while a pipeline that only reads the status marks the run successful.

Use set -e in a script you control, or connect dependent commands with && when a compact command remains readable:

ssh ops@db-01 'backup-db && upload-backup && prune-old-backups'

set -e is not magic. Shells have exceptions around conditionals, command substitutions, and some compound constructs. Do not write a long remote one-liner and assume that a single option makes every failure visible. Put nontrivial work in a remote script, give each operation a clear success condition, and test its behavior under failure.

Pipelines introduce another familiar trap. In many POSIX shells, this succeeds if the final program succeeds, even if an earlier producer fails:

collect-metrics | format-report > /var/tmp/report.txt

Some shells offer set -o pipefail, but /bin/sh may not support it. If the remote environment needs POSIX portability, avoid using a pipeline as the only error boundary. Write intermediate data to a temporary file, check the producer status, then consume it. Or execute the script under a shell whose pipefail behavior you explicitly require.

Also avoid ending a remote command with diagnostic output that can succeed after the real work fails:

apply-config
printf 'finished\n'

Without set -e or an explicit check, printf becomes the exit status. This mistake appears in hand-written incident commands because somebody wants a friendly final message. Print the message only after a condition check, or let the failed command terminate the script.

Quoting mistakes can make you verify the wrong machine

A local shell expands unquoted variables before SSH sends the command. That can make a command run against the wrong release, compare local output instead of remote output, or expose values in the local process list and logs.

This is wrong when you intend the remote host to evaluate $release:

ssh deploy@web-01 "test \"$(readlink /srv/example/current)\" = \"$release\""

The local shell evaluates $(readlink ...) before it starts SSH. You have now compared your workstation's /srv/example/current, if it exists, to a local variable, then sent the resulting text to the remote shell. The exit status may be zero. The check did not inspect the target host.

Keep the remote program in single quotes when it contains shell syntax that must run remotely. Pass untrusted or dynamic values as positional parameters rather than assembling shell text. For example:

release='2025.06.14'
ssh deploy@web-01 'sh -s -- "$1"' sh "$release" <<'REMOTE'
set -eu
release=$1
target=$(readlink /srv/example/current)
[ "$target" = "/srv/example/releases/$release" ]
printf 'verified=%s\n' "$target"
REMOTE

The quoted heredoc delimiter prevents the local shell from expanding the script body. The release value travels as a shell argument, where the remote shell can quote it correctly. This does not turn arbitrary input into a safe release name by itself. Validate allowed characters and expected formats before using user-controlled values in paths, commands, or database queries.

For automation, prefer a remote script with parameters over a growing string of nested quotes. Quoting failures are hard to spot in code review because the command can look plausible at a glance. They become much easier to diagnose when logs show the exact remote script version, parameter values that are safe to log, and the verification command's own output.

A useful SSH contract has three separate results

Treat every consequential SSH action as a small protocol with separate fields for transport, command result, and observed state. A caller needs all three to decide whether it can proceed, retry, or ask for help.

This Bash function demonstrates the shape. It captures stderr with stdout so a failed operation leaves diagnostic evidence, while it reports SSH transport failure separately from an unexpected success response.

run_remote_check() {
  local host=$1
  local expected=$2
  shift 2

  local output status
  output=$(ssh "$host" "$@" 2>&1)
  status=$?

  if [ "$status" -eq 255 ]; then
    printf 'ssh_transport=failed host=%s\n%s\n' "$host" "$output" >&2
    return 255
  fi

  if [ "$status" -ne 0 ]; then
    printf 'remote_command=failed host=%s status=%s\n%s\n' \
      "$host" "$status" "$output" >&2
    return "$status"
  fi

  if ! printf '%s\n' "$output" | grep -Fxq "$expected"; then
    printf 'remote_result=unexpected host=%s expected=%s\n%s\n' \
      "$host" "$expected" "$output" >&2
    return 1
  fi

  printf 'remote_result=confirmed host=%s\n' "$host"
}

Call it with a command that emits only a contract record after its own internal checks:

run_remote_check \
  deploy@web-01 \
  'RELEASE_TARGET=/srv/example/releases/2025.06.14' \
  '/usr/local/sbin/activate-release 2025.06.14'

Then run a follow-up read as a separate action. Keep it separate in logs and status reporting. If activation succeeds but the service check fails, operators should see that exact boundary. A single opaque deploy failed result forces them to re-run commands just to learn what already happened.

For commands that return structured data, return a small JSON object and parse it with a JSON parser. Do not use grep against JSON unless the output is deliberately a one-line sentinel and you do not need to interpret its fields. String matching against arbitrary JSON breaks when whitespace, ordering, or escaped content changes.

The contract should also name the target object. status=ok alone does not distinguish release 2025.06.14 from yesterday's release. Include the deployment identifier, hostname where relevant, object version, or operation ID. That small detail prevents a surprisingly common false positive: a check verifies that something healthy exists, but not that it is the thing the run just changed.

Observability must preserve the verification result

A log that only records the remote command and exit status leaves the most important question unanswered: did the caller independently observe the intended state? Record the verification action and its result beside the mutation.

For a deployment, a useful record might include the target host, release identifier, SSH status, command status, exact contract record, follow-up command status, and a short result from the health or state read. Do not log secrets, full authorization headers, or private command arguments simply because they help debugging. Design the command interface so that useful evidence is safe to retain.

Sallyport keeps an Activity journal for individual actions and a Sessions journal for agent runs, both projected from an encrypted, hash-chained audit log. That gives an operator a place to distinguish an SSH action that ran from a verification action that confirmed a result, instead of treating one successful call as the whole story.

Tamper evidence helps answer whether a recorded action changed after the fact. It does not convert a weak command contract into proof of a good outcome. sp audit verify can verify the log chain offline over ciphertext, but the action design still needs an explicit read-back check.

Keep the verification close enough to the write that another actor cannot silently replace the desired state in the gap. You cannot always eliminate races on a shared system. You can reduce them by using immutable release IDs, operation IDs, version checks, and APIs that support conditional updates. If the state can change again, record the observed version or timestamp and make later automation check it before acting.

Retries need reconciliation before repetition

A timeout, dropped connection, or interrupted CI job creates an unknown outcome. The remote host may have completed the change, may still be running it, or may have failed halfway through. The caller does not know merely because it stopped receiving output.

Do not solve that uncertainty with a blind retry. First run a read-only reconciliation command that classifies the remote state. For a release activation, inspect the current symlink and service health. For a migration, query a migration table. For a created resource, query by a caller-generated operation ID. For a secret rotation, check which version the consumers actually use before generating another one.

A good reconciliation command returns one of a small set of explicit outcomes:

• completed: the target state matches the requested operation.
• running: the operation still owns work and the caller should wait.
• absent: no evidence of the operation exists, so a retry may be appropriate.
• conflict: a different state exists and a human or higher-level controller must decide.

Do not force every result into success or failure. running and conflict are useful results. A system that calls them failures often retries work that should have been left alone.

Idempotency reduces the cost of legitimate retries, but people misuse the word. A command is idempotent when repeating the same request leaves the desired state unchanged after the first successful application. ln -sfn can be idempotent for a specific link target. "Create a new backup with the current time" is not. "Send an email" is not, unless the downstream system deduplicates with a stable message identifier.

Build the read-back check before you add retries. If you cannot state how to identify a completed operation, you cannot safely automate repetition after uncertainty. That is where SSH automation stops being a shell convenience and starts needing an operation model.

The first repair is to audit your green runs

Start with remote commands that mutate production state and currently report only a green exit code. For each one, write down the remote object that must change, the exact state that counts as success, the component that can read that state, and what a timeout leaves unknown.

Then change the command interface so it emits an exact result record after its own checks. Add a separate read-back action. Preserve both results in the run record. You will find commands that were never synchronous, scripts whose final printf hid an earlier failure, and deployment checks that queried the machine that launched the command rather than the machine that received it.

A zero exit status still has a place. It is the first gate, not the final verdict. Treating it that way produces automation that can explain what happened when the easy green signal turns out to be wrong.