SSH disconnect unknown remote state verification

An SSH disconnect does not mean the remote command failed. It means the client lost the evidence needed to classify what happened. That distinction is annoying when you ran uname, and expensive when you ran a database migration, a release, a credential rotation, or a command that calls an external API.

The fix is not a larger timeout. The fix is a remote verification sequence with a durable run identity, explicit state transitions, and an effect-specific proof of completion. Once that exists, a reconnect changes the question from "Should I run it again?" to "What does this run record say, and what did it change?"

A dropped connection leaves three honest answers

After an SSH client reports a reset, timeout, broken pipe, or abrupt EOF, the command has one of three states: it never started, it started and remains in progress, or it completed. The client-side exit code does not reliably distinguish them.

There are several boundaries between your shell and the remote program:

• Your local shell starts ssh.
• The client sends an SSH channel request and command bytes.
• The server accepts the request and starts a remote shell or program.
• That program performs its actual work.
• The program exits, and sshd sends output and an exit status back.

A network break after any of those boundaries can produce an error locally. If the break occurs before the remote program starts, nothing happened. If it occurs after the remote program commits a change but before the exit status returns, the change happened and your client still reports failure.

This is why a local message such as Connection reset by peer is transport evidence, not business evidence. It says that the client could not finish the SSH conversation. It does not say whether the remote operation ran.

OpenSSH's own configuration manual makes a related point about inactive channels: closing a session does not guarantee that associated shell processes have stopped. A channel timeout is therefore not a job-control mechanism.

The habit that causes damage is treating an ambiguous result as a failed operation. That habit is understandable because most command-line tools train us to read a nonzero exit status as "do it again." SSH transport failure breaks that shortcut.

Command delivery and command completion are different claims

A remote command has at least four claims worth proving: submission, start, completion, and effect. Teams often log one of them and assume they have all four.

Submission means the client attempted to send the command. Your local terminal knows this, but it is the weakest claim. Start means the remote wrapper created durable evidence before doing work. Completion means the wrapper recorded a final outcome. Effect means the intended remote or external state now matches the requested result.

A process listing proves less than people think. Seeing a PID might tell you that a similarly named process exists now. It does not prove that it belongs to your request, that it has not already committed the important part, or that a later retry will be safe. After a process exits, PID reuse makes stale records even less useful.

Exit codes have a similar limit. POSIX defines wait as a way for a shell to obtain the status of a child process it knows. That relationship exists inside the remote shell. Once the SSH connection disappears, the local shell has lost its path to that status. A later SSH session cannot recreate it from wait; it must read a record that the first run persisted.

Keep these statements separate in runbooks and automation output:

1. "The client could not confirm completion."
2. "Run r-20260722-1842-a91f began on the remote host."
3. "That run recorded exit status 0."
4. "The deployment marker reports release 2026.07.22.3."

The fourth statement may be the only one that answers the real operational question. A file-copy command needs a checksum or a final pathname with the expected content. A migration needs its schema version or migration ledger entry. A request to a payment or ticketing API needs an idempotency record at that API, not merely a local log line.

Put the run identity on the remote host before work starts

A durable run ID converts vague reconnection work into a lookup. Generate it before invoking SSH, pass it to the remote wrapper, and make every artifact live under a path derived from it.

Do not use only a timestamp. Two agents can start in the same second, clocks drift, and timestamps make poor opaque identifiers. Combine a timestamp with random data, or use a UUID generator available in your environment. The ID must be supplied again for verification and must appear in every meaningful log entry.

This shell fragment creates a run directory, writes the requested operation, records a start marker, and preserves both output streams. It expects a command to be passed after --. Keep the wrapper in a controlled location such as /usr/local/sbin/run-recorded, not copied ad hoc into every command string.

#!/bin/sh
set -eu

run_id=$1
shift
[ "$1" = "--" ]
shift

base=/var/lib/recorded-runs
run_dir="$base/$run_id"

case "$run_id" in
  *[!A-Za-z0-9._-]*|'')
    printf '%s\n' "invalid run id" >&2
    exit 64
    ;;
esac

if ! mkdir "$run_dir" 2>/dev/null; then
  printf '%s\n' "run already exists: $run_id" >&2
  exit 75
fi

umask 077
printf '%s\n' "$*" > "$run_dir/request"
date -u +%Y-%m-%dT%H:%M:%SZ > "$run_dir/started_at"
printf '%s\n' "started" > "$run_dir/state"
printf '%s\n' "$$" > "$run_dir/pid"

set +e
"$@" >"$run_dir/stdout" 2>"$run_dir/stderr"
status=$?
set -e

printf '%s\n' "$status" > "$run_dir/exit_status"
date -u +%Y-%m-%dT%H:%M:%SZ > "$run_dir/finished_at"
printf '%s\n' "finished" > "$run_dir/state"
exit "$status"

The mkdir call is doing more than housekeeping. Directory creation fails if that run ID already exists, so it acts as a simple create-once claim. That prevents two invocations with the same ID from silently doing the work twice. It does not solve concurrent work that uses different IDs, which needs a separate lock or an application-level constraint.

The order matters. The wrapper writes started_at, state, and pid before executing the payload. It records exit_status before switching state to finished. A verifier that sees finished without an exit status should treat the record as damaged, not successful. A verifier that sees a run directory but no started_at should treat it as an incomplete setup failure.

Do not write finished through a shell trap and call it done. A sudden host failure, storage failure, forced kill, or filesystem problem can prevent traps from running. A final marker is evidence when present, not proof that absence means the payload did not finish.

Verify a run in an order that cannot lie to you

Reconnect with a read-only status command first. Do not reconnect by launching the payload again with the same arguments and hoping the answer becomes obvious.

A useful verifier should classify the record into absent, running, finished, or damaged. This example uses the directory format above and prints facts a human or an agent can evaluate.

#!/bin/sh
set -eu

run_id=$1
run_dir="/var/lib/recorded-runs/$run_id"

if [ ! -d "$run_dir" ]; then
  printf '%s\n' 'state=absent'
  exit 0
fi

if [ ! -f "$run_dir/started_at" ]; then
  printf '%s\n' 'state=damaged reason=missing-start-marker'
  exit 2
fi

if [ -f "$run_dir/finished_at" ] && [ -f "$run_dir/exit_status" ]; then
  printf '%s\n' 'state=finished'
  printf 'exit_status=%s\n' "$(cat "$run_dir/exit_status")"
  printf 'started_at=%s\n' "$(cat "$run_dir/started_at")"
  printf 'finished_at=%s\n' "$(cat "$run_dir/finished_at")"
  exit 0
fi

if [ -f "$run_dir/pid" ]; then
  pid=$(cat "$run_dir/pid")
  if kill -0 "$pid" 2>/dev/null; then
    printf 'state=running pid=%s\n' "$pid"
    exit 0
  fi
fi

printf '%s\n' 'state=damaged reason=no-finish-record-and-pid-not-live'
exit 2

Run it as a new SSH command:

ssh ops@host /usr/local/sbin/check-recorded-run r-20260722-1842-a91f

Its output should look like one of these:

state=absent

state=running pid=48192

state=finished
exit_status=0
started_at=2026-07-22T18:42:19Z
finished_at=2026-07-22T18:47:03Z

The awkward damaged state belongs in the protocol. Leaving it out forces the verifier to convert missing evidence into a cheerful guess. If the host rebooted while the command ran, kill -0 will fail and no final marker will exist. The correct response is to inspect the intended effect and any application logs, then decide whether a reconciliation action is needed.

Do not have the verifier use ps | grep. It will match unrelated processes, command names change, and output formats vary. kill -0 is only a liveness hint for a recorded PID. It is not a completion proof, which is why the verifier checks final artifacts before consulting the PID.

A finished exit status still may not prove the desired effect

The wrapper's final record proves what the wrapper observed, not necessarily what the outside world accepted. This becomes obvious with commands that send requests.

Imagine a remote script that creates a DNS record through an API, then writes exit_status=0. The script may receive success from the API before a resolver sees the new record. A deployment script may exit successfully after submitting a rollout that later fails health checks. A database tool may report a successful connection while one statement inside a multi-step procedure commits and a later statement fails.

Every operation needs a postcondition that fits its effect. The postcondition should be safe to read repeatedly and specific enough to reject an old or unrelated result.

For a release, write the run ID into a release manifest and query the active version from the service. For a database change, query the migration table for both the migration identifier and its checksum. For a generated artifact, compare a precomputed SHA-256 digest after the file has been placed at its final pathname. For an API request, supply the provider's idempotency token when it offers one, then query the resulting resource by that token or by a request ID you stored.

The worst design is a script that emits "done" after it sends a request and then treats that word as evidence. The stdout file tells you what one process printed. A postcondition tells you what the system now contains.

This distinction also tells you when an operation cannot safely be automated over SSH alone. If the remote command calls a third-party service that lacks idempotency controls and cannot search for a prior request, an interrupted call may be impossible to classify. Put a human approval or a compensating process around it. More retries will not create missing evidence.

Idempotency is better than recovery theater

A verification protocol reduces uncertainty. Idempotent command design reduces the cost of uncertainty. You want both.

An idempotent operation reaches the same desired state when applied again with the same request. mkdir -p /srv/app/cache is close to that model. useradd deploy is not, unless the script first verifies the existing account has the expected properties. curl -X POST /orders is not idempotent unless the service understands an idempotency token and treats a repeated token as the same request.

Do not confuse "the command probably does nothing on the second run" with idempotency. A deploy command might overwrite a file the same way twice but trigger a restart both times. A migration tool might recognize its own history but still run dangerous initialization before it checks. Read the command's behavior and test the interruption case.

Build requests around a stable operation identifier. Pass the same ID to the remote wrapper and, where possible, to the target system. A remote release wrapper could create /var/lib/recorded-runs/$run_id/effect only after the active release endpoint reports the requested version. A provisioning API call could use run_id as its idempotency value. Then an SSH retry can ask both systems about the same unit of work.

There is a practical rule here: retry a read freely, retry a create only with a durable uniqueness constraint, and retry a multi-stage change only after its postcondition has classified the earlier run. That is slower than blindly resubmitting a command. It is much faster than cleaning up duplicate infrastructure.

Backgrounding the command moves the problem somewhere else

nohup, &, disown, tmux, screen, and service managers each solve a different part of the problem. None turns an uncertain remote request into a verified result.

nohup helps a process survive a hangup signal in common shell setups. A plain nohup task & still leaves you with output files, a PID, and no structured completion record unless you add one. It also gives the caller a new ambiguity: did the remote shell start nohup, or did the connection die before that?

tmux and screen keep an interactive environment alive. They work well when an operator needs to reconnect and inspect a long command manually. They work poorly as an automation contract because session names collide, scrollback is not a result schema, and a detached terminal does not tell another system whether the requested effect occurred.

A service manager is stronger when the work is truly a service or a queued job. For example, a remote command can submit a named unit, and a later query can inspect that unit's lifecycle and logs. Use that model when the host already has an operational owner for jobs. Do not bolt a service manager onto a five-second administrative command just to avoid writing a small run record.

The useful split is simple. Keep interactive repair work in a terminal multiplexer. Put scheduled or long-lived workload under a service manager. Use a recorded wrapper for imperative commands where an SSH caller needs a dependable answer after reconnecting.

Keepalives shorten the wait but cannot close the ambiguity window

OpenSSH client keepalives make dead connections visible sooner. They do not guarantee that a command was not accepted before the network path failed.

For hosts where a stuck client wastes time, a client configuration such as this is reasonable:

Host production-*
    ServerAliveInterval 20
    ServerAliveCountMax 3
    TCPKeepAlive yes

ServerAliveInterval sends application-level messages through the encrypted SSH channel when no data arrives. If the client does not receive enough responses, it exits rather than waiting indefinitely. OpenSSH documents this separately from TCP keepalives, which operate at the transport layer.

Use the setting to bound how long a caller waits before it starts verification. Do not describe it as command delivery assurance. The disconnect can still occur after the remote host accepts the command and before the client receives the outcome.

Multiplexing deserves the same caution. ControlMaster and ControlPersist can reuse an existing network connection for several SSH commands. That reduces setup cost, but it also means a broken master connection can affect several calls at once. The OpenSSH manual says a persisted master stays in the background after the original client exits, which is useful operationally but does not add completion evidence for a command sent over it.

For automation, set an explicit connect timeout, set liveness bounds that fit the environment, and make the verification path independent of the original SSH session. A fast failure is useful only if the next action is a status lookup rather than a blind retry.

Build the failure test before you need it at 2 a.m.

A protocol you have never interrupted is a design sketch. Test it with an operation that is safe but slow enough to cut the connection at different points.

Start with a payload that writes a numbered progress file, sleeps between stages, and writes a final effect marker. Launch it through the wrapper. Kill the local SSH client after the remote started_at marker appears, then reconnect and run the verifier. Repeat by terminating the remote payload before it writes finished_at. Finally, simulate a host restart if your environment permits it.

Your expected classifications should be explicit:

• Before the wrapper claims the run ID, verification returns absent.
• During payload execution, verification returns running.
• After normal completion, it returns finished with the recorded status.
• After forced interruption or host loss, it returns damaged, followed by a postcondition check.

Test duplicate submission too. Invoke the same run ID twice at nearly the same time. One call must win the directory claim; the other must return an unambiguous duplicate result without running the payload. Then try two different IDs that target the same resource. If that creates a race, the wrapper needs a resource-specific lock or the target system needs a uniqueness rule.

Keep run records long enough to cover your operational retry window. If a job may be retried a day later but the host deletes records after an hour, you built a timer into your uncertainty. Protect records from casual modification as well. The account that verifies a run should not be able to edit exit_status or replace the payload log. On shared systems, separate the submitter, runner, and reader roles where the operating model allows it.

When AI agents invoke SSH, credential isolation and command verification need to work together. Sallyport keeps SSH credentials in its vault and can record the action that an agent requested, while the remote wrapper supplies the durable answer about the job itself. The action record can tell you which process asked for the command; the remote run ID tells you what happened after the connection became uncertain.

The safe retry decision has four outcomes

After a disconnect, classify before you act. There are four useful outcomes, and only one of them is an automatic retry.

If the run is absent, the remote wrapper never created its record. You may submit the same operation with the same run ID if you trust the wrapper's create-once behavior. If the command could have been executed outside the wrapper, inspect the target first because the wrapper cannot prove what bypassed it.

If the run is running, wait or cancel through an operation-specific control path. Do not launch another copy. A timeout policy belongs in the payload or job manager, not in a second SSH invocation that races the first.

If the run is finished, evaluate its exit status and then inspect the postcondition when the operation has meaningful external effects. An exit status of zero plus a failed postcondition is a failed operation. Treat the effect check as authoritative.

If the run is damaged, stop calling it a retry problem. It is reconciliation work. Inspect logs, journal entries, target state, and any idempotency record. Decide whether to repair the partial effect, mark the operation complete, or submit a new run that explicitly handles the observed state. That decision may need a human because the missing record removed the proof that automation depends on.

The worthwhile change is small: every command that can hurt you when repeated gets a run ID, a remote start record, a remote final record, and a target-state check. SSH will still disconnect. Your automation no longer has to pretend it knows what happened.