SSH background jobs outlive a successful tool call

A zero exit status from an SSH tool call proves that the remote command reported success. It does not prove that every process caused by that command has stopped, finished its writes, or reached a successful outcome. If the command launched work in the background, the meaningful action may only be getting started when the tool records completion.

I treat that gap as an audit boundary, not a shell curiosity. An agent can run a deployment script, receive status 0, and move on while a detached migration still changes data. The tool transcript then tells the truth about the SSH channel but invites the wrong conclusion about the remote work. The fix is to name the lifecycle you intend to observe, test it under the same shell and terminal conditions the agent uses, and collect completion evidence from the system that owns the long-running process.

Zero describes the remote command, not its descendants

OpenSSH returns the exit status supplied for the remote command, or 255 when the SSH client itself encounters an error. RFC 4254 is even more precise: when the command at the other end terminates, the server may send an exit-status channel request and then close the channel. Neither document says that the server recursively waits for every descendant of that command.

That distinction matters whenever a shell executes an asynchronous list. POSIX defines a command terminated by & as asynchronous: the shell starts it and does not wait before continuing. If the shell has nothing else to do, it can exit successfully while the asynchronous child remains alive. The SSH status belongs to that shell.

There are at least four outcomes people casually call success:

• The SSH connection and authentication succeeded.
• The remote shell accepted and launched a command.
• The launched workload completed with status 0.
• The intended effect became durable and observable.

One integer cannot establish all four. A clean audit record must say which event produced it. I use ssh_command_exit_status for the channel result and reserve workload_result for evidence reported by the remote owner of the work.

The same warning applies when the remote executable daemonizes itself. A launcher can return 0 after a successful fork even if its child fails seconds later while opening a database, binding a port, or reading a configuration file. The launcher kept its contract; the auditor chose the wrong contract.

A twelve-second probe exposes the gap

You can reproduce the misleading success without a daemon, root access, or unusual shell settings. Run this against a disposable Unix account. The explicit redirections are important because they let the background process release the SSH channel while it keeps running.

ssh testhost 'rm -f /tmp/ssh-bg.done /tmp/ssh-bg.log; (sleep 12; date -u +%FT%TZ > /tmp/ssh-bg.done) > /tmp/ssh-bg.log 2>&1 < /dev/null & printf "launcher_pid=%s\n" "$!"'
printf 'ssh_status=%s\n' "$?"
ssh testhost 'test -f /tmp/ssh-bg.done; printf "done_status=%s\n" "$?"'
sleep 13
ssh testhost 'cat /tmp/ssh-bg.done'

A typical immediate result has this shape:

launcher_pid=41872
ssh_status=0
done_status=1
2026-07-24T10:14:05Z

The PID and timestamp will differ. The contradiction is the point: ssh_status=0 and done_status=1 coexist because they answer different questions. The shell successfully launched the asynchronous list, but the marker file did not exist yet.

Do not turn this example into production orchestration. /tmp marker files can collide, disappear, or be forged by another process with suitable access. The probe is useful because it makes the timing visible. A production completion record needs a unique execution identifier, protected storage, an authenticated writer, and a defined failure state.

Repeat the probe through the exact route your agent uses. A direct terminal command, a noninteractive SSH exec request, an SSH call with a pseudo-terminal, and a tool gateway can select different startup files, shells, and file descriptor arrangements. A test that skips those details has tested a neighboring system.

Open file descriptors can make a detached job look synchronous

Backgrounding and channel closure are separate mechanisms. A child that inherits the SSH channel's standard output or standard error may keep that channel readable even after the remote shell exits. The local ssh process can appear to wait for the child because the pipe has not reached end of file, not because SSH is supervising the child's result.

Compare these two calls and measure their elapsed time:

time ssh testhost 'sleep 12 &'
time ssh testhost 'sleep 12 > /tmp/sleep.log 2>&1 < /dev/null &'

On common OpenSSH and shell combinations, the first call may remain open until sleep exits, while the second returns promptly. Treat that as an observation to verify, not a portable promise. Shell implementation, server behavior, pseudo-terminal allocation, and the child program's own descriptor handling can change the result.

This accidental wait is weak evidence. The background process can close its descriptors early and continue working. It can fork a grandchild that closes them. It can send output through a socket or write directly to storage. Conversely, a helper that merely keeps stdout open can make the call look busy after the important work has failed.

File descriptors are still worth inspecting because they explain many inconsistent tests. On Linux, capture the remote PID and inspect descriptors while the SSH call is active:

pid=$(cat /run/user/$(id -u)/agent-job.pid)
ps -o pid=,ppid=,pgid=,sid=,stat=,etime=,args= -p "$pid"
ls -l "/proc/$pid/fd/0" "/proc/$pid/fd/1" "/proc/$pid/fd/2"

Record the parent PID, process group, session ID, state, elapsed time, command, and the targets of descriptors 0, 1, and 2. If /proc is unavailable, use the operating system's native process and descriptor tools. Do not reduce the test to pgrep name: names collide, wrappers change names, and a PID can be reused after exit.

nohup solves hangups, not ownership

nohup changes signal handling so the invoked command ignores SIGHUP. It does not place the command in the background. The GNU Coreutils manual says this directly and tells the user to add & for asynchronous execution. That qualification is often lost in copied deployment snippets.

Its redirection rules also surprise people over SSH. GNU nohup redirects standard input only when it is a terminal, sends standard output to nohup.out only when stdout is a terminal, and normally follows that choice for standard error. A noninteractive SSH command commonly uses pipes rather than a terminal, so nohup may leave those descriptors attached to the SSH channel.

These two commands therefore make different promises:

ssh testhost 'nohup /opt/jobs/rebuild-index &'
ssh testhost 'nohup /opt/jobs/rebuild-index > /var/log/rebuild-index.log 2>&1 < /dev/null &'

The second form explicitly disconnects the standard descriptors. It still does not tell you whether rebuild-index completed. nohup reports invocation failures such as a missing command, and otherwise its status follows the command it invokes. Once the shell backgrounds that invocation, the shell normally reports that it started the job, not the job's eventual status.

SIGHUP immunity is only one part of survival. The process may die because a login manager removes the session, a service manager kills the session's control group, the kernel invokes an out-of-memory policy, an administrator revokes the account, or the host reboots. A process can also survive perfectly and produce the wrong result. nohup supplies no identity, retry policy, resource boundary, durable status, or trustworthy completion record.

I still use nohup for small, disposable maintenance work when losing the result is acceptable and I am watching the host. I do not use it to turn an agent's SSH call into a managed production job. That recommendation remains popular because the snippet is short and usually survives a dropped terminal. It is wrong when anyone must later prove what finished.

Job control changes when a terminal enters the picture

Shell job control groups processes so an interactive user can suspend, resume, foreground, and background pipelines. A noninteractive shell usually runs without monitor mode, and an SSH exec request normally has no pseudo-terminal unless the client asks for one. Scripts that rely on jobs, %1, disown, or terminal-generated signals can therefore behave differently when an agent runs them.

POSIX ties job identifiers and known background PIDs to the current shell execution environment. Its wait utility can wait for those known processes, but a wait started in another shell has no inherited job table. This fails as an audit technique:

ssh testhost 'long_task & printf "%s\n" "$!"'
ssh testhost 'wait 41872; printf "wait_status=%s\n" "$?"'

The second call starts a new shell. Even if 41872 is still a live process, that shell does not know it as its child. POSIX specifies 127 for an unknown PID passed to wait. Permissions and PID reuse make attempts to reconstruct this relationship even less reliable.

Keep launch and wait in the same shell when synchronous completion is the contract:

ssh testhost 'long_task > /tmp/long-task.log 2>&1 < /dev/null & pid=$!; printf "pid=%s\n" "$pid"; wait "$pid"; rc=$?; printf "workload_status=%s\n" "$rc"; exit "$rc"'

This pattern returns the child's status and keeps the SSH action open. It works for a child that stays attached to that shell. If long_task forks and its original process exits, wait can still finish before the actual worker. Test the real executable, not a substitute sleep, before accepting this contract.

Pseudo-terminals add signal behavior and buffering changes. A terminal can deliver SIGHUP when its session ends, and background process groups that read from the controlling terminal can receive SIGTTIN and stop. Some programs switch to line buffering or emit different output when they see a terminal. Unless the command genuinely needs terminal semantics, automation should avoid allocating one and should set all three standard descriptors deliberately.

setsid detaches a process but does not produce evidence

setsid creates a new session and process group, initially without a controlling terminal. That is a stronger form of terminal detachment than ignoring SIGHUP alone. It explains why a child can outlive the shell and why terminal-generated signals stop following it.

It does not make the process supervised. After the original parent exits, another process may adopt the descendant. On a conventional host that may be PID 1; inside a container or service tree it may be a subreaper. The new parent relationship says nothing about the workload's success, and it may erase the easiest link back to the action that launched it.

A useful detachment test captures identity before the shell disappears:

ssh testhost 'run_id=agent-probe-20260724-1014; setsid sh -c '\''printf "%s\n" "$$" > /tmp/'"$run_id"'.pid; sleep 12; printf "complete\n" > /tmp/'"$run_id"'.state'\'' > /tmp/'"$run_id"'.log 2>&1 < /dev/null & printf "run_id=%s\n" "$run_id"'

Then query by the returned run ID, read the recorded PID only as a hint, and verify the process start time and command before acting on it. A bare PID is not durable identity. If the process exits and the kernel reuses its number, a later cleanup command can target unrelated work.

Double forking, setsid, disown, and closing descriptors are implementation techniques. Teams often mistake them for a job protocol because they make the terminal return. A job protocol answers different questions: Who owns the work now? How do I query it? What terminal states exist? Where is the exit reason? How do I cancel the whole process tree? What identifier joins the request, logs, effects, and audit event?

If the launch response cannot answer those questions, record it as a detached start, not a completed action.

Define three lifecycle events in the audit contract

An auditable remote action needs separate events for acceptance, channel completion, and workload completion. Collapsing them into one success boolean creates false certainty and makes incident reconstruction depend on shell history.

I use a record with this conceptual shape:

{
  "action_id": "act_01J3M8Q4",
  "remote_host": "worker-07",
  "launch": {"state": "accepted", "at": "2026-07-24T10:14:00Z"},
  "ssh_command": {"state": "exited", "status": 0, "at": "2026-07-24T10:14:01Z"},
  "workload": {"id": "job_8931", "state": "running", "result": null},
  "completion_source": "remote-job-manager"
}

The state names matter less than the separation. accepted means the remote owner validated and took responsibility for the request. exited means the SSH command ended. running means the lasting work has not reached a terminal state. Only the component that owns the workload should write succeeded, failed, or cancelled for that workload.

Make the transition rules explicit. A launch can fail before it creates a workload ID. An SSH channel can break after the remote system accepts the job, leaving the caller uncertain rather than failed. A workload can fail after a clean channel exit. Cancellation can be requested but not yet complete. An audit model that cannot represent unknown will eventually record a guess as fact.

Idempotency belongs in this contract. If the client loses the channel after submission, it should retry with the same action ID and ask whether the remote owner already accepted it. Starting a second migration because the first response went missing is worse than an untidy log.

Completion evidence should include the workload ID, terminal state, exit reason, start and finish timestamps, and the identity of the manager that observed the state. Add effect-specific proof where the risk justifies it, such as a deployed revision, completed backup manifest, or schema version. Do not use a log line containing the word done as the sole authority unless the logger and storage are part of the trusted job protocol.

Test the failure windows, not only the happy path

A useful test matrix varies how the process detaches, what happens to its descriptors, and when the connection or process fails. Run it on every supported host class because login managers, shells, and service managers change survival behavior.

Cover at least these cases:

• Foreground command, shell background job, nohup plus backgrounding, new session via setsid, and a program that daemonizes itself.
• No terminal and an allocated pseudo-terminal.
• Descriptors inherited, redirected to files, and closed by the child.
• Disconnect before acceptance, after acceptance but before the reply, and after the SSH command exits.
• Child exits nonzero, receives a signal, stalls, spawns a grandchild, and survives until explicit cancellation.

For every case, capture four clocks: client start, launch acknowledgement, SSH channel close, and remote terminal state. Capture the SSH status and workload result separately. Inspect the process group and session while the job runs, then prove whether cancellation reaches all descendants.

A compact harness can fail the test when status 0 arrives without terminal workload evidence:

result=$(ssh testhost '/usr/local/bin/job-submit agent-probe-42')
ssh_rc=$?
printf 'ssh_rc=%s response=%s\n' "$ssh_rc" "$result"
job_id=$(printf '%s\n' "$result" | sed -n 's/^job_id=//p')
test "$ssh_rc" -eq 0 && test -n "$job_id" || exit 1
/usr/local/bin/poll-job "$job_id" || exit 1

The example assumes job-submit returns exactly one job_id= line and poll-job authenticates its query, waits for a terminal state, and exits with the workload result. Those are contract requirements, not properties supplied by SSH. In a real harness, reject extra output, impose a deadline, preserve the uncertain state on timeout, and store the raw response for investigation.

Also test the observer. Stop the agent after launch. Restart the client machine. Rotate the SSH credential. Reboot the remote host if the job is supposed to survive reboot. If the only record of the job ID lives in one agent's context window, the system is not auditable.

A service manager is the usual owner for lasting work

When work should outlive an SSH command, hand it to a remote service or job manager and return its durable identifier. The manager should own the process group, collect output, enforce resource and cancellation behavior, persist status, and expose a query that distinguishes running from terminal states.

On a systemd host, a transient or templated service can provide a control group and journal identity. The systemd-run manual distinguishes asynchronous service startup from waiting for service termination, and it warns that a simple service may count startup as successful after a fork, before the requested program executes. I prefer Type=exec when execution failure must be visible, but that still proves startup, not eventual job success.

A templated unit might establish the ownership boundary like this:

[Unit]
Description=Agent job %i

[Service]
Type=exec
ExecStart=/usr/local/libexec/agent-job %i
StandardOutput=journal
StandardError=journal
KillMode=control-group
TimeoutStopSec=30s

Submit a unique, validated instance ID, then query the unit until it reaches a terminal state. Record ActiveState, SubState, Result, and ExecMainStatus, along with timestamps and the instance ID. Confirm how the workload behaves if it forks, because service type and program behavior must agree. Do not let arbitrary user input become a unit name or command argument without strict validation.

A queue, batch scheduler, container orchestrator, or application-specific job table can offer the same ownership boundary. Choose the owner already responsible for the workload's resources and recovery. SSH should submit and query. It should not impersonate a scheduler through an elaborate chain of shell operators.

For short work, keeping the command in the foreground and returning its actual status is simpler and often better. Detachment has a cost: another state store, another identity, cancellation semantics, retention, and reconciliation. Pay that cost only when the work genuinely must outlive the call.

The audit story ends at remote terminal state

An action gateway can accurately record the SSH call without knowing that a remote descendant exists. Sallyport logs the SSH action in its Activity journal and the agent run in its Sessions journal, so treat the call entry as evidence of the channel result, not as a census of processes on the host. The remote job ID and its terminal event still need to come back through an explicit, auditable action.

That division keeps each record honest. The gateway proves which agent run invoked SSH, which protected key authorized the action, what call occurred, and what result returned. The remote manager proves what happened after submission. Join both records with an action ID that the agent cannot silently replace between launch and query.

Do not label a detached launch completed in a UI or log. Use submitted or detached, display the workload ID, and keep the parent action open or visibly pending until a trusted observer records a terminal state. If observation times out, show unknown and require reconciliation. A red status may be inconvenient, but a green status based on the wrong process is dangerous.

Approval timing deserves the same precision. A human approving an SSH key use authorizes an attempt under the information shown at that moment. The click does not approve every future action of an unbounded descendant, and it does not certify the eventual effect. If a submitted job can run for hours or create further processes, show that fact before launch and bind the approval to the action ID, host, command intent, and remote job type. A per-call approval record and a workload completion record belong in the same chain, but they describe different decisions.

Preserve raw output around every boundary. Save the submission response before parsing it, capture stderr separately when the protocol allows it, and record whether a pseudo-terminal merged the streams. A parser should reject duplicate job IDs, control characters, truncated replies, and extra lines that could make one response look like another. The parsed fields support automation; the raw bytes support later review when the parser, shell quoting, or remote program turns out to be wrong. Keep secrets out of both forms.

A completion marker also needs an atomic publication rule. The worker should write its result to a temporary file in protected storage, flush the data when durability matters, and rename it into place only after the complete record is ready. The query side should validate the action ID, expected owner, file type, and state before trusting it. Better still, let a service manager or database expose status through an authenticated interface. A world-writable marker in /tmp demonstrates timing, but it cannot settle an incident.

Plan for uncertain delivery on both submission and cancellation. If the SSH connection disappears after the remote manager accepted a job but before the client received its ID, the correct local state is unknown. Reconnect with the original idempotency key and ask the manager to find that submission. Do not silently submit again. If cancellation loses its reply, query until the manager reports a terminal state and proves that its process group is empty. A sent signal is an attempt, not evidence that work stopped.

Reconciliation must survive the agent process. Store unresolved action IDs outside the transient conversation, assign an owner, and run a periodic query that closes or escalates old records. Define a deadline separately from failure: a job can exceed the caller deadline while remaining healthy and owned. The audit record should show that the caller stopped waiting, who still owns observation, and whether later completion arrived. Otherwise a timeout becomes another false terminal state.

Reviewers need the same vocabulary as the runtime. Search for actions where ssh_command.status is zero and workload.state is missing, running past its deadline, or unknown. Search for terminal jobs without a matching launch approval and for repeated submissions sharing an idempotency key. Those queries turn the lifecycle distinction into a control that can find gaps, rather than a paragraph engineers agree with and forget.

Give operators one action for reopening the remote record. The view should show the last observation time, the component that supplied it, and whether the status came from a live query or cached data. Never change unknown to failed merely to clear a queue. Preserve the uncertainty until the remote owner answers or an authorized reviewer resolves it with documented evidence. This may leave uncomfortable open records after an outage, which is accurate.

Test retention as well as creation. Workload records must remain queryable longer than the longest expected job and through the period when audits or incident reviews occur. If a service manager discards transient unit details immediately, copy the terminal result into the durable action record before collection. Logging a job ID that no system can resolve next week creates correlation without accountability.

Run the twelve-second probe in your real agent path, then repeat it with the workload that matters. If the SSH card turns green before the remote completion marker exists, you have found an audit gap. Keep the zero, because it is valid evidence about the command. Stop asking it to testify about work it never observed.