# Does remote process cleanup work after SSH cancellation? A cancelled agent run is not the same event as a stopped remote command. The local process can exit cleanly while the SSH transport is still alive, the transport can disappear while the remote shell keeps running, and the shell can die while its children continue in another process group. If you collapse those events into one status called "cancelled", you will eventually leave a database migration, package install, test worker, or deployment helper running after the agent has reported that it stopped. The fix is not a clever signal trap by itself. You need a cancellation contract that names the remote run, creates a killable boundary around its descendants, preserves enough output to diagnose an interrupted run, and has a remote-side answer for a controller that vanishes. Build and test that contract before you let an agent run commands with side effects. ## SSH cancellation has four separate hops A cancellation request must cross four boundaries: the agent decides to stop, the local supervisor stops or signals its SSH client, the SSH protocol carries a channel event or signal, and the remote host acts on that event. Each hop can fail independently. RFC 4254 separates these concepts. It defines a channel close message and separately defines a `signal` channel request for names such as `TERM`, `INT`, and `HUP`. A channel close is a transport event. It does not mean "send SIGTERM to every remote descendant." The RFC also says data sent before a close should be delivered if possible. "If possible" is doing a lot of work when a laptop sleeps, a network route drops, or a local process is force-quit. That distinction catches a common bad design: 1. An agent starts `ssh host long-command`. 2. The user presses cancel. 3. The agent runner kills its local child process. 4. The UI marks the job cancelled. 5. `long-command`, or one of its children, continues on the host. The fifth line is not a corner case. It is the default result whenever the server has no reason to terminate the command, or the command has detached from the session before the connection disappeared. A useful cancellation contract says exactly what the local side will attempt and exactly what the remote side owns: - The launcher creates one identifiable remote run with a random run ID. - The remote wrapper starts the workload in its own process group or session. - A normal cancel sends `TERM` to that process group and records the result. - The wrapper escalates to `KILL` only after a defined grace period. - A remote deadline or lease ends work when the controller never returns. - Output and final status survive the SSH stream. Do not call a task cancelled until you have one of two outcomes: a confirmed final remote record, or an explicit "state unknown" result. Pretending certainty after a connection loss makes incident response slower because everyone starts from a false premise. ## A remote PID is not enough to clean up children Killing the remote shell PID is only safe when the shell never forks, starts a pipeline, launches a background process, or invokes a tool that spawns helpers. That is a small set of commands. Consider this ordinary remote command: ```sh build-assets | tee build.log & wait ``` The shell has one PID. The pipeline has more processes. `tee` may still write its log after the shell exits. A compiler may start worker processes. A package manager may hand work to a service. If you send `kill -TERM "$shell_pid"`, you have terminated one member of a larger group and learned very little about the rest. Process groups give you the right unit of cancellation for a short-lived remote run. On Linux, each process belongs to a process group, and each process group belongs to a session. Terminal-generated signals go to the foreground process group, which is why terminal behavior can look more magical than it is. The Linux `setpgid(2)` documentation also makes clear that a child inherits its parent's process group unless something changes it. For an agent-controlled command, create a fresh session for the workload. The session leader normally has a PID equal to its PGID and SID. Then a negative PID in `kill` addresses the process group: ```sh kill -TERM -- -"$pgid" ``` The leading minus sign is the difference between terminating a process and terminating its process group. The `--` matters too. It prevents a malformed value from being parsed as an option. Do not blindly assume that the workload PID is its process group ID. Check it at launch. A shell wrapper, a service manager, or a program that calls `setpgid` can change the tree. This is the smallest inspection command worth keeping in your test kit: ```sh ps -o pid=,ppid=,pgid=,sid=,stat=,etime=,command= -p "$pid" ``` Typical output has this shape: ```text 24182 24177 24182 24182 Ss 00:03 bash ./worker.sh /tmp/agent-runs/6c4... ``` Here, PID, PGID, and SID all match. That is evidence that `kill -TERM -- -24182` targets the intended boundary. If the PGID does not match the run record, fail the launch instead of guessing. A process group still has limits. A child can call `setsid`, a container runtime can move a process elsewhere, and a workload can ask a service manager to run something outside the group. That behavior is sometimes legitimate. It also means your cancellation guarantee has ended at that handoff. Treat detached work as a distinct job type with its own identity, stop operation, and audit record. ## A cancellation wrapper needs a real cleanup path A remote shell wrapper should own the workload PID, trap expected termination signals, target the workload group, wait briefly, and write a final record. It should not use `pkill command-name`, scrape a loose process list, or kill every process belonging to an account. Those shortcuts work right up to the day two agent runs share a user, a hostname changes, or a command name matches someone else's work. This Linux-oriented fixture is deliberately plain. It creates a protected run directory, starts one workload in a new session, writes output to files, and terminates the workload's group when the wrapper receives `TERM`, `INT`, or `HUP`. ```bash #!/usr/bin/env bash set -Eeuo pipefail run_id=${1:?run ID required} shift run_dir="${HOME}/.agent-runs/${run_id}" umask 077 mkdir -p "$run_dir" child_pid="" child_pgid="" finished=0 write_status() { local state=$1 local code=${2:-} local tmp="$run_dir/status.tmp" printf '{"run_id":"%s","state":"%s","exit_code":"%s"}\n' \ "$run_id" "$state" "$code" >"$tmp" mv "$tmp" "$run_dir/status.json" } stop_group() { if [[ -z ${child_pgid:-} ]]; then return fi kill -TERM -- "-$child_pgid" 2>/dev/null || true for _ in 1 2 3 4 5; do if ! kill -0 -- "-$child_pgid" 2>/dev/null; then return fi sleep 1 done kill -KILL -- "-$child_pgid" 2>/dev/null || true } cancel() { local signal=$1 trap - TERM INT HUP write_status "cancelling:$signal" stop_group wait "$child_pid" 2>/dev/null || true write_status "cancelled:$signal" finished=1 exit 143 } trap 'cancel TERM' TERM trap 'cancel INT' INT trap 'cancel HUP' HUP write_status "starting" setsid "$@" >"$run_dir/stdout.log" 2>"$run_dir/stderr.log" & child_pid=$! child_pgid=$(ps -o pgid= -p "$child_pid" | tr -d ' ') if [[ "$child_pgid" != "$child_pid" ]]; then printf 'unexpected PGID for %s: %s\n' "$child_pid" "$child_pgid" \ >"$run_dir/stderr.log" kill -TERM "$child_pid" 2>/dev/null || true write_status "launch_failed" exit 70 fi printf '%s\n' "$child_pid" >"$run_dir/pid" printf '%s\n' "$child_pgid" >"$run_dir/pgid" write_status "running" set +e wait "$child_pid" code=$? set -e if [[ $finished -eq 0 ]]; then write_status "finished" "$code" fi exit "$code" ``` This wrapper prevents one specific failure: a cancellation signal reaches the wrapper, but it only kills itself and strands the workload. It does not promise to stop intentionally detached descendants. It should not make that promise. The Linux `setsid(2)` manual says that `setsid()` creates a new session and makes the caller the leader of a new process group, initially with no controlling terminal. The util-linux `setsid` command runs a program in that new session, forking when needed. That is why this is a practical boundary for a command run, not a magic cleanup switch. Keep the wrapper narrow. It should launch, record, stop, and report. Do not bury business logic in it. The workload should still have its own transaction handling, temporary-file cleanup, and idempotency rules. ## A clean transport disconnect is not a cleanup guarantee SSH users often infer too much from a terminal test. They run a command with a PTY, close the terminal, see a process exit after `SIGHUP`, and decide that disconnect cleanup works. Then an agent uses a non-interactive SSH channel without a PTY, and the behavior changes. A PTY creates terminal semantics. A terminal hangup can lead to `SIGHUP` delivery, but only under the conditions that govern controlling terminals and foreground process groups. The Linux manual describes `SIGHUP` as a hangup on a controlling terminal or death of a controlling process. That description does not say every SSH disconnection signals every process started through SSH. Non-interactive SSH is usually the better default for agents because it gives cleaner output and fewer shell startup surprises. It also removes any accidental dependency on terminal behavior. Use a PTY only when the remote program needs one, such as an old installer that refuses to run otherwise. Then document that the PTY is part of the command's behavior and test it separately. There are three disconnect cases worth naming: ### The client sends a deliberate cancel The local supervisor still has a live connection and can send a protocol signal, or it can open a separate authenticated control command that signals the recorded PGID. This is the best case. The remote wrapper receives `TERM`, cleans up, and writes `cancelled:TERM`. Do not rely on an SSH client receiving local `SIGINT` to do exactly this unless you have tested the client library and invocation you ship. A terminal client, an embedded SSH library, and an MCP tool may map local cancellation differently. Some will close a socket. Some will terminate the local process. Some can send an SSH `signal` request. Those are different implementations of an interface that users call "cancel." ### The local client crashes or loses its network The remote command may continue. The server cannot distinguish a temporary routing problem from a user who intends work to continue unless your protocol tells it how. A remote lease is the honest answer. At launch, write a `deadline_epoch` into the remote run directory. A local supervisor renews it while the run remains authorized. A remote watchdog checks it and calls the same process-group cleanup path after it expires. Pick a lease duration that fits the operation. A five-minute lease might fit a shell command. It may be reckless for a build with long but normal quiet phases. ### The remote host fails or reboots You may lose both process and final status. Do not report "cancelled" or "completed" merely because the SSH connection ended. Mark the run unknown until a later reconciliation reads the host's journal, deployment state, lock record, or application-specific result. The hard part is not emitting a status word. The hard part is refusing to emit a status word that your system cannot support. ## Partial output proves observation, not completion A stream answers "what bytes did the client receive so far?" It does not answer "what state did the remote command leave behind?" This mistake appears when a remote program prints `done` before flushing its last file, or when the network drops after the command completed but before the client received the SSH exit status. Keep two records: - `stdout.log` and `stderr.log` hold diagnostic output as the workload writes it. - `status.json` is a small final record written atomically after the wrapper observes exit or handles cancellation. The `mv` in the wrapper matters. Write a temporary status file in the same directory, then rename it into place. Readers see either the previous whole file or the new whole file. They should never parse half of a JSON document and invent a result. Output itself needs rules. A command can buffer output heavily when stdout is a file instead of a terminal. If progress matters, have the workload emit explicit line-oriented status to stderr or use an application-level progress file. Do not solve buffering by allocating a PTY for every command. That changes behavior and can merge stdout with stderr, which makes audit and failure analysis worse. Treat these cases differently in your agent UI or journal: | Remote record | Stream state | Meaning | | --- | --- | --- | | `finished`, exit code present | complete | The wrapper observed normal completion. | | `cancelled:TERM` | may end abruptly | The wrapper began cancellation and stopped its group. | | `cancelling:TERM` only | disconnected | Cleanup started, but the final record was not observed. Reconcile. | | `running` only, lease valid | disconnected | Work may still be running. Do not retry blindly. | | no usable record | disconnected | State is unknown. Check side effects before relaunching. | Avoid placing access tokens, unredacted configuration dumps, or credentials in these logs. SSH credential injection can keep the private key away from the agent, but a remote command can still print secrets that it read from its own environment or configuration. Output retention is part of the command design, not an afterthought. ## Test process trees instead of testing one sleepy shell `trap 'exit' TERM; sleep 600` is a weak cancellation test. It proves that one foreground shell can receive one signal. It does not test children, process groups, delayed cleanup, output persistence, or a remote shell that disappears at the wrong moment. Use a workload that creates a visible process tree and records each signal. Save this as `worker.sh` on a disposable Linux host: ```bash #!/usr/bin/env bash set -Eeuo pipefail run_dir=${1:?run directory required} note() { printf '%s pid=%s pgid=%s %s\n' \ "$(date +%s)" "$$" "$(ps -o pgid= -p $$ | tr -d ' ')" "$1" \ >>"$run_dir/worker.log" } trap 'note TERM; exit 143' TERM trap 'note INT; exit 130' INT trap 'note HUP; exit 129' HUP ( trap 'note grandchild_TERM; exit 143' TERM trap 'note grandchild_HUP; exit 129' HUP while :; do note grandchild_tick sleep 1 done ) & grandchild=$! note "started grandchild=$grandchild" while :; do note parent_tick sleep 1 done ``` Run it through the wrapper with a random run ID. In another SSH session, inspect the process tree and the run directory: ```sh run_id=cancel-test-$(date +%s) ssh host.example './remote-wrapper.sh '"$run_id"' ./worker.sh "$HOME/.agent-runs/'"$run_id"'"' ``` The exact quoting will differ in a real launcher. That is fine. What must not differ is the test evidence: you need the remote run ID, wrapper PID, workload PGID, log locations, and final status. Then test these failure paths one at a time: 1. Send `TERM` to the wrapper PID. Confirm that both parent and grandchild logged termination and that `ps` finds no process in the recorded PGID. 2. Send `TERM` directly to the workload PID. Confirm that its child behavior is not assumed safe. This test explains why the wrapper targets a group. 3. Kill the local SSH client without sending a remote signal. Confirm that the workload remains active until its remote lease expires. If it stops immediately, record why, such as PTY hangup behavior, rather than treating that result as universal. 4. Disconnect after the wrapper writes `cancelling:TERM` but before it writes its final record. Confirm that reconciliation can distinguish incomplete observation from a fresh running job. 5. Start two runs under the same account, cancel one, and prove that the other remains alive. This catches dangerous broad `pkill` and account-wide cleanup logic. Use `ps` and `pgrep -a -g "$pgid"` while testing, then repeat after the grace period. Check the remote status file and logs only after checking the process table. A status record that says "cancelled" while workers remain alive is a bug in your supervisor, not a harmless reporting mismatch. ## Parent-death signals help only inside a controlled worker Linux provides `PR_SET_PDEATHSIG`, which lets a process ask the kernel to send it a signal when the thread that created it terminates. It can be useful when you own a small native helper that spawns an immediate child and wants that child to stop if the helper dies. The setting survives `execve` in ordinary cases, but the manual documents important exceptions, including credential changes. It does not solve remote agent cleanup by itself. First, the SSH server process is not necessarily the parent you care about. Second, the signal applies to the process that set it, not automatically to all descendants. Third, a remote command that forks, double-forks, or hands work to another service leaves that relationship behind. Fourth, it is Linux-specific, which matters if your fleet has mixed Unix hosts. Use it only as a reinforcement where the process tree is yours. For example, a small Linux worker can set `PR_SET_PDEATHSIG` before it execs a single controlled child, while the outer wrapper still owns the process group and lease. That gives you two failure detectors with different scopes. It does not give you permission to skip the group boundary or remote final record. The same warning applies to `nohup`, `disown`, and `setsid` inside the workload. They are useful when someone deliberately wants work to outlive a terminal. They are incompatible with a promise that cancelling an agent run stops the work. Make the choice explicit at launch. ## A second control channel is often cleaner than killing the first one When an agent cancels an in-flight SSH call, its own local execution context may already be tearing down. Depending on that dying process to send a final protocol signal invites races. A separate supervisor process should own cancellation and reconciliation. One workable design looks like this: 1. The supervisor generates a cryptographically random run ID and invokes the remote wrapper. 2. The wrapper records its PID, PGID, start time, and status under a directory named for that run ID. 3. The supervisor records the run ID and remote host before it begins consuming output. 4. On cancel, the supervisor opens a fresh control action that reads the run record, verifies expected ownership and age, then signals the recorded PGID. 5. The supervisor polls the final status record until it sees a terminal state or reaches a reporting deadline. The control action must validate the run record before it signals anything. At minimum, check that the record directory belongs to the expected user, the PID still exists, the recorded PGID matches `ps`, and the start time is consistent with the process you launched. Linux can reuse PIDs. A stale PID file plus an unconditional `kill` is how a failed cleanup script terminates unrelated work weeks later. Do not put the cleanup command behind a generic agent instruction such as "kill the process for my previous task." The agent should receive an opaque run handle. The supervisor translates that handle into a narrowly scoped remote action. That design also makes auditing readable: a reviewer sees that run `6c4...` requested cancellation of PGID `24182` on one host, not that an agent formed an arbitrary kill command. For autonomous coding agents, Sallyport can execute SSH actions without exposing the SSH key to the agent. That keeps credential custody separate from the cancellation contract, which still needs the run handle, group check, remote lease, and final record. ## The right retry decision depends on the side effect A command that only reads a file can usually be retried after an unknown disconnect. A command that creates a user, applies a migration, rotates a certificate, or starts a deployment cannot. The SSH layer cannot tell you whether the operation became safe to repeat. Give side-effecting remote commands an idempotency token derived from the run ID. The remote program should store the token with the operation's result and return the existing result if it sees the same token again. If that is impossible, add a preflight query that identifies whether the requested change already happened. Do not use cleanup as a substitute for idempotency. Even a perfect `TERM` can arrive after a remote API accepted a request but before the command printed its response. Even a `KILL` can land after a database transaction committed. Process cleanup answers whether your worker is still executing. It does not roll back external effects. Keep three outcomes in the control plane: completed, cancelled with confirmed cleanup, and unknown. Unknown is uncomfortable, but it gives the next operator the information they need: inspect the remote state before issuing another action. That is far better than a green retry button backed by wishful thinking. A cancellation feature is ready when you can interrupt it at every boundary and explain the surviving process tree, the output on disk, the final record, and the retry decision. If you cannot do that on a disposable host, do not trust it against a production host at 2 a.m.