# SSH connection multiplexing: control socket risks SSH connection multiplexing can leave an authenticated route to a host open after the command that created it has returned. That behavior is deliberate. It is also easy to misunderstand when an automated task, an agent run, or a deployment job claims to be finished. I have seen teams investigate a supposedly closed maintenance window only to find that a local `ssh` master still held a live transport, a socket still accepted new clients, and a later command reused it without another interactive authentication. Nothing exotic happened. The configuration did exactly what OpenSSH documents. The team had treated the end of a shell command as the end of access. The useful distinction is between a completed **client command** and a terminated **authenticated transport**. Multiplexing makes those separate events. If you run autonomous work over SSH, your cleanup and evidence need to acknowledge that separation. ## A control socket can outlive the command that opened it SSH multiplexing uses one long lived SSH connection, called the master, to carry work for later SSH client processes. Those later clients are often called slaves in OpenSSH documentation. A slave does not repeat the normal connection setup and user authentication when it successfully contacts the local master through its control socket. A typical configuration looks harmless: ```sshconfig Host build-box HostName 192.0.2.44 User deploy ControlMaster auto ControlPath ~/.ssh/cm/%C ControlPersist 20m ``` The first `ssh build-box` creates a network connection and a Unix domain socket under `~/.ssh/cm/`. Later commands such as `ssh build-box 'uname -a'`, `scp`, and `sftp` can use that socket. With `ControlPersist 20m`, the master remains available for twenty minutes after its last session closes. That means this sequence is normal: 1. A task runs `ssh build-box 'apply-change'` and exits with status zero. 2. The master remains connected because `ControlPersist` tells it to remain available. 3. Another local process runs `ssh build-box 'read-status'` nineteen minutes later. 4. That process opens a channel through the already authenticated transport. The later command might be authorized by your local operating system and socket permissions, but it does not cause the remote host to see a fresh SSH authentication. A reviewer who looks only at an initial login record can easily assume the activity ended much earlier than it did. OpenSSH describes `ControlMaster` in the `ssh_config` manual as allowing multiple sessions over a single network connection, and describes `ControlPersist` as keeping the master open in the background. Read those together. `ControlPersist` is not merely a performance setting. It changes the period during which a local process can request new channels on an authenticated connection. For a human working at one machine, that trade may be sensible. For task scoped automation, it deserves an explicit owner and an explicit stop action. ## One authentication event is not one command event Teams often blur authentication, connection, channel, and command. They are separate things in SSH, and multiplexing exposes the difference. The initial master creates a TCP connection to the server, performs host verification, negotiates cryptography, and authenticates the account. Once authenticated, it can open SSH channels. A shell command, an interactive shell, an SFTP transfer, a local port forwarding request, and a remote port forwarding request all use channels or channel related requests on that transport. A new local `ssh` invocation that finds a usable control socket sends a request through that socket. The master decides whether to open the requested channel. It does not need the private key again. It does not need an agent prompt again. It does not need the server to receive another login attempt. This does not mean OpenSSH stores a reusable password in the socket. That description is sloppy and causes bad analysis. The security concern is an already authenticated transport with a local interface that can ask it to act. An attacker who can use the socket may not need the original credential at all. The distinction changes incident questions. "Did the key remain on disk?" matters, but it does not settle whether access remained. Ask these instead: - Did a master process remain connected to the remote destination? - Could another local process reach its control socket? - Did the master accept additional sessions, file transfers, or forwarding requests? - Who could run as the socket owner or cross the socket directory? - When did the master actually exit? A task runner that reports a command exit code answers none of those questions by itself. The code only describes the child process it waited for. ## Shared sockets turn local process boundaries into access boundaries A control socket is a local Unix domain socket. Its filesystem location and permissions decide which local processes can try to speak to the master. That is why a broad `ControlPath` can join unrelated work together even if the remote host aliases look tidy. Consider a CI runner account with this configuration: ```sshconfig Host * ControlMaster auto ControlPath /tmp/ssh-%r@%h:%p ControlPersist 1h ``` Job A connects as `deploy` to `app.internal`. It creates a master and a socket in `/tmp`. Job A then finishes. Job B, running under the same local account, connects to the same target. It can reuse Job A's authenticated transport if it can name and access that socket. This is popular because it speeds up repeated commands with almost no application changes. It is wrong for independent jobs because the reuse boundary is based on host, port, and user, rather than on the job that owns the authorization. A shared runner account turns that mistake into routine cross task access. The directory matters as much as the socket file. On Unix systems, a process needs directory search permission to reach a pathname. A socket stored in a directory with private ownership and mode 0700 gives a much clearer boundary than a shared temporary directory. The socket's own permissions still matter, but do not treat them as the entire control. Use a directory created for the task and owned by the executing account. A shell wrapper can do this without relying on a global SSH configuration: ```sh set -eu run_id="release-4821" cm_dir="$HOME/.ssh/task-control/$run_id" mkdir -p "$cm_dir" chmod 700 "$cm_dir" socket="$cm_dir/%C" ssh -o ControlMaster=auto \ -o ControlPersist=5m \ -o ControlPath="$socket" \ build-box 'id && hostname' ``` The `%C` token avoids a long literal name and reduces collisions among distinct connection parameter sets. The OpenSSH `ssh_config` manual defines it as a hash over connection details. It is useful, but it does not include your deployment ticket, agent session, or task ID. The parent directory supplies that missing boundary in this example. Do not put a literal control socket in a repository checkout, a broadly writable workspace, or `/tmp` merely because those locations are convenient. Convenience is how a local socket becomes an accidental shared capability. ## ControlPersist is a retention policy, not a cleanup plan `ControlPersist` tells SSH to keep a master available after client sessions close. It accepts `yes`, which keeps it running indefinitely in the background, or a time value such as `10m`. Both are retention choices. A timeout helps when a client crashes before cleanup. It does not prove that the task ended at the same time as the access. During the timeout, a process with socket access can request fresh work. An hour is especially hard to defend when a deployment task lasted two minutes. There are cases where a short timeout is reasonable. A controlled automation process may execute several commands in succession and benefit from a warm connection. In that case, make the timeout shorter than the expected idle gap, isolate the socket to that run, and terminate it on the success path. The timeout then acts as a failure fallback rather than the ordinary closure mechanism. The `ControlPersist yes` setting needs an owner with a long lived reason for access. An administrator's interactive workstation may have one. A disposable task does not. The other easy mistake is assuming that a command failure closes the master. A shell may return early after a failed remote command while the background master stays alive. A task cancellation can do the same. Cleanup must run after success, failure, and interruption, and it must record whether the shutdown worked. If your automation uses a trap, keep the cleanup narrow and verify its target. Do not blindly remove the socket pathname first. Removing a pathname can make discovery harder while the master process and connection continue to exist. ```sh cleanup() { ssh -S "$socket" -O exit build-box >/dev/null 2>&1 || true rmdir "$cm_dir" 2>/dev/null || true } trap cleanup EXIT HUP INT TERM ``` This pattern asks the master to exit before it tries to remove the directory. If `ssh -O exit` fails, preserve the directory and investigate rather than erasing the evidence. In production code, write the failure, process ID, and socket path to the task record. ## `exit` and `stop` have different operational meanings OpenSSH exposes control commands through `ssh -O`. Operators regularly use the wrong one because both names sound like cleanup. `ssh -O check host` asks whether a master is running and reports its process ID when one responds. `ssh -O exit host` asks the master to exit. For a completed task, `exit` is normally the command you want because it ends the reusable transport. `ssh -O stop host` tells a master to stop accepting new multiplexed sessions. Existing sessions continue. That can be useful when an operator needs to drain active work, but it does not meet a claim that all SSH access ended at task completion. A live master with active channels still has a live network connection. Use the socket path explicitly when diagnosis must not depend on the user's current configuration: ```sh ssh -S "$socket" -O check build-box # Master running (pid=41782) ssh -S "$socket" -O exit build-box # Exit request sent. ssh -S "$socket" -O check build-box # Control socket connect(...): No such file or directory ``` The words and exact error text vary by platform and OpenSSH version, so capture both standard output and standard error rather than parsing one sentence as a contract. The shape of the evidence matters: a successful check identifies an active master, a successful exit sends the termination request, and a later failed check supports the claim that no control socket answered. There is an awkward case worth handling. The socket can remain after a master crash, and a PID can disappear between checks. A stale pathname does not prove access still exists. Conversely, an unresponsive check does not prove a remote connection ended if you removed the pathname before examination. Check the process table and open Unix sockets before deleting anything. On macOS, `lsof` is often the fastest local inspection tool: ```sh lsof -nP -U | grep '/.ssh/task-control/' ps -p 41782 -o pid=,ppid=,lstart=,etime=,command= ``` Capture the output under the task record. The first command associates a Unix socket with a process. The second supplies parentage, start time, elapsed time, and invocation details. Treat `grep` as an interactive aid, not an audit control. A real collector should query and store the relevant records directly. ## Port forwarding makes an idle master more consequential A master that appears idle can still carry forwarding state or accept a later forwarding request. This is why counting shell commands alone gives an incomplete picture. Local forwarding exposes a local listener that sends traffic through the SSH connection. Remote forwarding asks the server to listen and send connections back through the client. Dynamic forwarding creates a SOCKS proxy. Each can outlive the command that established it depending on how the session and master were invoked. The OpenSSH manuals document control commands such as `forward` and `cancel` for forwarding requests when multiplexing is active. That is useful operationally. It also means a process that reaches the socket may request network paths beyond an ordinary shell command, subject to server policy and the master's state. For automated tasks, make forwarding an explicit exception. Record the bind address, local or remote port, destination host and port, and the cleanup result. Do not let a generic SSH wrapper quietly inherit `LocalForward`, `RemoteForward`, or `DynamicForward` directives from a user's broad `Host *` configuration. Inspect resolved settings before you trust an alias: ```sh ssh -G build-box | grep -E '^(controlmaster|controlpath|controlpersist|localforward|remoteforward|dynamicforward) ' ``` `ssh -G` prints the effective configuration after OpenSSH applies host matching and defaults. That check catches a surprisingly common failure: a task uses a simple host alias, but an included configuration file enables multiplexing or forwarding far away from the task's own configuration. The command may show more fields on different versions. Store the complete `ssh -G` output rather than only the lines you expected. A remote system may log a single source connection while several forwarded application connections pass through it. Network telemetry, server SSH logs, and task logs answer different parts of that story. None can replace the others. ## Server logs alone cannot reconstruct the local decision trail The remote SSH server sees the transport and whatever its logging configuration records about channels and commands. It cannot reliably tell you why a local process received permission to use a control socket, which task owned that socket directory, or whether another local process reused it after the original task ended. This is not a complaint about server logs. They are operating at a different point in the system. `sshd` can record authentication and connection events. A forced command wrapper or audit subsystem may record remote commands. Those records remain useful. They do not identify every local multiplex client by default because the server may see all channels as part of one already authenticated transport. Keep evidence on both sides of the boundary. A useful task record includes: - the complete resolved client configuration from `ssh -G`, with secrets excluded; - the remote hostname, address, account, host fingerprint result, and initial master PID; - the control socket path, private parent directory, and observed create and exit times; - each requested remote command, transfer, or forwarding operation with its exit status; - the `check` result before cleanup, the `exit` result, and post cleanup process and socket inspection. Add a task identifier to the directory name and the log record, not to a global control socket path shared by every task. The identifier connects local events without pretending it is a security boundary by itself. Hashing or signing the record after collection helps detect later editing, but it does not repair missing events. Collect lifecycle events as they happen. A record assembled from a shell history after an incident is weak evidence, especially when background masters and retries enter the picture. For agent driven work, distinguish the agent's intent from the executed SSH action. "Deploy version X" is intent. `ssh build-box 'sudo systemctl restart api'` is an action. The socket reuse record explains whether that action obtained a new transport or rode an existing one. Those are different audit facts. ## Task scoped sockets give cleanup an owner The safest pattern for short automated work is simple: use a unique control socket directory per task, permit reuse only within that task, and send an explicit `exit` request before the task reports completion. A practical wrapper needs a clear lifecycle. It should create the directory with restrictive permissions, write a record before the first connection, run commands with the same explicit `ControlPath`, shut down the master, verify the result, and only then remove the empty directory. A task that cannot verify cleanup should report cleanup failure even if its remote command succeeded. This example uses `mktemp` to avoid guessing a unique directory name: ```sh set -eu base="$HOME/.ssh/task-control" mkdir -p "$base" chmod 700 "$base" cm_dir=$(mktemp -d "$base/run.XXXXXX") chmod 700 "$cm_dir" socket="$cm_dir/%C" finish() { status=$? ssh -S "$socket" -O exit build-box >>"$cm_dir/cleanup.log" 2>&1 || \ printf '%s\n' 'master exit request failed' >>"$cm_dir/cleanup.log" ssh -S "$socket" -O check build-box >>"$cm_dir/cleanup.log" 2>&1 || true exit "$status" } trap finish EXIT HUP INT TERM ssh -o ControlMaster=auto \ -o ControlPersist=2m \ -o ControlPath="$socket" \ build-box 'deployctl apply release-4821' ``` The short `ControlPersist` setting covers a process that dies before its trap runs. It should not be read as permission for another task to reuse the master. The random directory blocks that reuse because the second task does not know or inherit the first task's path. There is one correction to make before adopting this exact sample. Do not log sensitive command arguments, environment values, or copied private data into `cleanup.log`. Audit records need enough detail to establish who did what, but they must not become a new secret store. Redact arguments at the wrapper boundary, where you still know their meaning. Sallyport routes SSH actions through its bundled stateless `sp-ssh` helper while keeping SSH keys in its encrypted vault rather than exposing them to the agent. That removes a common credential handling failure, but teams should still define the action boundary and retain records that show when the run ended. ## Global convenience settings defeat task boundaries A global `Host *` stanza often turns multiplexing on for every human shell, script, repository, and automation child process under an account. That is too broad when the same account runs tasks with different approvals or different owners. The setting may arrive through `Include` files, configuration management, a developer's personal dotfiles, or a build image. The command line can also override it. Never infer the active behavior from one visible config file. Use `ssh -G` with the exact host alias and user context that the task will use. If an automation environment cannot guarantee a private control path, disable multiplexing for that action: ```sh ssh -o ControlMaster=no \ -o ControlPath=none \ build-box 'maintenancectl status' ``` This costs a new connection and authentication for each invocation. That is a good cost for high consequence actions, rare break glass access, or work that crosses task and trust boundaries. The repeated authentication gives you a cleaner authorization event and makes lifecycle reasoning much less ambiguous. Do not mistake `ControlMaster=auto` for a guarantee that a process will only reuse its own connection. `auto` means the client will try to find a master at the configured path and create one if it cannot. The configured path decides whose connection it may find. Some teams argue that a shared socket is fine because all their jobs run under one service account. That argument only holds if every job with that Unix identity has the same authority to reach every target through that account, and if the team accepts that one job can inherit another job's live transport. Most mature environments do not actually want that. ## A clean task ending needs proof of transport closure Do not declare an SSH task complete when the last remote command returns. Declare it complete when the task has either closed its master and recorded proof, or reported that it could not do so. The final record should tell an investigator whether a later command had any path to reuse the connection. It needs a task identity, the resolved SSH configuration, the socket path, master process details, activity records, explicit control command results, and post cleanup observation. It also needs remote evidence for changes that continued independently of SSH, such as a service restart or a detached process. Set a standard that operators can execute under pressure: isolate the socket, inspect it before cleanup, request `exit`, verify that no master responds, and preserve the result. That procedure is more useful than a long default timeout because it turns an assumption about access into a checkable fact. When the work is sensitive enough that a leftover authenticated transport is unacceptable, do not optimize it with a shared master. Open a fresh connection, perform the action, close it, and keep the record. The extra seconds are cheaper than explaining why an access path survived after the task was supposed to be over.