Orphaned MCP processes can still leave work behind

A client crash does not tell you whether its MCP server stopped. It tells you only that one process died, or at least stopped responding. The distinction matters because stdio is a transport, not a kill switch for work that has already crossed a process boundary.

I have seen teams treat a pile of old helper processes as a housekeeping problem, then discover that one still held a cloud token, an SSH control socket, or a queue of work that no one had tied back to the failed run. The cleanup command was easy. Reconstructing who authorized the last action was the part they had skipped.

The right response has three separate jobs: identify the leftover process accurately, prove whether it still has a path to act, and preserve the record that connects its final calls to the session that owned them. Do those in that order. Killing first may stop the nuisance, but it can erase the best evidence of what happened.

A parentless process is a clue, not a verdict

An orphaned process is a child whose original parent exited and whose operating-system parent changed, often to PID 1. That is useful evidence, but it is not the definition of an unsafe MCP server. Process supervisors, shells, IDEs, and launch services can reparent healthy children during ordinary operation.

For a stdio MCP server, ask a narrower question: does this process still belong to a live client connection, or has it survived after the client that owned its stdin and stdout disappeared? That question combines process ancestry with open descriptors, elapsed time, and an activity record.

The Model Context Protocol documentation describes stdio as a local, process-spawned integration. The client starts a command and exchanges newline-delimited JSON-RPC over the server's standard input and output. That design gives a well-behaved server a simple end-of-life signal: when its input reaches EOF, the client side is gone.

EOF is evidence of a broken or closed transport. It is not evidence that every worker, child process, network connection, timer, or remote job has stopped. A server can receive a tool call, begin work, and then lose the client before it produces a JSON-RPC response. If the handler launched a subprocess or sent a request to a remote system, that work may have its own lifetime.

This is where people blur two different failures:

• An operating-system orphan has lost its original parent or detached from its expected process tree.
• A logical orphan has lost the session that gave its work meaning, even if its parent PID still looks normal.

The first can waste memory or keep a port open. The second can make an unwanted external change. You need to inspect both.

A process with PPID 1, an old elapsed time, and no open stdin is suspicious. A process still attached to a live terminal can also be dangerous if its owning client has silently wedged and it has a credential or reusable connection. Conversely, a process that remains after a crash may be harmless if it has no authority and cannot reach any action channel.

Do not build a rule that says "PPID 1 means kill." Build a record that says why this process exists, which run created it, what it can still reach, and what you did with it.

Start with a process inventory you can defend

Take a snapshot before you signal anything. You cannot recover a command line, process group, or open descriptor after you terminate the process, and PID reuse turns loose notes into guesswork surprisingly fast.

On macOS, begin with the full view rather than a clever one-liner that filters out the evidence you need:

ps -axo user,pid,ppid,pgid,stat,etime,command

Look for the commands your MCP client starts. That may be a direct server binary, an interpreter such as node or python, a package launcher, an SSH helper, or the sp mcp shim. Copy the relevant rows into an incident note before you narrow the search.

The fields answer different questions:

• pid identifies the process for this inspection only.
• ppid tells you whether its immediate parent still exists.
• pgid identifies the process group, which often reveals sibling helpers launched together.
• stat can show a sleeping, stopped, or uninterruptible process state.
• etime shows whether a supposedly recent run has been around for hours or days.
• command is often the only surviving record of the executable and arguments used to launch it.

Then inspect each candidate directly. Substitute the actual PID, and save the output with a timestamp in your incident record.

ps -o pid=,ppid=,pgid=,stat=,etime=,user=,command= -p 48271
lsof -nP -p 48271

The first command gives you a compact identity record. The second tells you what the process still has open. Pay close attention to file descriptors 0, 1, and 2. A normal stdio server usually has a read end for stdin and write ends for stdout and stderr. If stdin has reached EOF, the process may still display a pipe descriptor, so do not infer liveness solely from its presence. You want the wider picture: peer process, terminals, regular files, Unix sockets, TCP connections, and child workers.

lsof can also expose a mistake that process listings conceal. Suppose the server's parent is gone, but it still owns an established TCP connection to an internal API. That does not prove it can make a new request, but it gives you a concrete capability to investigate. If it owns an SSH control socket or a local Unix socket connected to a credential helper, treat that as a lead, not as an excuse to assume safety.

Check the process group before you act:

ps -axo pid,ppid,pgid,stat,etime,command | awk '$3 == 48271 || $2 == 48271'

This example assumes 48271 is the group ID you observed. Adjust it to the actual value. The output can show a wrapper, server, and helper child that a simple kill against one PID would leave behind. It can also prove that a process has no remaining siblings and is easier to isolate.

Avoid a common shortcut: searching only for node, python, or npx and killing every match. Those names are runtimes, not identities. A broad cleanup can break an editor extension, local test run, build task, or an unrelated automation job. Match the complete command line to the server configuration you expect, then inspect the parent and group around it.

If your client records the server command at launch, store that command next to the session record. It turns a later investigation from a fuzzy process-name search into a direct comparison.

The stdio pipe does not define the action boundary

A properly written stdio server should stop accepting requests when stdin closes. It should cancel work that has not started, close its transport, and exit after it deals with any bounded cleanup. The official TypeScript MCP SDK guide makes the same operational point in its shutdown discussion: closing a transport does not automatically drain in-flight tool handlers before the process exits.

That last part deserves more attention than it gets. A tool handler can be in one of several states when a client crashes:

1. It has not begun external work and can be cancelled cleanly.
2. It has sent a request but has not received a response.
3. It completed the remote change but lost the response path before it could report success.
4. It started a local child or remote job that outlives the handler.
5. It is blocked waiting for an external dependency and may resume later.

Only the first state is safely described as "nothing happened." The other four require records outside the dead client process.

Take a concrete example. A server receives a tool call that deploys a build. It writes the request to a remote build API, then the client crashes while the API is processing it. The server sees a broken stdout pipe. If it exits immediately, the build may still run. If it retries without an idempotency mechanism, it may start a second build. If it stays alive with a long-lived credential, it may poll, retry, or launch follow-up work after the original client is gone.

The failure is not that the server remained alive. The failure is treating transport closure as a complete statement about authorization, cancellation, and remote state.

For every action channel, define the boundary explicitly:

• What event stops new work from entering the server?
• What identifier lets you query or cancel work already sent out?
• Does the remote operation support idempotency or a request token?
• Which process still holds the credential after the client disappears?
• Where does the final result get recorded if the JSON-RPC response cannot be delivered?

If you cannot answer those questions for a high-impact tool, do not call it safe just because it uses stdio.

The same rule applies to SSH. An interactive-looking command can still have started a background process on the remote host. Closing the local client may close the local channel, while a remote command that detached from its shell continues. A server needs a command design that makes remote work observable and cancellable, not wishful thinking about terminal disconnects.

Prove whether the leftover process can still act

Process existence is not authority. You need to test the action path without provoking a real action.

First, identify where authority lives. A server that reads an API token from its own environment has a different risk profile from one that asks a separate broker to perform each action. A server with an SSH private key in a file has a different profile from one that delegates through a short-lived local helper. Write the answer down for the specific server under review.

Then inspect what the process still has open and reachable. lsof gives you a starting point, but it does not reveal every in-memory credential or every authenticated session. Pair it with your own action logs and, where available, the external system's audit trail.

A useful investigation sequence looks like this:

1. Record the candidate PID, command, PPID, process group, and open network or Unix sockets.
2. Find the most recent external action tied to that process or its session.
3. Check whether a new action occurred after the client crash time.
4. Revoke or disable the process's authorization path.
5. Observe whether the process attempts another action and whether the gateway rejects it.

Do not test by asking the orphaned server to perform a harmless-looking write. Many systems have no truly harmless write, and a test call can alter rate limits, create audit noise, rotate state, or trigger automation. Prefer a read-only health or identity endpoint if your design has one. Better yet, validate the denial from the gateway or credential broker after revocation.

There is a hard distinction here: a process that still has network access is not necessarily able to act, and a process with no visible network connection may still be able to act later. DNS resolution, a proxy, a local helper, a queued timer, or a child process can reopen the path. This is why the revocation test matters more than a snapshot of sockets.

If the server directly holds a reusable secret, your cleanup scope must include secret rotation or revocation. Killing the process removes one copy from memory, but it does not change what an attacker, a dumped process image, or a remote service can do with the same secret. Teams often avoid rotation because it creates work. That is understandable, but it is not a security argument.

A gateway model changes the investigation. The server can survive as an untrusted process, yet fail to make a new protected call because it never had the credential and no longer has an approved live session. Sallyport keeps API and SSH secrets in its encrypted vault and executes the action itself, so the agent does not receive the secret as plaintext or as a substitute value. That reduces the damage a stray stdio process can do, but you should still revoke a suspect run rather than assume a process crash revoked it for you.

The practical standard is simple: after you revoke the session or action path, an attempted protected call must be denied and the denial must be recorded. If you cannot demonstrate that, you have not confirmed containment.

Record the final calls before cleanup changes the story

An incident record needs stable identifiers, not a narrative assembled from memory. A PID is temporary and can be reused. A process command can change after an update. A session identifier and a tamper-evident audit record give you a much better anchor.

At minimum, create one row per suspicious process with these fields:

Observed at:
Client process PID and command:
MCP server PID and command:
Parent PID and process group:
Session identifier:
Authorization state:
Last successful action time:
Last attempted action time:
Action target and operation:
Result or remote job identifier:
Revocation time and operator:
Termination signal and exit result:
Follow-up required:

The important pair is the session identifier and the last attempted action. Many teams record only successful calls, which hides the most revealing moment in a crash investigation: the request that left the local machine but never received a result.

Keep three timestamps separate. Record when the client became unavailable, when the server's last known action started, and when you revoked authority. If you use only one "incident time," you cannot tell whether a call happened before the crash, during an uncertain window, or after containment should have taken effect.

Also preserve the action target. "Called cloud API" is not enough. Record the account or endpoint class, the method or SSH command category, and any request or job identifier that lets an operator search the remote service. Avoid storing sensitive request bodies in a general incident note. You need enough information to reconcile the side effect, not a second copy of every secret and customer record.

For a gateway with separate session and activity journals, use both. The session record answers who ran the agent process and whether that run remains authorized. The activity record answers which individual calls occurred and in what order. Those are different questions, and combining them into one broad event stream makes both harder to answer under pressure.

Sallyport projects both journals from one encrypted, hash-chained audit log. After you capture the relevant session and activity entries, run the offline integrity check:

sp audit verify

Save the command's exact result with the incident record. The verification does not decide whether the action was authorized or wise. It tells you whether the audit chain you are relying on still verifies without requiring access to the vault. That separation is useful when the person reviewing the incident should not receive credentials just to check the history.

Do not wait for a formal security incident to practice this recordkeeping. A routine crash review is where you find missing session IDs, ambiguous commands, and logs that only exist on the dead machine. Fixing those gaps during a calm cleanup is much cheaper than discovering them after a production write.

Revoke authority before you terminate the process

The safest order is revoke, verify denial, then stop the process. Reversing it feels faster because the process disappears immediately, but it can leave an authorization record live and make later correlation harder.

Start with the narrowest effective revocation. If the system tracks authorization per agent run, revoke that run. If a single credential was exposed to the server, disable or rotate that credential. If a remote job has a cancellation handle, cancel that job separately. These are different actions because they address different lifetimes.

Do not assume a server exit cancels remote work. A process can be gone while its request remains queued. Do not assume a remote cancellation stops a server. The process may retry or submit another request if it retains authority. Containment requires both the local and remote sides to agree that the run is over.

Once you have revoked the action path, verify the state with evidence appropriate to your setup. That could be a recorded denial in the gateway's activity journal, a failed authenticated request to a safe identity endpoint, or a remote audit entry showing the token disabled. The exact method varies. The principle does not: prove that a surviving process cannot make a protected call.

Then send a normal termination signal to the identified PID:

kill -TERM 48271
sleep 2
ps -p 48271 -o pid=,ppid=,stat=,etime=,command=

If the final command returns no process row, record that result. If it remains, inspect whether it is shutting down, blocked in I/O, or keeping child work alive. Before escalating, inspect the group and children again. You may need to terminate a known helper separately, but do not kill an entire process group blindly unless you have confirmed every member belongs to the same failed run.

Use kill -KILL only when ordinary termination fails and you have preserved the evidence. SIGKILL gives the process no chance to close files, cancel work, emit a final log, or remove temporary state. Sometimes that is the right tradeoff. Call it what it is: forced containment with possible incomplete cleanup.

macOS Activity Monitor can help when the command line is not enough. Apple documents a normal Quit action and a Force Quit action, and it can display processes hierarchically. The hierarchy view is useful for confirming parent and child relationships. It does not replace the incident record, because a GUI list does not preserve the session and action evidence you need later.

Make server shutdown a design requirement

Stdio servers should treat client disappearance as a first-class event, not an exceptional corner case. Client crashes are ordinary software behavior. Laptops sleep, terminals close, IDEs restart, updates interrupt processes, and agents can abort after a model error.

A server should have an explicit shutdown path with four properties. It stops accepting new requests when input closes. It tracks every started action with a correlation identifier. It gives active work a bounded chance to cancel or reach a known state. It exits after that bound rather than becoming a permanent background service by accident.

Do not confuse graceful shutdown with waiting forever. A server that receives EOF and waits indefinitely for an external API becomes an orphan with better manners. Put a deadline around cleanup, log what remains unresolved, and exit. The unresolved work must be discoverable through its external job identifier or action record.

Child management matters just as much. If a tool launches a compiler, package manager, SSH helper, browser driver, or command wrapper, the server must know whether that child should die when the server dies. Set up process groups deliberately. Capture child PIDs. On shutdown, terminate only children that belong to the request, then record whether they exited.

Avoid backgrounding work through a shell unless the tool contract explicitly calls for durable background work. A command such as some-command & creates a second lifetime that your MCP server may not observe. If you need durable work, submit it to a job system that returns a job ID, then expose status and cancellation through deliberate tools. Hidden background work is not durability. It is an audit gap.

Use idempotency where the remote service supports it. Give each external write a request identifier derived from the session and tool call, then record that identifier before sending the request. When the client crashes after submission, you can query the remote system instead of guessing whether to retry. When it does not support idempotency, document the ambiguity and require an operator decision before replaying a write.

A server should also write diagnostics to stderr, never stdout. Stdout belongs to MCP's newline-delimited JSON-RPC stream. A stray debug line can corrupt the protocol, trigger a client failure, and create the very crash pattern you are trying to clean up. That detail sounds small until a production server prints one library warning at startup.

Put ownership in the action record, not in a shell history

Shell history is convenient until it is missing, truncated, shared, or written after the process dies. The action record must carry the ownership information while the work is happening.

For each call that can touch an external system, attach enough context to answer four questions later: which agent run asked for it, which local process submitted it, which authorization decision allowed it, and which external operation resulted. If any one is absent, a client crash leaves a hole that an investigator must fill with inference.

Do not overload code-signing identity to answer every ownership question. It tells you who signed the executable, which is useful for deciding whether a process deserves approval. It does not identify the individual run, the prompt that led to the call, or the remote request. Use code-signing authority for trust at session start, then use a session identifier and per-call records for operational traceability.

The same warning applies to command lines. A command can tell you that sp mcp or a server executable started. It cannot reliably tell you which tool call was last, whether a user approved it, or whether the remote system accepted it. Treat the process list as supporting evidence, not as the audit source.

When a crash occurs, correlate in this order:

1. Find the agent run that owned the client process at the crash time.
2. Find the server process or process group that the run launched.
3. Locate the final activity entries for that run and compare their timestamps with the process snapshot.
4. Reconcile any unfinished external operation using its request, transaction, or job identifier.
5. Record the revocation and termination events beside the final calls.

This sequence prevents a familiar mistake: finding a stale PID, killing it, and later assigning its last API call to the wrong agent run because two sessions used the same server command. Commands repeat. Session records should not.

If your environment cannot make that correlation today, add it before you grant autonomous tools write access. Read-only experimentation can survive imperfect observability. Production changes cannot.

A crash drill exposes the gaps while the stakes are low

Run a controlled crash drill for each MCP server that can make external changes. Do it in a test account or with a read-only action path, and capture the evidence as though you were handling a real incident.

Start a normal client session and issue one action with a known correlation identifier. While the server is active, terminate the client abruptly. Then inspect the server PID, its PPID and process group, its open descriptors, and the action journal. Revoke the session. Confirm that a subsequent protected call is denied. Finally, terminate the server if it did not exit on EOF and reconcile the external action.

The drill should produce answers, not a pass or fail sticker. You want to know whether stdin closure reaches the server, whether child processes linger, whether remote jobs have cancellation handles, whether your logs identify the owner, and whether revocation changes behavior immediately.

Pay attention to timing. A crash that lands before request dispatch behaves differently from one that lands after a remote service accepted the request. Repeat the drill at both points if the server exposes enough instrumentation. The ugly boundary between "sent" and "confirmed" is where duplicate writes and false assurances live.

Write down the cleanup expectation for each server. A good expectation is specific: after input closes, the server stops accepting calls, exits within its configured timeout, leaves no owned helper processes, and creates an activity record for every started external action. A weak expectation is "the client normally cleans it up." Normal behavior is not a control.

Once you have done this a few times, stale processes stop being mysterious. They become a defined failure mode with a process snapshot, an ownership trail, a revocation action, and a cleanup rule. That is the standard to aim for. The process may crash. Your ability to explain and contain its final calls should not.