Why do MCP server startup failures look the same?

An MCP client can report that a server "failed to start" even after the operating system launched the process, the process read input, and the server had already contacted something outside the machine. That message is not a diagnosis. It is a bucket where launch failures, protocol failures, discovery failures, and sometimes tool failures get thrown together.

Treat startup as a sequence of evidence-bearing boundaries. If you cannot say which boundary the server crossed, you cannot tell an operator whether a retry is safe, whether a credential could have been used, or whether the client simply failed to render a healthy server. The fix is not a larger timeout. The fix is to separate the states and make each one observable.

A single red status hides four different failures

An operator needs four answers, in order: did the client start the configured command, did both sides complete MCP initialization, did the client receive a usable tool list, and did any code reach an external channel? Each answer proves something different.

A process can fail before it exists in the usual sense. The executable may be missing, the working directory may not exist, a package runner may fail before it invokes your code, or the child may exit immediately because a required environment variable is absent. Call this a launch failure. There is no MCP session, and your application code may not have run at all.

A process can also exist and still fail the protocol exchange. With stdio, the child process has stdin and stdout connected to the client. The server must read JSON-RPC from stdin and write only JSON-RPC messages to stdout. It then needs to answer the client's initialize request with a compatible protocol version and declared capabilities. The client follows with notifications/initialized. If that sequence does not complete, call it a handshake failure.

A successful handshake does not prove that the client learned about tools. A server may declare tool support but throw while registering tools, generate an invalid input schema, return a malformed tools/list result, or return an empty list because its own configuration disabled every tool. Call that a tool discovery failure. The client and server may both be healthy enough to exchange messages, yet there is nothing the agent can call.

Finally, a server may get through discovery and fail only when a tool runs. That is a tool execution failure. It belongs in a different incident record. If you collapse it into startup, someone will eventually retry a server that already sent an HTTP request or opened an SSH connection.

The Model Context Protocol documentation makes this separation visible even if many client interfaces do not. Its debugging guidance distinguishes process and configuration problems from protocol logging, and it warns that local stdio servers must keep stdout free of ordinary logs. The protocol's initialization lifecycle and the tools/list request are distinct exchanges. Keep that distinction in your own telemetry instead of accepting a client's generic label.

Launch failure ends before MCP exists

A launch failure means the client did not get a usable child process with a readable protocol stream. It does not mean the command "looked right" in a terminal.

Interactive shells conceal a lot. Your shell has a PATH, a current directory, language version managers, credentials, and dotfiles that a desktop application or agent subprocess may not inherit. A client might launch with / as its working directory on macOS. It might use a restricted environment. It may pass the command as an executable plus argument array rather than through a shell, which means shell aliases and redirection do nothing.

Capture the exact launch record before you try to reason about MCP:

run_id=run_01JX...
phase=launch
command=/usr/local/bin/node
argv=["/Users/dev/work/acme-mcp/dist/index.js"]
cwd=/
pid=84217
started_at=2026-07-22T14:03:12.417Z

Then capture one terminal event after the process exits or the handshake deadline expires:

run_id=run_01JX...
phase=launch
exit_code=1
signal=null
stderr=Error: ENOENT: no such file or directory, open './config.json'

That record settles a common argument quickly. The server did not "have an MCP problem." It assumed a relative path would resolve under the project directory, but the client launched it from /.

Use absolute paths for the executable, entry point, configuration files, and any files read during startup. The MCP debugging guide explicitly calls out undefined working directories for client-launched servers and recommends absolute paths. This is not stylistic caution. It removes a source of failures that only appear after someone installs the same configuration on another machine.

Do not declare launch successful merely because you received a PID. A PID tells you the kernel created a process. It says nothing about whether the program loaded, whether its stdout pipe is intact, or whether the process has already become a zombie waiting to be reaped.

A useful launch state machine is small:

not_requested
  -> spawn_requested
  -> spawned
  -> executable_ready
  -> handshake_pending

spawned means the parent received a child PID. executable_ready means the child wrote a deliberate, non-protocol readiness event to stderr after it loaded its configuration and installed its fatal-error handler. Do not send that event to stdout. In a stdio server, stdout is not a logging channel that happens to be nearby. It is the wire.

The readiness event should not claim that the server is connected to an API, database, or remote host. It should say only what it proves: the process reached its MCP transport setup. A line such as ready=true becomes harmful when teams quietly interpret it as "safe to call tools." Name the phase instead.

The handshake has a narrow definition

An MCP handshake failure starts after a usable process exists and ends before the initialization lifecycle completes. Do not call it a connection failure without checking the messages.

For a stdio transport, the first bytes on stdout matter. A plain startup banner can corrupt the stream before your server sees the request. So can a dependency that prints an update notice, a console.log, a Python print, a framework exception formatter, or a wrapper script that writes status text to stdout. The official MCP build and debugging guidance says the same thing plainly: for stdio servers, write logs to stderr, because stdout carries protocol messages.

This is the smallest useful trace for a healthy initialization:

{"direction":"in","id":1,"method":"initialize"}
{"direction":"out","id":1,"result":{"protocolVersion":"2025-06-18","capabilities":{"tools":{}},"serverInfo":{"name":"acme","version":"1.4.0"}}}
{"direction":"in","method":"notifications/initialized"}

The exact protocol version depends on the versions supported by the client and server. The important fact is that the server selected a version the client accepts, returned a valid result, and received the completion notification. Store a parsed event for each message, not raw credential-bearing payloads.

If your trace instead begins like this, the diagnosis changes:

stdout: Starting Acme MCP server
{"jsonrpc":"2.0","id":1,"method":"initialize",...}

The server may be perfectly capable of responding, but the client's JSON parser has already hit invalid input. A timeout after that point does not tell you that the server was slow. It tells you the transport was corrupted.

Another familiar failure looks healthier:

phase=handshake
initialize_received=true
initialize_response_sent=false
fatal_error=Cannot read properties of undefined (reading 'tools')

The child launched. It received the request. It failed while preparing a response. That is a server bug or an unhandled configuration assumption, not a bad client configuration.

Make the boundaries explicit in logs:

phase=handshake event=initialize_received run_id=run_01JX request_id=1
phase=handshake event=initialize_responded run_id=run_01JX request_id=1 protocol_version=2025-06-18
phase=handshake event=initialized_received run_id=run_01JX

If you write only connected=true, you erase the one bit of information that separates a response sent from an initialization lifecycle completed. Clients may close or restart after receiving the response but before sending the notification. That is operationally different from an initialization parser error.

Give handshake its own deadline. Start it when the process is spawned, or when the transport is ready if you can observe that moment. Stop it when notifications/initialized arrives. When it expires, report the last confirmed event, such as spawned_no_initialize, initialize_received_no_response, or response_sent_no_initialized. Those names let an operator inspect the right side first.

Tool discovery fails after the server is already reachable

A tool discovery failure means the client and server can speak MCP, but the client did not receive a usable answer to tools/list. It is often misreported as a startup failure because many clients discover tools immediately after initialization.

Do not assume that a blank tool panel proves an empty tools/list result. Some clients hide tools after schema validation fails. Some cache discovery results. Some request tools lazily, only when the agent begins a task. Others connect to an MCP server for resources or prompts and never ask for tools. Your evidence should capture both the request and the response.

A healthy discovery trace has this shape:

{"direction":"in","id":2,"method":"tools/list"}
{"direction":"out","id":2,"result":{"tools":[{"name":"issue_lookup","description":"Fetch one issue by identifier","inputSchema":{"type":"object","properties":{"id":{"type":"string"}},"required":["id"]}}]}}

A discovery record should include the number of tools and a digest of the normalized schemas. A digest helps you see whether two runs had the same advertised interface without storing sensitive descriptions or configuration. It also catches accidental changes where a tool still exists but its required parameters vanished.

Do not build tool definitions by contacting an external service during tools/list. That design turns discovery into a side effect, makes a client refresh look like an execution, and creates the worst possible incident question: "Did listing the tools change something?" Tool registration should be local and deterministic whenever possible.

A server may need configuration to decide whether it advertises a tool. Read that configuration at startup and record the result, but do not make tool discovery wait on a token refresh or an SSH probe. If a tool requires a credential, validate the presence of its local reference without using it. Defer the actual remote action until the client calls the tool.

This distinction matters for agent control. If an agent reaches Sallyport through sp mcp, a successful MCP discovery record shows only that the shim exposed callable operations, not that Sallyport performed an HTTP or SSH action.

There is one legitimate reason to return an empty list: the server has no enabled tools for the current configuration. Say that in a structured response or client-facing log. Do not crash during registration and leave the client to infer whether the tool set is intentionally empty.

phase=discovery event=tools_list_responded run_id=run_01JX tool_count=0 reason=no_enabled_tools

That record gives the operator a configuration problem to fix. A generic startup error gives them a superstition to repeat.

External reachability needs its own proof

The question "did the process reach an external channel?" cannot be answered by a PID, a successful handshake, or a populated tool list. You need an event at the boundary where your code attempts the external action.

Define external channel narrowly. For this purpose, it includes an outbound HTTP request, an SSH helper invocation, a database connection outside the local process, a cloud credential refresh, a message queue publish, or any call that can create an effect or reveal information outside the MCP session. Reading a local configuration file does not count. Loading a credential from a local protected store does not by itself count either. Sending it in a request does.

Record the attempt before the call starts, then record its outcome. Use an opaque action ID that can be joined to the MCP request ID and server run ID.

run_id=run_01JX phase=execution event=external_attempt action_id=act_8Qf tool=issue_lookup channel=https host=api.example.test
run_id=run_01JX phase=execution event=external_result action_id=act_8Qf status=200 duration_ms=184

Do not put authorization headers, bearer tokens, signed URLs, command arguments containing secrets, or full response bodies in those records. An incident log that leaks the credential it was meant to investigate makes the incident worse.

The placement of external_attempt is not academic. Put it too early and you will claim an external call happened when the code only assembled a request object. Put it too late and a timeout or process crash can leave a gap after bytes have already left the machine. Emit it immediately before the library call that can initiate network or SSH activity. If the library exposes a lower-level connection or request hook, record a second event there only if you can do so without confusing the meaning of "attempt."

A failure walkthrough shows why this matters. An operator adds an MCP server that reads an issue tracker token during module initialization and calls a "who am I" endpoint to validate it. The child process starts, writes a debug line to stdout, and corrupts the first MCP message. The client shows "server failed to start." The team restarts the client twice.

Without phase records, they conclude that no request left the machine because the server never appeared in the client UI. That conclusion is false. The module initialization call ran before the client sent initialize, and it contacted the issue tracker three times. The UI state said nothing about external reachability.

Move the identity check into a deliberate read-only tool, or make it part of the first action that genuinely needs the remote service. Then record it as tool execution. The server can launch, initialize, and list tools without touching the network. An operator can now distinguish "the server is available" from "the credential and remote service work." Those are separate facts and should remain separate.

A phase ledger turns vague incidents into testable claims

Build one ledger entry per run and append immutable phase events. You do not need a complicated rules engine. You need stable names, timestamps, and enough correlation fields to reconstruct what happened.

Use this shape:

{
  "run_id": "run_01JX",
  "server_name": "acme",
  "pid": 84217,
  "phase": "discovery",
  "event": "tools_list_responded",
  "request_id": 2,
  "tool_count": 4,
  "at": "2026-07-22T14:03:13.083Z"
}

The ledger should record events, not conclusions pasted into a string. phase=handshake and event=initialize_received can be counted, queried, and tested. message="MCP seems stuck" cannot.

Keep the state model deliberately boring:

1. spawn_requested, spawned, executable_ready, and exited belong to launch.
2. initialize_received, initialize_responded, and initialized_received belong to handshake.
3. tools_list_received and tools_list_responded belong to discovery.
4. tool_call_received, external_attempt, and external_result belong to execution.
5. revoked, terminated, and client_disconnected describe interruption, not success.

The distinction teams often blur is session establishment versus authority to act. A server can establish an MCP session without having permission to use a credential or connect to a remote system. If you treat those as the same state, an approval event can look like a connectivity event, and a denied action can look like a failed startup.

Keep authorization events beside the call they govern. For example, record authorization_requested and authorization_granted after tool_call_received but before external_attempt. Then an operator can say, with evidence, that the tool request arrived, the human denied it, and no external attempt occurred. That is much stronger than saying the request "did not complete."

Use a run ID that survives only for one child process. Do not reuse a server name as a correlation identifier. A client can start two copies of the same server, restart one after a timeout, and retain old tool metadata. Reused identifiers turn those separate attempts into a fabricated story.

Hash or redact values that reveal user data. You generally need a tool name, an endpoint host, a status class, an error class, and timing. You rarely need a query string, a request body, or a response. Operators need to prove the boundary crossed, not replay a user's data from logs.

Test the boundaries without trusting the full client

A full client is useful for integration testing, but it is a poor first witness. Its UI may compress errors, cache capabilities, restart children, and apply its own timeout. Test each boundary through a narrower path before you blame the server or the client.

Start with the exact command, environment, and working directory the client uses. Do not substitute npm run dev for the configured command. Do not run it from the project folder if the client launches from elsewhere. Redirect stderr to a file for inspection, but leave stdout untouched if another process will speak MCP through it.

For a stdio server, use the official MCP Inspector as the first protocol test. The MCP documentation recommends Inspector for testing servers across transports, and the Inspector project can launch a stdio command directly. It lets you observe the initialization exchange and invoke tools/list without guessing what a desktop client did with the result.

Then reduce the test to three checks:

1. Does the configured command remain alive long enough to receive initialize?
2. Does it return a valid initialize response and receive initialized?
3. Does tools/list return the expected tool names and schemas?

Only after those pass should you call a tool that reaches an external system. Pick a read-only action with a harmless target. Confirm that the execution ledger contains one external_attempt and one terminal result. If a call can mutate data, test it against a disposable environment or provide a dedicated dry-run operation that does not contact the production endpoint.

The official MCP Inspector repository is useful here because it makes the transport boundary visible. It is not a network interception proxy for your server traffic. It acts as an MCP client for the selected server and provides a browser interface for the test. That difference matters when you investigate transport corruption: Inspector can reproduce the client side of the protocol, but it cannot prove what a separate production client wrote to the pipe.

For HTTP transports, add HTTP evidence without confusing it with MCP state. Record the request method, endpoint path, status, session identifier when present, and whether the response contained JSON or began an event stream. A TCP connection or an HTTP 200 does not automatically mean the MCP initialization completed. Apply the same lifecycle records after the HTTP request reaches your server.

Keep a fixture server in your test suite that deliberately fails at each boundary. One fixture exits before reading input. Another writes hello to stdout before responding. A third answers initialize and then returns an invalid tool schema. A fourth lists one tool whose handler records an external attempt and returns a controlled error. If your client integration collapses all four into the same alert, fix the integration before a real server forces you to debug blind.

Timeouts and retries need phase ownership

A single startup timeout encourages the wrong repair. It lets a slow package download, an initialization parser error, a schema exception, and a remote API stall look identical. Use separate deadlines because each deadline belongs to a different owner.

The launcher owns the period from spawn_requested to spawned. The server and transport own the period from spawn to initialized_received. The server's registration path owns discovery. The tool handler and its remote dependency own execution. Name the timeout after its owner and emit the last confirmed phase event.

error=handshake_timeout last_event=initialize_received run_id=run_01JX

That is actionable. It tells the server maintainer to inspect response construction and stderr, not the remote API.

Automatic retry is safe only when the failed phase has no external effect. Retrying a failed spawn because the executable was temporarily unavailable may be acceptable. Retrying a discovery request is usually acceptable if discovery is local and pure. Retrying a tool call after external_attempt is dangerous unless the remote operation has a documented idempotency mechanism and you attach an idempotency value.

Do not hide retry behind a server restart. If startup code refreshes a token, creates a tunnel, sends a telemetry event, or validates a remote identity, a restart is already an external action. That is another reason to keep startup local and move remote work into explicit tools.

When a client kills a child after a deadline, emit an interruption event before termination if you can. The server may not flush it. The parent should record the kill request as its own event, including the last child event it observed. This leaves an honest record: the process may have been about to respond, but you do not claim it did.

Make startup boring before you make it fast

A good MCP server can start with no network, no credential use, no mutable side effect, and no ambiguity about its protocol state. It loads local configuration, installs its transport, replies to initialization, and advertises a deterministic interface. That behavior is easier to operate and safer to retry.

The most popular bad recommendation is to "verify everything on startup." It feels responsible because errors appear early. In practice, it mixes local configuration, identity, remote availability, and authorization into one opaque ritual. It also causes clients to retry external actions under a startup error label.

Validate what you can locally. Report remote reachability through an explicit tool or through the first operation that needs it. Keep the ledger phase names stable. Make stdout protocol-only for stdio. Test an intentionally broken server for every boundary you claim to observe.

When the next client says an MCP server failed to start, you should be able to answer four questions from one run record: whether the process launched, whether initialization completed, whether tools were discoverable, and whether anything reached the outside world. If you cannot answer all four, the status is still ambiguous.