Nested MCP requests: trace every external action safely

An agent can call a tool that calls another tool, which asks a helper to resolve an environment, which finally sends an HTTP request or runs an SSH command. If your records stop at the first tool name, you have a story about intent. You do not have an account of what happened outside the process.

Nested MCP requests need tracing that follows the causal chain all the way to each credentialed external action. That means recording more than an agent transcript and more than a conventional request log. You need a graph that can answer: which run caused this outbound call, what did the call resolve to, who approved it, and did it actually execute?

I have seen teams treat a neat tool trace as proof that their controls work. Then an incident asks which call altered a production resource, and the trace says only release_service. Friendly names are for operators. Audit evidence needs the concrete operation.

The external action is the record that counts

The last call in a nested chain often carries the consequence, so it needs its own durable record even when every earlier call already has a span. A tool that checks a branch name may be harmless. A later helper that uses the result to invoke a deployment endpoint is not the same event.

Separate three things that teams regularly collapse into one:

• A tool invocation is a request to execute a named capability with arguments.
• A trace span describes one unit of work and its place in a causal graph.
• An external action is a concrete operation that crosses a trust boundary, such as an HTTP request with injected credentials or an SSH command on a remote host.

Getting this wrong produces two opposite failures. Some teams create an audit record for every internal function call. Their journal fills with noise and nobody can identify the few calls that changed anything. Other teams log only a final success line. They lose the chain that explains which agent decision, lookup, and approval caused that call.

Give an external action a stable action_id. Connect it to the span that initiated it, but do not use the span ID as its identity. A span may cover local preparation, an HTTP attempt, a redirect, and response parsing. Those are useful details. The action record should remain legible after someone changes the implementation.

An action record must describe the destination after resolution. Recording environment=production or target=customer-api is not enough. Store the resolved host, port, protocol, request method or SSH command, and the credential reference that the executor selected. Store secret references, never secret values.

The distinction also changes how you assess success. A wrapper tool can return ok because it queued work. A lower layer may then fail before it opens a connection. Record an outcome for the wrapper span and an outcome for the external action separately. Those facts can disagree without either record being wrong.

MCP does not supply your whole call graph

MCP gives clients and servers a protocol for discovering and calling tools. It does not force every implementation to expose an internal call graph. A host can orchestrate several servers. A server can call local helpers. A tool can trigger a job that continues after the response. Your audit design must account for those shapes explicitly.

The Model Context Protocol specification describes tools/call as a request from the client to the server for a named tool and arguments. That is an interface contract, not a tracing contract. The protocol does not turn a local function call into an observable child event, and it does not define a universal parent field that every intermediary must preserve.

This matters when a team says that a tool "called another MCP tool." Sometimes that phrase means a genuine second protocol call. Sometimes it means one server invoked a library function with a similar name. Sometimes an agent host received a result, reasoned over it, and made a fresh call. The resulting graph may look similar, but its trust boundaries differ.

Treat these as distinct edge types in your records:

• protocol_call connects an MCP client request to a server tool invocation.
• local_call connects code inside one trusted process.
• delegated_job connects a request to work that another worker performs later.
• external_action connects a span to an HTTP or SSH operation.

Do not infer an edge type from a tool name. Record it where the handoff happens. That is where you know whether identity, credentials, and cancellation rules crossed into another process.

A delayed job deserves special care. If a tool queues a task and returns, keep the originating trace ID and root run ID with the job payload. When a worker later runs, create a new span for that worker execution and point it back to the original action plan. Do not pretend the worker remained inside the original request. Its execution time, identity, and authorization state may have changed.

Create identifiers at every trust boundary

A trace only helps if every participant can attach its work to the same causal chain without accepting forged history as fact. Generate your own identifiers when a request enters a component you control, and preserve upstream context as untrusted diagnostic input unless an authenticated peer supplied it.

The W3C Trace Context recommendation defines the traceparent header with a version, a 32 hexadecimal character trace ID, a 16 hexadecimal character parent ID, and flags. OpenTelemetry uses this format widely. Use it where HTTP or another transport can carry headers, because existing tracing tools understand it. Do not mistake compatibility for an audit model.

For MCP over stdio, there may be no HTTP header at all. Put equivalent context into your application envelope or maintain context in the process that dispatches the call. The mechanism matters less than two properties: each child operation must know its immediate parent, and the receiving component must record who handed it the context.

A minimal event shape can look like this:

{
  "event_id": "evt_01J8...",
  "time": "2025-03-08T14:32:11.214Z",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "00f067aa0ba902b7",
  "parent_span_id": "b7ad6b7169203331",
  "root_run_id": "run_8d43",
  "edge_type": "external_action",
  "actor": {
    "kind": "agent_process",
    "identity": "signed-process-identity"
  },
  "action": {
    "action_id": "act_5f17",
    "channel": "http",
    "method": "POST",
    "host": "deploy.internal.example",
    "path_template": "/v1/releases/{name}",
    "credential_ref": "ops-deploy"
  },
  "outcome": {
    "state": "sent",
    "http_status": 202
  }
}

The timestamp above is an example shape, not a recommended retention format. In a real system, use a clock format your log verifier can parse consistently. Keep raw endpoints and arguments out of broad telemetry exports when they contain tenant names, repository paths, or personal data. A template plus a separately protected forensic record often gives operators enough context without copying sensitive material into every dashboard.

A receiving service should not simply trust parent_span_id because an agent supplied it. Create a new local span, record the received value, and attach a field such as upstream_context_source=authenticated_mcp_client or upstream_context_source=unverified_input. That small distinction prevents a malicious caller from attaching its action to a benign run after the fact.

Record resolution, not only the tool arguments

The action that an agent asks for and the action the executor performs can differ after templates, defaults, aliases, redirects, environment lookup, and credential selection. Your records must preserve both sides, because the difference is often where an unsafe design hides.

Consider an agent call with these arguments:

{
  "tool": "publish_release",
  "arguments": {
    "environment": "prod",
    "release": "2025.03.08-rc2"
  }
}

A tool may translate prod into a base URL, select a credential, turn the release string into a request path, and add headers. The initial call does not prove the destination. A complete chain should expose a transition such as:

1. The agent invokes publish_release in run run_8d43.
2. The tool resolves prod to a specific allowed endpoint and selects credential reference ops-deploy.
3. The executor records act_5f17 immediately before it sends the request.
4. The executor records the response or transport error against that action.
5. The wrapper returns a result that cites act_5f17 without exposing credentials.

That sequence gives an investigator a route from an agent's request to a network operation. It also gives an approver something concrete to inspect before the executor sends anything.

Do not log complete authorization headers, cookies, private commands, or arbitrary request bodies as a substitute for good modeling. People do this under pressure because a raw dump fixes today's debugging problem. It creates tomorrow's credential leak. Capture a credential reference, a header name, a body digest, selected nonsecret fields, and an explicit redaction status instead.

SSH needs the same discipline. A record that says ssh deploy is too vague if the helper later expands a host alias, selects an identity, and constructs a remote command. Log the resolved host and port, the account name, the identity reference, a command template or command digest, and the exit status. If the command includes sensitive data, retain a protected forensic copy only when you have a clear reason and retention rule.

A retry is another attempt, not a footnote

Retries and fan-out turn a simple tree into a graph. If you force them into one span with a final success field, you erase the information that explains duplicate side effects and partial failure.

Use three IDs when an operation can retry: a trace ID for the overall run, an action ID for the intended logical operation, and an attempt ID for each actual send. Each attempt gets a span of its own. The action record then points to every attempt.

Suppose a tool sends a release request, times out after the remote service accepted it, and retries. The second request may create the same release twice if the remote endpoint lacks idempotency handling. A final 200 tells you very little. Your log should show that attempt one reached the network, ended in timeout locally, and attempt two received a response.

Use an idempotency token whenever the destination supports one. Derive it from the logical action ID, not from a transient span ID. That lets the remote service recognize a duplicate even when your executor restarts or your tracing library generates fresh spans.

trace_id=4bf92f... action_id=act_5f17 attempt=1 state=timeout bytes_sent=418
trace_id=4bf92f... action_id=act_5f17 attempt=2 state=completed http_status=200

bytes_sent helps distinguish a connection failure before request transmission from a timeout after the client wrote data, but it does not prove what the remote service committed. State that uncertainty in the record. Do not label the first attempt failed in a way that suggests the remote side did nothing.

Parallel work needs sibling spans, not a shared mutable trace field that workers overwrite. If one planning tool calls four environment checks, create four child spans and four separate results. If two branches lead to external actions, issue two action IDs. An operator must be able to revoke or investigate one branch without confusing it with its sibling.

Approval must attach to the resolved operation

A human approval is useful only when the reviewer can see the operation that will happen after resolution. Approving a broad label such as deploy gives a person almost nothing to judge, particularly when nested tools select the actual host and credential later.

Build approval input from the pending external action record: channel, resolved destination, operation shape, credential reference, process identity, and a concise impact description. Keep the trace and action IDs on the approval decision. When the executor sends the request, it must prove that it consumed the same approved action, not merely an approval from the same agent run.

Do not approve an entire chain just because its first tool looked harmless. A chain may begin with find_release and end with an SSH command that modifies a host. If the design permits later resolution to broaden what the chain can do, require a fresh decision at the outbound boundary.

This does not mean asking a human about every string concatenation inside a tool. It means putting the decision where a capability crosses out of the trusted execution path. That boundary gives people an intelligible prompt and gives your log a durable connection between consent and effect.

Sallyport takes this approach for its supported HTTP and SSH channels: it keeps credentials in its vault, executes the action itself, and can require approval for each use of a selected credential. The useful integration point is the resulting external action, not an agent's claim about what its nested helper intended.

Approval records also need expiry and binding rules. Bind a decision to the action ID, resolved target, credential reference, and argument digest. If any of those change between prompt and execution, discard the decision and ask again. A reusable approval token that follows the agent process into unrelated calls will eventually authorize something nobody read.

An audit journal needs order and verification

Conventional application logs help diagnose a failure, but ordinary logs rarely tell you whether someone removed the inconvenient line. For actions made by autonomous agents, preserve event order and make later alteration apparent.

A simple hash chain records each event's canonical serialized bytes, the previous event hash, and the new event hash. A verifier starts at the first retained record and recomputes every link. If an attacker changes, inserts, or removes a record in the middle, verification fails at the affected point.

Canonical serialization matters. If one process sorts JSON fields and another does not, equivalent records hash differently. Define field ordering, Unicode normalization, timestamp precision, absent-field handling, and byte encoding. Test those rules across every language that writes events. Most broken audit chains fail here, not in the hash function.

A verifier should produce results that an operator can act on:

$ audit verify journal.events
records_checked: 1842
first_sequence: 91001
last_sequence: 92842
chain: valid
signature: valid

When it finds damage, it should identify the first bad sequence and the expected versus observed predecessor hash. It should not attempt to silently repair the file. Repair destroys evidence about what went wrong.

Hash chains do not solve every threat. Someone who controls the signing material and the storage may rewrite a complete alternate history. Periodic signed checkpoints stored outside the writer's normal control reduce that risk. So do separate access paths for writing events and reading them. Be precise about the protection you have rather than calling any log immutable.

Keep the action event near the point of execution. A background collector that receives batches after the fact can lose the only record that proves an outbound call occurred. The executor should append a "planned" record before transmission and a completion record immediately after it receives an outcome. If it crashes between them, the incomplete pair tells you that the action may have escaped.

Test the trace against failures people actually cause

A trace design is credible only after it survives malformed context, lost workers, duplicate sends, and operator revocation. Happy-path demos hide exactly the edges that matter when an agent behaves unexpectedly.

Run a small test suite against a disposable endpoint or host. Make each test assert the audit graph, not only the tool response:

• Send a nested call with a forged parent ID and confirm the receiver labels that context as unverified.
• Force a timeout after request bytes leave the client, then retry and confirm that both attempts share an action ID.
• Queue work, restart the worker, and confirm that the resumed span links to the original run without pretending it ran continuously.
• Revoke authorization after planning but before execution and confirm that no external action record reaches sent.
• Make two sibling calls in parallel and confirm that neither branch adopts the other's parent span or result.

The third test catches a common lie in logs. Systems often report a single uninterrupted request even though a worker restarted hours later under a different process identity. That hides who actually performed the action. Record the worker's identity and start time as facts of the new span.

The fourth test catches another familiar failure. A tool obtains authorization before it resolves its final target, then uses that stale decision for the resolved call. Your test should alter the target or credential reference after approval and expect the executor to reject it.

Do not settle for screenshots of a trace viewer. Export the raw records, verify their chain, and write assertions over parent IDs, action IDs, destinations, authorization bindings, and outcomes. The viewer is a convenience. The event stream is the evidence.

Build the boundary first, then fill in the graph

Start at the code that performs HTTP and SSH operations. Make it accept explicit trace context, create an action record before it sends, create an outcome record afterward, and refuse to execute when authorization does not bind to the resolved operation.

Once that boundary behaves correctly, instrument the nested tool calls above it. You will know what information every parent must pass down because the executor will demand it. Working in the other direction often produces beautiful trees with no trustworthy leaf nodes.

Keep tool names, planning notes, and model reasoning separate from the external action journal. They can help someone understand why a run happened. They cannot substitute for a record of which remote system received a request, under which credential reference, after which approval, and with what result. That is the chain you will need when the agent's answer sounds plausible and the remote system says otherwise.