# Secret broker errors need proof before retries A secret broker must report what its records prove, not what an exception suggests. If it cannot prove that a remote operation stayed unattempted, it must not describe the call as a preflight failure. Once request bytes or an SSH exec request may have reached the other side, the outcome can be unknown even when the local error says "timeout" or "connection reset." That distinction decides whether an agent creates a second payment, rotates a credential twice, deploys the same release again, or safely repeats work that never left the machine. A useful error contract therefore carries two independent facts: where the broker stopped and what it knows about remote execution. One field for "transient" cannot carry both. The design below applies to HTTP calls and SSH commands. It also assumes the broker owns a durable action record. Memory-only state can improve an error message, but it cannot justify a retry after the broker or caller restarts. ## A cause is not an outcome Every failure needs a phase, an outcome, and a retry disposition. The phase tells an operator where to look. The outcome tells the caller whether the remote side may have changed state. The retry disposition tells automation what it may do now, based on recorded evidence and operation semantics. Use five public phases: - `validation`: the action was rejected before the broker selected or used a secret. - `credential_injection`: the broker could not obtain, authorize, or attach the credential. - `connection_setup`: name resolution, routing, TCP, TLS, SSH transport, host verification, or remote authentication failed before dispatch. - `remote_execution`: the broker dispatched the operation and is waiting for, or received, the remote result. - `result_delivery`: the broker recorded the remote result but could not return it intact to the caller. Those phases are diagnostic, not a retry policy. A `connection_setup` error can prove no application request was sent, while a broken reused connection can leave the broker unsure whether the peer received a write. A `result_delivery` error may accompany a known remote success. Treating both as a generic network error destroys the one fact the caller needs. Use four outcome states: - `not_attempted`: durable evidence shows the remote operation did not cross the dispatch boundary. - `rejected`: the remote system returned a complete, authoritative refusal and did not claim success. - `committed`: the broker has a complete, authoritative result for the operation. - `unknown`: dispatch may have happened, but the broker lacks a complete result that settles the operation. `committed` does not mean success. A command that exits with status 23 or an HTTP request that receives a complete 500 response has a known result. The remote application ran far enough to give an answer. Calling that "failed to execute" invites a duplicate. Retry disposition should be equally explicit: `never`, `after_correction`, `backoff`, `same_idempotency_key`, `reconcile`, or `fetch_result`. The broker computes it from evidence, method semantics, remote guarantees, and the state of its result journal. Callers should never infer it from an error string. Do not collapse `rejected` and `committed` into a single `known` value in the public contract. A rejection may permit a corrected request, while a committed result tells the caller to consume or reconcile that result. They share certainty about the attempt, but they lead to different workflows. Asynchronous acceptance needs one more piece of evidence without another outcome enum. A complete HTTP 202 response is a committed result for the submission, not proof that the queued job finished. Record the job identifier and status resource supplied by the service, then track the job as a separate operation. Retrying the submission because the job remains pending can enqueue it twice. This separation sharpens a distinction that many SDKs blur. Error cause answers "what broke locally?" Remote outcome answers "what might already have happened?" A retry engine that reads only the cause is unsafe. ## Validation and injection fail before dispatch Validation errors are genuine preflight failures only while the broker has not opened a channel that can carry the action. Reject malformed targets, unsupported methods, missing action fields, oversized payloads, unknown credential references, and forbidden header overrides before resolving a host or accessing a secret. Record `phase=validation`, `outcome=not_attempted`, and usually `retry=after_correction`. Automatic retry does not help a deterministic validation failure. Repeating the same invalid payload wastes capacity and can hide an agent loop. Return a stable code such as `INVALID_TARGET`, `UNSUPPORTED_ACTION`, or `PAYLOAD_LIMIT`, plus a field path that the caller can fix. Do not return the rejected secret reference if it contains sensitive naming information. Credential injection is still preflight when the broker fails before it releases any remote request bytes. Vault locked, approval denied, credential absent, unsupported injection mode, and local key decryption failure belong here. The outcome remains `not_attempted`, but the retry advice differs. A locked vault may allow a retry after user action; an approval denial should normally be `never` for that invocation; a missing secret needs correction. Keep credential material out of the error and the action record. Record the credential identifier or a nonsecret version tag, the injection mode, and the decision that stopped the call. Logging a rendered `Authorization` header to prove injection happened defeats the broker. There is a subtle cutoff. If the broker constructs a complete HTTP request with the credential in a private buffer and fails before writing it, the remote action is still unattempted. If it hands that buffer to a transport API and the API returns a partial or ambiguous write, injection succeeded and dispatch may have begun. Classify the failure by the last proved boundary, not by the function whose stack frame caught it. Preflight also needs a configuration snapshot. If validation uses one route definition and dispatch later reads a modified definition, the evidence no longer describes the action that ran. Bind the normalized target, credential version, permitted injection mode, and request fingerprint to the invocation before authorization. If any bound value changes, create a new invocation rather than mutating the old record. This matters during human approval. An approval card may sit open while an agent or configuration reload changes the body, host, or selected secret. The broker must approve the fingerprint it will dispatch and verify that fingerprint again immediately before the write. A mismatch is `validation/not_attempted`, not an excuse to send the newer request under an older approval. ## Connection setup needs a precise endpoint A new connection that fails before an application channel exists normally proves `not_attempted`. DNS failure, route failure, refused TCP connection, TLS certificate rejection, SSH host-key rejection, and SSH user-authentication failure all happen before an HTTP request or SSH command can run. Record the exact completed stage so a caller can distinguish a bad hostname from rejected credentials without seeing a secret. The phrase "connection failed" is too broad for a pooled HTTP client. When a broker checks out an existing connection, it has already completed setup. A write can fail because the peer closed an idle socket. The operating system may report a broken pipe after some bytes reached the peer, or after the peer received the whole request but before the client observed the close. That belongs to `remote_execution` with `outcome=unknown`, unless the transport supplies stronger proof. Do not use "zero bytes acknowledged by the local write call" as proof that the remote received nothing. A buffered API can accept bytes locally before a later failure, and a failed write says little about what the peer already read. The useful dispatch boundary sits at the first handoff to a transport that can deliver application data. Crossing it changes the default outcome to `unknown`. Connection setup also ends at different points for HTTP and SSH. For HTTPS, finish DNS, TCP, TLS, certificate verification, and any proxy tunnel before dispatch. For SSH, finish transport negotiation, host verification, user authentication, session channel creation, and any required environment setup. None of those proves that a later command did or did not start, but failure within them can prove that this command was never requested. Redirects create a second dispatch boundary. A complete 307 or 308 settles the first HTTP exchange, but following it creates a new request to another target. Revalidate the destination, credential scope, and method before that request. Never forward an authorization credential across origins merely because a client library follows redirects automatically. An HTTP proxy adds another observer but does not remove uncertainty. A successful tunnel only proves the proxy opened a path. A forwarding proxy may return a complete error about its own attempt, and RFC 9209 can describe where forwarding failed, yet the origin outcome may still be unknown. Store the hop that produced the evidence and avoid presenting an intermediary's certainty about its response as certainty about the origin's effects. A broker may retry connection setup internally when each attempt has its own record and no attempt crossed dispatch. It should cap the attempts and expose them in one action record: ```json {"attempts":[{"n":1,"stage":"tcp_connect","outcome":"not_attempted","code":"ECONNREFUSED"},{"n":2,"stage":"tls_handshake","outcome":"not_attempted","code":"CERT_EXPIRED"}]} ``` The final error should not erase earlier evidence. It should say the operation remained unattempted across both attempts and needs correction. ## Dispatch changes the burden of proof The moment of dispatch must be an explicit, durable state transition. Before the broker makes the first transport handoff, append `dispatch_started` to the action record and flush it according to the durability promise of the system. If the process dies after the write but before recording that transition, a restart may incorrectly call the operation unattempted. Strict write-ahead recording costs latency, so teams are tempted to record after sending. That recommendation is popular because the normal path gets faster and the code looks simpler. It is wrong for non-idempotent actions. The rare crash lands in the exact gap where the broker must choose between losing work and duplicating it. Write-ahead state alone does not prove the remote received the request. It deliberately moves uncertainty in the safe direction. After `dispatch_started`, the outcome begins as `unknown`. A later authoritative response can change it to `rejected` or `committed`. The broker never moves it back to `not_attempted`. For HTTP, dispatch begins before the first request byte enters the connection. Track whether the broker sent headers, sent a complete body, received response headers, and received a complete response body. These markers help diagnosis, but `request_body_sent=true` still does not prove that the application processed the request. Likewise, `request_body_sent=false` does not prove the application did nothing; a server can reject or act on headers before reading the whole body. For SSH, dispatch begins before the `exec` channel request enters the authenticated transport. Use `want reply=true`. RFC 4254 says the server then responds with channel success or failure, but channel success only means it accepted the request to start the command. It does not mean the command finished or that its effects can be repeated. A cancellation after dispatch is not a preflight failure. If the caller times out and closes the channel, the remote process may keep running. Report `CALLER_CANCELLED` as the local cause and preserve `outcome=unknown` until a recorded remote result settles it. Cancellation describes the caller's interest, not remote state. Batch requests need an outcome per member. If the broker sends five changes in one HTTP request and receives a complete response that settles only four, the envelope cannot safely assign one retry value to the batch. Record the parent transport result plus five child outcomes. Retry only the child whose records and remote contract permit it, or reconcile the whole batch when the remote API makes the changes atomic. The same rule applies to a shell script sent through SSH. One exit status covers the script process, not necessarily each side effect it attempted. If callers need action-level retry decisions, give each operation its own remote identifier and result record rather than trying to infer progress from stdout. ## A complete remote result settles execution An authoritative, completely framed response turns uncertainty into a known result. For HTTP, record the final status, selected nonsecret headers, the complete body or a body digest, and framing completion. For SSH, record command acceptance, stdout and stderr completion status, exit status or exit signal when present, and channel closure. RFC 9112 requires a client to record an HTTP response as incomplete when the connection closes early or chunked decoding fails. That rule deserves a stricter application in a secret broker: never expose a partial body as a complete remote result, even if the first bytes contain plausible JSON. Return the partial content only in a clearly marked diagnostic field, or discard it if it could contain sensitive data. HTTP status alone does not define whether repeating the application action is safe. A complete 401 proves the server rejected those credentials for that request, so the broker can mark the result `rejected`; retrying with unchanged credentials is pointless. A complete 429 or 503 can allow `backoff` when the method is safe to repeat and the response supplies suitable timing. A complete 500 is known, but the application may have changed state before producing it. Do not convert every 5xx into permission to replay a POST. RFC 9110 defines idempotency by the intended effect of multiple identical requests, and it permits automatic retry of idempotent methods after a communication failure. The useful qualification is "known to be idempotent." Method names are evidence, not magic. A misdesigned GET endpoint that triggers a deployment is unsafe despite the method token; a properly implemented PUT can be repeatable even though it changes state. Informational HTTP responses do not settle the action. `100 Continue` permits the client to send a request body; it says nothing about the final application result. Other 1xx responses likewise leave the invocation in progress. Only a complete final response, or a stronger application receipt whose contract the broker understands, can move the outcome out of `unknown`. Response completeness and response authenticity belong together. A perfectly framed response from the wrong TLS identity, an untrusted SSH host, or an unexpected proxy is not authoritative evidence about the intended target. Identity verification normally finishes during connection setup, but resumed sessions and connection pools still need to bind the verified peer identity to the action record. RFC 9209 defines `http_response_incomplete` for an intermediary that received only part of the next hop's response. Its recommended 502 status is useful for HTTP compatibility, but `502` alone loses the outcome evidence. Keep the broker's structured `outcome=unknown` beside any mapped status code. SSH has a comparable trap. RFC 4254 recommends that a server return `exit-status`; it does not require one. If the channel closes after stdout without an exit status or exit signal, the broker knows the stream ended but not whether the command succeeded. Return `REMOTE_RESULT_INCOMPLETE` and choose `unknown`, unless the action contract defines a different authoritative completion marker. ## Result delivery must not repeat execution Result delivery starts only after the broker has durably stored a settled remote result. If serialization to the caller fails, the MCP pipe closes, or the caller process exits, the remote operation does not become unknown again. Report `phase=result_delivery`, preserve `outcome=committed` or `rejected`, and set `retry=fetch_result`. This phase needs an invocation identifier that the caller can use to retrieve the stored result. A repeated action submission is not result retrieval. Keep those operations separate so a generic client library cannot accidentally turn a broken response pipe into a second remote call. The order of writes matters: 1. Finish and validate the remote response. 2. Append the settled outcome and result digest to the durable record. 3. Commit the retrievable result under the invocation identifier. 4. Deliver the result to the caller. If step 4 fails, steps 2 and 3 prove what happened. If the broker delivers first and journals later, a crash can leave the caller with success while the audit record says unknown. That is an audit defect, even if it does not cause an immediate retry. Large or streaming results need the same rule. Store chunks with sequence numbers and a final completeness marker. A caller may resume delivery from the last verified chunk, but the broker must not label the result complete until it has the terminator, declared length, or protocol-specific closure that proves completion. Caller acknowledgement is useful for retention, not for remote outcome. Mark `RESULT_DELIVERED` only after the caller-facing protocol confirms a complete handoff. If no acknowledgement exists, keep the result available until the retention policy expires and treat repeated reads as reads. Never rerun the action to reconstruct a result that the broker chose not to retain. Result storage can fail after remote completion. If the broker has the complete response in memory but cannot commit it, it knows more than a plain `unknown` suggests, yet the evidence will not survive a crash. Return `phase=result_delivery`, include `outcome=committed` only if the durability contract allows the current record to support that claim, and force immediate operator attention. The right fix is reserved capacity and failure-tested storage, not a retry of the remote action. ## Idempotency is a remote contract An idempotency key makes an unknown outcome retryable only when the remote service promises to bind that key to one logical operation. Generating a UUID in the broker and logging it does nothing by itself. The remote endpoint must accept the key, compare the request fingerprint, retain the first settled result for an adequate period, and return that result for a repeat. The broker should store four facts before dispatch: idempotency key, request fingerprint, remote scope, and expiry or retention information when the service publishes it. On retry, it must reuse the same key and identical fingerprint. Reusing a key with a changed body should fail locally with `IDEMPOTENCY_MISMATCH`. Do not silently add an idempotency header to endpoints that have no declared semantics for it. Some services ignore unknown headers. Others scope keys by account or route. A retry policy needs configured, reviewed knowledge of the remote contract, not hope based on a header name. Retention windows are part of that contract. If a service forgets keys after a day, a retry after that point can create a new effect while looking identical to the broker. Store the earliest safe expiry, stop automatic retries before it, and reconcile thereafter. When the service publishes no retention guarantee, treat the key as useful only within a conservative, configured window. Concurrency can defeat a correct single-threaded design. Two workers may read the same unknown record and both decide to retry with the same key. A proper remote deduplication contract should collapse them, but the broker should still take a lease on the invocation, record the retry generation, and allow one active attempt. That reduces load and keeps the journal intelligible. There are three safe paths from `unknown`: - Repeat an operation whose semantics are known to be idempotent. - Repeat with the same idempotency key under a verified remote deduplication contract. - Reconcile by querying remote state with a stable operation identifier, then decide whether any new action is needed. Everything else stops for review. That can feel conservative when an agent is waiting, but duplicated side effects cost more than a visible pause. Conditional HTTP requests can strengthen the contract. `If-Match` with a known entity tag can make an update fail if the resource has changed, while `If-None-Match: *` can prevent creating a second resource at the same target. They do not solve every duplicate because the endpoint's resource model still matters, but they provide server-enforced evidence instead of client guesswork. SSH commands rarely offer a protocol-level idempotency key. Put repeatability into the command's application contract: create a deployment under a unique release identifier, write with an atomic compare, or run a query that can confirm the intended state. Never assume a shell command is safe because it returned no output. ## The error envelope should carry evidence A caller needs a stable machine contract and a short human message. Keep transport-library exceptions in an internal diagnostic field because their names change across platforms and expose implementation detail. The public envelope should look like this: ```json {"invocation_id":"act_01J...","error":{"code":"REMOTE_OUTCOME_UNKNOWN","phase":"remote_execution","outcome":"unknown","retry":"same_idempotency_key","message":"Connection closed before a complete response was recorded."},"evidence":{"dispatch_started":true,"request_complete":true,"response_headers_received":false,"response_complete":false,"idempotency":{"key":"req_01J...","scope":"payments.create","fingerprint":"sha256:8b1...","remote_contract":"configured"}}} ``` Keep `code`, `phase`, `outcome`, and `retry` as closed enums. Add new evidence fields without changing their meaning. Callers can branch on the enums and show `message` to a person. They should not parse the message. Evidence must say how the broker knows, not merely repeat the conclusion. Useful fields include attempt number, connection identifier, dispatch journal sequence, request fingerprint, protocol completion marker, remote request identifier, response digest, exit status, and result-record identifier. Omit secrets, full authorization headers, private keys, and unfiltered remote bodies. Persist transitions as append-only events, then project the current status: ```text ACTION_ACCEPTED PREFLIGHT_VALIDATED CREDENTIAL_AUTHORIZED DISPATCH_STARTED REQUEST_SENT REMOTE_RESPONSE_STARTED REMOTE_RESPONSE_COMPLETE RESULT_COMMITTED RESULT_DELIVERED ``` An action that ends after `PREFLIGHT_VALIDATED` is provably unattempted. An action that ends after `DISPATCH_STARTED` but before an authoritative completion remains unknown. An action with `RESULT_COMMITTED` can survive failed delivery without another remote execution. The projection must reject impossible regressions. `unknown` may become `rejected` or `committed` when late evidence arrives, but `committed` cannot become `not_attempted`. A second observer can attach a reconciliation result, yet it must not rewrite the original attempt as if dispatch never occurred. Record monotonic sequence numbers rather than depending on wall-clock order. Clocks can move, and events from concurrent components can arrive late. Timestamps help operators correlate systems, but the journal sequence defines which durable transition happened first. Attach a source and local sequence to imported remote evidence instead of splicing it into history. Evidence also needs a declared confidence source. `transport_observed`, `remote_response`, `remote_query`, and `operator_attested` tell later code why an outcome changed. An operator may legitimately settle an unknown attempt after checking the remote system, but that fact should not masquerade as a response the broker received during the original connection. Audit integrity and outcome evidence solve different problems. A hash chain can prove that recorded events were not altered after the fact; it cannot prove the broker recorded every event or that the remote application honored a request. Sallyport's write-blind, hash-chained audit log and separate session and activity views provide the durable place to retain these transitions, while the action result still needs the phase and outcome contract described here. ## Retry code should be boring and testable The retry engine should consume the disposition already derived from evidence. It may add rate limits and attempt caps, but it must not upgrade an unsafe disposition because an exception looks temporary. ```text decide(record, operation): if record.retry == "fetch_result": return FETCH(record.invocation_id) if record.outcome == "not_attempted": if record.retry == "backoff": return RETRY_NEW_ATTEMPT return STOP_FOR_CORRECTION if record.outcome == "unknown": if operation.idempotent: return RETRY_NEW_ATTEMPT if record.retry == "same_idempotency_key" and operation.fingerprint == record.fingerprint: return RETRY_SAME_KEY return RECONCILE if record.outcome == "rejected" and record.retry == "backoff": return RETRY_WHEN_ALLOWED return RETURN_RECORDED_RESULT ``` Test transitions, not exception classes. Inject a failure before credential access, during TLS, before the first request write, after a complete request write, halfway through response headers, halfway through a framed body, after result commit, and during caller delivery. Kill the broker between every pair of durable transitions and verify that recovery never claims `not_attempted` after `DISPATCH_STARTED`. Add adversarial remote behaviors. Have a server apply the side effect and close without a response. Have it return 500 after committing. Have it honor an idempotency key, ignore one, and reject a reused key with a changed payload. For SSH, close after accepting `exec`, omit `exit-status`, and send an exit status before a broken output stream. The expected outcome should follow the evidence every time. Metrics should count phase and outcome separately. A spike in `connection_setup/not_attempted` points toward routing, certificates, or authentication. A spike in `remote_execution/unknown` demands reconciliation and may expose a remote reliability problem. Combining them as "broker failures" hides both the operational cause and the duplicate-action risk. Do not let a friendly SDK erase the contract. If it must throw exceptions, attach the complete envelope and make automatic retry opt in only for `backoff` or `same_idempotency_key`. A caller should have to write conspicuously unsafe code to repeat an `unknown/reconcile` action. Recovery tests should include competing workers and stale leases. Pause one worker after it acquires a retry lease, let the lease expire, and start another. When the first resumes, its generation check must stop it before dispatch. Without that check, a perfectly classified outcome can still produce concurrent duplicates. Treat retry budgets as part of the action record, not process-local counters. Restarts must not reset the attempt count or the remote key's expiry. Once the budget ends, return the last evidence and require reconciliation; changing the error code to a generic "max retries exceeded" would discard the safer diagnosis. The hardest state is supposed to remain uncomfortable. When the record says dispatch began and no authoritative completion arrived, the broker does not know the remote outcome. Preserve that fact, reconcile it, and refuse to turn missing evidence into permission.