# Do your agent secret leakage tests prove anything? An agent does not need to print a token to have received it. If a bearer token enters its environment, command line, tool payload, transcript, child process, or crash artifact, the boundary has already failed. A polite model response does not repair that mistake. The test you want is not "did the agent avoid repeating the secret?" It is "can we run real actions through a test gateway, collect every artifact the agent side can produce, and show that none contains a unique working credential?" That claim is narrow enough to test and strong enough to catch the failures that show up in production. Use disposable accounts, disposable hosts, and a new canary for each run. Keep the secret in the gateway configuration only. Then make the agent perform successful HTTP and SSH work, deliberately trigger failures, and inspect the agent side as if you expected a leak. That is the posture that finds the ugly paths. ## A clean answer is not evidence of a clean boundary An agent can receive a secret without ever placing it in natural language. The common leak is a tool implementation that tells the agent to call an API itself, then puts an `Authorization` value in tool input. Another is a subprocess launcher that inherits `API_TOKEN` because nobody replaced its environment before execution. A third is an SSH helper that writes a temporary identity file where the agent can read it. These are different failures, but they share one consequence: the agent process has material that lets it act outside the gateway. Once that happens, approval prompts and audit logs describe only part of the risk. The agent can copy the secret into a repository, send it to another service, or leave it in a transcript that somebody exports later. Keep two claims separate: - **Action isolation** means the agent asks for an operation and receives the operation's result, while another component injects the credential and performs the network or SSH exchange. - **Secret non delivery** means the credential and usable derivatives never enter the agent process or files that the agent can read. Teams often test the first claim and assume the second. That is how an agent can successfully call a test API through a gateway while still receiving the token in a debug field or inherited environment variable. Apple's Process documentation is blunt about inheritance: a subprocess gets its environment from the process that launches it unless the launcher changes it before launch. Its APIs also expose command arguments and environment data to the process itself. That is why a child process is not an innocent implementation detail in this test. It is another observation point. The test standard should read like this: > Given a fresh credential canary stored only in the gateway, an agent can complete specified HTTP and SSH actions, but no agent reachable process, transcript, log, failure output, or collected diagnostic artifact contains the canary or an identifiable encoding of it. Do not promise that a black box test proves a secret never occupied any byte of memory in the gateway. It cannot. It does establish the boundary users care about: the agent never receives a usable credential through the interfaces and artifacts it controls. ## A test credential needs a job and a fingerprint A test token should do one useful thing and nothing else. For HTTP, create a test account that can call a single endpoint, such as `GET /whoami` or `POST /echo-action`, and returns a harmless account identifier. For SSH, create a restricted account on a disposable host and allow a small command set that writes a server side event record. Do not use a token like `test-token` and then search for it. Short, predictable markers create false matches and make encoding checks pointless. Generate a distinct string for every test execution. Put a recognizable prefix in it, followed by random data, so a human can identify a failure without mistaking ordinary output for a secret. For example, a harness might make a canary shaped like this: ```text sallyport_probe_7M3jP4Fqk2rV9dN8xC5a ``` That string is a credential value, not an identifier printed to the agent. Keep a separate public label for logs, such as `run-2026-07-22-ssh-04`. The label may appear in transcripts. The canary must not. Use distinct canaries for each channel and each failure path. Reusing one HTTP token across every test turns a single leak into a mess of stale matches. It also makes it hard to tell whether a later artifact came from the current run or a previous failed cleanup. A practical fixture has four pieces: 1. An HTTP test account whose server returns a fixed nonsecret response after valid authentication. 2. An SSH test account whose forced command records an action identifier and returns a fixed message. 3. A gateway credential entry containing the fresh canary and no agent accessible copy. 4. A manifest outside the artifact directory that maps the run label to the canaries used in that run. The manifest is sensitive test material. Store it where the agent cannot read it, and delete the canaries after the run. The test account should reject those credentials after cleanup, even if a failure left a copy in a local archive. Sallyport lets the gateway hold test HTTP and SSH credentials while the agent receives action results rather than those credentials themselves. The server side record matters. It proves that authentication took place, which stops a bad test from passing because the action never ran. A response such as `authenticated action accepted for run-2026-07-22-ssh-04` gives the agent enough evidence of success without echoing its authentication input. ## Launch capture catches the leak before the first tool call Capture the agent's initial arguments and environment at the launch boundary. This test catches secrets passed by shell scripts, CI configuration, editor integrations, wrappers, and convenience launchers. It also catches a mistake that people make after adding a gateway: they leave the old `API_TOKEN` export in place because the new path appears to work. Start the agent through a wrapper you control. The wrapper writes an exact snapshot of its own argument vector and environment to a protected test directory, then replaces itself with the agent executable. Replacing the process matters because it records the values supplied to the actual agent launch, not a guessed reconstruction after several layers have started. This small Python wrapper is enough for a local test harness: ```python #!/usr/bin/env python3 import json import os import pathlib import sys out = pathlib.Path(os.environ["PROBE_LAUNCH_RECORD"]) out.parent.mkdir(parents=True, exist_ok=True) record = { "argv": sys.argv[1:], "environment": dict(os.environ), } out.write_text(json.dumps(record, sort_keys=True), encoding="utf-8") os.execvp(sys.argv[1], sys.argv[1:]) ``` Launch it with a deliberately sparse environment. Include only what the agent needs to find its executable, its temporary directory, the MCP endpoint or stdio shim, and the path where the wrapper writes its record. Do not inherit the developer's entire shell by habit. A full shell environment imports unrelated credentials, cloud configuration, package registry tokens, and old SSH settings that can make the test fail for the wrong reason. The expected launch record has this shape: ```json { "argv": ["agent-command", "run", "tests/agent-task.txt"], "environment": { "HOME": "/private/tmp/agent-home", "PATH": "/usr/bin:/bin", "PROBE_LAUNCH_RECORD": "/private/tmp/probe/launch.json" } } ``` The exact paths do not matter. The important result is that the record does not contain the HTTP or SSH canary, its base64 form, URL encoded form, or the name of a file holding a private key. Do not redact this record before the scanner sees it. Redaction belongs in reports intended for people. The raw record is the evidence. If a release test records only a cleaned version, it can prove that the redactor works while hiding the very leak you needed to catch. A launch snapshot has a limit: it tells you what the process started with. It cannot tell you whether a later tool call puts a secret into a child environment or a temporary file. That is why the next tests force the agent to create work after startup. ## Child processes are part of the agent boundary Agents frequently launch formatters, package managers, test runners, Git commands, SSH clients, and scripts. If the agent can launch a child, that child can write its environment and arguments to disk, return them as output, or pass them to a network request. Treat every child as agent reachable unless you have a hard technical reason not to. Give the agent a harmless task that runs a probe program after it completes a gateway action. The probe prints its own arguments and environment in a machine readable form. Because it is a child of the agent side runtime, it observes the values that the runtime propagates at that point. Use a probe that writes to a controlled directory rather than returning a giant environment dump in the model conversation. You are testing for a leak, not inviting one into the transcript. ```python #!/usr/bin/env python3 import json import os import pathlib import sys path = pathlib.Path(os.environ["PROBE_CHILD_RECORD"]) path.parent.mkdir(parents=True, exist_ok=True) path.write_text( json.dumps( {"argv": sys.argv, "environment": dict(os.environ)}, sort_keys=True, ), encoding="utf-8", ) print("child probe completed") ``` Ask the agent to perform an authenticated action first, then invoke the probe with a boring argument such as `after-http-action`. Run the same sequence after SSH. If the gateway implementation injects a credential into an environment variable for a helper and lets that helper become a descendant of the agent, this test will find it. Check argument paths separately. Developers know that environment variables leak, but command arguments are often worse because process inspection, shell history, error formatting, and diagnostic collectors can record them. Apple documents that a process can access its own arguments through `CommandLine.arguments`, and `ProcessInfo` exposes both arguments and environment. That makes secrets passed in argv immediately observable to code running inside the process. Do not accept an argument like `--token-file=/private/tmp/secret` merely because the token bytes are absent. The test should inspect that file path too. If the agent can read the file, it has the secret. If the file is readable only by a separate gateway process and never travels through agent controlled working directories, record that fact in the test setup. For SSH, inspect for more than private key text. Fail if you find an identity file path, an agent socket that exposes the test identity, a generated `known_hosts` record containing private material, or a command line that includes a password. A private key stored in a temporary file is still a private key, even when the agent only receives the path. ## Successful actions need hostile transcripts A happy path that returns `200 OK` proves almost nothing. It proves only that somebody made a request. Make the agent request a real action through the gateway, then preserve every transcript and tool trace produced on the agent side. The HTTP test should request an endpoint that confirms the authenticated test account without reflecting request headers. A useful response is a fixed object such as this: ```json { "account": "gateway-test-http", "accepted": true, "request_label": "run-2026-07-22-http-01" } ``` The agent can reason from that response. It does not need the bearer token, the authorization scheme, an injected header name, or a scrubbed token prefix. If the action interface returns a request object for debugging, make that a separate test target because it is a likely leak. A result field called `request_headers` is a design bug unless it is guaranteed to omit credential material before it crosses the agent boundary. For SSH, have the host accept a fixed command such as `report-status `. The server records the authenticated account, requested command, and label. It returns a response such as `status recorded`. The agent should not receive the private key, an SSH agent export, or a transcript of an authentication exchange. Save these artifacts from the agent side: - The original agent prompt and model transcript. - Raw MCP messages or equivalent action requests and responses. - Standard output and standard error from agent controlled commands. - Tool debug logs, retry logs, and structured event files. - Files written under the agent workspace, temporary directory, and configured cache directory. Collect them before a cleanup routine removes evidence. Then scan the exact bytes, not just text decoded as UTF 8. A secret can appear in JSON escapes, percent encoding, base64, a compressed trace, or a file where invalid bytes make a casual text search skip the match. Sallyport's `sp mcp` shim makes a good subject for this test because the test can exercise the same regular MCP path an agent uses while keeping the credentials in the app's vault. Do not mistake a redacted transcript for proof. A log line reading `Authorization: [REDACTED]` may be fine for operator output, but the raw event object that produced it could still contain the token. Capture before display formatting, then test the formatter separately. Those are two different obligations. ## Error handling is where secret isolation usually breaks A gateway can keep the success response clean and still leak a secret when something goes wrong. Error paths attract debug context, request reconstruction, exception chaining, and retry messages. Run them on purpose. Start with HTTP failures that occur at different points: 1. Make the test server return a nonsecret `401` after receiving a valid canary. The agent should learn that authentication failed, not the header value sent. 2. Make the server return `500` with a body that includes the public run label. The gateway may return a bounded error body, but it must not append request headers or a curl equivalent. 3. Close the connection after the gateway has prepared authentication. This catches low level exceptions that include request objects in their descriptions. 4. Return malformed JSON after successful authentication. Parsers often include the offending response or surrounding context in an exception. 5. Use a DNS name that does not resolve for one test route. This catches retry and endpoint diagnostic output. Then run SSH failures that occur before and after connection setup. Use a host with the wrong host identity, a remote command that exits with a nonzero status, and a forced command that returns a controlled error. Do not test a wrong private key by sending a test key back to the agent or by making the server log it. The agent only needs to see a classification such as `connection rejected` or `remote command failed` plus a safe request label. Test the locked state too. While the vault is locked, an action must fail before network authentication occurs. The response may say that authorization is unavailable. It must not contain a token placeholder, a path to credential storage, an SSH identity filename, or the number of characters in a secret. After unlock, repeat the action and require the server side success record. This pair catches implementations that build a credential request before checking the lock. A useful failure fixture has assertions on both sides: ```text Agent side: the canary is absent from every collected artifact. Gateway side: the attempted action has the expected safe error classification. Server side: the expected request occurred, or did not occur for a locked vault test. ``` That last assertion prevents false confidence. If a test expects a retry error but the gateway rejected the call earlier for a configuration mistake, it may pass the leak scan without exercising the dangerous code. Do not send raw exceptions across the boundary. Error objects should contain an action identifier, a safe category, a human useful message, and perhaps a retry hint. They should not serialize the request configuration that caused the exception. The desire for easy debugging makes full request dumps popular. It is still the wrong default when a credential injector owns the request. ## Crash artifacts deserve an intentional failure A crash is not a normal API result, which is why teams skip it. That is a mistake. A developer may attach a crash report to an issue, a support script may archive it, and a diagnostic system may collect related logs. If a secret lands there, you have built a delayed leak rather than a safe boundary. After a successful HTTP action and again after a successful SSH action, cause an agent side probe process to abort. Keep the crash target separate from the gateway. You want to test whether the agent side had inherited or recorded secret material, not whether deliberately crashing the credential holder exposes its private state. On macOS, collect the report through Console or the test environment's diagnostic collection path, then scan the unedited file. Apple describes crash reports as detailed records of application state and advises analyzing the entire operating system report. Its crash report documentation also notes that reports contain process and environment information such as process identity, path, parent process, timing, and thread state. Absence of the canary in a macOS crash report does not prove it was never present in memory. A standard crash report is not a complete memory dump. That limitation is not a reason to skip the test. It means you should state the result correctly: the report did not expose the canary, and the agent process did not receive it through the other tested channels. Also scan application logs and support archives made around the crash. Apple warns developers not to include privacy sensitive information in logs. Treat that as a requirement for your own fixture: if an exception printer records a request object, the crash test should fail even if the operating system crash report is clean. If you run related tests on Linux later, add core dump metadata and journal output to the artifact set. The `systemd-coredump` manual documents fields that can store a crashed process's command line and environment. A test suite that only scans the core file but ignores its metadata misses a straightforward leak path. ## Scan bytes, encodings, and split values A simple recursive grep is better than nothing, but it misses the forms that show up in JSON, URLs, stack traces, and binary bundles. Build a scanner that reads files as bytes and searches several deterministic transforms of each canary. At minimum, generate these needles for every canary: ```text raw bytes base64 text URL encoded text JSON escaped text hex text first half and second half separated by one newline ``` The split case catches log wrappers that fold long values. The encoded cases catch systems that serialize structured data before writing it. Do not scan only for a token prefix. A prefix test can pass if the token is truncated after enough characters to remain usable, and it can fail on an unrelated identifier. A compact scanner can report file path, transform name, and byte offset without printing the secret itself: ```python import base64 import json import pathlib import urllib.parse secret = bytes.fromhex("73616c6c79706f72745f70726f62655f5837") needles = { "raw": secret, "base64": base64.b64encode(secret), "url": urllib.parse.quote_from_bytes(secret).encode(), "json": json.dumps(secret.decode()).encode(), "hex": secret.hex().encode(), } for path in pathlib.Path("artifacts").rglob("*"): if not path.is_file(): continue data = path.read_bytes() for name, needle in needles.items(): offset = data.find(needle) if offset >= 0: raise SystemExit(f"secret match: {path} transform={name} offset={offset}") ``` The code intentionally reports an offset rather than the matching bytes. A test failure should not create a second leak in CI output. Store a tightly controlled forensic copy only if your incident process requires it, and keep it outside normal build logs. Scan archives after unpacking them into a protected temporary directory. Scan compressed data too when practical, because a raw byte scan cannot see a canary inside a compressed payload. If your telemetry client batches events, collect the batch before it leaves the test machine. It is useless to discover later that the scanner inspected only local files while an encoded token had already gone to a third party collector. Keep an allowlist for expected public labels, not for secrets. If a test starts failing because a field includes the test account name, decide whether that name itself grants access. Do not add broad exclusions until the scanner turns green. Every exclusion is a hole you will forget to revisit. ## The report must show both absence and action A good test report answers four questions without asking the reader to trust your interpretation. First, which gateway action succeeded or failed? Show the public run label, action type, and server side observation. Second, which artifacts did you collect? List the launch snapshot, child snapshot, transcript bundle, workspace tree, error outputs, and crash artifacts. Third, which secret transforms did the scanner search? Fourth, did any scanner read fail because of a permission problem, bad encoding assumption, or skipped archive? A skipped artifact is not a pass. Mark it as incomplete and fail the suite unless you have a documented reason that it is outside the agent boundary. This rule irritates people during CI setup. It also prevents the predictable situation where a test reports success because it silently could not open the directory that held the leak. Keep the gateway's action record separate from the agent artifact bundle. The gateway audit can prove that a credential protected action occurred. The agent bundle can prove what the agent received. Combining them into one convenient export makes an unnecessary new place for sensitive information to travel. For release checks, make the pass condition strict: ```text PASS only when the server confirms the intended action, all required artifacts were collected, and no raw or transformed canary appears in agent reachable material. ``` Then add negative controls. Run one deliberately broken fixture that passes the canary through an environment variable or a fake debug response. The scanner must fail it. A test that never demonstrates its ability to catch a known leak is theater. Run the suite whenever someone changes credential injection, process launch code, MCP transport, error formatting, logging, support collection, or SSH handling. Those changes look unrelated in a review, but they are where credentials escape. Keep a smaller version in ordinary integration tests and reserve the crash and archive collection for scheduled or release runs if they are expensive. The standard is not difficult to state: the gateway may use a secret to act, but the agent may not acquire that secret as data. If your test only watches what the agent says, it leaves too much untested. Make the agent act, make it fail, make it spawn a child, make it crash, and scan what remains.