Timing side-channel tests should treat denial as a result

An agent does not need a secret value to learn something dangerous. If it can repeatedly call an action gateway and sort outcomes by duration, it may learn whether the vault is locked, whether a credential name exists, whether a request passed authorization, or whether the gateway reached a remote service.

That is enough information to change its behavior. A coding agent that learns a credential exists can keep asking for access to that service. A compromised agent can use the result as reconnaissance. The leak may look harmless in a single call. It becomes useful when the caller controls inputs, repeats requests, and can measure a clock precisely.

MITRE records this class as CWE-208, Observable Timing Discrepancy. Its description is appropriately broad: security-relevant internal state can escape when operations take observably different amounts of time. The usual example is password checking. An agent gateway has a less familiar version of the same problem: the caller is already inside the developer's workflow and may make many structured tool calls without tiring out.

The fix is not to make every action take exactly the same time. That promise is usually false, especially once an HTTP request or SSH connection leaves the machine. The job is narrower and more practical: identify states that an agent should not be able to distinguish, make their local handling comparable, and keep a differential test suite that catches new fast paths.

Treat vault state as data the caller must not infer

A locked vault, a missing credential, a denied action, and a bad remote credential are different operational conditions. They are not automatically different facts the agent needs to learn.

Start by writing down what each caller is allowed to know. This sounds obvious, but teams often skip it and let implementation order decide the disclosure policy. An early vault lookup exposes credential existence. An early authorization check exposes whether a process has an approved session. An immediate local error exposes that the vault is locked. Each shortcut may be reasonable in isolation. Together they give the caller a state probe.

For an action gateway, separate these questions:

• Can this process ask for an action at all?
• Is the vault currently available to execute it?
• Does a credential mapping exist for this action?
• Did the remote service accept the credential?
• Can the caller see any of those answers directly, or only receive an action result?

The important distinction is between authorization state and credential state. Authorization answers whether a particular agent run may invoke an action. Credential state answers whether the gateway has material to execute that action. If you collapse those internally, a timing result can accidentally tell an agent that a credential exists merely because the gateway checked for it after approval.

Sallyport's vault gate is intentionally absolute: while the vault is locked, every action is denied. That is a clear security boundary, but it still deserves timing tests because a caller can observe how the denial is produced. The goal is not to pretend a locked vault can complete a remote request. The goal is to ensure that local denial handling does not create an easy, repeatable signature that reveals more than the caller needs.

OWASP makes the same point in its Authentication Cheat Sheet. Generic error text alone does not close an enumeration leak if one failure path does more work than another. The guidance uses authentication, but the engineering lesson carries over: a uniform message wrapped around different execution paths still leaks through elapsed time.

Four outcomes need two different timing contracts

Trying to equalize every outcome is a popular recommendation because it is simple to say. It is also wrong once a gateway can call networks. You need two timing contracts, not one impossible universal duration.

The first contract covers local outcomes. These are states where the gateway should refuse before it sends a request: the vault is locked, session authorization is absent or denied, an action has no configured credential, or a per-call approval is declined. For states that are meant to reveal the same amount of information to the agent, run them through a common local response envelope.

The second contract covers dispatched outcomes. A valid credential and an invalid credential can both reach the same HTTP test server or SSH test host. Their total times include transport, connection reuse, server work, and response delivery. You cannot make the public internet constant-time. You can make sure the gateway injects, sends, records, and maps both outcomes through comparable paths, then use a controlled server to keep the remote part stable during the test.

This produces a useful matrix:

Do not compare a missing credential directly with a successful call to a third-party API and declare failure because one is faster. That test only proves that a request which never left the machine is faster than an internet request. You already knew that.

Instead, test the boundaries separately. A local caller should have trouble separating the local states you intend to hide. A dispatched caller should have trouble separating a good test credential from a bad test credential based on gateway overhead. The test server can intentionally return distinct status codes after the same delay, so the semantic result stays different while the timing experiment stays useful.

There is one hard limit: if a caller receives a detailed error saying "credential missing," it no longer needs timing to know the credential is missing. Timing defenses cannot rescue an API that freely reports the secret state. Decide the allowed error semantics first.

A fast failure becomes an oracle when callers can repeat it

A timing leak rarely arrives as a dramatic difference. More often, someone adds a sensible early return during a refactor.

Picture a gateway with this order of operations:

1. Parse the agent's request.
2. Look up the named credential in the vault index.
3. Check whether the agent process has session approval.
4. Build and dispatch the request.

A missing credential exits at step two. An existing credential with an unapproved process continues to step three. The external response could be identical in both cases: "action unavailable." Yet the second path has a vault-index hit, an authorization lookup, audit preparation, and perhaps an approval-card setup. A caller can run each input many times and rank the results.

The leak gets worse when the agent can choose credential aliases. It can probe a wordlist of names such as staging, production, deploy, or the names it found in a repository. It does not need a successful action. It needs only a fast and a slow bucket.

The defensive order depends on your disclosure policy, but a safer shape looks like this:

1. Validate the request without secret-dependent branches.
2. Apply the absolute vault gate.
3. Apply process or session authorization.
4. Resolve the action and credential only after the caller has passed the controls that should precede that knowledge.
5. Put local failures that must remain indistinguishable through the same logging, error shaping, and response completion work.

Do not read this as an instruction to perform secret work for an unauthorized process. You should not decrypt a credential or construct a real authorization header merely to waste time. The point is to avoid state-dependent shortcuts at the observable boundary, not to turn denials into secret access.

A common bad patch is sleep(100ms) before every denial. It makes a demo graph prettier, then creates three problems. The duration often differs after scheduler noise and cache behavior. The delay burdens legitimate users. Most importantly, a repeated caller can average random or fixed padding away if the underlying paths still differ. Equalizing required work is more dependable than sprinkling delay on top of unequal work.

Measure at the boundary the agent actually sees

Instrumentation inside the gateway is useful for diagnosis, but it is not the primary security measurement. The agent observes the time between sending a tool call and receiving its final result. That is where your end-to-end differential test starts and stops.

Use a dedicated test driver that behaves like an agent client. It should start with a known process identity, send one request, wait for one complete response, record a monotonic duration, and store the outcome label outside the measured request. Do not print labels during the run. Console output, tracing exporters, and debug logging can distort short local paths enough to hide the regression you are looking for.

A practical sample record needs more than one number:

{
  "case": "vault_locked",
  "sequence": 184,
  "elapsed_us": 12746,
  "result_class": "local_denial",
  "connection_mode": "fresh",
  "run_id": "test-run-7"
}

Use a monotonic clock. Wall clocks jump when time synchronization runs and are poor instruments for subsecond comparisons. Record microseconds or nanoseconds if the platform offers them, then report milliseconds when people read the results. Precision in the file does not mean accuracy in the measurement, but it prevents you from throwing information away before analysis.

Run a warm-up phase before collecting samples. The first calls may start the app, load code, initialize a vault handle, create a connection pool, or populate a cache. Those effects are operationally real, yet they often swamp the steady-state difference you need to inspect. Test cold start separately if a cold start is visible to agents. Do not quietly discard inconvenient first-call behavior because it makes charts uglier.

Randomize the order of cases. If you run 500 locked-vault calls and then 500 missing-credential calls, thermal state, garbage collection, background activity, and connection reuse become confounders. Interleave the cases using a seeded shuffle so failures can be reproduced.

Also alternate fresh and reused connections when the client protocol supports both. A leak that disappears with a warm connection can still matter if an agent creates short-lived clients. A leak that appears only with reuse can expose a cache keyed by credential existence.

Build a controlled upstream that separates semantics from delay

Invalid credentials are the test case teams tend to get wrong. They point their test harness at a real service, send a deliberately bad token, and compare it with a successful call. The provider's rate limits, regional routing, TLS session resumption, and abuse controls become part of the result. That is not a gateway timing test.

Build a small fixture server under your control. It should read a known test credential from the same header shape your gateway injects, wait for a fixed amount of work, and return a different response body and status for good versus bad credentials. It must not shortcut the invalid path.

For example, this fixture contract is enough:

Request header: Authorization: Bearer test-good
Response: 200 {"fixture":"accepted"}

Request header: Authorization: Bearer test-bad
Response: 401 {"fixture":"rejected"}

Both requests: wait until the same server-side target duration has elapsed

The target duration should be longer than ordinary local scheduling jitter but short enough to keep the suite quick. Choose it from measurements on your own test machine rather than copying a number from a blog post. What matters is that both fixture branches run the same parse, clock, wait, and response path before they differ semantically.

For HTTP, run the fixture on loopback when you want to isolate the gateway and client. Run it on a controlled remote host in a second job when you want to see how normal network noise affects detection. Keep those reports separate. A loopback result tells you whether local implementation changed. A remote result tells you whether your test remains sensitive under realistic transport variation.

For SSH, use a test account and a command that exits in a known way after authentication. Do not test by repeatedly failing against a production bastion. SSH servers may deliberately slow failures, lock accounts, or add logging work. Those are sensible defenses, but they make your measurement a story about the server rather than the gateway.

Sallyport sends HTTP requests with bearer, basic, or custom-header credential injection, and uses its bundled stateless sp-ssh helper for SSH. Those are separate channels and need separate fixtures. A result that looks even for an HTTP header path says nothing about an SSH setup path that resolves keys, starts a helper, and negotiates a connection.

Compare distributions, then try to classify the state

Averages conceal timing leaks. If 90 percent of calls take 12 milliseconds and 10 percent take 80 milliseconds because a cache misses only for an existing credential, the average may look close enough to another case while an agent can exploit the fast cluster.

For each case, retain the complete set of samples. Report at least the median, lower and upper percentiles, minimum, maximum, and sample count. Plotting a histogram is often more revealing than a table because it exposes two clusters immediately.

Then make the test adversarial. Give a deliberately simple classifier only elapsed duration and ask it to guess the hidden label. Start with a threshold classifier. It selects one cutoff, for example "under X microseconds means missing credential," and reports accuracy on data it did not use to choose that cutoff. If a one-threshold classifier repeatedly beats the naive baseline by a wide margin, you have an observable signal worth investigating.

A small Python analysis script can make this regression visible without pretending it proves cryptographic constant time:

from statistics import median

samples = {
    "vault_locked": [...],
    "credential_missing": [...],
}

for name, values in samples.items():
    ordered = sorted(values)
    p10 = ordered[int(len(ordered) * 0.10)]
    p90 = ordered[int(len(ordered) * 0.90)]
    print(name, {"n": len(values), "p10": p10,
                 "median": median(values), "p90": p90})

best = None
all_values = sorted(set(samples["vault_locked"] + samples["credential_missing"]))
for cutoff in all_values:
    correct = 0
    total = 0
    for label, values in samples.items():
        for value in values:
            guess = "vault_locked" if value <= cutoff else "credential_missing"
            correct += (guess == label)
            total += 1
    score = correct / total
    if best is None or score > best[0]:
        best = (score, cutoff)

print({"best_training_accuracy": best[0], "cutoff_us": best[1]})

Split samples into training and holdout groups before selecting the cutoff. Otherwise the script will overfit noise and congratulate itself. If you add a richer classifier, keep it as a secondary diagnostic. A complicated model can find tiny patterns that no practical agent can use, while a plain threshold exposes the embarrassing leaks engineers actually introduce.

Do not set a universal pass condition such as "all medians must be within five milliseconds." That threshold has no meaning across machines and test setups. Use a baseline recorded in your controlled environment, inspect whether distributions overlap as expected, and fail when a change creates stable separability between states you intended to hide.

Constant-time comparison solves a smaller problem than this one

Developers often reach for a constant-time equality function when they hear "timing attack." That function matters for comparing equal-length secrets, signatures, MACs, and tokens. It does not make an action gateway's full request path constant-time.

The Go crypto/subtle package provides ConstantTimeCompare for byte slices with equal contents. It is the right kind of primitive when the secret comparison itself needs data-independent behavior. But it cannot equalize a branch that returns before opening the vault, a branch that renders an approval card, or a branch that makes a network connection.

Use constant-time comparison where you compare sensitive, fixed-format values. Then review the surrounding control flow. A clean comparison inside a path that immediately returns for a missing credential still leaves an oracle about credential presence.

This is also why you should avoid calling the whole problem "constant time." For a desktop gateway that can make HTTP and SSH calls, end-to-end constant time is neither achievable nor necessary. The requirement is narrower: sensitive local states must not create a conveniently classifiable timing signal for the caller.

MITRE's CWE-208 examples include early exits during password checks and different work for existing versus nonexistent accounts. The pattern is the same even when there is no password comparison at all. The observable difference comes from the decision path, not from a single unsafe equality operator.

Approval flows need an explicit measurement boundary

Per-session approval and per-call approval introduce a human into the timeline. A whole-call duration will include the time it takes someone to notice a card, read the process identity, decide, authenticate with Touch ID, and click. That duration varies for reasons unrelated to vault state.

Do not attempt to hide it with padding. You will make the interaction worse and still fail to make people behave on a clock.

Instead, split the flow into automatic and human-controlled portions. The automatic portion starts when the agent sends the call and ends when the gateway has either produced a denial without prompting or presented an approval request. Measure that portion across states. The human-controlled portion begins at presentation and ends at approval, rejection, timeout, or process exit. Log it for operations, but do not use it as a timing side-channel equivalence target.

You also need a test mode that establishes a stable approval state before sampling dispatched calls. Otherwise your "valid credential" distribution includes an approval card once, then skips it for the rest of the session. That produces a huge first-sample outlier and masks more subtle differences.

A good suite has distinct cases for these questions:

• Does a locked vault deny before any approval-dependent credential lookup occurs?
• Does an unapproved process get the same pre-prompt treatment regardless of whether an action has a configured credential?
• After a session is approved, do valid and invalid test credentials take comparable gateway paths to the controlled upstream?
• Does a per-call approval rejection expose extra credential-dependent work before the prompt?

Sallyport identifies a new agent process by its code-signing authority in the approval card, and its per-session authorization lasts only while that process runs. The timing suite should create fresh agent processes deliberately when it tests the first-call behavior, then hold one known process alive when it tests an approved session. Mixing those modes makes the result impossible to interpret.

Audit work must not create a secret-dependent fast path

Audit logging often causes the last timing leak because it is treated as bookkeeping rather than part of the security boundary. A denied request may record only a terse event. A successful request may allocate a detailed record, hash it into a chain, and write it to encrypted storage. That difference may be legitimate if success is already observable. It is dangerous when two local denials disclose different amounts of hidden state.

Decide which fields each outcome can safely record, then make equivalent local failures perform equivalent audit work. You do not need to write fictional credentials or send dummy requests. You do need to avoid a branch where "missing credential" bypasses the journal while "locked vault" performs a heavier lookup and record write, if the agent should not distinguish them.

Keep audit verification out of the request path. Verification is an operator action with a different purpose and a different timing profile. Sallyport's audit chain can be verified offline over ciphertext with sp audit verify, without a vault key. That design keeps verification from requiring a secret read, but it does not remove the need to test request-time behavior around logging.

Add internal spans around request parsing, authorization, vault gating, credential resolution, audit append, dispatch, and response mapping. Do not expose these spans to the agent. When the end-to-end test finds a separation, compare aggregate span durations across cases to locate the newly divergent stage.

The mistake is to add detailed tracing only after a timing regression appears. Add the instrumentation early, guard it behind a test or local diagnostics setting, and ensure the normal request path does not emit synchronous logs whose cost depends on a credential or outcome.

Put differential tests beside the code that adds branches

A timing suite belongs in the same change review as authorization tests. The branches most likely to regress are ordinary maintenance changes: a missing-configuration shortcut, a cache for credential aliases, a new audit field, a better error message, or a refactor that moves vault lookup ahead of process approval.

Run a compact local suite on every change that affects vault access, authorization, action resolution, error mapping, transport setup, or audit writes. It can use loopback fixtures and a moderate sample count. Run a longer randomized suite in a quiet controlled environment before releases, with cold and warm modes recorded separately.

Have the test output something a reviewer can act on. This is better than a green check with an unexplained aggregate score:

case pair: vault_locked vs credential_missing
samples: 400 per case, randomized order
median: 12.7 ms vs 13.1 ms
p10-p90: 11.8-14.0 ms vs 11.9-14.3 ms
holdout threshold accuracy: 51.4%
result: within baseline

And this is better than burying the problem under a generic performance failure:

case pair: vault_locked vs credential_missing
samples: 400 per case, randomized order
median: 4.2 ms vs 19.6 ms
p10-p90: 3.9-4.7 ms vs 18.2-22.1 ms
holdout threshold accuracy: 99.1%
result: investigate credential lookup before authorization

The second report does not claim an attacker will always get 99.1 percent accuracy on a busy workstation. It tells the engineer that the local code has created a clean, repeatable separation. That is enough to block the change until the branch order or response envelope is fixed.

Do not hide the result with an arbitrary delay. Move secret-dependent work behind the correct gate, eliminate needless state-specific work, and retest from the agent's boundary. The useful outcome is not a perfectly flat graph. It is an agent that cannot turn a stopwatch into an inventory of what sits behind the vault.