Rotate credentials behind an action gateway with proof

Replacing an expiring API token or SSH identity should be boring. It becomes dangerous when every agent, shell profile, repository secret, and local tool owns its own copy. You do not rotate one credential in that setup. You hunt for an unknown number of copies, hope you found them all, and break work in places nobody thought to test.

An action gateway changes the job. The gateway owns the credential, performs the external request, and returns the result to the agent. Rotation then has one secret-bearing location, one controlled cutover, and evidence that separates an agent session from the identity it used. That separation matters most during an incident, when people otherwise confuse "this process sent the request" with "this process possessed the token."

I have seen teams call a rotation complete because a new value appeared in a secret manager. Then an old deploy key remained in an account's authorized keys file, or a forgotten local environment variable kept an expired API token alive. A rotation is complete only when the replacement has proven the intended action path and the retired identity cannot authenticate anymore.

Rotation is an identity change, not a string replacement

Credential rotation replaces one authenticating identity with another and proves that access moved as intended. Pasting a new token into a field is only one operation inside that change.

Keep four things distinct in your notes and your tooling:

• The external identity: an API token, client secret, SSH key pair, or service account credential accepted by a provider.
• The credential record: the encrypted entry that stores the identity and its use details.
• The authorization target: the API account, repository, machine account, host, or network endpoint that accepts it.
• The agent session: the particular process that asked for an action.

People routinely merge the first and fourth items. That mistake produces bad incident reports. An agent session may have called an endpoint through a gateway, but it did not need to read the bearer token to do so. Conversely, a leaked token may be used by a process that never appears in your agent journal. Those are different investigations with different containment steps.

The same distinction applies to expiry. A provider may expire a token at a stated time, while your gateway record remains perfectly healthy as stored data. The record does not expire by magic. You must replace the external identity before the provider rejects it, validate the new identity, and retire the old one.

NIST Special Publication 800-57, Part 1 treats cryptoperiods as more than calendar reminders. Its discussion ties a cryptoperiod to exposure, use, and the risk of compromise. That is the right instinct for API and SSH credentials too. A token that grants broad write access, gets used often, or has unclear ownership deserves a shorter planned life than a narrowly scoped identity for one read-only task. Do not turn that into a ritual that changes strings every Friday while preserving excessive scope.

A clean rotation statement should read like this: "The build agent's session used credential record deploy-api-prod for this request. That record changed from provider token ID ending in 4K2 to provider token ID ending in P9M. The old token was revoked after the new record completed the expected action." It should not include the token itself.

Put the secret in one place before its expiry date forces the issue

You cannot make rotation controlled while credentials still live in agent prompts, repository files, shell variables, and copied configuration. Consolidation comes first, even when the expiration date feels close.

Start with an inventory that follows use, not just storage. Ask each team which external calls and SSH destinations an agent can make. For every answer, record the provider account or host account, credential type, owner, intended action, scope, expiry behavior, and whether the credential appears anywhere else. The last field is where the unpleasant work begins.

A repository scan can find obvious mistakes, but it cannot prove absence. Look for environment variable names, configuration keys, deployment templates, copied private key paths, and documentation that tells people to paste a token into an agent configuration. Also inspect the agent's tool definitions. If a tool accepts token, api_key, authorization, or raw private key content as an argument, the agent can still carry secrets even if another system holds a duplicate.

For HTTP, the intended shape is simple: the agent supplies an action and ordinary request data, while the gateway selects the stored credential and injects the authorization material only when it sends the request. An agent tool call might conceptually contain this:

{
  "credential": "deploy-api-prod",
  "method": "POST",
  "url": "https://api.example.internal/releases",
  "headers": {"Content-Type": "application/json"},
  "body": {"revision": "a81c2f"}
}

It must not contain a bearer token, even under a friendly field name. Returning an authorization header in a tool result is the same failure in reverse. Redact it at the action boundary, not later in a chat transcript.

For SSH, the agent should request a connection through a named credential record and destination. It should not receive a private key blob, a temporary private key file, or an instruction to search ~/.ssh. A private key in an agent's working directory survives in caches, editor history, archives, and sometimes a commit. I have cleaned up enough of those files to consider "temporary" a word with no security meaning.

Sallyport keeps API keys and SSH keys in an encrypted vault inside the macOS app, then executes HTTP and SSH actions without exposing the secret to the agent. That arrangement gives you a single record to change, but it does not remove the need to find and remove older copies that existed before the move.

Give every credential one owner and one purpose

A credential can have more than one legitimate user, but it still needs one accountable owner and one stated purpose. Shared ownership is usually a polite name for nobody checking expiry, scope, or retirement.

Name records so an operator can identify the authorization boundary without seeing secret material. billing-write-prod says more than token-final-2. github-deploy-repo-a says more than automation-key. Include the environment when it affects the target, and avoid embedding a person's name when the credential belongs to a service function. People leave; service purpose should remain legible.

Do not put unrelated targets behind one shared token because it reduces the number of records. That shortcut is popular because a single renewal seems easy. It causes three failures:

1. You cannot tell which target forced the rotation.
2. A scope increase for one use broadens access for every other use.
3. Revoking the identity during an incident interrupts unrelated work.

A record can support a closely related set of actions against one provider account when the scope and owner are truly the same. Treat that as an explicit decision, not a default. If the build publisher and the support export job need different permissions, they need different identities even if both call the same API.

SSH needs the same discipline. An SSH key tied to a deployment account on one host group should not double as a key for production database access because both happen to use SSH. Put restrictions on the server side where possible: a dedicated account, a constrained command, a source restriction when your network supports it, and a clear authorized key comment. A comment does not enforce access, but it makes the surviving old key visible when someone reviews authorized_keys under pressure.

OpenSSH's authorized_keys manual documents options such as command=, restrict, and from=. Those options are useful only if you test the exact command path that automation requires. I have watched a well-meaning restrict entry break port forwarding that a deployment job quietly depended on. That is a good failure to find during planned rotation rather than at midnight. Do not add every available restriction blindly; add the restrictions that match the action you have named.

Use an overlap window that has a scheduled end

Keep old and new credentials active together only long enough to validate the replacement and recover from a failed cutover. Overlap is a safety device, not a permanent operating mode.

Some providers permit multiple API tokens or multiple active client credentials. Create the replacement first, record an identifier that is safe to retain, and load it into the gateway record. Then make a harmless action through the same route the agent will use in normal work. A read endpoint, a draft creation in a test project, or a request that retrieves the caller identity can work, provided that action checks the required scope and target account.

Do not rely on a generic "token valid" endpoint if the agent normally publishes releases or modifies tickets. A token can be valid but lack the write scope, point at a sandbox account, or fail because the gateway injects the wrong header type. Test the actual method, URL family, account, and payload shape with a harmless object.

Set the old credential's revocation time before you test. If you cannot name that time, you have no overlap window. You have created another permanent credential.

A practical sequence looks like this:

1. Create the new provider credential with the required scope and a known expiration policy.
2. Update the one gateway record, retaining the previous credential only for the declared overlap.
3. Run one narrow action through an authorized agent session or an operator-controlled test session.
4. Verify the result at the provider and inspect the action journal for the expected session and destination.
5. Revoke or remove the old provider credential, then repeat the narrow action once more.

The final test after revocation is not ceremony. It catches the embarrassing case where your test silently used the old token because an environment variable, a proxy configuration, or a different credential record took precedence. That failure happens more often than people admit.

For a provider that allows only one active API credential, you cannot have a true overlap. Schedule a change window, capture a baseline action before the switch, replace the secret in the gateway, run the narrow test immediately, and keep an account owner available to issue a replacement if the provider rejects it. Do not "solve" the lack of overlap by placing the new token in agent configuration ahead of time.

Validate the path an agent actually takes

A rotation test must exercise the gateway, credential selection, request construction, remote authorization, and result handling. Testing each component alone leaves gaps exactly where credential mistakes hide.

For HTTP, make the test request sufficiently specific that you can recognize it in provider logs. Use an idempotency token where the API supports one, or create a clearly labeled disposable object. Check that the provider reports the intended account or principal. Then confirm that the action journal has a matching call, target, outcome, and session reference without printing credentials.

A generic shell test is useful for isolating provider behavior, but it proves less than people think:

curl -sS -D /tmp/headers.txt \
  -H "Authorization: Bearer $NEW_TOKEN" \
  https://api.example.internal/v1/whoami

Expected output often has a 200 status in /tmp/headers.txt and a body that identifies the service account. That tells you the provider accepts the token. It does not prove that your action gateway injects the same header, chooses the correct record, or prevents the agent from seeing $NEW_TOKEN. Use this only under an operator's control and remove the variable from the shell when finished. Never paste a real token into a ticket or a saved terminal recording.

For SSH, test against the same hostname, user, and command shape that automation needs. The following public-key check helps diagnose server authorization without revealing private material in output:

ssh -o BatchMode=yes -o IdentitiesOnly=yes \
  [email protected] 'id && test -w /srv/releases && echo write-ok'

BatchMode=yes makes an authentication failure fail instead of waiting for an interactive password. IdentitiesOnly=yes prevents an SSH client from trying every unrelated key loaded in an agent. Do not take a successful ssh deploy@host login as proof that a deployment action works. The remote account might accept a shell while rejecting the command, directory, or forced-command restriction used by the job.

The OpenSSH ssh manual explains that IdentitiesOnly limits identities offered by the client. That option catches a common false green result on developer machines: the local agent succeeds with a personal key in an authentication agent, while the new automation key would fail in production. In a gateway design, perform the equivalent check through the gateway's SSH path, where the selected stored key is unambiguous.

Test failure as well as success. Deliberately try a request with an unauthorized method or an SSH command outside the intended account's permission. You want a clear denial at the remote system and an accurate journal entry. If a supposedly narrow credential succeeds at an unrelated action, stop the rotation and reduce its scope before you retire the old identity.

SSH rotation fails on the server when old public keys survive

Replacing an SSH private key in a vault does not revoke the old key. The server still accepts the old identity until you remove its public key from every authorization source that trusts it.

SSH complicates inventory because public keys can appear in several places: ~/.ssh/authorized_keys, an identity management service, a cloud instance metadata setting, a configuration management template, or a provider's deploy-key interface. Find the authoritative source before changing anything. If configuration management rewrites authorized_keys, an emergency manual deletion may return on its next run.

Generate a replacement identity with your approved tooling and store its private half only at the gateway boundary. Install the new public half alongside the old one. Give each entry a comment that identifies purpose and rotation date, then test it through the intended route. Once it works, remove the old public entry from the source of truth and confirm the server rejects it.

You can inspect a public key fingerprint without exposing a private key:

ssh-keygen -lf deploy-release-ed25519.pub
# 256 SHA256:exampleFingerprint deploy-release-2025 (ED25519)

The shape of the output matters more than the sample fingerprint: bit length, fingerprint, comment, and key type. Put the actual fingerprint in the change record. Do not put the public key's surrounding authorization options in a vague screenshot. Copy the exact server-side rule into your reviewed configuration so another operator can see whether it contains a forced command or a source constraint.

Then perform a negative test using the old key before destroying your last controlled copy of it. The server should reject it after removal. If you cannot conduct that test because you no longer have the old private key, inspect the authoritative authorized-key source and provider audit records instead. State that limitation in the record. Pretending you tested a revocation is worse than documenting an incomplete check.

Avoid rotating an SSH key by overwriting a file at the same path and restarting an unknown client. Long-running processes can hold connections open, SSH agents can offer an old identity, and a helper may cache a file descriptor. A stateless action helper reduces this kind of ambiguity because each connection starts with a known selection. The important property is observable behavior, not a favorite implementation language.

Keep authorization and rotation separate

Approval to let an agent run an action is different from approval to use a particular credential on every call. Treating them as the same control either creates approval fatigue or leaves sensitive identities too loose.

A session control answers: "May this newly started agent process make actions during this run?" It is useful for catching a new process before it gets external access. A per-call control answers: "May this particular credential be used right now?" It is appropriate for a payment action, a production release, or a credential whose scope makes each use worth a human check.

Do not ask a human to approve every harmless status read merely because the credential rotation process made everyone nervous. People approve repeated identical prompts without reading them. Use frequent approval for identities whose individual action needs judgment, and use a clearly bounded session approval for the rest. Rotation should not train operators to click through warnings.

Sallyport's decision ladder keeps the vault gate absolute, then authorizes a new agent process for its session by default, with an option to require an approval for each use of a selected credential. During rotation, that lets an operator permit the test session while requiring explicit confirmation on the new production credential until the cutover is settled.

The vault gate has a separate role. A locked vault should deny actions regardless of what an agent session had permission to do earlier. That gives an operator a hard stop during a suspected leak or a change that starts behaving unexpectedly. It does not revoke the provider credential. After locking access, still revoke or disable the external identity if someone may have copied it outside the gateway.

Keep a short record of who approved an exceptional production test and why. Do not turn approval notes into a diary. A session identity, time, credential record, target, and change reference are enough to connect the approval to the rotation event.

Audit evidence must answer two different questions

A useful rotation record can answer both "what did the agent do?" and "can we trust the record?" A plain application log often answers neither well after someone has edited files, deleted a line, or mixed routine calls with an incident.

The first question needs operational fields: session identity, code-signing authority or process identity where available, time, action type, target, credential record name, outcome, and a provider-side request reference if one exists. The journal should not retain bearer tokens, passwords, private keys, or full secret-bearing request headers. Logs that contain credentials turn every reader of the logs into another credential holder.

The second question concerns integrity. Append-only wording is not enough if an administrator can rewrite yesterday's log. A hash chain makes each entry depend on the preceding entry, so a verifier can detect removal or alteration when it checks the chain. It does not prove that an action never occurred, and it does not make false inputs true. It does make quiet edits much harder to pass off as the original record.

This is where separate session and activity journals pay off. A session journal answers which agent process received authority and whether someone revoked that run. An activity journal answers which individual HTTP or SSH actions occurred during it. If an API token rotates at 14:00, you can examine the calls that used the record before and after the cutover without treating every session approval as proof of every request.

Sallyport projects both views from an encrypted, write-blind audit log and can verify its chain offline with sp audit verify, without requiring the vault key. Run verification before and after a sensitive rotation, and preserve its result with the change record. A successful verification says the recorded chain is internally consistent; it does not replace review of the actual destinations and outcomes.

For a high-risk rotation, capture this evidence in one compact entry:

• Why the credential changed and who owns the authorization target.
• Safe identifiers for old and new provider credentials, plus the planned overlap end.
• The narrow validation action and its provider-side result.
• The agent session and activity references involved in validation.
• Proof or documented limitation for revocation of the old identity.

That record lets a later investigator distinguish a normal planned change from an unexplained new credential. It also exposes a missing revocation check before months pass.

Treat a suspected leak as containment before replacement

When you suspect that an agent saw or exported a secret, stop access first. Creating a replacement token without disabling the exposed one leaves the old path available to whoever copied it.

Lock the action gateway if you need immediate local containment, revoke active agent sessions that should no longer operate, and disable or revoke the provider credential. Preserve the relevant session and activity evidence before cleanup jobs erase context. Then issue a new identity with scope limited to the work that must resume.

Do not wait for perfect proof that the value leaked. A token in an agent transcript, shell history, repository commit, build log, or pasted chat is enough to treat it as exposed. For an SSH private key, remove the matching public key from every trusted source and search for copies of the private material in the places automation writes artifacts. Rotating a private key while leaving its old public key authorized is not containment.

After service returns, identify how the secret crossed the boundary. The usual causes are painfully ordinary: a tool parameter accepted raw credentials, a debug log printed request headers, a developer copied a local environment file into an agent workspace, or a fallback client bypassed the gateway. Fix that route before declaring the incident closed. Otherwise the replacement merely starts its own countdown to the next leak.

Make expiry work a scheduled operational test

A calendar reminder should trigger a repeatable test, an ownership check, and a revocation decision. It should not produce a panicked request for someone to paste a new token into whichever configuration file last worked.

Review each credential record before its provider expiry date. Confirm that the named owner still owns the external account, the documented purpose still exists, the scopes still match the action, and the selected agent sessions still need access. If the answer to any of those checks is no, retire the identity instead of rotating it.

For identities that remain necessary, rehearse the narrow validation action early enough to discover provider account changes, new SSH restrictions, or altered API permissions. Keep the record of that rehearsal separate from the actual rotation so nobody mistakes an old successful test for proof that today's replacement works.

The uncomfortable test is the one that catches the most defects: after revoking the old credential, repeat the exact action that the agent depends on and verify the journal attributes it to the intended session and record. If that test fails, you have a contained failure with an owner, evidence, and a known rollback path. That is a much better place to be than discovering an expired credential in the middle of an autonomous deployment.