Separate SSH credentials for safer production access

One SSH credential that reaches development, staging, and production turns a small convenience into broad remote authority. It also makes approvals dishonest: a person may approve a harmless-looking staging task while the same credential can open a production session minutes later.

Split SSH credentials by system class and by risk. Then make the split visible in the client configuration, enforce it on the servers, and test the denied paths as seriously as the allowed ones. I have watched teams call this access control when they merely gave one key three names in ~/.ssh/config; that is not a boundary.

One credential creates one failure domain

A credential shared across environment classes gives every holder the combined privilege of those classes. If an engineer's workstation, an automated task, or an approval path can use it, compromise of that one path reaches the highest environment that accepts the public key.

The usual defense is that the same people administer every environment. That misses the point. The people may overlap while the situations do not. Development work often involves unreviewed scripts, test data, temporary machines, and broad experimentation. Production work should have a smaller set of acceptable processes, more careful review, and a clear record of who connected and why.

This is also why a single credential with separate host aliases does nothing useful. These entries look organized:

Host dev-db
  HostName dev-db.internal

Host prod-db
  HostName prod-db.internal

But if both hosts accept the same SSH public key, either alias can authenticate with the same private key. A typo in an automation variable or a copied command can cross the boundary without any new authorization decision.

Separate credentials change the outcome. A stolen development credential should fail at staging and production. A process approved to use a staging credential should have no private material or signing capability that lets it authenticate to production. Those are different controls. The first limits a breach. The second keeps a legitimate but over-broad process from doing the wrong thing.

Do not confuse this with password rotation. Rotation replaces a credential over time. Segregation decides where a credential can work at all. You need both, but rotation cannot repair a credential that was deliberately accepted everywhere.

Environment names are not sufficient boundaries

Development, staging, and production are useful labels only when they correspond to different trust decisions. A host called staging can contain production-like data, send real email, hold a signing secret, or connect to a live payment endpoint. Conversely, a production monitoring host may need less authority than a staging database administrator.

Classify SSH access by the effect of a session, not by a hostname prefix. I normally begin with four questions:

• Can this account read customer or regulated data?
• Can it alter a live service, deployment, firewall, or DNS record?
• Can it retrieve another credential or impersonate another service?
• Can it pivot to systems with more authority?

If the answer changes across hosts, the credential scope should change too. This often produces more useful groups than the familiar three-environment model: application development, test systems with sensitive data, release hosts, production read-only diagnosis, and production administration.

A separate Unix account often belongs in the design as well. An account called deploy can own the release directory and accept a restricted command. An account called ops-read can inspect logs without editing service units. An account called admin can perform maintenance under a higher approval standard. Separate accounts give the server a place to attach permissions and give logs a meaningful subject.

Do not let separate account names fool you into accepting one shared credential. If the same public key appears under dev, deploy, and admin on different machines, the private key remains a cross-environment credential. Separate accounts and separate credentials solve different parts of the problem.

Give each person and process its own identity

Each human operator and each automation process needs a distinct SSH identity within its permitted environment class. A team-wide private key makes incident response slower because revoking one person's access means replacing it for everybody. It also makes logs nearly useless when several people authenticate as the same account with the same public key.

For a human operator, create separate private keys for the allowed scopes. Use names that state the scope, not vague names such as id_ed25519_new or server-key-final.

mkdir -p ~/.ssh/identities
chmod 700 ~/.ssh/identities

ssh-keygen -t ed25519 \
  -f ~/.ssh/identities/id_ed25519_dev_alex \
  -C dev-alex

ssh-keygen -t ed25519 \
  -f ~/.ssh/identities/id_ed25519_stage_alex \
  -C stage-alex

ssh-keygen -t ed25519 \
  -f ~/.ssh/identities/id_ed25519_prod_alex \
  -C prod-alex

The comment helps people inspect a public-key list, but it does not enforce anything. The server decides access from the public key, the account, and any authorization rules. Treat comments as labels for operators, not security metadata.

Automation deserves the same discipline. A release job should use a credential issued to that job and environment, not an engineer's production identity copied into a secret store. If several jobs need access, give them separate identities unless they have the same owner, target set, and command authority. A credential should answer a simple question in a post-incident review: which process used this?

Hardware-backed credentials can make theft of a private key file harder, but they do not repair a broad authorization design. A hardware-backed credential accepted by all environments still grants access to all environments. Scope first, then choose how to protect each private key.

Client configuration must prevent identity spillover

OpenSSH will try identities from configuration and, unless constrained, identities offered by an SSH agent. That behavior is convenient until an agent holds a production identity and a connection intended for staging succeeds with it.

The OpenSSH ssh_config manual describes IdentitiesOnly as a control that limits identities used for public-key authentication to the configured identity files and certificates, even when an agent holds other identities. Set it for every scoped host alias. Give each alias one explicit identity file and do not rely on the order in which an agent happens to offer keys.

Host dev-*
  User devops
  IdentityFile ~/.ssh/identities/id_ed25519_dev_alex
  IdentitiesOnly yes
  ForwardAgent no

Host stage-*
  User release
  IdentityFile ~/.ssh/identities/id_ed25519_stage_alex
  IdentitiesOnly yes
  ForwardAgent no

Host prod-*
  User admin
  IdentityFile ~/.ssh/identities/id_ed25519_prod_alex
  IdentitiesOnly yes
  ForwardAgent no

Use aliases that make the environment hard to miss in a terminal history. For example, prod-api-01 is better than api-01 when both development and production contain an API host with similar names. Do not hide the target behind a generic alias such as server.

Check the final configuration that OpenSSH will use. This catches wildcard collisions, included files, and a forgotten global setting.

ssh -G prod-api-01 | grep -E '^(hostname|user|identityfile|identitiesonly|forwardagent) '

The output should have this shape:

hostname prod-api-01.internal
user admin
identitiesonly yes
forwardagent no
identityfile ~/.ssh/identities/id_ed25519_prod_alex

The path may expand differently on your machine. What matters is that the production alias resolves to only the production identity and that identitiesonly says yes.

A frequent failure appears after someone runs ssh-add for convenience. Their agent now holds several identities. Without IdentitiesOnly yes, the client tries them against a host until one works. Servers commonly limit authentication attempts, so this can cause confusing failures. Worse, a broad credential can quietly make the connection succeed, and the operator may never notice they used the wrong scope.

Server authorization must mirror the split

Separate private keys only work when each server accepts the matching public key and rejects the others. Put the development public key on development hosts, the staging public key on staging hosts, and the production public key only where production access is intended.

This rule sounds obvious, yet the bad pattern is common during an urgent setup: somebody copies an entire authorized_keys file to a new host. That file often contains years of stale identities, former contractors, deployment keys, and a general administrator key. The new production host now inherits access decisions nobody reviewed.

Build each account's authorization list from its job. For a deployment account, use OpenSSH restrictions that fit a noninteractive transfer or deploy operation. The OpenSSH authorized_keys manual documents the restrict option, which turns off port forwarding, agent forwarding, X11 forwarding, and PTY allocation unless another option permits them. A restricted deployment entry can look like this:

restrict,from="198.51.100.0/24" ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAI... prod-release-job

Replace the example network with an address range you actually control. Do not add from= merely because it looks strict. If the job originates from changing addresses or a hosted runner pool, an inaccurate source restriction causes an outage and encourages somebody to remove all restrictions under pressure.

Use forced commands only for narrowly defined automation. A forced command can stop a deployment credential from obtaining a shell, but it also becomes a maintenance contract. The command must validate its input, choose paths safely, and log the request. Do not apply a forced command to an administrator account and then assume the account is safe; people will eventually need a shell and work around it.

For human production administration, a separate account plus a separate public key is usually clearer than elaborate authorized_keys options. Restrict the account through normal host permissions, record its sudo activity where applicable, and remove the public key when the person no longer needs access.

SSH certificates help only when their claims stay narrow

OpenSSH certificates can make short-lived access easier to issue and revoke operationally. A certificate authority signs a user's public key, and servers trust the authority rather than storing every individual public key. That can reduce the work of updating large fleets.

They do not remove the need to separate authorities. A certificate with a production principal should not also grant development and staging access just because one operator works across all three. Issue distinct certificates, use different principals, or use separate certificate authorities when the administration and risk boundaries justify it.

The OpenSSH certificate protocol distinguishes user certificates, principals, validity intervals, and critical options. That structure is useful only if the server checks principals and you keep the issuance rules narrow. A certificate valid for every account on every host simply turns a private key into a broadly trusted bearer identity with an expiry date.

Short certificate lifetimes help with lost devices and departed staff, but they are not a substitute for revocation during an active incident. You still need a way to stop acceptance of a compromised identity or remove its authorization promptly. Plan that mechanism before issuing certificates, and test it on a real host.

Certificates also do not solve agent misuse. If an automated process can obtain a production certificate whenever it asks, that issuance service is the production boundary. Protect it with the same care you would give a production private key.

Agent forwarding crosses a boundary you cannot see

SSH agent forwarding lets a remote host ask your local agent to sign authentication challenges for other hosts. The remote machine does not receive the private key, but a process running as the remote account can use the forwarded agent while your session remains open.

That makes forwarding particularly dangerous for a production session that lands first on a jump host. If the jump host is compromised, or if an untrusted process runs under that account, it can request signatures from every identity your forwarded agent offers. The result may be lateral access that your original connection plan never mentioned.

Keep this default in the client configuration:

Host *
  ForwardAgent no

Then create a single, explicit exception only where a maintained workflow requires it. Before adding that exception, ask whether ProxyJump, a dedicated bastion credential, or an operation performed locally would remove the need. ProxyJump transports the SSH connection through an intermediate host; it does not expose your local agent to that host in the same way.

Do not accept the claim that forwarding is safe because the private key stays on the laptop. A signing oracle can be enough to authenticate elsewhere. The distinction matters during a compromise: protecting a file is not the same as constraining what can use the credential.

Approval scope must match credential scope

An approval that authorizes a new process to use an SSH credential should identify the credential's environment and the process that requested it. A prompt that says only ssh command requested asks a human to infer too much from too little.

This is where teams often blur two controls. Credential scope answers where an identity can authenticate. Approval scope answers which running process may exercise that identity. You need both. Separate production credentials prevent a development process from reaching production by accident; a process approval prevents an unknown local process from exercising an available production credential.

Per-command approval sounds safer, and people often ask for it after an uncomfortable incident. It usually fails under routine maintenance because repeated prompts train operators to approve without reading. Reserve per-use confirmation for credentials whose use itself demands a human decision, such as a high-impact production administration identity. Give ordinary work a session authorization that ends when the process exits, then keep the credential's server-side scope narrow.

Sallyport keeps SSH credentials in its encrypted vault and can require authorization for a new agent process or approval on every use for a selected credential. That does not replace separate server authorization, but it makes the process boundary explicit without handing the credential to the agent.

An AI coding agent deserves stricter separation than an interactive terminal because it can issue many commands quickly and may follow a bad instruction. Give it development access by default. If it needs staging, use a staging-only identity and a separate approved run. Treat production access as a distinct operation with a named target and a narrow purpose.

The failure usually starts with a harmless exception

Consider a team with one ops SSH key accepted by development, staging, and production hosts. An engineer loads that key into an SSH agent for a production maintenance window. Later, a local build helper opens an SSH connection to a staging host to collect logs.

The helper has no explicit IdentityFile setting and IdentitiesOnly is absent. OpenSSH tries identities from the agent. The shared ops key succeeds because staging accepts it. The helper now has a staging session authenticated with an identity that also works on production.

A second mistake follows. The staging host has agent forwarding enabled because somebody needed a one-off connection last month. A process on that host can request signatures through the forwarded agent. It reaches a production host with the same ops identity. The original engineer approved a production maintenance session, but an unrelated helper and a staging host inherited its authority.

No exploit requires a leaked private key in this sequence. The design allowed a broad identity, the client selected it opportunistically, and forwarding extended its reach. Logs may show valid authentication all the way through, which is why teams mislabel it as user error instead of fixing the access model.

A split design breaks the chain in several places. The helper uses only a staging credential. Production rejects that credential. Agent forwarding stays disabled. The production identity is available only to an explicitly approved production process. Any one of those controls helps; together they make the wrong connection fail early.

Rotation and emergency revocation need a practiced order

Rotation works when you add a replacement before removing the old identity, verify the exact route, then revoke the old one. Production rotations fail when somebody tests only from their laptop while the real deployer connects from a different account, network, or automation runner.

For a normal rotation, use this order:

1. Create a replacement credential in the same narrow scope as the old one.
2. Add its public key or certificate authorization to the intended account and hosts.
3. Test the real command from the real process path, including any jump host.
4. Remove the old authorization and confirm that the old credential now fails.
5. Record the replacement fingerprint, owner, scope, and removal date in the access record.

Keep a break-glass path, but do not make it a second permanent administrator credential copied to every workstation. Store it separately, limit who can activate it, and test it under controlled conditions. A break-glass process that nobody has exercised becomes a frantic guessing exercise during an outage.

Emergency revocation is different. If a production credential may be exposed, remove its public key or stop accepting its certificate immediately, then replace it. Do not wait for the scheduled rotation window because an attacker will not honor your calendar. The tradeoff is operational disruption, which is why narrow credentials pay off: revoking a production deployment identity should not stop development work.

Test denied access and read the evidence

Access separation is complete only when the wrong credential fails in a way you can explain. Test with an intentional negative case after every significant change: use the development identity against a production host and verify public-key authentication fails. Then test the expected credential and inspect the account name and target in the connection record.

Use verbose client output during testing, not as permanent habit:

ssh -vvv -o IdentitiesOnly=yes \
  -i ~/.ssh/identities/id_ed25519_dev_alex \
  [email protected]

You should see the client offer the development public key and the server refuse it. Do not paste this output into tickets without checking it first; verbose SSH logs can disclose hostnames, usernames, and authentication details.

Server logs should let you answer who authenticated, which account they used, which public-key fingerprint the server accepted, and from where the connection originated. If several people or processes share one identity, the log cannot recover that missing attribution later.

Review for boundary drift: a development key added to a production account, an old deployment credential still accepted, a production key loaded into a general-purpose agent, or a wildcard client rule that overrides IdentitiesOnly. These changes often arrive as temporary fixes. Temporary SSH access has a habit of surviving long after the emergency that created it.

Start by inventorying every public key accepted on production hosts and labeling each one by owner, process, purpose, and scope. Any entry that cannot survive that four-part description should not remain authorized.