Is your SSH host key rotation ready?

An SSH host key should change on your schedule, with its replacement already trusted. If the first time developers hear about a rotation is the red warning in their terminal, the rotation plan has already failed. That warning cannot tell them whether the server changed legitimately or someone intercepted the connection. A hurried message telling everyone to delete a line from known_hosts destroys the evidence they need to decide.

A safe rotation has four distinct acts: publish the old and new fingerprints through a trusted channel, make both keys available long enough for clients to learn the new one, remove the old key during a defined window, and prove that clients reject the retired identity. I have seen teams do the server work correctly and still train their developers into the worst possible habit: treating host verification as an obstacle. The operational job is to make the secure path ordinary before the warning appears.

Host identity is separate from user authentication

A host key proves which SSH server answered; a user key proves who is asking to log in. Rotating one does nothing to rotate the other. This distinction sounds basic, yet incident runbooks routinely mix authorized_keys, personal SSH keys, host certificates, and /etc/ssh/ssh_host_* files into one vague instruction to "rotate SSH keys."

RFC 4253 puts server authentication inside the transport handshake. The server signs the exchange hash with its private host key, and the client checks that public key against a trusted source such as known_hosts, a host certificate authority, or authenticated SSHFP records. Passwords, user certificates, and public user keys come later. A developer can present a perfectly valid user credential to an impostor if the client has stopped checking the server.

That is why the changed-host warning is deliberately severe. It marks a broken continuity claim: the name the developer typed now presents a different identity. Planned replacement, a rebuilt virtual machine, a load balancer pointed at the wrong pool, DNS tampering, and active interception can all look the same at that moment. The client has no access to your change ticket.

Write the scope down before generating anything. Record every hostname and alias people use, every nonstandard port, each address that appears directly in scripts, bastion routes, CI runners, deployment agents, and any shared GlobalKnownHostsFile. [host]:port is a different known-hosts name from host, while aliases can hide the canonical destination. A rotation that covers git.example.net but misses git.internal will look intermittent even when both names reach the same machine.

Also inventory every host-key algorithm currently offered. One server may hold Ed25519, ECDSA, and RSA host keys at once. OpenSSH negotiates an algorithm with the client, so two developers can connect to the same daemon and pin different public keys. Rotating only the key observed on your laptop does not establish the server's complete identity set. Use the configured HostKey entries and verified public-key files as the authoritative inventory, then compare that inventory with what supported client groups actually negotiate.

Publish both fingerprints before changing the server

Publish the current and replacement fingerprints while the current key is still serving traffic. The publication must travel through a channel whose trust does not depend on the SSH host being changed. A signed operations repository, an authenticated internal status page, a managed device configuration, or a separately administered DNSSEC zone can work. A message copied from the same possibly intercepted SSH session cannot.

Generate fingerprints from public-key files on a trusted administrative machine, not by asking the production network what it currently serves. OpenSSH displays SHA-256 fingerprints by default. The command and output shape are:

$ ssh-keygen -lf ssh_host_ed25519_key.pub -E sha256
256 SHA256:<base64-fingerprint> host.example.net (ED25519)

Publish more than the short digest. For each identity, include the hostname and port, algorithm, SHA-256 fingerprint, status (current, new, or retired), first-use time, retirement time, and the person or system that approved the record. Label time zones. If aliases share the key, list them. If different nodes behind one name intentionally present different keys, publish the entire permitted set and explain why it is a set.

Do not use ssh-keyscan as the authority for the replacement. Its manual is unusually direct: building a known-hosts file from unverified scan output exposes users to interception. The tool is useful for collecting what an endpoint presents, but you must compare that result with a fingerprint obtained through a separate trusted path. Discovery and verification are different jobs.

A useful publication record looks like this:

Host: build.example.net:22
Algorithm: ssh-ed25519
Current: SHA256:<old-fingerprint>
New: SHA256:<new-fingerprint>
Overlap begins: 2026-08-10 15:00 UTC
Old key removed: 2026-08-17 15:00 UTC
Old key marked revoked: 2026-08-17 15:30 UTC
Owner: Platform operations

The dates are illustrative, but the ordering is not. Publishing after the server changes converts planned work into an unplanned trust decision. Publishing only the new value removes the comparison developers need when they inspect an existing pin. Keep the old record visible as retired after the change so investigators can recognize stale machines and unexpected reuse.

Fingerprints are public identifiers, not secrets. The private host key stays protected on the server, but teams should distribute the public key and fingerprint widely enough that verification does not depend on finding one available administrator during the window. Treat approval of the publication as a security change: two source files with matching fingerprints are stronger than a screenshot pasted into chat.

An overlap lets clients learn the replacement safely

Serve the old and new host keys together before removing the old one. During that overlap, an established client authenticates the server with the old trusted key and can then learn the additional key through OpenSSH's [email protected] extension. The continuity comes from the old key, so the new key does not arrive as an unauthenticated assertion.

The OpenSSH ssh_config manual describes UpdateHostKeys specifically as support for graceful rotation. It also lists limits that matter in real fleets. The client accepts additional keys only after authentication with an already trusted or explicitly accepted plain key, using a user known-hosts file. It does not learn them through this path when the server was authenticated by a host certificate or only through a global known-hosts file. The default can also turn off when the user overrides UserKnownHostsFile or enables VerifyHostKeyDNS.

Do not assume the default. Inspect effective client configuration for representative hosts:

$ ssh -G build.example.net | grep -E '^(updatehostkeys|userknownhostsfile|stricthostkeychecking) '
stricthostkeychecking ask
updatehostkeys true
userknownhostsfile ~/.ssh/known_hosts ~/.ssh/known_hosts2

On the server, keep separate private files and declare both. File names here are examples; use paths that match your operating system and configuration management:

HostKey /etc/ssh/ssh_host_ed25519_key_old
HostKey /etc/ssh/ssh_host_ed25519_key_new

Run sshd -t before reload. The sshd manual says this mode checks both configuration validity and key sanity, which catches unreadable files and malformed settings before the daemon rereads them. Reload rather than killing active sessions unless your service manager documents another procedure. Then connect from a clean test client that already trusts only the old key, and inspect its isolated known-hosts file after a successful authenticated session.

Multiple configured keys do not guarantee that every client negotiates the same one. Algorithm preference, client age, custom HostKeyAlgorithms, certificates, and global trust stores affect the result. Keep the overlap for at least one normal connection cycle for each managed client class, not an arbitrary number of hours. A laptop fleet that connects weekly needs a different window from CI workers that connect every few minutes.

Measure adoption without pretending you can see every personal known_hosts file. Managed clients can report whether the new public key exists in their distributed trust file. For unmanaged clients, track successful connections by client version where your privacy policy allows it, publish an explicit verification command, and retain the overlap long enough for ordinary usage. Silence is not proof that the new key propagated.

The change window needs states and stop conditions

A change window should define observable states, owners, and rollback conditions. "Rotate at 3 p.m." is not a plan because it says nothing about the overlap, client readiness, or the moment the old identity becomes unacceptable. Put the trust transition on the same timeline as the server deployment.

Use five gates:

1. Publication complete: independent reviewers reproduce every old and new fingerprint from approved public-key files.
2. Overlap live: the server offers both identities, its configuration passes sshd -t, and clean clients authenticate through the old pin while learning the new one.
3. Readiness met: managed trust stores contain the replacement, test clients cover supported operating systems and routes, and the help desk has the exact expected fingerprints.
4. Cutover complete: the server no longer offers the old private key, fresh sessions succeed with strict checking, and monitoring finds no unexpected node still presenting it.
5. Revocation proven: a test endpoint that presents the old identity is rejected, and the retired fingerprint remains published with its revoked status.

Give one person authority to stop the cutover. Stop if any production node presents an unpublished key, if a route reaches a node outside the inventory, if a supported client cannot learn or receive the replacement, or if rollback would require restoring a private key that the incident response team considers compromised. A routine lifecycle rotation may roll back to the old key during the overlap. A compromise response cannot treat the compromised key as a safe rollback target.

Separate service rollback from trust rollback. You may restore an earlier server package without restoring its retired host identity. Keep known-good configuration, replacement private-key access, and console or provider access available so the team can repair SSH without weakening verification. If SSH is the only way to fix SSH, the window has a hidden single point of failure.

The window must cover long-lived multiplexed sessions. OpenSSH connection sharing can keep a master connection alive while new shells reuse the existing transport. Those shells do not perform a fresh host-key exchange, so they cannot prove the post-cutover identity. Close test control masters with ssh -O exit host where applicable, and ensure verification probes establish new TCP connections.

Time the retirement precisely and keep the communication open beyond it. Developers returning from leave will hold stale pins. Their expected outcome is a documented failure plus a verified update path, not an exception. The support response should compare fingerprints first, identify the stale entry with ssh-keygen -F, and replace it from the approved record.

Rehearse the warning with an isolated trust file

Test the rotation without touching anyone's real ~/.ssh/known_hosts. An isolated file makes each state reproducible and prevents a successful test from depending on keys your workstation learned months ago. Use a staging endpoint that can present the production key sequence, or run a temporary sshd on a separate port under the same configuration constraints.

First, create known_hosts.test from the approved old public key. Construct the line from the reviewed artifact, not live scan output. Then connect to the staging endpoint while pinning its logical production name with HostKeyAlias:

$ ssh -F /dev/null \
  -o HostKeyAlias=build.example.net \
  -o UserKnownHostsFile=./known_hosts.test \
  -o GlobalKnownHostsFile=/dev/null \
  -o StrictHostKeyChecking=yes \
  -o UpdateHostKeys=yes \
  -p 2222 test-host.example.net true

The first run should succeed while the endpoint offers both keys. Inspect the file with ssh-keygen -F build.example.net -f ./known_hosts.test and confirm that it now contains the approved replacement. Fingerprint each returned public-key field and compare it with the publication record. A zero exit status alone proves connectivity, not the identity set.

Next, configure the test endpoint to offer only the new key and establish a new connection. It should succeed because the client learned the replacement through the authenticated overlap. Restore a snapshot containing only the old pin and repeat against the new-only endpoint. With StrictHostKeyChecking=yes, that attempt must fail. This negative case proves that unmanaged clients which missed the overlap receive a hard stop rather than silent acceptance.

Finally, point the test at an endpoint that presents an unrelated key. Capture the failure text and exit status for the runbook. Do not sanitize the test until it always passes; a changed-key rejection is the behavior you are protecting. Confirm that scripts propagate the nonzero exit instead of swallowing it in a retry loop.

Never make StrictHostKeyChecking=no the rehearsal fix. The current OpenSSH manual says that setting allows changed keys to proceed subject to restrictions, while accept-new still refuses changed keys. Even accept-new does not solve rotation by itself because a replacement for an already known name is a changed key. Supply authenticated trust material or use the overlap mechanism.

Revocation must produce a rejection, not a cleanup tip

Removing the old private key from the intended server proves only that one server stopped offering it. Revocation means clients reject that public key if it appears again, including on a forgotten node or an attacker-controlled endpoint holding a stolen private key. Deleting the old line from known_hosts does the opposite: it removes the memory needed to recognize the retired identity.

OpenSSH known-hosts files support an @revoked marker. A matching key marked this way must not be accepted and produces a warning when encountered. Managed fleets can distribute a global revoked entry, while smaller environments can maintain a dedicated revocation file or generated trust bundle. Use the exact approved old public key, and scope the hostname pattern deliberately:

@revoked build.example.net ssh-ed25519 <old-public-key-data>

A Key Revocation List is useful when the set grows or when host certificates are involved. Generate a test KRL from the old public key, then query it before deployment:

$ ssh-keygen -k -f revoked-hostkeys.krl ssh_host_ed25519_key_old.pub
$ ssh-keygen -Q -f revoked-hostkeys.krl ssh_host_ed25519_key_old.pub
ssh_host_ed25519_key_old.pub (<comment>): REVOKED

The query returns a nonzero status for a revoked key, so automation must interpret the status intentionally rather than treating it as a failed test. Configure a test client with RevokedHostKeys pointing at the KRL, connect to a test endpoint serving the old identity, and require rejection. Then connect to the new-only endpoint and require success. Both halves matter: a malformed or unreadable revocation file can deny every connection rather than only the retired identity.

ssh-keygen -R host is useful for editing hashed user files, but it is not revocation. It removes all entries for the name, including a valid replacement, and leaves the next connection to establish trust again. Use it only after comparing the stored entries with the published record and then adding the approved new key. A support script that runs -R followed by unauthenticated ssh-keyscan has automated trust surrender.

Retain revoked public material. You can destroy an uncompromised retired private key according to your retention policy, but keep its public key, fingerprint, approval record, and revocation test result. Those artifacts let responders identify an old image that returned to service months later.

Automation should fail closed without becoming brittle

Noninteractive jobs need a maintained trust source, not disabled checking. CI runners, deployment agents, and autonomous coding agents often encounter the rotation first because they open short-lived connections all day. When their image bakes in one host key and nobody owns its renewal, the usual emergency patch is StrictHostKeyChecking=no. That patch turns an availability mistake into an authentication failure.

Choose a trust delivery model for each automation class. Bake both approved keys into an image during overlap, mount a centrally managed global known-hosts file, return approved lines through KnownHostsCommand, use verified SSHFP with DNSSEC, or trust a host certificate authority. The OpenSSH manual says KnownHostsCommand output uses the normal known-hosts line format and runs in addition to user and global files. It also terminates the connection if the command fails, which is the right default but requires the trust service to meet the job's availability needs.

Keep batch behavior explicit:

Host build.example.net
    BatchMode yes
    StrictHostKeyChecking yes
    UserKnownHostsFile /etc/company/ssh_known_hosts
    UpdateHostKeys no

UpdateHostKeys no is deliberate in this managed-file example. Configuration management owns the file, so allowing each ephemeral worker to edit it creates state you cannot preserve or audit. In a developer-managed user file, UpdateHostKeys yes may be the better choice. The same directive can be right or wrong depending on who owns trust distribution.

Test aliases, jump hosts, direct IP connections, and nonstandard ports as separate identities. A ProxyJump route still authenticates the destination host, and the jump host has its own pin. Containers may mount a read-only known-hosts file. Sandboxed agents may run a different SSH binary or ignore the user's configuration directory. Run ssh -G target inside the actual execution environment to see the effective settings rather than reading a workstation config and assuming parity.

For agent-driven SSH, keep credentials and host trust as separate controls. Sallyport executes SSH actions through its bundled sp-ssh helper while SSH keys remain in its encrypted vault, so an agent does not receive those secrets. That protects credential custody; your rotation plan still needs an authenticated host identity source and a tested rejection path.

An automation failure during the window should name the expected and observed fingerprints without printing private material. Do not automatically retry with weaker settings. Stop the job, preserve its stderr and effective configuration, and route the mismatch to the owner named in the publication record.

Run the same automation test from a disposable worker that starts with no home directory state. Give it the exact managed trust bundle intended for production, invoke the real job command, and archive ssh -G output alongside the result. This catches an easy lie in rotation testing: a runner appears healthy because a previous interactive session populated a writable file that the image specification never declared.

Test the failure path by withholding the replacement from a copy of the bundle after the server becomes new-only. The job must stop before it sends a remote command, and its wrapper must preserve SSH's exit status. Then distribute the replacement and repeat. If a deployment framework converts every SSH failure into a generic timeout, fix that observability gap before the window because responders need to distinguish rejected identity from network loss and user-authentication failure.

Watch for fleet configuration that expands trust more than intended. A wildcard known-hosts line can make one approved key valid for many names, while shared host keys can make separate machines indistinguishable to clients. Both designs may be intentional, but the rotation record should state the scope. Prefer a host-specific line when machines have distinct identities, and test the exact hostname string used by the job.

Finally, pin the SSH binary and configuration contract used by autonomous jobs. A base-image update can change defaults at the same time as the host identity changes, making the root cause hard to isolate. Capture the client version, effective host-key algorithms, trust-file digest, and logical hostname before and after the cutover. These are public operational facts, and together they show whether a failed job saw the wrong server, lacked the new key, or could not negotiate an offered algorithm.

DNS and host certificates change the distribution job

SSHFP and host certificates can reduce per-host pin management, but neither removes the need to plan trust rotation. They move the stable trust anchor. Use them when you can operate that anchor better than thousands of independent known-hosts entries.

RFC 4255 defines SSHFP records and requires authenticated DNS data before a client treats a fingerprint as trusted. In practice, that means DNSSEC validation must hold from the client to the record. An ordinary unsigned DNS record is useful for comparison but cannot establish identity against an attacker who can alter DNS answers. OpenSSH's VerifyHostKeyDNS yes implicitly trusts only secure matches; insecure results still require the normal confirmation path.

Publish old and new SSHFP records during overlap, wait for DNS caches and managed resolvers to observe them, then remove the old record when the old private key leaves service. Generate records from approved public files with ssh-keygen -r hostname -f public_key_file. Verify the served record from the same resolver path clients use, including validation status. A low TTL shortens cache persistence but does not repair broken DNSSEC or an incorrect record.

Host certificates let clients trust a host CA entry such as @cert-authority *.example.net instead of pinning every host key. When a server receives a new certified host key with the correct principals and validity interval, clients can accept it under the existing CA. This is cleaner for large dynamic fleets, but the CA becomes a powerful trust anchor. Protect its private key, constrain issuance, log certificate serials and principals, and rehearse CA revocation separately.

Do not deploy a host CA merely to make one awkward rotation disappear. It adds issuance, expiry, principal naming, and CA rollover work. A small stable fleet can operate explicit pins well. A large ephemeral fleet often benefits from certificates because instance identity changes more often than organizational authority.

Whichever model you choose, document where trust begins. UpdateHostKeys does not apply when a host certificate performed authentication, and a client using a global known-hosts file will not learn additions through the user-file mechanism. Mixing models without recording precedence creates rotations that work on one laptop and fail in CI.

Close the rotation with evidence that survives the window

A completed rotation leaves proof of what changed, who approved it, what clients accepted, and whether the old identity was rejected. Keep the generated public keys, reproduced SHA-256 fingerprints, sshd -t result, effective server configuration, publication revision, client test matrix, cutover timestamps, revocation artifact, and captured negative test. Store no private key in the ticket.

Record the actual server state after the window. Query every node through each production route and compare what it presents with the approved set, but remember that network collection is an observation, not its own trust source. Compare it against the signed or otherwise authenticated publication. Check autoscaling images and powered-off recovery nodes too; stale host keys often return through replacement capacity rather than the machine changed during the window.

The Activity journal and Sessions journal in Sallyport can retain a tamper-evident record of SSH actions an agent performed, projected from its encrypted hash-chained audit log. sp audit verify checks that chain offline over ciphertext without a vault key, which can add action evidence to the change record without replacing the host-key tests described here.

Set a follow-up date based on fleet behavior. Look for clients still presenting failures for the retired fingerprint, nodes serving unexpected identities, and scripts that added bypass flags during the incident. Remove temporary overlap configuration and test endpoints. Keep the revocation active for as long as the retired private key could plausibly survive in backups, images, or unauthorized copies.

A host-key warning should remain rare and alarming. Good rotation work does not suppress it. Good work arranges the trust transition early enough that expected clients never need to ignore it, then proves the warning still stops a connection when the retired identity comes back.