TLS certificate rotation needs two-sided proof

A certificate authority can issue exactly the certificate an AI agent requested while the service keeps presenting the old one. I have seen rotation jobs celebrate a downloaded PEM file, a successful secret update, or a green reload command, then discover at the worst possible hour that one listener never changed. Issuance success and deployment success are different claims, so they need different evidence.

A sound rotation records what the CA issued, deploys that exact artifact, and opens a new TLS connection to observe what clients receive. The final comparison must use stable certificate identity, the intended hostname, and the actual network path. If the agent cannot produce both sides of that comparison, the change is still in progress.

Completion needs an explicit evidence contract

A rotation is complete only when an issued-certificate record matches observations from every endpoint in the deployment scope. Write that rule before giving an agent credentials or commands. Otherwise the agent will optimize for whatever its tool reports as success, which may be a file write rather than a client-visible change.

The issuance record should contain the order or request identifier, requested DNS names, certificate serial number, SHA-256 fingerprint, validity interval, issuer, and a digest of the public key. It should also identify where the full chain and private key were stored without copying the private key into the log. A certificate fingerprint identifies the whole leaf certificate; a public-key digest lets an operator tell whether a reissued certificate kept or replaced the key.

The deployment record answers different questions: which target received the artifact, which configuration references it, which process accepted a reload, and when that action occurred. A successful control-plane API response belongs here, but it is not proof of service. Treat it as evidence that an attempted transition reached the deployment system.

The observation record comes from a fresh connection made after deployment. It needs the requested hostname, resolved or forced address, port, observed serial and fingerprint, verification result, timestamp, and verifier location. The location matters because an internal probe, an external probe, and a probe behind a corporate proxy can reach different termination points.

Keep the completion predicate mechanical:

complete = issuance.valid
        && deployment.accepted
        && every(expected_endpoint,
                 observation.chain_valid
                 && observation.name_valid
                 && observation.leaf_fingerprint == issuance.leaf_fingerprint)

This makes an awkward state visible: the certificate exists and deployment was accepted, but the served fingerprint still differs. Call that state deployed_unverified, not complete. Precise state names prevent a dashboard or agent summary from quietly merging two claims.

ACME issuance does not prove deployment

ACME proves that a certificate authority accepted an order and issued a certificate after the required authorization, but the protocol does not prove that your application serves that certificate. RFC 8555 describes four main client steps: submit an order, prove control of the requested identifiers, finalize with a certificate signing request, then wait for and download the certificate. The boundary ends at retrieval.

That boundary is easy to miss because many ACME clients run a deploy hook after renewal. The hook sits outside the CA transaction. It can fail because a file path changed, a container mounted the old secret, a reload targeted the wrong process, a remote API accepted an asynchronous update, or a node was unreachable. The order can remain perfectly valid through all of those failures.

Record the final ACME order object or the equivalent CA response. For an ACME order, status: valid and a populated certificate URL prove issuance. Capture the identifiers and the returned leaf fingerprint after parsing the downloaded certificate. Do not use the mere presence of a new file as the identity of the artifact; temp files, symlinks, and repeated filenames make paths poor evidence.

Certificate Transparency also does not close the loop. A log entry can show that a public certificate was issued for a name. It says nothing about whether your load balancer, ingress controller, web server, or CDN now presents that certificate. CT is useful independent evidence of issuance, not a deployment monitor.

This distinction changes error handling. An issuance failure means the agent never obtained the intended credential. A deployment failure means it obtained a sensitive credential that may now exist in storage but is not serving. The second case often needs cleanup, retry, or rollback logic, and it deserves a separate alert.

Validate the artifact before it reaches a listener

Before deployment, confirm that the leaf certificate has the requested names, expected validity, intended issuer, and the public key that matches the private key. This gate catches malformed bundles and wrong-file errors without touching production. It does not replace the live probe.

RFC 9525 makes the identity rule sharp: clients construct acceptable reference identifiers independently, then compare them with identifiers in the certificate. For ordinary DNS services, the relevant names live in the subjectAltName extension. The RFC explicitly rejects falling back to a domain-like Common Name. An agent that checks only subject=CN=... can approve a certificate that modern clients should reject.

Inspect the leaf artifact with real output fields:

openssl x509 -in leaf.pem -noout \
  -serial -fingerprint -sha256 -dates -issuer -subject \
  -ext subjectAltName

A typical output shape is:

serial=03A17C...
sha256 Fingerprint=6B:19:8A:...
notBefore=Jul 24 08:15:00 2026 GMT
notAfter=Oct 22 08:14:59 2026 GMT
issuer=C=US, O=Example CA, CN=Example Issuing CA
subject=CN=api.example.net
X509v3 Subject Alternative Name:
    DNS:api.example.net, DNS:www.example.net

Use openssl x509 -checkhost api.example.net -noout as a machine gate when your OpenSSL version supports it. OpenSSL documents -checkhost specifically for matching a certificate to a host. Parse the exit status, not friendly output text that may change between versions.

Compare public keys without exposing private material. The following commands hash the DER-encoded public key from each side, so matching lines show that the certificate and private key form a pair:

openssl x509 -in leaf.pem -pubkey -noout \
  | openssl pkey -pubin -outform DER \
  | openssl dgst -sha256

openssl pkey -in private-key.pem -pubout -outform DER \
  | openssl dgst -sha256

Never put private-key.pem contents, command tracing, or an environment variable containing the key into agent context. Let a constrained executor perform the comparison and return the digest and exit status. The agent needs the result of the action, not the secret used to perform it.

A successful file write still needs activation

Deployment has two operations: place the new material atomically, then make the TLS terminator load it. Agents routinely perform the first and infer the second. Long-running servers usually keep parsed certificates in memory, so replacing a file on disk does not necessarily change new handshakes.

Stage files beside their destination, apply ownership and restrictive permissions, validate the configuration, then rename into place. A rename on the same filesystem avoids exposing a partly written PEM file. Keep the previous working files or a versioned secret reference until live verification passes; immediate cleanup destroys the fastest rollback.

Activation depends on the terminator. The nginx documentation says a HUP makes the master validate the new configuration, start new workers, and gracefully retire old ones; if applying the configuration fails, nginx continues with the old configuration. That behavior is safe for availability, but it creates a perfect false positive if an agent records only that it sent HUP. Apache's graceful restart similarly checks syntax and starts a new generation while old connections finish. Managed load balancers and CDNs may accept an update and finish it later.

Capture the activation command or API operation ID, its exit status, and the target process identity. Then inspect service-specific status or logs for acceptance. Do not turn a fixed sleep into proof. A 30-second pause sometimes hides eventual consistency and sometimes wastes 29 seconds; polling a documented state with a deadline is clearer.

Activation failures should stop the workflow before public verification retries flood the service. If syntax validation fails, if the process identity changed unexpectedly, or if the control plane reports a terminal error, preserve the old certificate and mark deployment failed. If activation reports success but the live endpoint remains old, keep probing within a bounded window because graceful replacement and distributed propagation take time.

Verify what a client receives, including its name

The decisive check opens a new TLS connection, sends the intended Server Name Indication value, validates the chain and hostname, and records the leaf certificate. Looking at a local file cannot answer what a client receives. Reusing an existing keep-alive connection cannot answer it either, because the TLS handshake already chose a certificate.

Use OpenSSL with both SNI and hostname verification:

openssl s_client \
  -connect api.example.net:443 \
  -servername api.example.net \
  -verify_hostname api.example.net \
  -verify_return_error \
  </dev/null 2>/dev/null \
| openssl x509 -noout -serial -fingerprint -sha256 -dates -issuer

The -servername option controls SNI, which selects the virtual host on many shared listeners. The -verify_hostname option checks the identity, and -verify_return_error makes a verification error stop the command instead of merely printing diagnostics and continuing. OpenSSL's -showcerts can display the chain sent by the server, but the documentation notes that this is the server-supplied list, not a verified chain by itself.

Do not add -k, --insecure, or an equivalent skip-verify flag to make automation pass. The curl manual states that a normal TLS request checks both CA trust and whether the certificate matches the hostname. An insecure probe can confirm a fingerprint while missing the exact broken-chain or wrong-name condition that users will hit.

A fingerprint match alone is also insufficient. A server might present the intended leaf with an incomplete intermediate chain, causing some clients to fail. Your completion gate should require a successful trust-path and name check plus the expected leaf identity. Where revocation checking or Certificate Transparency policy is part of your real client profile, run that profile too rather than claiming the basic OpenSSL probe covers it.

Run each observation in a clean process or explicitly disable connection reuse. Record stderr on failure, but sanitize it before it enters agent context. The failure category should remain clear: connection, handshake, trust path, identity, or fingerprint mismatch.

Every TLS termination point is a separate target

A hostname is not a deployment scope. The scope is the set of places that can terminate TLS for that hostname: load-balancer addresses, CDN points of presence, ingress replicas, regional front doors, IPv4 and IPv6 paths, and direct origin listeners when clients can reach them. One passing connection proves one route at one moment.

Build the expected endpoint inventory from infrastructure state, not from whatever DNS answer the verifier happened to receive. Dynamic DNS and anycast make exhaustive public sampling difficult, so define a defensible set: each configured listener or certificate attachment in the control plane, plus probes from the client regions that matter. If a vendor owns the edge fleet, verify the vendor's deployment status and sample externally from more than one vantage point.

For address-specific tests, curl's --resolve is safer than replacing the URL with an IP address because it keeps the original hostname for SNI and certificate verification:

curl --fail --silent --show-error \
  --resolve api.example.net:443:192.0.2.18 \
  --output /dev/null \
  https://api.example.net/health

Repeat that request for each known address, including IPv6 addresses in brackets where supported. Follow it with the OpenSSL fingerprint probe against the same address while keeping -servername api.example.net. A successful HTTP status checks more than TLS, so choose a cheap endpoint that does not mutate state; the certificate observation remains the completion evidence for this rotation.

Be precise about proxies. A probe from a developer laptop may see a corporate TLS inspection certificate. A probe inside the cluster may bypass the public CDN. Record the verifier location and network path, and reject an observation when its issuer or fingerprint shows that it never reached the intended termination point.

During a rolling change, expect a mixture of old and new fingerprints. That is a valid transient state, not an excuse to declare success based on a majority. Completion requires every endpoint in scope to converge, or an explicit decision to remove a failed endpoint from service.

Make the agent emit comparable records

An AI agent should produce structured evidence that another program can compare without interpreting prose. Free-form summaries such as "renewal succeeded and the site looks good" erase target identity, timestamps, and mismatches. Keep the agent's explanation for humans, but make state transitions depend on typed fields.

A compact event pair can look like this:

{"type":"certificate.issued","rotation_id":"rot_7f2","order_id":"ord_91c","dns_names":["api.example.net"],"serial_hex":"03A17C","leaf_sha256":"6B:19:8A:...","spki_sha256":"9f4c...","not_before":"2026-07-24T08:15:00Z","not_after":"2026-10-22T08:14:59Z"}
{"type":"certificate.observed","rotation_id":"rot_7f2","host":"api.example.net","address":"192.0.2.18","port":443,"leaf_sha256":"6B:19:8A:...","chain_valid":true,"name_valid":true,"observed_at":"2026-07-24T08:19:22Z","vantage":"external-us-east"}

Use one rotation_id across issuance, deployment attempts, observations, retries, rollback, and closure. Keep the CA order ID and infrastructure operation ID as separate fields. Combining them into one generic job ID makes incident reconstruction needlessly hard.

Retries need their own semantics. An agent may lose the response after the CA accepted an order or after a control plane accepted an attachment. Repeating the request blindly can create extra certificates, trigger rate limits, or start competing deployments. Give every mutating call an idempotency token derived from the rotation ID and phase, and query the remote operation before attempting a replacement. Record whether a response describes a new action or a replay of an existing one.

Freshness deserves an explicit rule too. A probe that passed an hour before deployment proves nothing about the resulting state, even if it happened to observe the same fingerprint during a previous attempt. Require observed_at to follow the accepted activation event and set a maximum observation age when closure runs. Use a trusted workflow clock or verifier timestamp rather than time supplied in model-generated prose.

Treat certificate identity and service identity as separate fields in the record. The fingerprint answers "is this the issued leaf?" The hostname check answers "is that leaf valid for the service the client requested?" The chain result answers "can this verifier build an acceptable trust path?" Collapsing those booleans into tls_ok makes troubleshooting depend on rerunning the test after evidence has already disappeared.

The endpoint inventory should also carry a revision. If infrastructure adds a listener halfway through rotation, closure must either adopt the new revision and require its observation or bind itself to the approved earlier revision and raise a follow-up. Silently reading a moving list creates a race in which the agent can finish just before an unverified endpoint appears. Store the inventory revision beside the closure result so a reviewer can reconstruct what "every endpoint" meant.

Evidence collection must be safe to repeat. Probes should use read-only routes, short connection timeouts, and bounded concurrency. A health URL that warms a cache or changes session state is a poor certificate probe; the TLS handshake occurs before the HTTP request, so a minimal request or direct handshake is enough. Rate limits still matter when a fleet has thousands of addresses, so schedule probes by target and retry only failures.

Finally, separate observation from interpretation. The probe reports bytes parsed from the handshake and the verifier's result. The comparator decides whether that observation satisfies the current rotation. This split lets you fix a comparator bug and replay stored evidence without touching production, and it stops an agent from rewriting an inconvenient mismatch into a reassuring summary.

The comparator should reject observations taken before the accepted deployment attempt, observations for an unexpected hostname or address, and evidence older than the current rotation. It should also compare normalized binary fingerprints rather than display strings, since colons and letter case vary. Preserve the original display value for operators.

Do not let the agent decide which evidence fields to omit. Define the schema in the workflow and fail closed when required evidence is absent. Signed events can strengthen provenance, but signatures do not repair a weak statement. A perfectly signed "command exited zero" still does not say which certificate a client received.

Store failed observations as evidence too. A wrong fingerprint, expired certificate, or name mismatch explains why the job stayed open and helps distinguish slow propagation from a bad attachment. Overwriting failure with the eventual success leaves an audit trail that cannot explain the outage window.

The familiar failure is a split deployment

A split deployment starts quietly. An agent obtains a new certificate for api.example.net, records the ACME order as valid, updates a secret called api-tls, and receives a successful response from the orchestration API. It then checks the secret metadata, sees the new resource version, and closes the ticket.

The public hostname resolves to two load balancer addresses. One controller watches api-tls in the production namespace and loads the new certificate. The other listener references a secret with the same name in a legacy namespace. No API call failed because the agent updated a real object, just not every object that controls the hostname.

Most users land on the first address and see the new certificate. A smaller path still serves the old leaf. Near expiry, failures appear intermittent, and retries seem to cure them because DNS or load-balancer selection sends the next connection elsewhere. Operators lose time blaming caches even though TLS certificates are selected during each new handshake.

Two-sided proof exposes the split immediately. The issuance record says fingerprint 6B:19:8A:.... Observations against the two addresses return that value and 41:D0:72:.... The rotation remains deployed_unverified, and the mismatching observation names the address that needs repair.

The popular recommendation to "check the certificate in a browser" is weak automation advice. A browser can reuse connections, hide the selected address, apply a different trust store, or sit behind inspection. It is a useful independent spot check during an incident, but it should not drive completion. Address-pinned, hostname-verified probes produce evidence an agent can compare and repeat.

After fixing the legacy attachment, probe both addresses again with fresh connections. Keep the first mismatch, the corrective deployment event, and the passing observations. That history proves what changed and why the workflow eventually closed.

Privileged actions need narrow authority and durable attribution

Certificate rotation gives an agent access to CA APIs, secret stores, and production reload paths, so the executor should expose narrow actions instead of raw credentials. The useful boundary is an operation such as "submit this CSR," "attach artifact fingerprint X to listener Y," or "probe host H through address A." Giving the model a reusable API token adds risk without improving its reasoning.

Per-action results should include the remote operation ID and evidence fields while excluding credentials and private-key bytes. Separate authority for issuance, deployment, verification, and rollback. A compromised or confused agent that can request a certificate should not automatically gain permission to attach it everywhere.

Sallyport can keep API and SSH credentials in its encrypted vault while the agent performs the HTTP and SSH actions through the app, so the credentials never enter agent context. Its Activity and Sessions journals derive from a write-blind encrypted, hash-chained audit log, which fits the need to attribute issuance, deployment, and verification calls without pretending those call logs replace the certificate observations.

Put human approval where impact rises. Routine renewal against a predeclared hostname and listener may fit session authorization, while a per-call approval is sensible for adding a new name, changing the key, touching a wildcard, or invoking rollback. Approval should bind the displayed target and artifact fingerprint; "allow certificate update" is too vague to catch a wrong listener.

Audit the verifier as carefully as the deployer. If the same unrestricted agent can alter endpoint inventory, perform deployment, and mark its own incomplete sample as sufficient, the evidence contract is cosmetic. Compute expected targets from a controlled source and let the closure check refuse missing observations.

Close only on convergence, and keep rollback ready

The closure action should run a deterministic comparison over immutable issuance, deployment, and observation events. It should not ask the agent whether the rotation "seems done." When all expected endpoints present the issued leaf through a valid, name-matched chain, record the exact predicate result and close the rotation.

Set deadlines around propagation rather than hiding it. During the allowed window, retry only endpoints that have not converged and use bounded backoff. At the deadline, fail the rotation with the mismatching targets attached. Whether to roll back then depends on validity and service health: an old certificate with ample lifetime may be safer temporarily, while an expired or revoked old certificate demands forward repair.

Rollback has the same proof burden. Reattaching the previous artifact is a deployment attempt, and the workflow must observe the previous fingerprint live before declaring recovery. If the incident involved a compromised private key, rollback to the compromised certificate is not recovery; issue with a fresh key and revoke according to the incident plan.

Retain both certificates until connection draining and rollback policy allow cleanup. A graceful reload can leave old workers serving established connections, but new probes should reach the new generation. Delete old private material through the secret system's controlled path, then record that cleanup as a post-completion action rather than a prerequisite for serving the new certificate.

The first change to make is small but strict: remove "certificate downloaded" or "secret updated" as a terminal success state. Require an issued fingerprint on one side and fresh, hostname-verified fingerprints from the full endpoint inventory on the other. An agent may perform every step, but only matching evidence gets to say the rotation is complete.