TLS certificate failures: an agent response procedure

A certificate error in an agent API call is a stop signal, not a minor networking annoyance. The call may carry a bearer token, a signed request, a customer record, or instructions that change production state. If the client cannot establish who owns the other end of the connection, it has no business sending any of them.

This gets mishandled because a human under pressure sees an expired certificate and reaches for an insecure flag. An autonomous agent makes that shortcut worse. It can retry quickly, follow an instruction hidden in a repository issue, and repeat the same unsafe call with more than one credential. A good response procedure stops the run, separates facts from guesses, and restores access only after someone proves the intended endpoint is again the endpoint receiving traffic.

A certificate warning changes the trust decision

TLS does two jobs that people too often merge into one. It encrypts traffic, and it verifies the server's identity. Encryption without identity verification merely gives an attacker a private conversation with your client.

RFC 6125 describes how an application checks a service identity against a reference identifier, usually the DNS name the client meant to reach. RFC 5280 describes certificate path validation: the client builds a chain from the presented leaf certificate through intermediates to a trust anchor. Both checks matter. A certificate can have a valid signature and still belong to api-attacker.example, not api.example. It can contain the right hostname and still chain to an issuer your organization never approved.

For agent traffic, the consequence is concrete. A bearer token injected into an HTTPS request becomes usable by whoever terminates the connection. A request signed with a client credential can authorize state changes. An API response from an impostor can instruct a later action or poison an agent's working context. The fact that the request was encrypted does not soften any of this.

Treat these messages as security-relevant until the team establishes their cause:

• certificate has expired or is not yet valid
• hostname or subject alternative name does not match
• unable to get local issuer certificate or unable to verify first certificate
• self-signed certificate in certificate chain
• certificate verification failed after a proxy, DNS, or network change

Do not classify every failure as an active attack. A forgotten intermediate certificate and a laptop clock set wrong are ordinary operational failures. The response starts from uncertainty because the same error class also appears when somebody redirects traffic or installs an unapproved interception proxy.

A successful TCP connection proves only that something answered at an address. A successful TLS handshake with verification disabled proves even less. The server identity check is the point at which a credentialed API call earns permission to leave the machine.

Stop the affected run before collecting clues

The first operational action is to prevent additional credentialed calls to the failing destination. Suspend the agent process, revoke its active session if your action layer supports revocation, or remove its permission to make the action. Do this before spending an hour debating whether the certificate looks familiar.

Preserve the original error exactly. Record the full hostname and port, the request method without its authorization header, the timestamp with timezone, the client runtime and version, and the process that initiated the call. Capture which network the machine used and whether the failure began after a deploy, certificate renewal, VPN change, proxy rollout, DNS update, or device management change.

Keep this evidence out of loose chat transcripts if it includes customer paths or request bodies. Never paste authorization headers, cookies, client private keys, or full environment files into an incident ticket. A certificate investigation does not justify creating a second secret leak.

Use a short incident record that makes ownership explicit:

1. Name a responder who can halt the agent and a responder who owns the endpoint or network path.
2. Mark whether the call had reached a credential injection point. If it had, assume the credential might have been exposed until the TLS peer is identified.
3. Preserve the agent session identifier and the failed action record, then prevent automatic retries for that destination.
4. Set a review point for restoration. The person who needs the API to work should not be the only person deciding that a new root certificate is trustworthy.

This is not ceremony. I have seen teams spend most of an incident testing commands while a background worker kept retrying the same failed endpoint every few seconds. The diagnostic work became part of the exposure. Stop the behavior first.

Do not let the agent "repair" the trust problem by editing global runtime settings. Prompts that suggest curl -k, verify=False, a blanket custom CA bundle, or disabled Node.js verification deserve the same response as a request to print an API token. The agent cannot determine whether a surprise certificate is an approved change just because a web search says the error is common.

Preserve the certificate without sending a secret

You can inspect a server certificate without authenticating to the API. Use an unauthenticated health endpoint if one exists, or establish a handshake and stop before any application request. Run the check from the same machine and network that saw the failure, because proxy settings, enterprise trust roots, and split DNS often differ across machines.

This command asks for the intended server name through SNI, checks the requested hostname, shows the certificates sent by the peer, and fails on verification trouble:

openssl s_client \
  -connect api.example.com:443 \
  -servername api.example.com \
  -verify_hostname api.example.com \
  -verify_return_error \
  -showcerts </dev/null

A clean result ends with output shaped like this:

Verification: OK
Verify return code: 0 (ok)

On failure, save the output as restricted incident evidence. The useful fields include the subject, issuer, Not Before and Not After dates, subject alternative names, and each certificate in the chain. The PEM blocks let the endpoint owner compare what your machine received with what the server should present. They are public certificates, but treat the record carefully because hostnames and internal issuers can still disclose infrastructure details.

Then test normal client behavior without an authorization header:

curl --verbose --fail --show-error \
  https://api.example.com/health

The verbose transcript exposes the TLS version, selected certificate, and verification message. Redact request paths if they reveal tenant names. Do not add -k to make this command complete. The failed verification is the evidence you need.

A common diagnostic error is leaving out -servername. Many hosted endpoints select certificates based on SNI and return a default certificate when the client omits it. That creates a hostname mismatch that never affected the real client. Another error is using openssl s_client without -verify_hostname, then treating a chain result as proof that the hostname matched. It is not.

Record the resolved address separately with your normal DNS diagnostic tool. Do not make address matching the entire test. Large providers legitimately use many addresses, while an attacker can control a DNS answer that looks plausible. The certificate name, issuer path, network path, and approved change record together tell the story.

Classify the failure before choosing a repair

Certificate errors look similar in a log but require different fixes. Put the evidence into a category before changing trust configuration. Otherwise teams solve the visible error and leave the original routing or deployment fault untouched.

An expired certificate usually means the endpoint operator missed renewal or failed to deploy the renewed leaf certificate. Check the validity interval from the captured certificate against a trusted clock. A clock error can make a current certificate appear expired or not yet valid, so verify the client time before asking a provider to rotate anything.

A hostname mismatch means the server did not prove ownership of the name the client requested. The usual causes are a wrong API base URL, an SNI configuration error, a default virtual host, stale DNS, or a proxy certificate for a different name. Do not accept an alternate name because it resembles the intended one. api.example.com and api.internal.example.com can represent different services and trust boundaries.

An unknown issuer can mean the client lacks a legitimate private root or intermediate certificate. It can also mean an unapproved TLS inspection device issued a substitute leaf certificate. Compare the issuer and chain with the endpoint owner's published or recorded deployment information through an independent channel. A chat message from the same system that produced the error is not independent confirmation.

An incomplete chain is usually the endpoint's fault. Servers must send the leaf certificate and required intermediate certificates. Clients should already possess only their trust roots. Installing a missing intermediate locally may hide the broken server configuration for one machine and leave every other caller broken. Ask the server owner to serve the correct chain.

A self-signed leaf certificate needs extra suspicion. It may be normal on an internal development service, but it carries no public identity proof on its own. A managed private CA can work well if administrators distribute its root through an approved device or workload trust store and record who controls that CA. A random self-signed certificate emailed by an endpoint owner is not a trust program.

Certificate revocation deserves care. Many clients do not reliably perform online revocation checks in every environment, and network failures can affect those checks. Do not claim a certificate is safe merely because a basic handshake returns 0 (ok). If an owner reports a compromised private key or revoked certificate, block the destination, rotate exposed credentials, and get a replacement deployment in place.

Separate the server path from the proxy path

A managed proxy can legitimately terminate TLS and issue substitute certificates, but that fact must be explicit. If a machine suddenly starts seeing an unfamiliar corporate issuer, do not label it harmless because the subject contains your company name. Confirm the proxy rollout, its approved root certificate, its scope, and whether the API's data and credentials may pass through it.

Check the calling environment for proxy configuration without dumping values that may contain credentials. On a Unix-like shell, this prints only the names of configured proxy variables:

for n in HTTP_PROXY HTTPS_PROXY ALL_PROXY NO_PROXY http_proxy https_proxy all_proxy no_proxy; do
  [ -n "${!n}" ] && printf '%s is set\n' "$n"
done

A proxy setting can explain why the same API works on a mobile connection but fails on an office network. It does not approve the proxy. Compare results on an authorized alternate network only when your organization's rules allow it. If the certificate issuer changes with the network, record that result and hand it to the network owner.

Also inspect name resolution and routing changes. A stale DNS record after a migration may direct calls to an old load balancer that presents a retired certificate. Split-horizon DNS may return an internal address with a private certificate to office devices and a public address elsewhere. These are fixable designs, but the client must trust the issuer appropriate to the route and still match the requested hostname.

Do not route around the issue by hardcoding an IP address. HTTPS identity checks use DNS names, and a direct IP can bypass intended routing, fail SNI selection, or mask a DNS control problem. An IP override is occasionally useful for a tightly controlled diagnostic test, but it needs an endpoint owner's approval and should never become the agent's permanent configuration.

Proxies also create a data governance question that a certificate error alone cannot answer. If the proxy terminates TLS, it can read API bodies and injected authorization material. The security team must decide whether that inspection is allowed for that API. The endpoint working again says nothing about whether the new path is acceptable.

Keep private CA trust narrow and reviewable

Private certificate authorities solve a real problem for internal APIs, development environments, and controlled service meshes. They fail when teams distribute roots informally and forget which processes trust them. A root certificate grants its holder broad power to issue identities inside its trust domain, so adding one is an administrative security change, not a compatibility tweak.

Install an approved private root through the managed operating system, container image, or application trust store that owns the call. Record the root's subject, fingerprint, owner, intended DNS suffixes, approval date, and removal path. Restrict the trust scope when the runtime allows it. A development CA should not quietly become trusted by every production agent on every machine.

Avoid trusting a single leaf certificate as a shortcut for a missing CA process. Leaf trust breaks at renewal and encourages people to paste PEM strings into repositories. It also makes it hard to distinguish a deliberate endpoint rotation from an unexpected substitution. If you control both ends, create a private CA process with protected signing material, documented issuance, and planned rotation instead.

Mutual TLS adds a separate identity direction. In ordinary HTTPS, the client validates the server certificate. In mTLS, the server also asks the client for a certificate. A failure such as "alert certificate required" or "bad certificate" may mean the client did not provide its own certificate, presented one from the wrong issuer, or lacks the matching private key. It does not mean that disabling server verification would help.

Keep client private keys outside prompts, shell snippets, and agent workspaces. Give the controlled caller access to the client credential through an operating system store or another protected mechanism. If a client certificate or its private key might have passed through an unverified TLS connection, revoke or replace it according to the issuing CA's procedure. Treat it differently from a server certificate, but do not treat it casually.

Certificate pinning is another area where good intentions cause outages. Pinning a leaf certificate binds the client to a particular renewal artifact, so a normal rotation can halt every agent. Pinning can make sense for a narrow, high-risk integration when operators maintain backup pins and test rotation. Most teams get better results by enforcing hostname verification, managing approved trust roots, and monitoring certificate expiry.

Restore access only after an independent check

Restore the agent only when someone has verified the repair outside the failed agent workflow. The check must use the actual hostname, expected trust store, and a client that performs normal verification. A browser visit is often insufficient because browsers and command-line clients may use different proxy settings and trust stores.

For a public API, verify that the presented chain leads to an expected public trust root, that the subject alternative names include the configured hostname, and that the endpoint owner confirms the deployment or renewal through a recorded change. For a private API, verify the approved private root, issuer, hostname, and expected network route. Capture the successful command result beside the original failure.

Then run one low-impact authenticated action through the same path the agent will use. Choose a read-only endpoint or a harmless identity check where the API provides one. Watch the action log and server-side audit record if available. Confirm the destination hostname, response status, and the credential identity used. Do not restore a broad batch job as the first test.

If the earlier failure left any chance that a token, signed request, or mTLS credential reached an impostor, rotate that credential before resuming normal work. Teams often resist rotation because the evidence is incomplete. That is backward. Incomplete evidence is why a credential that crossed an unverified connection has to be treated as exposed.

Write down the exact repair, not a vague conclusion such as "TLS fixed." A useful closure says that the endpoint operator deployed an intermediate certificate, that the managed trust store received a specified approved root, or that a proxy configuration was removed. It says who verified the hostname and which agent runs resumed. That record prevents the next responder from reintroducing an insecure override when the same certificate renews.

Make the safe path easier than the bypass

The procedure will fail if secure handling takes hours and the bypass takes one environment variable. Build the agent integration so it cannot choose an insecure TLS mode, cannot edit global trust configuration, and cannot see long-lived credentials while troubleshooting. Put connection settings, approved hostnames, and trust material under normal change review.

Sallyport can keep API credentials in an encrypted vault and perform HTTP calls for an agent, so an agent does not receive the credentials themselves. Its session and call records can also preserve who attempted the action, but a human still needs to decide whether a changed certificate is an approved change or a dangerous route.

Alert on repeated validation failures and on any attempt to invoke known bypasses. A failed call should include enough structured context for a responder to identify the hostname, failure class, and agent run without including a secret. Give agents a fixed instruction: stop on certificate validation errors, report the exact error and endpoint, and request human review. Do not ask them to find a workaround.

The first repair worth making is usually boring: remove any insecure verification switch already present in scripts and make the normal client use a managed trust store. Then test certificate renewal before production does it for you. Teams that rehearse one renewal and one unexpected-issuer incident will respond faster than teams whose only TLS plan is a command copied from a forum.

A certificate warning has done its job when it interrupts a request that should not proceed. Keep it that way.