Expired API tokens: recovery rules for long agent jobs

Long agent jobs fail in a particularly stupid way when token expiry has no owner. The agent sees a 401, repeats the same call, consumes rate-limit budget, and sometimes turns one uncertain write into several. That is not an authentication glitch. It is a recovery design failure.

Plan for expiry as an expected state change. Give renewal to one component, classify failures before acting, bind retries to the semantics of the operation, and leave a record that lets an operator see whether the remote side already accepted the work. An agent should never have to guess whether it can mint credentials or whether it is safe to send a write again.

Expiration is a state change, not an exceptional outage

An expired API token tells you that authorization for a short-lived credential ended; it does not tell you that the task failed. A long-running job may have completed ten remote operations before the next request encounters expiry. The recovery code must preserve that distinction.

Teams often collapse four different events into one branch called auth_failed. That shortcut creates bad behavior because each event needs a different response:

• Expiry means the token lifetime ended and the authorized renewal owner may request another access token.
• Revocation means a user, administrator, or provider withdrew the grant. Renewal may fail by design.
• Invalid request authentication can mean a malformed header, wrong credential type, issuer mismatch, or wrong audience.
• Insufficient permission means the identity remains valid but cannot perform this operation.

OAuth names these distinctions for a reason. RFC 6750 specifies the invalid_token bearer-token error and says a resource server uses a 401 response with a WWW-Authenticate challenge when the token is expired, revoked, malformed, or otherwise invalid. That is useful protocol guidance, but it does not grant your client permission to refresh blindly. The resource server only reports that it rejected this request.

A job also has two timelines. Its work timeline records what it has discovered, calculated, created, and confirmed. Its authorization timeline records the credential generation that let it act. When a token expires, preserve the work timeline and move the authorization timeline to renewing or blocked. Do not reset the whole job and call that recovery.

This distinction matters most for agents because they make a chain of dependent calls. Suppose an agent creates a change request, uploads an artifact, then loses access before it can attach the artifact. Restarting from the first instruction can create a second change request. A durable checkpoint after each confirmed remote effect makes the agent continue from the missing attachment rather than replaying the entire plan.

Use explicit states rather than a boolean such as authenticated:

ready -> executing -> authorization_expired -> renewal_in_progress
renewal_in_progress -> executing
renewal_in_progress -> authorization_blocked
executing -> outcome_unknown
outcome_unknown -> reconciled -> executing

outcome_unknown deserves its own state. A connection can die after the service commits a write but before the caller receives the response. Token renewal cannot resolve that uncertainty. The job must query by its operation identifier, idempotency token, or a provider-specific lookup before it attempts the write again.

One component must own renewal

The client that holds the refresh credential should own access-token renewal, and the agent should request an action rather than receive renewable credentials. This rule sounds restrictive until two agents hit expiry at once.

If every worker carries a copy of a refresh token, each worker can renew independently. They race, they produce a trail full of unrelated credential events, and refresh-token rotation can invalidate a token another worker still holds. More importantly, you have turned every process that can read a task into a long-lived identity holder.

Put a credential broker or action gateway between agents and the provider. The broker stores the refresh credential or service credential, obtains short-lived access tokens, attaches one only when it executes a request, and returns the response. The agent receives neither an access token nor a refresh token.

This split gives each actor a clear job:

• The agent decides which allowed operation it wants to perform and supplies the operation inputs.
• The gateway checks whether the session can make the request, selects the credential reference, and executes the call.
• The renewal owner refreshes once when the provider reports a classified expiration failure.
• An operator handles a revoked grant, a new consent requirement, or a credential that needs human presence.

Do not make the agent the renewal owner merely because it can call an OAuth endpoint. Capability and authority are different things. An agent that can ask for a calendar event or deploy an artifact does not automatically need permission to extend a person or service identity.

RFC 6749 describes refresh tokens as credentials issued to the client and used to obtain new access tokens. Read "client" literally in your architecture. If your agent is not the registered client, it should not inherit the client's refresh token just because it produces the API request.

There are legitimate cases where a job itself owns renewal. A narrowly scoped machine workload with its own registered client, its own storage boundary, and no human delegation can do that. Even then, one renewal coordinator should serve all concurrent operations for that identity. Use a mutex or a single-flight mechanism keyed by credential reference. The first failing call renews; other calls wait for the result instead of stampeding the token endpoint.

A simple ownership record prevents vague designs:

{
  "credential_ref": "billing-write-prod",
  "renewal_owner": "action-gateway",
  "access_token_lifetime": "provider-defined",
  "refresh_allowed": true,
  "reauthorization_owner": "on-call-operator",
  "concurrent_refresh": "single-flight"
}

The record contains a reference, never the credential itself. It also names who must act when refresh no longer works. If nobody can answer that question before deployment, the job will answer it badly at night.

A 401 needs evidence before it triggers renewal

Renew only when the response and the credential record support an expiry diagnosis. Treating every 401 as expiry masks configuration defects and can generate a long chain of pointless refresh attempts.

Start with the provider's documented error body, headers, and token format expectations. Some APIs return OAuth-compatible WWW-Authenticate values. Others return JSON error codes. Some put authentication failures behind a gateway that uses a different status code. Your classifier should use the documented signals for that provider, then fall back to a safe terminal error.

This is a workable classification contract:

{
  "http_status": 401,
  "provider_code": "invalid_token",
  "www_authenticate": "Bearer error=\"invalid_token\"",
  "credential_ref": "reports-read",
  "token_generation": 17,
  "decision": "renew_once"
}

A response can qualify for renew_once only if all of these hold: the request used a credential that your gateway issued or selected, that credential has a renewable path, the provider signal matches the documented expiration or invalid-token condition, and this job has not already renewed generation 17.

Use a different result for each failure class. A malformed header belongs in configuration_error, where a developer can inspect the request builder. An audience mismatch belongs in credential_binding_error, where someone must fix the token request or resource configuration. A revoked grant belongs in reauthorization_required, where the system stops external actions and tells the right operator exactly which identity needs consent. A 403 belongs in permission_denied; refreshing it is cargo-cult behavior.

Clock problems cause a surprising share of false diagnoses. A client that computes local expiry can reject a usable token early, while a client with a drifting clock can send an expired one. Record the issuer-provided expiry when you receive the token, keep a small safety margin, and use a trusted system clock. Do not let every agent calculate its own expiry from a decoded token claim. That duplicates protocol logic and invites disagreement.

Do not inspect a token payload merely to decide whether you can trust it. A JSON Web Token can contain an exp claim, but decoding its base64url payload does not verify its signature, issuer, audience, or revocation status. Use it as a hint only after the component that received it has completed the provider's documented validation. Opaque access tokens offer no payload to inspect, which is another good reason to make the caller rely on response handling rather than token archaeology.

Retry limits protect the remote system and your evidence

After a classified expiry, allow one coordinated renewal and one controlled replay. More retries do not improve authorization; they mostly hide a broken renewal path and make the audit trail harder to read.

The replay rule depends on what the operation can do. A read request normally tolerates a replay after renewal. A write request needs stronger evidence because the remote service may have processed it before the expiration response, timeout, or connection loss reached the caller.

Classify operations when you design the action interface:

HTTP method names do not settle this. PUT often has idempotent intent, but a provider can attach an email notification or asynchronous downstream action to it. POST can be safe if the provider supports an idempotency field. Read the endpoint contract and test the actual behavior.

For operations with an idempotency token, create it before the first network call and persist it with a canonical request fingerprint. On every retry, send precisely the same token and logically identical payload. Do not generate a fresh token after a 401. A fresh token tells the provider that this is a new operation, which defeats the point.

{
  "operation_id": "job-84f3/create-draft",
  "idempotency_token": "a stable random value stored before send",
  "request_fingerprint": "method, path, normalized body hash",
  "attempt": 1,
  "authorization_generation": 17
}

The phrase "normalized body hash" matters. If a retry builder changes a timestamp, array ordering, or generated label, it can silently turn the same idempotency token into an incompatible request. Some providers reject that mismatch. Others handle it inconsistently. Keep the first serialized request body or canonicalize it once and reuse it.

Rate-limit and network retry logic must share this budget. An agent that uses three network retries, then a renewal retry, then three more network retries has created seven chances to duplicate or overload an operation. Define one operation-level attempt budget. For example, a safe read might allow an initial call, one renewal path, and one replay. A payment-like action may allow an initial call and then reconciliation only.

Log every suppressed retry too. Operators need to see that the system intentionally stopped after renewal_attempted=true, rather than assuming the agent crashed. That record also gives you a clean signal when a provider changes an error format and your classifier starts refusing a recovery it once permitted.

Checkpoints let a job resume without inventing its past

A long agent task should persist completed remote effects and pending intent separately, because a renewed token cannot tell you what happened before the failure. The common bad pattern stores only a chat transcript or an agent's final plan, then asks the agent to reconstruct state after an interruption.

Use a task journal with records that a program can reconcile. Each intended external call needs a stable operation ID. Each confirmed result needs the provider's resource ID, version or ETag if available, and the request fingerprint. Each uncertain result needs the lookup rule that resolves it.

A compact checkpoint might look like this:

{
  "task_id": "release-2025-04-17-42",
  "completed": [
    {"operation_id": "create-change", "remote_id": "CR-819", "version": "6"},
    {"operation_id": "upload-bundle", "remote_id": "asset-552"}
  ],
  "pending": {
    "operation_id": "attach-bundle",
    "request_fingerprint": "POST /changes/CR-819/assets body-sha256:...",
    "reconcile": "list assets for CR-819 and match asset-552"
  },
  "authorization_state": "authorization_expired"
}

The job does not need to save every intermediate thought. It needs sufficient facts to determine the next safe remote action. Keep secrets, bearer headers, and refresh responses out of this journal. Those materials belong to the credential owner, not to general task storage.

Version conditions matter during a pause. If the task read document version 6 before expiry and resumes an hour later, another actor may have changed it. Use ETags, revision numbers, conditional headers, or provider-specific concurrency fields when available. If the condition fails, tell the agent that the old plan no longer applies. Do not renew the token and overwrite a newer state because the task believes it owns the world.

This is where autonomous work needs a boundary around judgment. A job may safely resume an upload whose destination and hash it already recorded. It should not casually re-plan a deployment, modify an approval record, or choose a different target after its original context has aged. Mark those operations as requiring fresh confirmation after recovery.

The recovery response must tell the agent what it can do

An action gateway should return a structured recovery result, not a vague authentication sentence that invites the agent to improvise. The result should tell the agent whether the call ran, whether the gateway renewed authorization, whether replay is permitted, and whether a human must intervene.

A useful result separates execution status from credential status:

{
  "operation_id": "attach-bundle",
  "execution_state": "not_sent",
  "authorization_state": "reauthorization_required",
  "retry_allowed": false,
  "credential_ref": "release-api",
  "operator_action": "Reauthorize the release-api connection, then resume task release-2025-04-17-42",
  "safe_resume_from": "attach-bundle"
}

not_sent means the gateway stopped before it handed the request to the network client. outcome_unknown means it cannot make that claim. Do not let both cases collapse into failed. The first can wait for reauthorization. The second must reconcile against the remote service before anything retries.

Agents also need a narrow vocabulary for recovery. Give them outcomes such as completed, renewed_and_replayed, needs_reconciliation, reauthorization_required, permission_denied, and configuration_error. Each outcome should map to one permitted behavior. For example, an agent can continue after renewed_and_replayed; it can run a documented read-only reconciliation after needs_reconciliation; it must stop external writes after reauthorization_required.

Avoid returning raw provider responses as the only signal. Raw details help diagnostics, but agents can interpret them poorly, especially when providers use inconsistent prose. Keep the raw response in a protected diagnostic record and return a stable machine-readable decision to the caller.

A good error message names the identity reference and the blocked operation without exposing secret material. "Credential release-api needs reauthorization before attach-bundle can run" tells an operator where to act. "Unauthorized" tells nobody anything.

Refresh credentials need tighter controls than access tokens

A refresh credential deserves stronger protection because it can usually outlive the access token it replaces. Do not solve token expiry by distributing this longer-lived credential to every agent workspace, build directory, environment variable, or transcript.

OAuth 2.0 Security Best Current Practice, RFC 9700, recommends refresh-token rotation or sender-constrained refresh tokens for public clients. The exact provider support varies, but the security lesson applies even when your provider uses a different protocol: a stolen renewable credential has a much longer window for misuse than an ordinary short-lived bearer token.

Keep the renewal material inside the action gateway's encrypted credential store. Restrict which action definitions can select it, require an explicit human approval where the action warrants one, and make reauthorization a separate operator action. A locked credential store must deny work rather than allowing agents to fall back to copied secrets in configuration files.

Sallyport follows this model for supported HTTP and SSH actions: its encrypted vault retains the credential, while the agent requests an action through its MCP shim and receives only the result. That division is useful because the agent cannot print a refresh credential into its own context when recovery goes wrong.

Be disciplined about refresh responses. Some providers rotate the refresh credential and invalidate the old value on use. The renewal owner must replace the stored value atomically before it releases waiting calls. If it writes the new access token but loses the replacement refresh credential, the job may work briefly and then fail permanently at the next expiry.

Never log these fields: Authorization, access token, refresh token, client secret, signed assertion, or full token endpoint body. Masking is not enough when systems copy raw request objects before the masker runs. Design the logger to accept credential references and token generation numbers instead of secret-bearing structures.

Human approval should resume authority, not create a retry storm

Human approval helps only when it maps to a clear decision: permit this agent run, permit this sensitive call, or restore a revoked authorization. A generic "retry" button after expiry often turns an operator into a rubber stamp for an unclear action.

Separate the approvals. Reauthorization gives the credential owner a new grant or a usable renewal path. Session authorization decides whether this particular agent process may request actions. Per-call approval decides whether a sensitive operation may happen now. Those are different decisions, and merging them produces either noisy prompts or excessive standing permission.

When a credential needs reauthorization, show the affected credential reference, the identity or connection label, the blocked operation, and the task checkpoint. Do not show the token itself. Once the operator restores authorization, the gateway should resume only the pending operation recorded in the checkpoint. It should not silently replay every failed call from the agent's transcript.

Sallyport's fixed decision ladder is a sensible fit for this boundary: a locked vault denies every action, new agent processes need session approval by default, and selected credentials can require approval on every use. The controls do not try to infer intent from a pile of rules, which helps when an expired credential interrupts an otherwise legitimate run.

Approval fatigue is usually a design error. If an operator sees a dozen prompts because ten parallel calls all notice the same expiry, the renewal coordinator failed to collapse the event. Present one reauthorization request, make other calls wait, and report the resulting decision to each task.

Do not use approval to paper over an unknown write outcome. The right prompt in that case asks whether the operator wants the system to reconcile the remote state, not whether they want to retry. A person can authorize a duplicate by mistake just as easily as an agent can.

Expiry tests must include uncertain writes and concurrent workers

A token-renewal test that returns one synthetic 401 before a read proves very little. The failures that hurt teams appear at boundaries: during parallel work, after a write commits, when refresh rotation occurs, or when a grant has been revoked.

Build a test provider or controllable HTTP fixture that records received operation IDs and can inject failure at defined points. Your assertions should inspect both the remote-effect record and the local audit record. A successful final response alone can conceal a duplicate create.

Run these cases before you trust a long-running agent:

1. Expire the access token before a read. Confirm that only one worker renews and all waiting reads use the replacement generation.
2. Expire it before an idempotent write. Confirm that the gateway renews once and replays with the original operation ID and request body.
3. Drop the response after the provider records a non-idempotent write. Confirm that the job enters outcome_unknown, runs its lookup, and does not create a second effect.
4. Revoke the grant before renewal. Confirm that every dependent action stops with reauthorization_required and that no loop calls the token endpoint repeatedly.
5. Return a rotated refresh credential, then interrupt storage. Confirm that the gateway detects the incomplete replacement and blocks future renewal rather than using a stale copy.

Test time explicitly. Inject a clock into the credential component so you can place expiry just before request construction, just after header construction, and while a request waits in a queue. Sleeping until a real token expires makes tests slow and leaves the important timing cases untested.

Finally, test the audit path without access to the vault. You should be able to verify that the sequence of attempted calls and renewal decisions has not changed, even if you cannot decrypt every record. Sallyport's sp audit verify checks its hash chain over encrypted audit data without needing a vault key, which is the right shape of verification for an incident where credential access may remain locked.

A job that handles expiry well does less after authorization fails. It stops, classifies the failure, lets one owner renew, reconciles uncertainty, and resumes only the recorded operation. That restraint prevents duplicate side effects and leaves an operator with evidence instead of a pile of retries.