AI agent API rate limit: retries that stay safe

An agent that hits a quota should slow down, account for every extra attempt, and stop before it turns a temporary refusal into a wider operational mess. “Retry on failure” is adequate advice for a person clicking a button. It is dangerous advice for a process that can issue requests faster than anyone can watch.

The hard part is not sleeping for a few seconds. The hard part is preserving the meaning of the original task when the first request may have failed before it reached the provider, after the provider completed it, or because several agent runs exhausted one shared allowance. A safe design separates those cases and makes human intervention a defined outcome rather than an embarrassing exception.

A 429 is a scheduling instruction, not an error to hammer

HTTP 429 means the server is refusing the request because the client sent too many requests in a period the server controls. RFC 6585 defines the status code and says a response representation should explain the condition and may include a Retry-After header. That wording matters: a 429 does not tell an agent that repeating the exact request immediately has any chance of success.

An agent should first capture the provider, endpoint, credential or account identity, request class, response status, and any rate-limit headers. It should then place the work in a delayed state. Do not let the language model decide this in prose after every failure. The transport layer needs deterministic behavior, because a model under task pressure will often try a nearby endpoint, change a filter, or create a second request path. Those improvisations can multiply calls while producing the same refusal.

A refusal often applies to a bucket wider than the current request. Providers commonly set limits per account, project, token, IP address, endpoint family, or a combination of those. One agent may receive 429 while another agent using the same credential continues working briefly. That does not prove the first one can retry safely. It may mean the provider has multiple buckets, or that the second process is consuming the last available capacity.

Keep a local ledger for each known scope. If the service documents a per-token limit, group requests by token. If it gives only account-level guidance, assume every worker for that account shares the bucket until evidence says otherwise. A local ledger will not perfectly mirror the provider's internal counter, but it prevents your own workers from competing blindly.

The IETF RateLimit Fields specification, RFC 9333, defines RateLimit-Limit, RateLimit-Remaining, and RateLimit-Reset. These fields describe quota state, but they do not erase a 429 already received. Use them to pace future requests and to avoid filling the queue faster than the provider can accept it. Treat their scope according to the provider's documentation, because a header alone may not reveal whether it applies to one endpoint or an entire account.

A useful state record looks like this:

{
  "provider": "billing-api",
  "scope": "account:ops-team",
  "request_class": "write:invoice",
  "next_allowed_at": "2025-04-08T14:12:31Z",
  "remaining": 0,
  "reset_after_seconds": 60,
  "source": "HTTP 429 Retry-After"
}

The agent runner should read this record before dispatching another call. Sleeping inside one request handler is acceptable for a small command-line run. A queue with a visible next_allowed_at is better once several workers, tools, or resumed tasks exist. It lets the scheduler choose other work instead of keeping a process occupied and gives an operator a clear explanation for delay.

Retry budgets stop a small failure from becoming a flood

A retry budget puts a hard ceiling on the extra work a task may create after a failed call. Count both attempts and elapsed wait time. A fixed count alone fails when the service asks clients to wait minutes; a time limit alone fails when a fast retry loop burns through an account allowance in seconds.

Use separate budgets for each task and each shared provider scope. A task budget answers, “How much uncertainty may this one objective tolerate?” A scope budget answers, “How much disruption may all current work impose on this provider?” If ten tasks each receive three retries, the account may still get thirty extra requests. That is exactly how a supposedly conservative policy becomes a burst.

For ordinary read requests, I use a small budget with a deadline shorter than the business value of the answer. For writes, I spend much less budget on blind retries. The cost of waiting for a human can be lower than the cost of duplicating a payment, sending duplicate notices, or applying a deployment twice.

Represent the rule as data that the agent cannot casually override:

request_classes:
  read:
    max_attempts: 4
    max_wait_seconds: 90
    retry_statuses: [408, 429, 500, 502, 503, 504]
  idempotent_write:
    max_attempts: 3
    max_wait_seconds: 120
    retry_statuses: [408, 429, 502, 503, 504]
  uncertain_write:
    max_attempts: 1
    max_wait_seconds: 0
    retry_statuses: []
shared_scope:
  max_delayed_requests: 25
  max_concurrent_requests: 2

This fragment prevents a familiar failure: an agent receives a timeout after submitting a write, assumes nothing happened, and retries against a provider already under strain. The uncertain_write class forces a status check or escalation instead. It does not reward the agent for persistence when persistence changes the outcome.

The budget must charge every actual attempt, including retries started by libraries. I have seen retry controls declared in an application while an HTTP client, a workflow runner, and a proxy each retried beneath it. The resulting request count looked mysterious until someone read all three defaults. Pick one layer to own retries. Configure every other layer to report errors upward without retrying, or make its behavior explicit and include it in the total budget.

Budget exhaustion should produce a terminal result with context, not a generic failure. The result should say whether the original request was sent, whether a response arrived, how many attempts occurred, how long the task waited, and the next safe action. An agent can then continue with unrelated work or present a precise escalation rather than repeatedly asking itself to “try again.”

Backoff needs jitter and a ceiling

Exponential backoff reduces pressure after repeated refusal by increasing the delay between attempts. Jitter prevents many workers that failed at once from returning in the same synchronized wave. Both belong in the client, even if the provider publishes a reset time, because internal concurrency can create its own thundering herd.

For attempt number n, where the first retry is n = 1, calculate a cap and choose a random delay within it:

import random

BASE_SECONDS = 1.0
MAX_SECONDS = 60.0

def retry_delay(attempt_number: int) -> float:
    cap = min(MAX_SECONDS, BASE_SECONDS * (2 ** attempt_number))
    return random.uniform(0, cap)

This is full jitter. It avoids the trap of having every worker sleep exactly 2, 4, 8, and 16 seconds. Equal fixed delays are popular because they make logs easy to read. They also make coordinated retries easy to predict, which is the opposite of what a busy service needs.

When a response supplies Retry-After, that value takes precedence over a locally calculated shorter delay. RFC 9110 permits Retry-After as either a delay in seconds or an HTTP date. Parse both forms. If the date is already in the past because the machine clock differs from the provider's clock, apply a modest minimum delay rather than retrying in a tight loop.

Do not use backoff as a substitute for rate pacing. Backoff starts after a failure. A token bucket, leaky bucket, or simple scheduler limit controls the rate before failure. If a provider allows a known number of requests during a known interval, pace requests below that published allowance and reserve room for interactive work. The scheduler should also limit concurrency. Twenty simultaneous requests can consume a short interval's capacity before any worker reads a RateLimit-Remaining response.

A practical dispatch rule is simple: before sending a call, check the shared scope's next allowed time and available concurrency slot. After a 429, update the scope record before any retry is scheduled. That ordering matters. If workers schedule retries before publishing the refusal, each one may conclude it owns the next slot.

Cap the delay and cap the total wait. An uncapped exponential curve can defer a task for hours and leave a stale run alive long after its assumptions expired. The agent should fail the task or hand it to a scheduler after its deadline, with the original input preserved for later review.

Idempotency decides whether a retry is safe

An idempotent request has the same intended effect when repeated. That does not mean every request that uses HTTP PUT is harmless in every application, and it does not mean every POST is dangerous. RFC 9110 describes methods such as PUT and DELETE as idempotent in intent, while POST is not generally idempotent. The provider's actual API contract decides the operational risk.

Read requests usually tolerate retries, provided the agent accepts that the returned data may have changed. A delete request may also tolerate a retry if deleting an already absent resource yields a known, acceptable result. Creating a payment, invitation, support ticket, purchase order, or deployment often does not tolerate blind repetition. These are the calls that deserve the most skepticism after a timeout or connection reset.

The field routinely blurs two different states:

• A request that definitely did not reach the provider is safe to resend if the operation itself permits it.
• A request whose outcome is unknown needs reconciliation before the agent sends another side effect.

A connection failure after the client writes bytes does not prove the provider failed to act. The server might have completed the operation and lost the response connection. The agent cannot infer the outcome from the absence of a response.

When a provider supports an idempotency key, generate one stable token per logical action and reuse it for every retry of that action. Do not generate a fresh token on each attempt. A fresh token tells the provider that each retry is a separate operation, which defeats the protection.

POST /v1/transfers HTTP/1.1
Idempotency-Key: transfer-7c41b5b9-7f5b-4f51
Content-Type: application/json

{"source":"acct_17","destination":"acct_42","amount":12500,"currency":"USD"}

Store the token with the task and request payload before sending the first call. If the runner restarts, it must recover the same token. Store the provider's returned operation identifier too. That identifier lets a later reconciliation call ask about the original action without recreating it.

If the API offers no idempotency feature, look for a create-then-query pattern. The agent can attach a client-generated external reference in the payload, then search by that reference after an uncertain failure. If the API offers neither idempotency nor a reliable lookup, do not automate repeated writes. Require a human to inspect the provider's records. This feels slower until the first duplicated side effect lands in a system that cannot cleanly undo it.

Separate rate limits from outages and bad requests

A retry policy that treats every non-success status alike will hide defects and waste quota. Classify the response before choosing a delay. The status code, response body, provider error code, and request method all matter.

Use this practical split:

• 429 means pace down, honor provider guidance, and charge the shared rate-limit budget.
• 408, connection resets, and some 5xx responses may merit a bounded retry, but write operations still need an idempotency or reconciliation path.
• 400, 401, 403, 404, and 422 usually need a corrected request, changed authorization, or a human decision. Repeating them is wasteful.
• 409 requires resource-specific handling. It may mean a duplicate, a version conflict, or a lock held by another workflow.
• A provider-specific quota-exhausted error may require waiting until a billing or daily reset, which differs from a short burst limit.

Do not treat 503 Service Unavailable as an interchangeable form of 429. A 503 says the service cannot serve the request at that time; a 429 says the client exceeded a limit. Both might carry Retry-After, but 429 should cause your scheduler to reduce local throughput for the affected scope. A 503 may instead be regional, endpoint-specific, or provider-wide. Preserve the distinction in metrics and operator messages.

Agent behavior can cause invalid-request storms too. A model may repeatedly call an endpoint with an unsupported filter after receiving 422, or retry 401 after the credential was revoked. Put a circuit breaker around repeated identical failures. For example, if the same endpoint, method, and normalized error code fail several times in one run, stop that path and return the error details to the agent as a constraint. Do not allow it to vary whitespace, reorder JSON fields, and pretend it is exploring new options.

A normalized request fingerprint should exclude secrets and volatile headers. Include the method, endpoint template, stable payload fields, and API error code. This lets the runner identify a loop without logging credentials or full sensitive bodies.

Shared credentials need one queue, not polite agents

A shared API credential creates a coordination problem that individual agents cannot solve through good intentions. Each process sees only its own requests unless a scheduler gives them a common budget. Per-agent backoff lowers the chance of a burst, but it does not allocate a scarce account-wide quota fairly.

Place an outbound queue in front of the shared credential scope. Assign priority deliberately. A human-approved production change may need precedence over background inventory work. A long-running agent that discovers ten thousand records must page through them at the rate the provider permits, rather than filling the queue with every page request at once.

The queue needs cancellation. If an agent's parent task ends, discard its delayed requests before they wake up. Otherwise a stopped task can continue consuming quota later and make the audit trail confusing. Cancellation must also release any reserved concurrency slot.

Use a request coalescing rule for safe reads. If five agent runs ask for the same immutable record within a short interval, perform one request and distribute the result to all waiting tasks. Do not coalesce reads where freshness changes the meaning, such as a balance or an approval status. The point is to remove accidental duplicates, not to build a cache that returns stale facts to an agent making decisions.

Rate limits often expose a deeper planning issue. An agent that calls a detail endpoint once per item may be following its instructions exactly while using the wrong access pattern. Before adding retries, look for batch endpoints, pagination controls, conditional requests, webhooks, export jobs, or a server-side search endpoint. These changes reduce calls before the provider has to refuse them.

Sallyport can keep API credentials outside the agent process while recording the individual outbound calls, but the caller still needs queueing and retry controls. Credential separation reduces secret exposure; it does not change the provider's quota.

Human escalation should preserve the uncertainty

A human should take over when the system cannot establish whether a side effect occurred, when the remaining wait exceeds the task's deadline, when the shared quota is exhausted, or when repeated limits point to an account configuration issue. Escalation is not a generic “API failed” notification. It is a compact case file that lets someone decide without reconstructing the run from scattered logs.

Send the operator these facts:

• the task's requested outcome and the exact logical action that stopped;
• the provider, endpoint, method, account scope, and sanitized request fingerprint;
• attempt timestamps, statuses, Retry-After or rate-limit headers, and budget consumed;
• whether the operation has an idempotency token, provider operation ID, or a reconciliation query;
• the next safe options, such as wait, query status, increase quota, alter the plan, or cancel.

Do not put a raw request body in an approval card by default. It may contain personal data, internal document contents, or a value that looks harmless until it reaches the wrong person. Show a concise summary and make detailed access an intentional audit action.

Approval should request a decision, not merely consent to “continue.” For an uncertain write, offer “query existing operation,” “retry with the same idempotency token,” “cancel,” and, where justified, “send a new operation.” The last option should state plainly that it may create a second side effect. People make better calls when the choices describe the actual risk.

An approval for continued access and an approval for a particular external action are different controls. A session may remain authorized while an agent still needs per-call review for a payment, production write, or repeated request after a limit event. Keep those decisions separate in both the interface and the log.

When reviewing an escalation, start with reconciliation. Query the provider by idempotency key, external reference, or operation identifier. Only retry after the query shows no completed action, or after the provider guarantees duplicate suppression. This order is slower than a blind resend by one network round trip. It is faster than repairing a duplicate that entered a downstream ledger.

Audit records must explain the attempted action and the wait

A useful audit record answers more than “did the request return 200?” It should show what the agent attempted, what authorization allowed it, what the remote service returned, and how the retry controller reacted. Without the decision record, a sequence of calls can look like careless repetition even when the scheduler obeyed a documented Retry-After value.

Record an event before dispatch, then append result and scheduling events. Keep the request metadata free of plaintext credentials. A compact sequence might read:

14:11:02 action_requested  task=sync-482 method=POST route=/records
14:11:02 action_sent       attempt=1 idempotency=rec-91f2
14:11:03 action_result     status=429 retry_after=30 scope=account:ops
14:11:03 retry_scheduled   attempt=2 due=14:11:33 budget_wait=30
14:11:33 action_sent       attempt=2 idempotency=rec-91f2
14:11:34 action_result     status=201 provider_id=r_893

That sequence separates a logical action from its transport attempts. If a human asks why two POST calls occurred, the record shows they shared one idempotency token and followed a provider-directed delay. If the second response had timed out instead, the next event should be reconciliation_required, not another automatic action_sent.

Tamper-evident logging has a practical benefit during incident review: it lets a team verify that a run did not erase inconvenient attempts after the fact. Sallyport projects its session and call journals from an encrypted hash-chained audit log, and sp audit verify checks the chain offline without needing a vault key. That is useful when an operator needs to distinguish a provider throttle from an agent loop or a late manual retry.

Logs also need retention and access boundaries. A request path, provider account identifier, and timing pattern can reveal sensitive operational activity even when the token itself never appears. Capture enough to reconstruct the decision, then limit who can search and export the record.

Test the failure paths before an agent finds them in production

A rate-limit design is incomplete until a test proves that it stops calls, preserves idempotency, and escalates uncertainty. Mocking a single 429 is not enough. You need concurrent workers and failures that occur after the remote service may have acted.

Run this sequence in a test environment or against a controllable fake API:

1. Start three agent tasks that use one simulated account scope and issue the same safe read request.
2. Return 429 with Retry-After: 10 to the first request, then verify the scheduler delays all work in that scope instead of letting the other two continue at full speed.
3. Return success after the delay and check that retry times differ slightly rather than arriving as one burst.
4. Send a write with a stable idempotency token, simulate a connection drop after the fake API stores the record, and verify the runner queries status rather than issuing a new create.
5. Exhaust the configured time budget and verify that the operator receives the sanitized escalation record and that canceled work never wakes up later.

Measure outbound attempts at the fake API, not only function calls inside the agent. Internal counters miss retries added by HTTP libraries or wrappers. Test a restart too: stop the runner after the first uncertain write, restore its persisted state, and confirm that it resumes reconciliation with the original idempotency token.

Do not reward the agent test harness for eventually obtaining a success response. Reward it for making the right number of calls, waiting when instructed, and leaving an ambiguous write unresolved until it has evidence. An agent that “gets the job done” by duplicating external actions has not passed the test.

The first control to implement is a shared retry budget with explicit terminal outcomes. It turns rate limiting from an invitation to keep trying into an operational decision with a record, a deadline, and a person who can intervene when the remote state is uncertain.