# Status endpoint polling: how agents wait without loops An agent that starts an external job must not keep asking "are we there yet?" until a provider, a budget, or a human gives up. Status endpoint polling needs an explicit contract: what counts as progress, when the next request may occur, when waiting ends, and who decides what happens after that. I have seen apparently harmless status checks turn into hundreds of calls because a job identifier was valid, the endpoint kept returning HTTP 200, and nobody told the agent that "running" was no longer an acceptable answer after a deadline. The fix is not clever scheduling. It is making waiting a bounded action with evidence and an escalation path. ## A running status is permission to wait, not permission to act A nonterminal job state permits another observation later. It does not permit the agent to fetch output, start dependent work, retry the original submission, or widen its authority. That distinction matters because asynchronous APIs often return a successful HTTP response for every state. A response such as this says the status endpoint worked. It does not say the job succeeded. ```json { "job_id": "exp_71c", "state": "running", "updated_at": "2025-04-18T10:24:00Z" } ``` Treat the HTTP result and the job result as two separate facts. The first answers, "Did the provider answer this request?" The second answers, "May the workflow advance?" Teams blur them constantly, then an agent downloads an incomplete export or posts a result that never existed. Write a small state table for every provider integration. Do not infer meaning from a field name such as `status`; providers use the same word for very different lifecycles. A useful table has these categories: - **Pending states** permit a later status request, such as `queued`, `running`, or `processing`. - **Success states** permit the specifically named follow-up action, such as retrieving a result URL. - **Failure states** stop the run and preserve the provider error. - **Cancellation and expiry states** stop the run without resubmitting unless a human explicitly asks for it. - **Unknown states** stop the run because an integration cannot safely guess whether `paused`, `awaiting_review`, or a newly added value is harmless. The status response should also be checked for contradictions. A job that says `succeeded` but has no required result reference is not ready to consume. A job that says `running` after its own reported expiry time needs escalation, not more faith in polling. Keep the job ID, provider account context, original request fingerprint, and the expected terminal states together. If the agent loses that association, it may poll the wrong job after a restart or accidentally treat a job from an earlier request as the current one. ## Fixed intervals create synchronized pressure A fixed interval looks tidy in code and behaves badly in a fleet. If fifty agents submit work near the same minute and each polls every ten seconds, they tend to hit the provider in clumps. The clumps remain after a brief outage because every agent retries on the same clock. Use an increasing delay with jitter. Start with a short delay only when the provider normally finishes quickly or exposes fresh status immediately. Increase the wait after each pending response, cap it, and vary the actual delay per run. A practical schedule can use this rule: ```text base_delay = 5 seconds max_delay = 120 seconds attempt = number of completed polls raw_delay = min(max_delay, base_delay * 2^attempt) actual_delay = random value between 50% and 100% of raw_delay ``` The random range matters. A deterministic sequence of 5, 10, 20, 40, and 80 seconds merely moves synchronization to wider waves. Full jitter, where the random value can fall anywhere from zero to the cap, also works in some systems. I prefer a lower bound for job polling because a run that repeatedly chooses near-zero delays starts to resemble a retry storm. Do not apply a generic backoff schedule without considering job duration. A document conversion that usually finishes within a minute benefits from early observations. A batch export that reports an estimated completion time should not receive a status call every few seconds just because the agent feels idle. If the API provides a `next_check_at`, `poll_after_seconds`, or similar field, treat it as provider guidance. Validate it before accepting it. Reject negative values, absurdly long waits that exceed your operation deadline, and timestamps that cannot be parsed. The agent can wait until the indicated time or its own deadline, whichever comes first. A delay is not a promise that the agent will make a request at that exact instant. A local process may sleep, restart, lose connectivity, or resume much later. On wake, first check whether the deadline has passed. Do not compensate for missed intervals by sending several status calls in a burst. ## A deadline and a request budget catch different failures Every polling run needs both a wall-clock deadline and a maximum number of status requests. Add a separate cap for transport failures if the provider is remote or unreliable. A deadline controls how long the workflow may remain unresolved. It prevents an agent from keeping a stale task alive through a weekend because the API still says `queued`. Choose it from the business effect of delay, the provider's documented retention period, and the time after which a human should make the decision. Do not derive it only from average runtime. Averages hide the jobs that get stuck. A request budget controls the pressure the agent places on the provider and the credential it uses. It catches a scheduling bug even if time passes slowly, and it limits cost when an API meters calls. The budget should include status calls made after network ambiguity. If you do not charge those calls, an agent can burn through the provider quota while convincing itself it made only a few attempts. A failure budget has a narrower job. Count connection refusals, DNS errors, TLS failures, and 5xx responses that prevent the agent from learning job state. A single timeout does not prove the job failed. Repeating the same failed request for an hour does not prove patience. Use an operation record similar to this one, stored durably before the first poll: ```json { "operation_id": "report-export-2025-04-18-01", "provider_job_id": "exp_71c", "started_at": "2025-04-18T10:20:00Z", "deadline_at": "2025-04-18T11:00:00Z", "max_status_requests": 12, "max_transport_failures": 3, "status_requests_used": 0, "transport_failures_used": 0, "last_known_state": "queued" } ``` The values are examples, not defaults for every provider. Twelve checks in forty minutes may suit a slow export. It would be silly for an operation that normally completes in three seconds and dangerous for one that the provider asks clients to check once every fifteen minutes. Check the deadline before a request, not only after one. Otherwise a process that wakes late can make an unauthorized extra call. Check the request budget immediately before scheduling, then increment it immediately before transmission. That ordering matters when a process crashes between scheduling and sending. You want an occasional unused reservation, not an invisible extra request. ## Stop conditions must be executable, not aspirational "Stop if it takes too long" is a note for a human, not a condition an agent can enforce. Define stop conditions as predicates over the current record and response. Stop successfully only when the provider reports an accepted terminal success state and every field needed by the next action passes validation. If the next action downloads an artifact, validate the artifact reference before declaring success. If the next action affects another system, record the terminal response before performing it. Stop with failure when the provider reports a terminal failure, when the response cannot be parsed, or when a state falls outside the integration's allowlist. Unknown state handling deserves the same seriousness as an authorization denial. A provider can add a `needs_payment`, `manual_review`, or `blocked` state without warning. Guessing that it means "wait" turns a software change on their side into an endless loop on yours. Stop with timeout when the deadline arrives, even if the status has just changed. The agent should report the final observed state, but it should not grant itself another full delay because it saw what looks like progress. If a longer wait is acceptable, make that a new authorization decision with a new deadline. Stop with budget exhaustion when the next status request would exceed the allowed count. Do not reset the count merely because the agent restarted, changed sessions, or received a new prompt. The provider sees one caller and one job, not your internal process boundaries. Stop with ambiguous delivery after a status request times out if the failure budget is exhausted. The agent cannot know whether the provider received that request, but status requests should be safe to repeat. If the endpoint changes state, charges money, or refreshes an artifact on every GET, it is not a status endpoint in the operational sense. Treat it as an action and require stronger controls. This is where teams make an expensive mistake: they assume GET means harmless. HTTP defines GET as a safe method in the semantic sense, meaning the client did not request state change. That is a contract for the server to honor, not a magical property of a URL. Verify provider behavior with a test account and their documentation before classifying a call as observational. ## Respect the protocol signals that providers already send HTTP has a few signals that should change an agent's polling behavior immediately. Ignoring them because the loop has its own timer is rude to the provider and usually makes recovery slower. RFC 9110 defines `Retry-After` as an indication of the minimum time a client should wait before making a follow-up request. The field can contain a delay in seconds or an HTTP date. Parse both forms. For a status request that gets a 503 with `Retry-After: 120`, do not use your normal thirty-second backoff and try again early. Wait at least two minutes, provided the operation deadline permits it. RFC 6585 defines HTTP 429, Too Many Requests, and says a response may include `Retry-After`. That wording leaves providers room to omit the header, which is why the agent still needs its own backoff. A 429 without guidance should increase delay sharply and consume the failure or rate-limit budget you defined. A run that keeps receiving 429 should escalate. It should not keep stretching its timer indefinitely. For 202 Accepted, inspect the response body and headers for a location, job ID, and stated status resource. Do not construct a guessed URL from a submission endpoint. Some providers use a result location, some use a status location, and some return the final representation later. Follow the documented contract. For 404, do not always assume the job never existed. A freshly submitted job can be eventually consistent, and a completed job can disappear after retention expires. The original provider contract decides which explanation is plausible. If the documented behavior does not explain it, classify it as an integration failure and escalate with the job ID and timestamps. For 401 or 403, stop polling. Retrying authorization failures with the same credential adds noise and can trigger provider defenses. The human action is to inspect access, credential rotation, account scope, or the gateway configuration. The polling action is over. For 5xx responses and network failures, use the transport failure budget. Preserve the response code, request timestamp, and any provider request ID. Those details matter when support or an operator needs to distinguish a lost response from a job failure. ## Escalation should ask for a decision, not dump a log An agent needs a defined point where it stops waiting and hands the situation to a person or another explicitly authorized workflow. Escalation is not a decorative notification after the agent has already retried every possibility. Make the escalation reason machine-readable. Use categories such as `deadline_exceeded`, `request_budget_exhausted`, `rate_limited`, `unknown_state`, `authorization_denied`, or `provider_failure`. Each category should determine the permitted next action. A human who sees `unknown_state` may approve a temporary pause while someone checks provider changes. A human who sees `authorization_denied` should not approve another identical request. The escalation message should contain enough context to decide without exposing secrets: ```text External job needs a decision Operation: report-export-2025-04-18-01 Provider job: exp_71c Last state: running Elapsed time: 40 minutes Status requests: 12 of 12 Last HTTP result: 200 at 10:58 UTC Stopped because: request_budget_exhausted Safe options: extend waiting once, cancel at provider, inspect provider console ``` Do not phrase the choice as "continue?" That invites an operator to approve an indefinite loop. Offer bounded options. "Extend by fifteen minutes with four additional checks" states the cost and creates a new limit. "Cancel at provider" should only appear if the integration has a documented cancellation action and the operator understands its effects. Escalate earlier when the eventual result has a short usefulness window. A deployment validation that arrives after the deployment window closes may not deserve more polling. A tax document export may be worth waiting through a provider incident. The technical state alone cannot decide that priority, so encode the workflow's business deadline separately from the provider job deadline when they differ. Avoid automatic resubmission after a timeout unless the provider offers idempotency and you persist the idempotency token. A status polling failure does not prove the original job failed. Resubmitting can create duplicate invoices, duplicate emails, duplicate deployments, or competing exports. People call this "self-healing" right up to the point they have to clean it up. ## A polling controller needs durable state and one owner A reliable polling controller persists its record and ensures that only one worker owns a job at a time. Without those two properties, restart recovery and parallel agents will generate duplicate checks or contradictory follow-up actions. The owner can be a process lease, a database lock, or another durable mechanism appropriate to your environment. The mechanism must expire if the worker dies, and a new owner must reload the complete record before it sends anything. A plain in-memory boolean is not ownership once a second agent process exists. Persist after every meaningful event: job submission, scheduled next check, sent status request, received response, state transition, and escalation. You do not need a separate journal for every debug detail, but you must recover the facts that affect authority and budgets. This pseudocode shows the ordering that prevents most accidental loops: ```text load operation if operation is terminal or escalated: exit if current_time >= operation.deadline_at: record timeout and escalate exit if operation.status_requests_used >= operation.max_status_requests: record budget exhaustion and escalate exit if current_time < operation.next_poll_at: schedule wakeup and exit acquire ownership lease reload operation increment status_requests_used and persist send one status request persist response metadata if response has terminal success and required result fields are valid: record success else if response has terminal failure or unknown state: record stop reason and escalate else if response requires waiting: calculate next_poll_at with provider guidance, backoff, and jitter persist next_poll_at else: record integration failure and escalate ``` The second load after acquiring ownership is deliberate. Another worker may have finished the job while this worker was waiting for the lease. Skip it and you will eventually see two agents retrieve or publish the same result. Do not hold a lease while sleeping. Hold it only while you update the record and send the one request, if your ownership design allows that safely. A lease held for the full duration of a slow external job becomes an orphan problem when a laptop sleeps or a process dies. A status endpoint should remain read-only from the agent's perspective. Keep output retrieval, cancellation, and downstream publication as separate actions with their own records. Combining all of them inside one polling function is how a harmless timer turns into a hidden workflow engine. ## Human approval belongs at the escalation boundary Most routine polls do not need a human click if they stay within the original approved operation, credential scope, deadline, and request budget. Requiring approval for every read teaches people to approve without reading and delays work for no security gain. Approval does belong when the agent requests a new authority boundary: more time than the original deadline, a higher request budget, a cancellation call, a resubmission, a different credential, or use of an output that failed validation. Those are changes to the operation, not ordinary observations. Sallyport can keep the API credential out of an MCP-capable agent while the app performs the status call and returns the result. Its per-call key setting fits operations where every use of a particular credential, including a status request, needs an explicit approval, but that setting cannot repair an unbounded polling design. Choose the approval model based on consequence. A low-risk provider status call may remain inside a per-session authorization. A privileged status API that reveals sensitive job metadata may need per-call approval or a narrower credential. In either case, define the controller's limits before you decide how to authorize a request. Do not let a prompt decide whether a deadline extension is harmless. The request should name the current state, elapsed time, attempted calls, proposed extra budget, and expected effect of continuing. A reviewer can then reject an extension that would miss a release window or produce stale data. ## Audit the decision to wait, not only the request A request log tells you that an agent called `/jobs/exp_71c`. It does not tell you whether the call was allowed by the polling plan, whether the agent ignored `Retry-After`, or whether the job had already exceeded its deadline. Record the controller decision with every call: current state, next scheduled time, delay source, request count, deadline, response code, parsed job state, and resulting decision. The delay source can be `provider_retry_after`, `provider_poll_hint`, `local_backoff`, or `manual_extension`. That small field saves a great deal of argument after an incident. A useful audit sequence reads like a story: ```text 10:20:00 submitted job exp_71c, deadline 11:00:00, budget 12 10:20:05 polled, state queued, next poll 10:20:14 from local backoff 10:20:14 polled, state running, next poll 10:20:31 from local backoff 10:20:31 received 503, Retry-After 120, next poll 10:22:31 from provider guidance 10:22:31 polled, state running, next poll 10:23:48 from local backoff 10:58:00 polled, state running, request budget exhausted, escalated ``` That record makes a bad controller obvious. If timestamps show one request every second despite a provider delay, you have proof. If an agent claims a job timed out but its final state was `succeeded`, you have something concrete to fix. Sallyport projects agent sessions and individual calls from an encrypted, hash-chained audit log, and `sp audit verify` can verify that chain offline over ciphertext. That kind of evidence is most useful when your own operation record explains why each observed call happened. Do not measure polling quality by whether jobs eventually finish. Measure whether every job reaches a terminal result or a bounded escalation, whether agents honor provider waiting instructions, and whether an operator can reconstruct a disputed decision. The first controller change I would make in an uncontrolled loop is simple: persist a deadline and request budget before the agent makes its first status request.