Asynchronous API jobs: traceable agent workflows

An agent that submits a long-running API request needs a workflow, not a loop that keeps calling endpoints until something looks finished. Creation, observation, cancellation, and result retrieval each have different failure modes. If you collapse them into one prompt and a few retries, you will eventually submit work twice, lose a completed result, or claim a cancellation that never took effect.

The hard part is not making an HTTP request. It is preserving intent across an agent restart, a network timeout, a provider outage, and a human who says "stop" after the remote service has already begun work. Build around a durable local job record and make every external call answerable later: what did we ask for, which remote job owns it, what state did we observe, and what did we do next?

A create request must establish ownership

A create endpoint starts an asynchronous operation and returns before the operation reaches a terminal state. Its first response must give the agent enough information to continue without repeating the request. In a well-designed API, that means a job ID, an initial state, and either a status URL or an endpoint convention for retrieving the job.

Do not infer that a request was accepted because a socket connected or because the client timed out after sending bytes. The only useful proof is a server response, or a later lookup that binds the original logical request to a job. Network delivery is uncertain at exactly the moment a client most wants a simple answer.

Give each logical operation a local operation ID before the agent makes the request. This is your identifier, not the provider's. Store it with the request body or a normalized fingerprint of that body, the intended target, the idempotency token, timestamps, and the caller that authorized the work. Write this record durably before sending the request.

A minimal record can look like this:

{
  "operation_id": "op_01J7Q5X4D4PA3D",
  "request_fingerprint": "sha256:4f8b...",
  "idempotency_token": "idem_5b5c76c7",
  "remote_job_id": null,
  "state": "create_pending",
  "created_at": "2025-03-08T14:22:11Z",
  "create_deadline": "2025-03-08T14:24:11Z",
  "result_deadline": "2025-03-08T15:22:11Z"
}

The fingerprint catches a subtle but frequent error: an agent retries a create call after changing a parameter. That is a new operation, even if a human would describe it as "the same task." A token tied to one request shape must not quietly authorize another.

A creation response might be:

HTTP/1.1 202 Accepted
Location: /v1/jobs/job_7ad2
Content-Type: application/json

{
  "job_id": "job_7ad2",
  "state": "queued",
  "status_url": "/v1/jobs/job_7ad2"
}

As soon as the response arrives, update the durable record with remote_job_id, the observed state, and the response metadata. Only then can the agent move into observation. If the API returns a synchronous success instead, record that result under the same operation. The workflow should support both paths without pretending they mean the same thing.

Idempotency is for uncertain delivery, not general retries

An idempotency token tells the server that repeated delivery of one logical create request must not produce repeated work. It does not make every request safe, and it does not repair an API that never implemented idempotency on its create endpoint.

The token belongs in the create request according to the provider's contract. Some APIs accept an Idempotency-Key header. Others require a request field. Use the documented form and generate a token with enough entropy that unrelated operations cannot collide. Preserve the same token until you have resolved the original operation.

POST /v1/reports HTTP/1.1
Content-Type: application/json
Idempotency-Key: idem_5b5c76c7
X-Trace-ID: tr_0830d3

{
  "account": "acct_218",
  "range": {"start": "2025-02-01", "end": "2025-02-28"},
  "format": "csv"
}

The safe retry sequence is narrow:

1. Generate the token and persist the operation record.
2. Send the create request with that token.
3. If the response is lost or the client times out, retry the identical request with the identical token.
4. If the server returns the original job, save its ID and continue.
5. If you need different input, close or cancel the old operation if possible, then create a new record and token.

This distinction matters because agents naturally rewrite requests while reasoning. A changed date range, destination, account, or output format changes the effect. Reusing a token after such a change creates an argument between client and server: a careful server rejects the mismatch, while a less careful one may return an old response that no longer matches the agent's intent.

The HTTP standard makes the relevant distinction clear. RFC 9110 defines idempotent methods as methods whose intended effect of repeated identical requests is the same as one request. POST is not idempotent by default. A provider can add idempotency behavior to a POST endpoint, but the client must treat that as an explicit application contract, not an HTTP law.

A popular bad recommendation says, "Retry POST only once." The number of retries is beside the point. One duplicate submission can send a payment, provision an environment, or launch an expensive batch. Retry a create request whenever the deadline and provider guidance allow, but only with a token that gives the server a way to recognize the original operation.

A timeout leaves the job state unknown

A create timeout does not mean the service rejected the request. It means the agent did not receive a definitive answer before its own deadline. The server may have accepted the request, may still be processing it, or may never have received it.

This is the failure that exposes weak agent workflows. An agent sends a create request, waits thirty seconds, receives no response, and sends a fresh request with a fresh token. Now two reports run. The second one can finish first, which makes the incident hard to spot until someone compares charges, exports, or downstream changes.

Keep an explicit create_pending state. When the call fails ambiguously, record the error class, timestamp, and attempt count, but do not discard the operation. Then use the API's reconciliation path. Providers expose this in different ways:

• A retry using the same idempotency token may return the original acceptance response.
• A list or search endpoint may filter by a client request reference.
• A status lookup may accept a client-supplied operation ID.
• A provider may document a supportable query for recent creations by a request identifier.

If none exists, the API cannot give a client reliable at-most-once creation semantics across a lost response. Say that plainly in your design. You can reduce duplicates with a local outbox and restrained retries, but you cannot prove that a retry did not create more work.

Treat a response that contains an unfamiliar job ID with the same token as an incident-worthy contract violation. Do not overwrite the old ID. Preserve both response records, stop automatic activity for that operation, and require a human decision. Quietly selecting one is how audit trails become fiction.

Use deadlines that distinguish communication from work. The create deadline governs how long the agent will try to establish a remote job ID. The result deadline governs how long the business process will wait for completion. A job can survive a brief create response timeout and still have hours to finish. Combining both into one timer causes agents to abandon recoverable work or retry it at the wrong moment.

Polling needs backoff, ownership, and a stop time

Polling is safe when one durable workflow owns the job and every poll records an observation. It becomes abusive when several agent runs rediscover the same job and all poll it independently.

Put the remote job ID in one record and assign a lease to the process that currently owns observation. The lease can be a database row with an expiration timestamp, a queue message with visibility rules, or another durable concurrency control. If the worker crashes, a later worker can take over after the lease expires. Without ownership, retries and restarts multiply the number of status calls.

Honor Retry-After when the API sends it. If the API gives no guidance, use capped exponential backoff with jitter. The exact ceiling depends on how quickly the business needs an answer and on the provider's rate limits, but the shape should avoid synchronized bursts.

attempt 1: wait a randomized interval near 2 seconds
attempt 2: wait a randomized interval near 4 seconds
attempt 3: wait a randomized interval near 8 seconds
later attempts: keep increasing until the configured cap

Do not calculate the next delay from an agent's prose summary. Store the next poll time in the job record. That lets a restarted worker resume the schedule, and it gives an operator a clear explanation for why the agent is waiting.

A status response should update only observed facts. For example:

{
  "job_id": "job_7ad2",
  "state": "running",
  "updated_at": "2025-03-08T14:26:40Z",
  "progress": {"completed": 146, "total": 500}
}

Record state, the provider timestamp if supplied, the retrieval time, the raw response reference, and the next action. Do not convert a vague progress field into a promise that the job will finish. Providers often report progress late or in batches. Progress helps operators; terminal state controls the workflow.

Set a result deadline and make its expiry a state, not an excuse to forget the job. result_timed_out means the agent stopped automatic polling because its agreement expired. It does not mean the remote job stopped. If the action has real cost or side effects, keep enough information to reconcile later and decide whether cancellation is appropriate.

Webhooks can improve latency, but they do not remove the status loop. Providers can retry callbacks, deliver them out of order, or fail to deliver them. Verify the callback according to the provider's documentation, deduplicate it with an event ID when available, update the same job record, and perform a final status read before declaring success.

State transitions must reject wishful thinking

A job state machine protects the workflow from an agent that interprets words too generously. Define your local states and permitted transitions before connecting tools to the API. Remote services use different names, but your record should make uncertainty visible.

A practical local model is:

create_pending -> accepted -> observing -> result_collecting -> succeeded
create_pending -> create_unknown -> reconciliation
accepted or observing -> cancel_requested -> cancelling -> cancelled
accepted or observing -> failed
observing -> result_timed_out

The arrows are rules, not a diagram for documentation. A worker must reject a transition that lacks evidence. It cannot mark succeeded because it saw a progress value of 100. It cannot mark cancelled because it sent DELETE /jobs/job_7ad2. It cannot move from failed back to observing unless the remote API explicitly supports a retry or resume operation and the new action is separately recorded.

Keep remote state and local state separate. cancel_requested describes a local fact: the agent sent a cancellation request and awaits confirmation. cancelled describes a remote fact: the service reported a terminal cancelled state. This small distinction saves a great deal of confusion during an incident.

Use an append-only history for transitions. Each entry needs the operation ID, actor, time, previous local state, next local state, triggering request or response, and reason. A compact history is enough:

{
  "at": "2025-03-08T14:29:02Z",
  "actor": "worker-3",
  "from": "observing",
  "to": "cancel_requested",
  "cause": "human_request:req_91af",
  "remote_job_id": "job_7ad2"
}

Avoid a single mutable status field as your only record. It tells you what the workflow believes now, but not why it believed that five minutes ago. When a remote API later returns a surprising state, history tells you whether the provider changed, the agent repeated a call, or an operator intervened.

Cancellation needs confirmation and a damage boundary

Cancellation is a request to stop future work. It cannot undo work that the provider already committed, and some providers allow a race where a job finishes at the same time the cancel request arrives. Build the workflow around that reality.

When a human or policy decides to stop a job, record the cancellation intent first. Include who made the request, why, and the expected effect. Then call the documented cancel endpoint using the stored remote job ID. Persist the response even if it says only that the server accepted the request.

Keep polling after cancellation. The acceptable terminal outcomes usually include cancelled, succeeded, and failed. A completed result after a cancel request is not automatically an error. It may be the honest outcome of a job that crossed its commit point seconds earlier. The workflow must report that sequence accurately rather than rewrite history to match the desired outcome.

Some operations need a damage boundary separate from cancellation. If an export job writes a file, cancelling the job may leave a partial file. If a provisioning job creates resources, cancellation may leave some resources created. The API contract should state whether it offers cleanup, rollback, or partial-result details. If it does not, treat cancellation as an operational control, not a transaction.

Do not issue repeated cancel calls from every polling worker. Store cancel_requested, make the cancel operation idempotent if the provider permits it, and let the lease owner own follow-up. Repeating a harmless request wastes capacity; repeating a cancellation with side effects can muddy the remote audit log.

A cancellation deadline also helps. After a reasonable, documented wait, move to cancellation_unconfirmed instead of claiming success. Escalate with the remote job ID, trace ID, request history, and any provider request IDs. That package lets a human or provider support team see the actual sequence without reconstructing it from chat messages.

Result collection is a separate action

A terminal success state means the remote work completed. It does not guarantee that the result has been fetched, validated, stored, or delivered to the next system. Treat collection as its own recorded action.

First fetch the result using the job ID or result reference supplied by the API. Validate the expected content type, schema, checksum, size, or record count where the provider gives one. Store the result reference and validation outcome in the job record. If the result is large, store a durable location and integrity data rather than copying opaque content into an event log.

Then decide whether collection itself needs idempotency. Many result endpoints are safe reads. Others generate a temporary download, consume a one-time artifact, or mark a job as delivered. Read the contract. An agent that treats every GET as harmless can still trigger a provider-specific state change.

Do not use a successful HTTP status as the only validation. A report endpoint can return a valid file containing an error row. An image batch can return a manifest with failed items. A data export can complete while omitting records the API says the caller cannot access. Validate against the business expectation that caused the job to exist.

For batch work, record item-level outcomes when the provider supports them. A terminal job can contain 498 successes and two failures. Calling that simply "succeeded" forces the next agent to rediscover the partial failure from the result payload. Your local final state can remain successful while the result summary carries counts and a list of failed item references.

Close the operation only after collection meets its contract. succeeded should mean the intended result is available and verified according to your rules. If the provider completed but collection failed, use a distinct local state such as result_unavailable or result_validation_failed. The remote job may be done, while your workflow is not.

Trace IDs connect actions, while audit records establish facts

Use a trace ID for every operation and send it on creation, status reads, cancellation, and collection when the API accepts custom headers. Pair it with the remote job ID as soon as you know it. The trace ID connects events inside your systems; the job ID lets the provider find its own work item.

Do not overload either identifier. A trace ID should not become an idempotency token because one operation may involve several requests with different retry rules. A job ID should not become your authorization record because the provider generated it after your local decision to act.

Your audit record should answer questions that logs often cannot: which agent process initiated the operation, which human approval covered it, which credentialed action occurred, and whether anyone edited the history after the fact. Write a concise intent record before the create call and append observations afterward. Preserve request IDs and sanitized response metadata. Never place bearer tokens, passwords, private keys, or full sensitive payloads into a general log.

Sallyport can execute HTTP API calls without exposing stored credentials to the agent, and its sessions and individual calls provide a tamper-evident trail that can be verified with sp audit verify. That protects credential custody and action evidence; your workflow still needs its own operation record because only it knows whether job_7ad2 belongs to the requested business task.

A trace becomes useful during a bad day only if every component records it consistently. Put it in the local job record, the agent's execution context, request headers where permitted, audit annotations where permitted, and operator tickets. Do not fabricate a new trace for each poll. Those are child events of the same operation.

A reference loop handles the ordinary failures

The workflow below keeps the hard decisions explicit. It assumes a provider with an idempotent create contract, a status endpoint, and a cancellation endpoint. Adapt the endpoint names, but do not remove the persistent state transitions.

load operation by local operation ID

if no operation exists:
    create and persist record with fingerprint and idempotency token

if remote job ID is absent:
    send create with the stored token
    if response confirms job ID:
        persist job ID and move to observing
    if response is ambiguous:
        move to create_unknown and reconcile using the stored token
    if response rejects request definitively:
        move to failed

while local state requires observation and result deadline has not passed:
    acquire lease for the operation
    read remote status
    append the observation
    if cancellation was requested and remote state is nonterminal:
        send cancellation once and record the attempt
    if remote state is terminal:
        collect and validate result if appropriate
        persist final local state
    otherwise:
        persist next poll time and release lease

if the deadline expires before a terminal observation:
    move to result_timed_out and preserve the reconciliation record

This loop has no magical retry count because retry limits depend on the provider, the cost of the operation, and the caller's deadline. It does have a rule that matters more: every retry refers to one stored operation, and every externally visible action changes that operation's history.

Test it with failure injection before you hand it to autonomous agents. Drop the create response after the server accepts it. Kill the worker after saving the job ID but before scheduling the first poll. Return a terminal result during a cancellation race. Deliver the same webhook twice. Restart with a stale lease. If the workflow cannot explain and recover from each case, it is not ready to initiate expensive or consequential jobs.

The first implementation task is unglamorous: create the durable operation record and refuse to issue a create request without it. That one constraint forces the agent to retain intent, makes duplicate prevention possible, and gives everyone a factual record when the remote system behaves imperfectly.