Prevent duplicate API writes when AI agents retry requests

AI agents make retry mistakes faster than people do. A person who sees a timeout may pause, inspect the ticket queue, and decide what happened. An agent often sees an exception, follows an instruction to retry, and sends a second write before the first request has finished somewhere beyond the network boundary.

That behavior creates duplicate tickets, repeated payment attempts, duplicate user invitations, and two deployments of the same change. The fix is not telling the agent to be careful. You need an API contract that preserves the identity of one intended action across retries, detects changes hidden behind a reused identifier, and asks for fresh human confirmation when the consequence deserves it.

Retries are normal, duplicate effects are optional

A timeout does not mean the server did nothing. The server may have created the ticket and lost its response on the way back. It may still be processing the request. A load balancer may have accepted the connection while the upstream service never saw the bytes. The client cannot infer the outcome from a socket error.

This is why the usual agent instruction, "retry on network errors," is incomplete. It treats every uncertain result as a failed action. For write endpoints, an uncertain result has three possible states:

• the server did not receive the request
• the server accepted the request and completed the work
• the server accepted the request and has not completed it yet

The same retry should be safe in all three states. If it creates a second effect in the completed state, the endpoint has an unsafe retry boundary.

HTTP does not solve this for you. RFC 9110 defines idempotent methods as methods whose intended effect stays the same after one or more identical requests. It names PUT, DELETE, and the safe methods. The RFC also says a client may retry an idempotent request after a communication failure. That is useful, but it does not make every endpoint that uses a PUT route safe. A server can attach email delivery, credit issuance, or a deployment trigger to a PUT handler and repeat that side effect unless it designs for it.

POST needs an explicit agreement. Many APIs use POST for actions because the server assigns resource identifiers or because the request means "perform this business operation." An agent can retry such a request only when the API says how it identifies one operation across attempts.

Separate transport retries from business retries. A transport retry resends the same operation because the outcome remains unknown. A business retry starts another operation because the first one reached a known terminal failure. Conflating them produces the classic incident report: the agent retried successfully, twice.

An idempotency token identifies one intended action

An idempotency token is a client-generated opaque identifier that means "all requests carrying this value are attempts to perform this one action." The agent creates it before the first request, persists it with its task state, and sends the identical value on every retry.

The token must belong to the logical action, not to an HTTP attempt. If an agent creates a support ticket, loses the response, and makes another request with a new token, the API has no basis to recognize a retry. It should create a second ticket because the caller told it this was a second operation.

Use a random, high-entropy value. A UUID is common, but any format works if callers do not guess values and the server treats the token as opaque. Put it in an Idempotency-Key header or a documented request field. A header keeps the operation identity separate from the business payload and makes it easier for middleware to carry it through logs and tracing.

A practical request looks like this:

curl -X POST https://api.example.test/v1/tickets \
  -H 'Authorization: Bearer $TOKEN' \
  -H 'Content-Type: application/json' \
  -H 'Idempotency-Key: 81b59b1a-9e75-4de7-a53b-1bb50969c83c' \
  -d '{"project":"ops","title":"Rotate staging certificate","priority":"high"}'

On the first accepted call, the server records the token, a canonical fingerprint of the request, the operation state, and eventually the response it will replay. If a later request carries the same token and the same fingerprint, the server returns the prior result instead of creating another ticket.

The client needs a durable place to keep the token. An agent that stores it only in its current prompt or process memory loses the operation identity after a restart. Store it beside the task record, job record, or workflow checkpoint. If a human asks the agent to make a second, deliberately separate ticket with the same text, the agent must mint a new token because the human expressed a new intent.

Do not make the token equal to a mutable task name, a timestamp, or a natural-language request. Those values collide, change between retries, or expose information in logs. Opaque identifiers are boring. That is exactly why they work.

A fingerprint catches changed retries

A token answers whether the caller claims two requests are one operation. A request fingerprint answers whether those requests actually mean the same thing. You need both.

Suppose an agent first asks a deployment API to ship commit a1b2c3 to staging. It times out, reads a newer task note, and retries with the same token but commit d4e5f6. If the server blindly replays the first response, it hides an agent error. If it executes the second body, it lets one operation identifier authorize two different deployments.

Canonicalize the meaningful parts of the request and hash the result. Most APIs include the HTTP method, a normalized route, the authenticated account or tenant, and the canonical JSON body. Some include selected headers when those headers change the business effect. Exclude volatile tracing headers, connection metadata, and the idempotency header itself.

JSON demands care. Raw byte hashes fail when equivalent JSON uses a different property order or whitespace. A canonical representation sorts object properties, preserves array order, uses a defined number format, and omits fields the server assigns. Better still, fingerprint the validated command object after the API parses defaults and rejects unknown fields. That matches the operation the server will execute, not an arbitrary input encoding.

For example, this pseudocode records a digest after validation:

command = validate_create_ticket(request.body)
canonical = canonical_json({
  "method": "POST",
  "route": "/v1/tickets",
  "account_id": authenticated_account.id,
  "command": command
})
fingerprint = sha256(canonical)

When a token already exists, compare fingerprints before you return or wait for any prior result. If they differ, reject the request with a conflict response. Include the stored operation identifier and state, but do not echo protected request details to an unauthorized caller.

A fingerprint is not a duplicate detector by itself. Two users can legitimately file two identical tickets. A payroll service can legitimately issue equal payments to two employees. Hashing a payload and deduplicating every match silently drops valid work. Scope deduplication to the idempotency token, then use business-specific uniqueness rules where the domain actually requires them.

Cryptographic hashes make accidental collisions impractical when you use a modern function such as SHA-256. They do not prove caller intent. The token carries intent; the fingerprint enforces consistency. Teams that treat these as interchangeable usually end up with a dedupe rule they cannot explain when it rejects a legitimate request.

The server must claim the token before it acts

An idempotency table that records results only after the side effect completes still has a race. Two concurrent retries can both check the table, see nothing, create two tickets, then race to store a result. I have seen this disguised as a flaky agent problem when the actual defect was a missing uniqueness constraint.

The server must atomically claim the token before it performs irreversible work. Put a unique constraint on the scope and token, commonly an account identifier plus the idempotency token. In one transaction, attempt to insert a row with the fingerprint and an in_progress state. The request that wins owns execution. Every other request reads the existing row.

A simplified table might contain these fields:

create table idempotency_operations (
  account_id text not null,
  token text not null,
  fingerprint text not null,
  state text not null,
  response_status integer,
  response_body jsonb,
  created_at timestamptz not null,
  primary key (account_id, token)
);

The primary key is doing real work here. Application code that checks first and inserts later leaves a gap large enough for concurrent workers, queue redelivery, and impatient retries to pass through.

After the claim, the handler performs the business action and writes the final response to the operation row. Later matching requests receive that saved status and body. This gives callers a stable answer, even when the original handler already succeeded but the connection died before it could reply.

The awkward case is a request that owns a row but dies midway through its work. Do not delete the row just because a worker timed out. Another worker may still be finishing, or the external provider may already have accepted the operation. Mark the operation as pending or unknown, record enough data to investigate, and let callers ask for its status. A repair job can resolve stale records only when it understands the downstream system's state.

For work that crosses a database and an external API, use an outbox pattern or a provider-side idempotency token. A database transaction cannot roll back an email, a payment, or a cloud deployment after it leaves your process. Write the intent and an outbox event in one local transaction, then have a worker send the event with a stable downstream operation identifier. That design gives recovery code something concrete to replay without inventing a second action.

Confirmation must bind to the exact operation

Human confirmation prevents a different failure: an agent might have permission to act, but the proposed action could be surprising, too broad, or repeated after its context changes. A generic "allow deployment" button does not address that. It lets an agent substitute one deployment for another under the same approval.

A useful confirmation names the target, the operation, the consequence, and the operation identifier. For a production deployment, show the environment, artifact or commit reference, affected service, and whether the action can roll back. For a payment, show the payee, amount, currency, and invoice reference. For a ticket, show the destination project and title.

The confirmation record should bind to the request fingerprint and expire when the proposal stops being current. If the agent changes the body after a person approves it, the fingerprint changes and the system must ask again. Reusing approval after a changed request is a quiet form of privilege escalation, even when nobody intended it.

Do not force a person to approve every low-risk retry. That turns a correct idempotency design into approval fatigue. The first approval can authorize the one fingerprinted operation, and matching retries can use that approval because they cannot change its meaning. A changed payload needs another decision.

Some teams rely on a chat message such as "Proceed?" and call the reply an approval. That fails under pressure because the record often lacks the exact parameters, and the agent can misread a later reply as consent for an earlier request. Put the operation identifier into the confirmation record and require the executor to verify it before sending the write.

A simple approval payload makes the binding visible:

{
  "operation_id": "op_3f8c",
  "idempotency_token": "81b59b1a-9e75-4de7-a53b-1bb50969c83c",
  "fingerprint": "e5c7...",
  "expires_at": "2025-06-14T15:30:00Z",
  "approved_by": "user_42"
}

Treat confirmation as authorization for a particular command, not as permission to improvise around a category of commands. That distinction keeps a retry safe without giving an agent a blank approval it can reuse later.

Ticket systems need a business-level duplicate check too

Idempotency tokens stop duplicate transport attempts, but ticket systems have another source of duplication: agents may start separate operations that describe the same issue. A monitoring alert arrives twice, two agent runs read the same incident channel, or a scheduler wakes after a crash and replays a task without its original state.

Do not solve that by deduplicating on title text. Ticket titles vary enough to miss duplicates, and identical titles can refer to separate incidents. Instead, decide what identity means in the ticket domain. It may be an alert event identifier, an incident identifier, a repository issue reference, or a compound value such as service plus alert fingerprint plus incident window.

Make that business identifier explicit in the API:

{
  "source_event_id": "alert-7c91",
  "project": "operations",
  "title": "Certificate expiry alert",
  "description": "Alert event alert-7c91 crossed its threshold."
}

The ticket service can enforce uniqueness for source_event_id within the intended scope. A second agent run then gets the existing ticket identifier instead of adding another item to the queue. This is separate from idempotency. The two calls may have different idempotency tokens because they came from two different agent processes, yet they represent the same upstream event.

Agents should search before they create only when the search result has a stable identity they can trust. Search-by-title workflows are tempting because they require no API changes. They break as soon as indexing lags, query ranking changes, or an agent paraphrases the title. Put the uniqueness rule where the write happens, and return a clear response that says whether the API created or reused a ticket.

Be careful with automatic comments and status changes. An operation that finds an existing ticket might still append a duplicate comment or reopen a resolved incident. Give each meaningful sub-action its own identifier, or make the write command express the whole desired state. Vague "update this ticket" endpoints are difficult to retry safely because nobody can tell which part of the update already ran.

Payment writes require an outcome query, not optimism

Payment actions have a stricter standard because a duplicate charge harms a customer even if you refund it later. The application should send one stable idempotency token to its payment provider and retain the provider transaction reference with the local operation record.

When the client times out, it must treat the payment as unknown. It should query by the provider reference, merchant reference, or idempotency token if the provider exposes that lookup. It should not start another payment attempt because the agent received no success response.

There are two operations people often collapse into one: creating a payment intent and capturing funds. They can have different retry behavior. A service may safely create or retrieve one payment object using a token, then require a separate explicit capture action after checks pass. Model the business states openly instead of hiding them behind a single endpoint that attempts everything on every call.

Amounts need canonical handling before fingerprinting. Convert values into the smallest supported currency unit or another exact representation before the request reaches the dedupe layer. Do not hash a floating-point display value and expect equivalent calculations to compare reliably. A payment request should also include an invoice or order reference when the domain has one, because that gives staff a way to identify duplicate intent beyond network retries.

A provider's idempotency feature does not absolve your own API. Your application still needs to stop two agent tasks from initiating two distinct provider requests for the same order. Put a uniqueness constraint on the order's payable state, use a local operation record, and make the agent query that record after uncertainty.

Refunds deserve the same care. "Retry refund" can mean retrying the same refund request, or it can mean initiating another partial refund. Keep a stable identifier for each refund instruction and record the amount already requested. If the agent needs to issue a second refund, make that a new, explicitly authorized instruction with a new identifier.

Deployments need immutable references and a release lock

A deployment retry is safe only if it names the same release. Branch names such as main and mutable tags such as latest do not meet that standard. A retry after a timeout can resolve the same name to different code, then appear to succeed while deploying something the approver never reviewed.

Use an immutable artifact digest, commit identifier, or version that your release system guarantees will not change. Include it in the request fingerprint and confirmation. If an agent submits the same idempotency token with a changed artifact reference, reject it as a conflict rather than treating the second request as an updated retry.

You also need a concurrency rule for the environment. Two separate operations can legitimately carry two different tokens and still conflict because both target production. A release lock, optimistic version check, or deployment queue can serialize those changes. Idempotency does not decide which of two distinct deployments should win. It only stops one deployment from running twice.

Consider this failure sequence. The agent starts deployment dep-118 for commit a1b2c3 and the deployment controller accepts it. The agent loses the response, assumes failure, and starts dep-119 with commit d4e5f6 because a newer commit appeared. Both jobs now modify the same environment. A token would have stopped only a true retry of dep-118; the release lock or expected-environment-version check stops the conflicting second plan.

The deployment API should expose an operation status resource that reports queued, running, succeeded, failed, canceled, or unknown. Agents should poll that status after a timeout. They should not infer completion from a missing response or from a log line that lacks the operation identifier.

Rollback needs its own operation identifier and approval. Treating rollback as a retry of the deployment hides a meaningful change in intent. It can be automatic under a documented safety rule, but it must leave a record distinct from the original release.

Agent tools should preserve operation identity across the boundary

An agent tool interface should make safe behavior easier than unsafe behavior. Give the agent one action that accepts a stable operation identifier, a payload, and a declared retry mode. Return a result that says whether the service created work, replayed a prior result, found an operation in progress, or rejected a changed retry.

Avoid tools that silently generate a new idempotency token on every invocation. They look convenient in a demo and fail during the first real timeout. If the tool owns token generation, it must return the token immediately and persist it where a later invocation can retrieve it. In most systems, the workflow layer should own the token because it understands which calls belong to one user-requested action.

Sallyport can keep API credentials outside the agent while the agent submits the intended HTTP action through its MCP connection. That separation helps with credential exposure, but the downstream API still needs idempotency behavior. A protected credential does not turn an ambiguous POST into a safe retry.

Make the agent's retry rule explicit in its tool contract:

if response is a known success:
    record operation complete
if response is a timeout or connection failure:
    query operation status using the same token
    retry only with the same token if the API permits it
if response says fingerprint conflict:
    stop and request a new operation or human review
if response is a known business failure:
    do not retry until the task changes

Do not let the agent use exponential backoff as a substitute for state. Backoff limits pressure on a service, which matters, but it does not answer whether the last write succeeded. The agent must preserve the operation identifier before it waits.

Logs must prove what happened after a disputed write

When a customer says they were charged twice or an engineer finds two tickets, you need to answer four questions: which agent run issued each request, which token it used, what fingerprint the server calculated, and what outcome the downstream service returned. General request logs often omit at least one of these.

Log an operation record at the boundary where the API accepts the action. Include the authenticated principal, token, fingerprint, request route, operation state transitions, response reference, and the upstream provider reference when one exists. Keep secrets and full sensitive bodies out of routine logs. A fingerprint lets you compare requests without storing every private field in every log system.

An append-only audit record helps when an agent has authority to make external writes. The record should distinguish attempted, approved, sent, accepted, completed, and replayed. These are not interchangeable states. A retry that receives a stored prior response should say replayed, not created, or your operators will count it as a second action.

Sallyport records agent sessions and individual actions in journals derived from an encrypted hash-chained audit log, and sp audit verify can verify the chain offline. That can establish what passed through the action gateway. Pair that evidence with the receiving service's idempotency records, because the receiving service determines whether it executed the business action.

Test the disputed-write path before trusting it. Force the server to complete a request and drop the response. Send concurrent copies of one token. Restart the agent between attempts. Reuse a token with a changed payload. Kill a worker after it claims a token and before it records completion. A design that survives only clean success responses has not solved duplicate writes.

Start with the write endpoint that hurts most when repeated. Add a stable token, atomically claim it before any side effect, bind it to a fingerprint, and give callers a status lookup for unknown outcomes. Then make the agent carry that identifier until it can prove the operation reached a terminal state.