How concurrent AI agents collide on production accounts

Two autonomous runs on one production account do not become safe because each run has a different task description. They share a mutable system, and every plan they make can become stale before the next API call. If both can change the same resource, you need an ownership boundary that the service enforces.

The usual failure is quieter than a spectacular outage. One agent adds a member to a group while another replaces the group's full membership from an older read. Both requests return success. The later request removes the member that the first one added. Each run followed its instructions. Your API accepted an invalid sequence.

Treat an agent run as an untrusted concurrent client with real credentials and imperfect timing. Give it a narrow area to own, make write requests conditional on the version it read, and record enough context to explain a rejected or accepted change later. Human review remains useful, but it cannot substitute for a receiver that detects stale state.

Separate a task boundary from a write boundary

A task boundary says what an agent was asked to accomplish. A write boundary says which mutable objects it may change. Those are different things, and teams get hurt when they treat them as interchangeable.

"Update the staging deployment" sounds narrow. It may still touch a shared image tag, a release pointer, a traffic rule, a DNS record, a database migration ledger, and a notification channel. An agent can obey the task wording while colliding with a release run that owns one of those objects.

Define ownership in terms the receiving service can check. Good boundaries name durable resource identifiers, not a loose category of work:

• one environment and one deployment record
• one tenant or customer account
• one pull request and its branch
• one incident ticket and the resources named in its change set
• one maintenance window with an explicit list of targets

Avoid boundaries such as "backend work" or "production cleanup." They are labels for people. They do not tell an API which write should fail.

A useful ownership record contains the run ID, resource ID, allowed operation, and expiry. Keep it close to the service that owns the resource. If a deployment controller owns the release pointer, that controller should validate who may advance it. A spreadsheet, a chat message, or a prompt instruction cannot block a request arriving after the person who wrote it has gone home.

Build a small conflict map before granting write access

For each automated job, list the resources it reads, writes, deletes, and uses as a shared default. Then mark each pair of jobs that can write the same identifier or one job's input can be changed by the other. This is not bureaucracy. It exposes collisions that role-based permissions conceal.

For example, an agent that rotates a service token and an agent that updates integration configuration may never call the same endpoint. The configuration writer may read the current token reference, then publish its full configuration document after the rotation has changed that reference. The conflict sits in the document's version, not in an identical command.

If you cannot describe a run's write set, do not give it broad production write permission. Make it prepare a proposal, or constrain it to a resource namespace until you can describe the boundary.

A successful response can still erase a correct change

Last-write-wins behavior is a data-loss policy when clients send complete representations. It looks harmless in demos because each client reads and writes immediately. Agents often spend minutes inspecting logs, generating a plan, asking for approval, and retrying a call after a timeout.

Consider a service with a notification-policy resource. It returns this representation to Agent A:

{
  "id": "prod-alerts",
  "version": 41,
  "destinations": ["[email protected]"],
  "severity": "high"
}

Agent A plans to add a backup destination. During review, Agent B changes severity from high to critical and successfully writes version 42. Agent A then sends a full replacement based on version 41:

{
  "destinations": ["[email protected]", "[email protected]"],
  "severity": "high"
}

If the endpoint accepts this request, it silently undoes B's change. Neither agent needs a bug. The API allowed an old observation to overwrite a newer fact.

Partial updates reduce the surface area but do not abolish the problem. A patch that adds a destination can still violate a new quota, an updated routing policy, or a deletion that happened after the read. The service must decide whether the patch remains valid against current state.

This is why "we only let agents use PATCH" is not a concurrency design. It is a smaller write shape. You still need a condition that connects the write to the state the agent observed.

Make every state-changing request conditional

Optimistic concurrency control is usually the right first defense for agent writes. The client reads a version, ETag, generation number, or revision token. The client sends it back with the intended update. The service accepts the write only if the current value still matches.

RFC 9110 defines If-Match for exactly this class of request. A server evaluates the condition before applying the method. If the entity tag no longer matches, the server rejects the method with 412 Precondition Failed. That is not an API inconvenience. It is the server refusing to pretend that an outdated plan remains correct.

A conditional HTTP update can look like this:

GET /v1/notification-policies/prod-alerts

HTTP/1.1 200 OK
ETag: "41"
Content-Type: application/json

{"destinations":["[email protected]"],"severity":"high"}

The agent carries that ETag into its write:

PATCH /v1/notification-policies/prod-alerts
If-Match: "41"
Content-Type: application/json
Idempotency-Key: run-7f3c-add-backup

{"destinations":["[email protected]","[email protected]"]}

If another writer has produced ETag "42", return a clear rejection:

HTTP/1.1 412 Precondition Failed
Content-Type: application/json

{
  "error": "stale_version",
  "resource": "notification-policy/prod-alerts",
  "expected_etag": "41",
  "current_etag": "42",
  "retryable": false
}

Do not label this error retryable if the agent can blindly resend the same body. A retry must begin with a fresh read and a new decision. It may discover that the desired result already exists, that the latest policy makes the change invalid, or that a person needs to choose between two competing outcomes.

For databases, use the equivalent predicate in the mutation itself. A typical update checks the version in the WHERE clause and treats zero affected rows as a conflict:

UPDATE notification_policy
SET destinations = :destinations,
    version = version + 1
WHERE id = :id
  AND version = :observed_version;

Never read a version, then issue an unconditional update in a second operation. The check and state change must occur together at the authority that stores the state.

Idempotency stops duplicates, not disagreement

Teams often put an idempotency key on an endpoint and declare concurrent writes handled. An idempotency key prevents the same logical request from producing its effect twice. It does not tell the service whether two different requests are compatible.

A network timeout makes this distinction concrete. An agent sends a request to create a deployment, but loses the response. Retrying with the same idempotency key should return the original result rather than create a second deployment. That is duplicate suppression.

Now take two agents that each select a different release candidate for the same production environment. They send different bodies and different idempotency keys. Both requests can be perfectly idempotent while one should still lose because the release pointer has changed.

Use both controls on important write endpoints:

• An idempotency key binds retries and duplicate deliveries to one completed operation.
• A version precondition rejects a write whose decision rests on stale resource state.
• A server-side invariant checks rules that must hold even for a current write, such as a maximum number of active credentials.

Store the idempotency key with a request fingerprint and the completed response. If the caller reuses the key with a different body, reject it. Returning the first response for a different operation creates a debugging mess and can conceal a client bug.

Be strict about expiry. A service should retain a key long enough to cover its actual retry behavior, but an idempotency store is not a permanent command history. The audit log is where you keep the history.

Use leases only for work that cannot overlap

Some actions take long enough that optimistic checks alone make the user experience poor. A database migration, a destructive reconciliation job, or a cutover may involve many dependent writes. In those cases, give one run a short-lived lease on the resource.

A lease must have an owner, an expiry, and a fencing value. The fencing value matters because an expired worker may wake up and continue after another worker acquired a new lease. Every protected write must carry the lease's monotonically increasing token, and the service must reject a token older than the last one it accepted.

Without fencing, a lock service can tell Agent A that its lease expired, but it cannot stop a delayed request from A reaching the database. The destination service has to reject it. This is the part teams skip when they say they have a distributed lock.

Keep leases narrow and short. Do not lock "production" for an entire autonomous investigation. Lock migration/customer-1842 or release/prod-eu, and make the run renew its lease only while it is still making progress. A lease should expire safely if the agent process, its laptop, or its network disappears.

Do not use a lease to cover ordinary configuration edits that support version checks. Long locks turn routine changes into queues, then people learn to bypass the lock. A 412 response followed by a fresh plan is cheaper than an outage caused by a stale lock holder.

Agent identity must survive the gateway

A shared production token gives every run the same name at the service. After a collision, you can see that the token acted, but you cannot tell which process planned the change, which approval covered it, or which run to stop. That makes cleanup slow and revocation broad.

Give each agent process a distinct session identity, then pass a stable correlation identifier to every destination request. The destination service should log the identity, run ID, request ID, target resource, observed version, outcome, and its own resulting version. Do not bury this information in prose inside a commit message.

Sallyport keeps API and SSH credentials out of the agent process while it executes the action, which helps preserve a boundary between an agent's planning context and the secret itself. Its per-session authorization can identify a newly started agent process before that process begins a run. That authorization is a control over who may act, not a substitute for destination-side preconditions.

Do not let the agent choose its own effective identity in an arbitrary header. Have the gateway or destination service bind the identity from an authenticated session. Otherwise, a run can claim to be the deployment coordinator after the fact, and your logs become theater.

For SSH, the same principle applies even though the wire protocol differs. Use separate principals or restricted accounts for distinct classes of work. Make remote command logs include a run identifier, and avoid a single shared shell account that can edit every application directory.

Approval timing is not transaction timing

A person can approve an agent's request and still approve a write that becomes wrong ten seconds later. This is normal in a concurrent system. The approval speaks to authority and intent at the time of review. It does not freeze the resource.

The dangerous design asks a person to approve a broad sentence such as "update production configuration," then lets the agent perform a sequence of reads and writes whenever it reaches them. A safer design presents the target and intended effect, then has the service enforce the version or lease when the write arrives.

When a precondition fails after approval, do not automatically reuse that approval for a changed plan. The agent should report the conflict in concrete terms: which resource changed, which version it observed, what field changed if the service can determine it, and whether its proposed result is still needed. A person can then approve a new action, or the agent can make a safe no-op after rereading.

Per-call approval is appropriate for operations where each use is materially risky, such as deleting a production credential or changing an externally visible routing rule. For normal batches of narrow, conditional writes, approval per run is often more useful because it lets the operator inspect the run's identity and scope without creating a reflexive click-through habit.

Do not mistake a pile of approvals for control. If operators cannot see the resource ID, operation, and current conflict result, they are approving a sentence while the service does the real work elsewhere.

A conflict response needs a defined owner

A rejected stale write is a successful safety outcome, but only if the run knows what to do next. "Retry on error" is the wrong default. It turns a disagreement into an automated race.

Classify each write path before allowing autonomous execution. The class determines who resolves a conflict:

The point is not to make agents timid. It is to distinguish recomputation from judgment. An agent can safely retry a report generated from current inputs. It should not choose between two approved production versions, two access decisions, or two different rollback plans just because it saw a 412.

Make conflict responses machine-readable. Include the resource identity, current version, conflict category, and whether the endpoint allows a fresh automatic attempt. A vague 409 with an HTML error page pushes the agent toward guesswork.

Test the collision you expect to happen

Do not wait for production traffic to prove that your checks work. Build a test that pauses a run between read and write, lets a second run change the same resource, then releases the first run. Verify four outcomes:

1. The first write fails without changing the resource.
2. The response identifies a stale version rather than a generic server error.
3. The agent does not resend its old body automatically.
4. Your journals can connect both attempts to their run identities and approvals.

Run the same test with a timeout and retry to prove that the idempotency behavior is separate. These are different failure paths, and they need different expected results.

Audit both the attempted action and the resulting state

A production account needs two records after an agent collision: the command path and the authoritative resource history. Gateway logs explain who requested an action and through which approved session. Service logs explain whether the state changed, what version won, and why a request failed.

Do not settle for an activity record that says "PATCH succeeded." Record the resource identifier, method, request correlation ID, precondition sent by the client, idempotency key or a safe reference to it, response status, and resulting ETag. If your service keeps field-level history, capture the changed fields there instead of trying to infer them from an agent transcript.

Sallyport projects its Sessions and Activity journals from an encrypted hash-chained audit log. If you use it for agent actions, run this check when you investigate a disputed sequence:

sp audit verify

The command verifies the chain offline over ciphertext and does not need the vault key. It can establish whether that local journal remained intact; compare it with the destination service's request logs before you claim to know what happened.

Retention and access rules matter here. An agent transcript may include flawed reasoning or copied operational details, while a request journal should be a compact factual record. Keep the evidence needed to reconstruct authority and state transitions, then limit who can browse it.

Parallelism belongs in independent resource sets

You do not need a single global queue for every autonomous run. You need a rule that lets independent work proceed while making shared mutation explicit. Partition by tenant, environment, repository branch, service, or another resource namespace that the service can verify.

A practical production design has a coordinator that assigns each run a write set and grants credentials or gateway access only for that set. The coordinator does not decide whether every change is wise. It prevents two workers from arriving with overlapping authority by accident. The receiving services still enforce versions and invariants because coordinators fail, assignments drift, and people launch emergency work outside the normal path.

When an action spans multiple resources, resist the urge to call it one atomic change if the services cannot actually transact together. Record the intended state, order the writes so later steps can validate earlier ones, and define compensation before execution. A compensation action also needs a current-state check. Rolling back to an old snapshot can erase a legitimate change that happened after the original run.

The first production test should be deliberately boring: pick one shared configuration resource, start two agent runs from the same version, and make them propose incompatible edits. If the service accepts both, fix that endpoint before you give either run a larger remit. Autonomy gets less interesting after a collision, and that is exactly why you should force the collision in a controlled test first.