Does approval queue call order survive a race?

A human approval means little if a later agent call can jump past an earlier one because its card happened to get clicked first. When two writes compete for the same remote state, the gateway must assign their order before a person touches either card, then preserve that order through dispatch and into the audit record.

This is easy to miss because the happy path has one card, one click, and one successful response. The defect appears when two agent processes send requests within the same moment, a reviewer approves the visible cards in an inconvenient sequence, and the target accepts whichever request reaches it first. I have seen teams call that behavior concurrency. It is just an undefined decision made after the human thought they had controlled it.

Order starts when the gateway accepts the call

Call order is the sequence in which the action gateway accepts requests into a lane for the same conflicting resource. It is not the order cards render, the order a person clicks, the order a network connection opens, or the order remote responses return.

Give each accepted call an immutable ticket such as 42, 43, and 44. Store the ticket with the request description, the calling process identity, the target, and the lane identifier. The ticket must exist before the gateway asks for approval. Otherwise the interface can only report which request the reviewer happened to see first, which is not enough to reconstruct a race.

A lane is the scope in which changing order can change the result. Two requests that both overwrite the same deployment environment belong in one lane. Two calls that append to the same remote incident record belong in one lane. A request to fetch package metadata and a request to update a separate test service may not need to wait on each other. Calling every action globally ordered feels safe, but it turns an unrelated slow request into an outage for every agent.

The difficult part is choosing the lane honestly. A target hostname is often too broad. A bare endpoint path is often too narrow. PATCH /documents/7 and POST /documents/7/publish affect the same document even though their paths differ. If the gateway lacks enough information to derive that relationship, put those actions in the same configured lane. Do not pretend the remote service will repair an ordering promise the gateway never made.

The admission record should have a shape close to this:

ticket=42 lane=release-prod event=accepted action=write-release caller=agent-a
ticket=43 lane=release-prod event=accepted action=write-release caller=agent-b

Those lines answer a precise question later: which call did the gateway take responsibility for first? They do not claim that ticket 42 completed first. A slow target can make ticket 43 finish later or earlier depending on the permitted execution model. The gateway must state that model instead of allowing the journal to invent one after the fact.

An approval is a decision, not permission to overtake

A reviewer deciding on request B before request A does not make B earlier. It only means B has satisfied one condition while it waits for its turn.

This distinction gets blurred because approval interfaces usually treat every card as an isolated prompt. That works for an action with no ordering consequence. It fails for competing writes. If the interface lets both cards remain actionable, a click on B must move B to an approved waiting state, not directly to the dispatcher. The dispatcher checks the head ticket in the lane. It dispatches only the earliest ticket that has an allow decision.

A gateway has three sensible outcomes for the head ticket:

• Approval permits dispatch of that ticket.
• Rejection records a terminal refusal, then releases the next ticket.
• Expiry or explicit cancellation records a terminal outcome, then releases the next ticket.

There is a fourth behavior that causes trouble: a later approval dispatches immediately because its earlier neighbor has no decision yet. It seems friendly because the reviewer gets a fast result. It also changes the remote sequence according to a moment of user interface timing. A person may have opened A to inspect its parameters while approving B as a harmless housekeeping call. The system then sends B first, despite presenting the pair as a queue.

The user interface should make the state visible. The head card can offer Approve and Reject. Later cards can accept a decision, but their status should say that they are waiting behind ticket 42, or the interface can hold their controls until earlier tickets resolve. Either approach can preserve order. The first gives the reviewer more control; the second is harder to misunderstand. What the interface must never imply is that every positive click causes immediate execution.

Sallyport's per session authorization and per call approval controls make the request boundary visible, but order still needs a ticket assigned before the approval decision. A reviewer cannot evaluate a queue if the product records only an unstructured collection of cards.

Reproduce the race with writes that leave a trace

A useful test sends two independent agent processes toward one write lane and makes the target record arrival order. Do not use two reads, two idempotent status checks, or two requests that update unrelated records. Those tests can pass while the queue permits overtaking.

Use a disposable HTTP endpoint that accepts a POST body and appends the received ticket to a file or database table. It should return the ticket it received. The target does not need authentication for this drill if it runs only on a local test address; the point is to test the gateway's approval and dispatch path, not credential injection.

This small Node server produces a plain arrival log:

const fs = require("node:fs");
const http = require("node:http");

http.createServer((request, response) => {
  let body = "";
  request.on("data", chunk => { body += chunk; });
  request.on("end", () => {
    const item = JSON.parse(body);
    fs.appendFileSync("arrival.log", `${item.ticket} ${item.value}\n`);
    response.writeHead(200, { "content-type": "application/json" });
    response.end(JSON.stringify({ received: item.ticket }));
  });
}).listen(8787);

Configure one approved action that sends POST /write to that endpoint and carries the request ticket in its JSON body. Start two fresh agent processes rather than asking one process to issue two calls serially. Each process submits the same action with a distinct value, for example A and B. Arrange for both calls to require approval so the cards remain pending together.

The expected initial evidence is boring and specific:

Sessions
42 accepted agent-a write A
43 accepted agent-b write B

Approval cards
42 write A
43 write B

arrival.log
(empty)

If the cards appear in reverse ticket order, stop there and investigate. A reverse visual order does not prove wrong execution, but it invites the reviewer to act on a false story. If only one card appears at a time, record whether the second request already has a ticket and whether the journal says it waits behind the first. Hiding the second card is fine. Hiding its existence is not.

Run this drill many times. Start A first, then B first. Start them as close together as your test harness allows. Add a short delay after one process enters the gateway to make scheduling less accidental. The test needs repeated overlap because a single successful run often reflects a convenient thread schedule rather than an enforced rule.

Card order, dispatch order, and completion order differ

The test should collect three orders, because they answer different questions and should not be collapsed into one column.

Card order is the order in which the reviewer can see pending decisions. It should follow ticket order within a lane, even if the application draws the cards on separate event loop turns. A card may show an arrival timestamp, but the ticket is the decisive field.

Execution order is the order in which the gateway releases accepted actions to the HTTP or SSH channel. For a strictly serialized write lane, the target should observe ticket 42 before ticket 43. The gateway should emit a dispatched event immediately before it hands the request to the channel. Do not infer dispatch from a response event. A target may receive a write, alter state, and then lose the connection before it sends a response.

Completion order is the order in which responses, timeouts, or transport errors return. It can differ from dispatch order if the system permits overlapping calls in separate lanes. Even in one lane, an asynchronous implementation can record completion late because it flushes logs after network cleanup. Completion order is useful operational information, but it must not overwrite causal order.

A compact test record makes the distinction hard to dodge:

42 accepted
43 accepted
42 card-shown
43 card-shown
43 approved
42 approved
42 dispatched
42 succeeded
43 dispatched
43 succeeded

The fact that 43 approved appears before 42 approved is expected in this run. The fact that 42 dispatched appears before 43 dispatched is the contract. If the application logs only final successes, both facts disappear and an investigator cannot tell whether the user approved B first, whether the dispatcher overtook A, or whether the target reordered two already released requests.

RFC 9110 distinguishes safe HTTP methods from methods that request state changes. That distinction matters here: a gateway may tolerate looser ordering for observations, while writes need an explicit concurrency contract. RFC 9110 does not supply a useful total order across two independent client connections. The gateway owns that decision once it places approval in front of the call.

Approve the second card first and hold it

The strongest basic race test deliberately approves B first. It tests whether the implementation treats approval as a state transition or as a direct send button.

Start with tickets 42 and 43 pending in the same lane. Approve ticket 43. The card or journal should change to something equivalent to approved, waiting for 42. The target's arrival.log must remain empty. Then approve ticket 42. The target should receive 42 A followed by 43 B, and the activity record should show both dispatches in that sequence.

Repeat the same test with ticket 42 rejected. The expected target log contains only B. The journal still has both tickets:

42 accepted
43 accepted
43 approved
42 rejected reason=user
43 dispatched
43 succeeded

A rejection is an action outcome, not an absence of action. If the journal omits it, someone reviewing the log later sees B execute with no explanation for the missing earlier request. That invites a bad conclusion: perhaps an attacker bypassed approval, perhaps the app lost data, perhaps a reviewer approved something they did not see.

Then test a timeout. Let A's approval expire while B has already been approved. The application should record A's expiry once, make B eligible, and dispatch B. It must not produce both expired and rejected because a background timer and a late click raced each other. Pick one terminal decision with an atomic compare and swap on the ticket state. The losing event should see that the ticket has already resolved and do nothing.

Finally, cancel B after approving it but before A resolves. B should never dispatch. Its terminal state should say cancelled, and resolving A later must not revive it. This catches a common queue bug: the dispatcher stores an old list of approved tickets, then sends an entry after a cancellation handler removed it from the visible queue.

A good test asserts each result separately. Do not settle for one assertion that the target received the expected final state. A final state can look correct after the wrong sequence if B overwrites A, retries hide a duplicate, or the target applies its own conflict rule.

Preserve the ticket across retries and slow targets

A request keeps its place in line when the channel retries it. Creating a new ticket after a connection failure changes the meaning of the approval and can let a later request overtake it.

Suppose ticket 42 dispatches, the TCP connection closes before the gateway receives a response, and the target may or may not have applied the write. Ticket 43 waits. The gateway has several policies available: report unknown outcome and stop, retry with an idempotency token if the target supports one, or require a new human decision. It does not have permission to quietly drop 42 and send 43 as though 42 never existed.

The right policy depends on the remote operation. An endpoint that accepts an idempotency identifier can make a retry safe enough to reuse the original ticket. A blind command over SSH often cannot. In that case, report an unknown result for 42, keep the lane blocked or explicitly abandon the lane after human intervention, and record why. Releasing 43 automatically can compound damage when 43 assumes 42 failed.

Test this with a target that receives A, writes its arrival record, and closes the response before completing the HTTP exchange. The gateway should preserve a trace like this:

42 accepted
42 approved
42 dispatched attempt=1
42 outcome=unknown
43 accepted
43 approved waiting-for=42

Whether ticket 43 eventually runs is a documented operational choice. The unacceptable behavior is a journal that claims 42 failed before the gateway has evidence, followed by a successful B that relied on that claim.

Slow requests expose a separate implementation error. A dispatcher that holds a mutex while waiting for the remote response may serialize everything by accident, including unrelated lanes and user interface work. A dispatcher that releases its order state too early lets the next ticket race ahead. Keep the lane's admission and dispatch state small, persist the dispatch event, hand the request to the channel, then wait for the result without allowing another ticket in that lane to pass when strict serialization applies.

The audit journal must retain causality

A hash chained audit record proves that retained records were not silently altered; it does not make a confusing event sequence understandable. The event model still needs enough information to explain a concurrent approval race.

Record an acceptance event before any card appears. Record an approval decision with the ticket and the actor or interaction method. Record dispatch before the HTTP request or SSH command enters its channel. Record the terminal channel outcome without replacing any earlier event. Each event needs its own append position in addition to wall time.

Wall clocks are useful for debugging but cannot define order by themselves. Two events can share a timestamp resolution, clocks can move, and an application may queue writes from different threads. An append position or sequence number establishes the order in which the journal accepted each event. The per request ticket establishes the intended order in the lane. Keep both.

For the two write drill, compare these facts:

• Acceptance positions show 42 before 43.
• Approval events may show 43 before 42.
• Dispatch positions show 42 before 43 when both calls succeed.
• The target arrival file shows 42 before 43.
• Terminal outcomes attach to the same tickets rather than replacing them.

Sallyport projects Sessions and Activity views from one encrypted hash chained audit log, and sp audit verify can check that chain offline over ciphertext. That makes event selection especially important: verification can show that records have not been modified, while the ticket and event types tell a human what actually happened.

Do not sort the visible journal by completion time and call it history. That presents a network race as a decision sequence. Sort the primary timeline by durable append position, then show ticket number and timestamps beside it. A filtered view for one agent process should retain original positions so a reviewer can see that another request waited between two events without needing to guess.

FIFO applies to conflicting effects, not every byte

A FIFO queue is a promise about effects that conflict. It is not a reason to force every network operation through one thread or to delay harmless reading behind an unresolved write.

Start by classifying each action. A deployment command that mutates a shared environment needs a lane. An HTTP POST that creates a billing action needs a lane, perhaps per account. A fetch that reads a static artifact can often run independently. An SSH command that only reports disk space may be observational, but be careful: commands often have hidden effects through shell startup files, temporary files, or remote command wrappers. Treat ambiguous commands as writes until you can describe their effects.

The popular alternative is to say that every approval is independent because the reviewer can inspect the request. That is attractive because it removes queue design. It fails when the reviewer understands each request individually but has no indication that B assumes A has already happened. Human review does not repair missing serialization context.

The opposite error is a single global queue. It makes order simple, then makes the application feel stuck when one remote host stalls. Separate lanes need clear names, stable resource selection, and journal fields that identify them. If you cannot explain why two actions share a lane, you cannot test the ordering promise either.

Do not use per call approval as a substitute for lanes. Requiring a click for every use helps a human evaluate each action. It does not define whether two approved actions can pass one another. Those are different controls with different failure modes.

Make the race a release criterion

A queue bug rarely appears as an obvious crash. It appears later as an unexplained remote state, an approval that seemed to apply to the wrong action, or a journal that cannot settle an incident review. Treat the two write drill as a release test for every action type that can change shared state.

The release criterion should check more than success responses. Generate overlapping calls, reverse the approval sequence, reject the first request, let it expire, cancel the second request, and force an unknown result after dispatch. For each case, save the ticket sequence, the card states, the dispatch events, the target arrival log, and the journal entries.

When one of those records disagrees, resist the urge to call it a display issue. A display issue may be the first visible sign that separate parts of the application chose separate definitions of order. Fix the contract at admission, then make the card, dispatcher, target test, and journal report the same contract.

The first practical action is small: add an immutable ticket to one conflicting write lane and make your test approve the later ticket first. If that request reaches the target before its earlier neighbor, the approval queue does not yet control the work it claims to approve.