# Agent activity ordering for reliable incident reconstruction Agent activity ordering decides whether an incident report explains what happened or merely displays a pile of timestamps. When an autonomous coding agent can call APIs and open SSH sessions, investigators need to establish which process acted, what it attempted, what completed, and what result the agent received before it chose its next action. A list sorted by time is not enough. Clocks drift, requests overlap, responses arrive out of order, and a timeout can conceal an action that succeeded remotely. Build records that preserve session context, a durable local order, and distinct lifecycle timestamps. Otherwise, the first hard incident will turn your audit trail into an argument about guesses. ## Time alone cannot establish action order A timestamp says when one clock observed an event. It does not prove that event happened before every other event with a later timestamp. This distinction sounds academic until two calls leave an agent in rapid succession, one remote service stalls, and the second call returns first. Sorting `completed_at` gives you response order, not agent decision order. Each call has several meaningful moments. The agent decides to invoke a tool. The gateway accepts the request. It authorizes the request. It dispatches work to an HTTP service or SSH helper. The remote side may accept it. A response returns. The gateway gives that result to the agent. Treating all of that as one event removes the evidence needed to explain a failure. Keep a local, monotonically increasing `call_sequence` within every session. Assign it when the gateway accepts a call, before network work begins. That number answers a bounded, useful question: "In what order did this gateway accept calls from this agent process?" It does not claim to describe remote execution order. Honest scope is better than a broad claim you cannot defend. Use a second durable sequence for the audit journal itself. Calls from different sessions can overlap, and authorization, revocation, vault lock events, and verification events must live in the same evidence stream. A journal sequence tells an investigator the write order of the recorder. A session call sequence tells them the intent order within one agent run. Neither field replaces the other. Do not use timestamp precision as a substitute for sequence numbers. Adding more decimal places only records a more precise reading of a clock. It does not solve a clock adjustment or establish a total order across concurrent writers. RFC 3339 defines a useful interoperable representation for wall-clock timestamps, including an explicit UTC offset. Use its UTC form, such as `2025-03-08T21:14:03.482Z`, for exports and human review. RFC 3339 does not promise causal order. That is your recorder's job, and it requires sequence fields and clear event boundaries. ## A session identifies the acting process, not a vague task A session should bind an ordered run to the specific agent process that received authority to act. It should begin when that process establishes a connection and end when the process exits, loses its channel, or an operator revokes it. Do not make a session mean "work on ticket 184" or "the afternoon deployment." Those labels can help search, but they do not define an execution boundary. The session record needs enough identity to answer questions investigators actually ask: which executable connected, who signed it, which local user launched it, what transport connected it, and when its authority began and ended. Record identifiers that the operating system or connection can attest. Do not let the agent write its own identity claims into the authoritative fields. That separation matters when someone copies a tool configuration into a different process. The action request can say it intends to operate on a particular repository. The gateway should record the process identity it observed. During an incident, the second statement carries more weight. A practical session record might contain: ```json { "event_id": "01JNRQ2Q9Y9J0R3E5P8F7K2X4M", "journal_sequence": 8124, "event_type": "session.opened", "occurred_at": "2025-03-08T21:14:02.901Z", "session_id": "sess_7f31c4", "process": { "pid": 48102, "code_signing_authority": "observed signing authority", "local_user": "developer account" } } ``` The agent should receive an opaque session handle, not authority to choose the session identifier or alter its metadata. An agent can still attach its own run label, repository path, or task reference in a separate declared-context field. Mark those values as agent supplied. That label can explain intent, but it must never overwrite observed process facts. Per-session authorization has a second investigative benefit. It records a human decision tied to a bounded process run. If that run later makes a damaging call, reviewers can see the authority event that preceded it and the session close or revoke event that ended it. An approval that silently applies to later processes creates a gap that no amount of call logging can repair. ## One call needs a lifecycle, not one completion line A useful call record preserves the lifecycle of one attempt. It does not collapse an attempted request, a network dispatch, a remote result, and the agent-visible result into a single vague "success" or "failure" field. Start with an immutable `call_id` and the session's next `call_sequence`. Record an accepted event before you contact the outside world. If policy or an approval blocks the request, that accepted event and the denial event still matter. They show intent and control behavior without pretending the external action occurred. For an allowed HTTP action, capture these separate event boundaries: 1. `call.accepted` records the ordered request at the gateway. 2. `call.authorized` or `call.denied` records the control decision. 3. `call.dispatched` records that the gateway handed the request to its network client. 4. `call.result_received` records the transport outcome or remote response. 5. `call.result_returned` records the outcome delivered back to the agent. The names can differ, but the semantics should not. A result received is not always a result returned. The gateway may redact a response, reject malformed data, lose the agent connection, or encounter an internal failure while preparing the result. Investigators need to see that gap. Keep `request_started_at`, `dispatched_at`, `result_received_at`, and `result_returned_at` when those moments occur. Use null for a moment that did not happen. Do not invent an end time when a process crashed. Record a later recovery event that says the recorder found an unfinished call. This example shows the shape of a completed request without exposing a bearer token or full response body: ```json { "event_id": "01JNRQ3M8W7P0Q4R6S9T1V2X3Y", "journal_sequence": 8131, "event_type": "call.result_received", "occurred_at": "2025-03-08T21:14:05.841Z", "session_id": "sess_7f31c4", "call_id": "call_00017", "call_sequence": 17, "channel": "http", "target": "api.internal.example/v1/releases", "method": "POST", "dispatch_event_id": "01JNRQ3G2A...", "outcome": { "transport": "response", "http_status": 201, "response_digest": "sha256:..." } } ``` The request and response digests let you compare retained evidence without placing secrets or large sensitive payloads in every journal reader's hands. A digest does not make a secret safe to log. Low-entropy values, predictable identifiers, and short tokens remain guessable. Exclude credentials at capture time, then decide which payload fragments your incident process genuinely needs. ## Retries and timeouts create the hardest ambiguity A timeout means you do not know whether the remote side acted. It does not mean the remote side did nothing. Teams repeatedly get this wrong because application logs often treat timeout as a simple error and the next retry as a replacement for the first attempt. Consider an agent that creates a release through an HTTP request. Call 41 receives a request timeout after dispatch. The agent reads that failure and sends call 42, a retry. Later, the remote service processes both requests. If your journal overwrote call 41 with a final status of "retried," investigators will see one successful request and miss the duplicate action. Give each network attempt its own `call_id` and its own `call_sequence`. Add `retry_of` when an attempt directly follows an earlier attempt. Preserve the agent-visible cause for the retry, such as timeout, connection reset, or a received retryable status. The relationship lets an investigator trace the chain without flattening it. A complete sequence can look like this: ```text sequence 41 accepted 21:14:11.024Z create release, request r_8d2 sequence 41 dispatched 21:14:11.027Z sequence 41 result_received 21:14:41.031Z timeout sequence 41 result_returned 21:14:41.034Z timeout returned to agent sequence 42 accepted 21:14:42.112Z retry_of call_00041, request r_8d2 sequence 42 dispatched 21:14:42.115Z sequence 42 result_received 21:14:42.490Z HTTP 201 sequence 42 result_returned 21:14:42.493Z HTTP 201 returned to agent ``` The repeated request reference is only useful if the remote API supports an idempotency mechanism or another stable operation identifier. If the service accepts an idempotency key, generate and record a nonsecret key that remains constant across retries for one intended operation. If it does not, record that the retry risk remains unresolved. Do not claim idempotence because the payload looks similar. SSH adds a different problem. A command can execute remotely and the connection can fail before the client receives output or an exit code. Record the command dispatch, the connection identity, the host reference, and the observed termination state. Label an interrupted SSH command as "outcome unknown," not "failed." A later command that checks remote state may reduce uncertainty, but it does not rewrite the original outcome. Do not turn all failures into terminal events. An authorization denial is terminal for that call because no external dispatch occurred. A local DNS failure may be terminal for the attempt. A timeout after bytes leave the machine has an unknown external outcome. These categories drive different incident decisions. ## Record two kinds of time and explain their limits Wall-clock time makes a timeline readable across systems. Monotonic time measures elapsed time without changes caused by network time synchronization or a manual clock adjustment. Collect both where the operating system provides them, and state what each means in your schema. For every journal event, record an RFC 3339 UTC `occurred_at` value. For events inside a live session, also record `monotonic_ns` measured from that process's chosen monotonic-clock origin. Do not compare monotonic readings from separate machines unless you have explicitly established a shared reference. They are local measurements. A clock correction can produce confusing records such as this: ```text journal 901 wall 21:19:07.900Z monotonic 5562019921 call accepted journal 902 wall 21:18:58.104Z monotonic 5562026310 call dispatched ``` The wall clock moved backward. The journal sequence and monotonic value still show that dispatch followed acceptance. The export should retain the original timestamps rather than silently sorting and rewriting them. Add a recorder event when the operating system reports a material time change, if you can observe it. That event gives reviewers a reason for the discrepancy. NIST Special Publication 800-92, Guide to Computer Security Log Management, advises organizations to synchronize clocks and to define log data requirements before an incident. That guidance is correct, but synchronized clocks alone do not give you order inside an agent run. Clock synchronization improves correlation with a remote API, a CI service, or host logs. Your local sequence still establishes the recorder's order. Remote timestamps deserve their own fields. An HTTP `Date` header, a provider request ID, and a server-generated event time are external claims. Preserve their source and exact value. Do not copy them into `occurred_at`, and do not use them to renumber your local journal. A remote timestamp can help reconcile systems later, but it might reflect a queue, a different clock, or a response-generation time. Duration fields also need a precise definition. `gateway_duration_ms` can mean the time from acceptance to result returned. `network_duration_ms` can mean dispatch to result received. Write the definition next to the schema. Otherwise a report that says a call took 30 seconds cannot tell whether the delay happened before dispatch, at the remote service, or after the response returned. ## The audit writer must choose order before it publishes results You cannot reconstruct order if concurrent workers write records whenever they happen to finish. Give the audit writer one append path that assigns a journal sequence, captures the event time, links the prior record, and commits the record before the system tells the agent that an externally meaningful state change occurred. This does not require one giant lock around all network activity. Calls can run concurrently. The recorder only needs a narrow serialized commit point. When a worker reaches an event boundary, it submits an event to that writer. The writer assigns the next durable journal sequence. The resulting order reflects commit order, which you must name accurately in documentation and exports. The failure pattern is familiar. Worker A accepts call 17 and starts a slow request. Worker B accepts call 18 and finishes quickly. If workers append only their completion records, the journal begins with call 18's success. An investigator cannot tell whether call 17 was in flight, never sent, or omitted. The accepted and dispatched events for call 17 close that gap. Hash chaining adds tamper evidence to the committed sequence. Each entry contains the digest of the previous committed entry and a digest of its own canonical contents. Canonicalization matters. The same data must produce the same bytes before you hash it. Specify field order, UTF-8 encoding, timestamp representation, null handling, and number formats. "We hash the JSON" is not a specification because ordinary JSON object order is not a security property. A conceptual entry might use these fields: ```json { "journal_sequence": 8131, "event_id": "01JNRQ3M8W7P0Q4R6S9T1V2X3Y", "previous_hash": "sha256:9c7d...", "record_hash": "sha256:04b1...", "payload": {"event_type": "call.result_received"} } ``` A valid chain tells you that retained entries connect without an undetected change, assuming the verifier has the expected chain anchor. It does not prove completeness if an attacker controls the recorder and can prevent it from writing a record. Do not sell hash chaining as magic. It makes alteration visible; it cannot record an event that the recorder never observed. Sallyport projects its Sessions and Activity journals from one encrypted, hash-chained audit log. Its `sp audit verify` command verifies the chain offline over ciphertext, without requiring a vault key. That design keeps the session view and individual-call view tied to one ordered source instead of asking investigators to reconcile two separate logs. ## Build a timeline that preserves uncertainty An incident timeline should show facts, observations, and unresolved outcomes separately. A polished narrative that converts unknowns into confident verbs may feel useful during a tense review, but it creates a false record that later evidence can contradict. Suppose an agent process received approval at 09:00:00. It issued an SSH command at 09:03:14. The client lost its connection at 09:03:16. At 09:03:18, the agent used HTTP to query the target system and found a changed configuration. That evidence supports several explanations: the SSH command completed, another actor changed the state, or an earlier queued task took effect. The timeline must state which conclusion the evidence supports and which it does not. Use this form in incident notes: | Order | Time | Evidence | Supported statement | |---|---|---|---| | 444 | 09:03:14.120Z | `call.dispatched` | Gateway sent the SSH command to the helper. | | 445 | 09:03:16.202Z | transport disconnect | Gateway did not receive an exit status. | | 446 | 09:03:18.810Z | HTTP query response | The queried configuration differed at this time. | | 447 | 09:03:19.001Z | `call.result_returned` | Agent received the query result. | Avoid writing "the SSH command changed the configuration" unless you have direct evidence that joins the command to the remote effect. Remote audit records, a unique operation ID, or a response containing a durable server-side request ID can provide that join. A nearby timestamp does not. Investigators also need to know what the agent saw when it made later decisions. This is why `result_returned` deserves its own event. If the remote response arrived but the agent disconnected before receiving it, a later agent action did not follow that response. If the response reached the agent, it may explain a dangerous branch in its behavior. Present both a session lane and a call lane in the incident view. The session lane shows open, approval, revoke, lock, and close events. The call lane shows accepted, authorization, dispatch, and outcome events. A flat list remains available for verification, but the two views answer different questions without mixing them. ## Secret handling must survive the incident review Audit trails often fail at the moment they become most useful because someone wants to log full headers, shell environments, and response bodies "just for this one investigation." That decision can turn a contained agent incident into a credential exposure. Capture action identity without secret material. For HTTP, record the method, normalized host and path, credential reference or key label, safe header names, request digest, response status, provider request ID when present, and a carefully chosen response summary. Never record an authorization header, a raw API key, a private key, or a complete environment dump. For SSH, record a host reference, the account reference if policy allows it, a normalized command representation, the command digest, connection status, and exit status when received. Commands themselves can contain secrets. If your workflow permits arbitrary shell text, use a protected evidence store for narrowly authorized review, or record only a redacted form plus a digest. Do not pretend a command journal is harmless because it lacks passwords. Sallyport keeps API and SSH credentials in its encrypted vault and executes the external action without passing those credentials to the agent. That removes one common reason an agent transcript and its logs become a secret dump, but the target, request body, command arguments, and response can still be sensitive. Control access to raw records separately from verification. A responder may need to run a chain verification without permission to read encrypted call detail. A security reviewer may need session and target metadata but not payload material. This separation makes incident response less dependent on copying an entire journal into chat, tickets, or spreadsheets. When you export evidence, include the schema version, export time, journal sequence range, verification result, and redaction rules used. Retain the original protected journal under its normal controls. An export is a working copy, not a replacement for source evidence. ## Test the record with a deliberately messy run A happy-path demo proves almost nothing about incident reconstruction. Test the conditions that make ordering ambiguous: concurrent calls, delayed responses, clock changes, process exits, denials, revocations, and a timeout followed by a retry. Run a controlled exercise with two allowed external targets. Make the first call wait before returning. Start a second call after the first dispatches. Interrupt a third call after dispatch. Then revoke the session and verify that later calls receive a denial. Export the journal and hand it to a colleague who did not write the scenario. Ask that reviewer to answer five questions using the export alone: - Which process received authority, and when did that authority end? - What order did the gateway accept calls in that session? - Which calls reached the dispatch boundary? - What result did the agent receive before each later call? - Which outcomes remain unknown rather than failed or successful? If they must ask you what a field means, fix the schema or export documentation. If they infer a remote effect from a timeout, fix the outcome labels. If they cannot distinguish a retry from a new operation, add the relationship and operation identifier. Keep the exercise artifacts. They become regression tests when you change a client library, introduce concurrency, adjust retention, or add a new channel. Ordering bugs often arrive through innocent refactors because developers focus on whether actions still work, while the evidence path quietly changes its commit timing. The incident will not wait for a cleaner logging design. Assign a session sequence at call acceptance, commit lifecycle events through one ordered writer, preserve both wall-clock and monotonic time, and let unknown outcomes remain unknown. Those choices give investigators a sequence they can defend instead of a timeline they have to explain away.