Agent action timeline: compare timestamps without lies

An agent action timeline can tell a convincing lie even when every system logged an honest timestamp. The lie enters when an investigator treats a local clock display, an API server receipt, and an audit record as interchangeable evidence of one moment.

Store timestamps as observed, with numeric offsets and their source. Then compare them as separate clocks with separate meanings. This takes a little more data than a single created_at field, but it prevents the usual incident report where an agent apparently acted before it received approval, or an API request appears to finish before it began.

One timestamp cannot describe an action

An action has more than one meaningful time. An agent can decide to call an API at one instant, send bytes later, reach the server later still, and receive a result after the server finishes its work. Each event can matter when someone asks whether the agent exceeded its authority.

A typical record contains at least these claims:

• The agent process says when it started the action.
• The receiving server says when it accepted the request.
• The receiving server may say when it committed or completed the work.
• The action gateway says when it received and released the request.
• A person may see a local time in a user interface.

Those are not competing versions of a single field. They describe different points in a causal sequence. If you collapse them into one normalized timestamp at collection time, you throw away the distinction that explains queues, network delay, retries, approval waits, and long running calls.

I have seen teams label an API audit entry as the time an agent "did the thing" when the entry only showed request acceptance. That error becomes expensive when the endpoint queued work and performed the change much later. The agent may have stopped before the change occurred, but its request still caused it.

Use names that state the event. agent_action_started_at, gateway_received_at, server_received_at, and server_completed_at force a reader to ask what happened at each point. A vague field named timestamp encourages people to invent an answer later.

An offset preserves an instant, a zone explains the display

A numeric UTC offset turns a wall clock reading into a specific instant. A named time zone explains the civil time rules that produced the reading. You often need both, but they solve different problems.

Consider these two values:

2025-11-02T01:30:00-04:00
2025-11-02T01:30:00-05:00

The clock face says 1:30 AM in both cases. They are an hour apart. In North American daylight saving fallbacks, local time repeats, and an unqualified value such as 2025-11-02 01:30:00 leaves an investigator unable to determine which instant occurred.

RFC 3339 addresses this directly. Its timestamp form uses a full date and time plus either Z for UTC or a numeric offset. The RFC also permits an offset of -00:00 to mean that the source knows the time but does not know the local offset. That distinction is useful evidence. Do not silently rewrite -00:00 as Z; UTC claims a fact that the source did not make.

A zone identifier such as America/Los_Angeles is still worth keeping when a human approval, support ticket, or screen recording refers to local office time. It lets an investigator reproduce the calendar rules in force for that place. It does not replace the offset. Zone rules can change, and the same zone has different offsets across the year.

Store the received timestamp as a string, retain its original offset, and derive a UTC instant for sorting. Do not retain only a rendered local string. Rendering belongs at the edge, where a human chooses a display zone.

Local, server, and audit clocks answer different questions

The local time answers what the operator or agent host believed the time was. The server time answers when a remote service observed or performed work. The audit time answers when the system of record accepted an event. Investigators should compare all three, not nominate one as universal truth.

Start with the event boundary. If an agent requests POST /deployments, its local action time may support an account of intent. The server receipt time can establish when the remote service became responsible for the request. A server completion time may establish when a deployment state changed. An audit entry can establish when your control point observed the attempt and whether it approved it.

Network latency creates normal gaps between these values. Queues create larger ones. Retries complicate matters because a client may use one action identifier for several attempts, while the server records each attempt separately. A timeline that only displays the first local time hides all of that.

Do not use a browser display, terminal prompt, or screenshot clock as a tie breaker unless you know how that device synchronized time. Those displays often help explain what a person believed, but they rarely settle a close ordering dispute.

A practical comparison uses three columns in the investigation worksheet:

The rows should retain their own times even after you add a calculated UTC sorting column. A clean spreadsheet with one time column looks convenient, but it conceals the evidence trail.

Daylight saving changes create duplicate hours and missing hours

Daylight saving transitions expose timestamp shortcuts because they break a human assumption: every local minute occurs once and every day has the same length. Neither assumption holds.

During a fall transition, a local hour repeats. During a spring transition, an hour does not exist. A parser that accepts an unqualified local timestamp must choose a rule, reject the input, or guess. Guessing is unacceptable in an audit path.

The following record is usable because it carries both a precise instant and the civil context that generated it:

{
  "event_id": "act_8f3c",
  "event": "authorization_granted",
  "observed_at": "2025-11-02T01:14:22-04:00",
  "zone": "America/New_York",
  "instant_utc": "2025-11-02T05:14:22Z",
  "clock_source": "agent_host"
}

The instant_utc field is derived, so preserve the original observed_at string as well. If a parser later changes, or a bug affects conversion, you can rerun the derivation and explain the difference. Treat derived values as analysis output, not as a replacement for source evidence.

A zone abbreviation such as EST makes matters worse. Abbreviations are ambiguous across regions and do not reliably state daylight saving status. Use an IANA zone identifier for civil context and a numeric offset for the instant. If a source only emits an abbreviation, record it exactly and record that its meaning remains unresolved.

Recurring schedules need their own rule. Store the schedule as a local time plus an IANA zone, then calculate each occurrence under that zone's rules. Store an action that actually ran as an offset timestamp. A schedule says when a job was intended to run; an event record says when it ran.

Clock drift turns precise looking order into false certainty

Millisecond precision does not mean millisecond accuracy. An unsynchronized laptop can produce timestamps with six decimal places while being minutes away from a server. Sleep, network loss, virtual machines, and bad synchronization all produce this problem.

Separate precision from uncertainty in your data model. Precision is the number of digits recorded. Uncertainty is the interval in which you think the actual instant falls. A host timestamp of 10:00:00.123Z with an uncertainty of two seconds should not settle a one second ordering dispute with a remote API.

Measure clock offset when you can. Capture the time from a trusted reference before an agent run and again afterward, then store the observed difference. If the reference is remote, account for request transit time. A simple midpoint estimate works for rough operational work when you retain the measurement rather than presenting it as an exact correction.

For example, a collector sends a request at local 10:00:00.000, receives a trusted response at local 10:00:00.200, and the response says 10:00:00.150Z. The server time occurred sometime within the round trip. The local midpoint is 10:00:00.100, so the host appears about 50 milliseconds behind under the usual symmetric transit assumption. That assumption may fail, which is why the useful output is a range, not a declaration of truth.

Monotonic clocks solve a narrower problem. A monotonic clock measures elapsed time within one running process and does not jump when wall time changes. Record a monotonic start value and duration alongside wall time when you need to prove that action B followed action A within the same process. Do not convert monotonic values into UTC or compare them across hosts.

NTP documentation makes a similar practical distinction: synchronization estimates offset and dispersion rather than granting perfect time. Treat those estimates as part of your evidence when the ordering is close enough to matter.

Keep raw evidence and normalized time in the same record

A defensible event schema preserves what each participant actually reported and makes analysis reproducible. The following JSON shape works for an agent action without pretending that all fields originate from one clock.

{
  "action_id": "a91c2d7e",
  "attempt": 2,
  "agent": {
    "process_id": "p_4b71",
    "started_at": "2025-04-18T14:07:12.481-07:00",
    "zone": "America/Los_Angeles",
    "monotonic_start_ms": 9184421,
    "clock_uncertainty_ms": 750
  },
  "gateway": {
    "received_at": "2025-04-18T21:07:12.661Z",
    "authorized_at": "2025-04-18T21:07:14.034Z",
    "result_released_at": "2025-04-18T21:07:14.882Z",
    "audit_event_id": "aud_3e90"
  },
  "server": {
    "request_id": "req_7c19",
    "received_at": "2025-04-18T21:07:14.301Z",
    "completed_at": "2025-04-18T21:07:14.649Z",
    "status": 201
  },
  "normalization": {
    "sort_instant_utc": "2025-04-18T21:07:12.481Z",
    "method": "RFC3339 offset conversion"
  }
}

This schema draws a boundary that teams often blur: an action identifier connects records, while a timestamp orders one event within that action. Reusing an identifier after a retry is sensible. Reusing one timestamp for every stage is not.

Record the server request ID even when you already have an internal action ID. During an investigation, the server's own identifier often provides the only dependable way to distinguish a timed out request from a request that never left the client.

Avoid storing only epoch milliseconds unless the producer guarantees UTC and documents its clock source. Epoch values sort easily but lose the original offset, display context, and sometimes the unit. If you accept them from third parties, record the unit explicitly and retain the received representation.

A timeline needs intervals when evidence overlaps

When two sources have uncertainty, calculate intervals rather than forcing an order. This prevents a familiar mistake: an investigator sees 10:03:01.010 and 10:03:01.400, sorts them, and states that the first event caused the second despite two machines having unknown clock offsets.

Suppose the agent reports an action start at 21:07:12.481Z with 750 milliseconds of uncertainty. Its possible interval runs from 21:07:11.731Z through 21:07:13.231Z. The gateway reports receipt at 21:07:12.661Z with an uncertainty of 20 milliseconds. The intervals overlap, so the timestamps alone cannot prove the gateway received the call after the claimed start. The protocol sequence can still support that conclusion, but the wall clocks do not.

State the reason for every ordering claim. These are different conclusions:

• "The gateway recorded receipt after the agent emitted the request" follows from protocol evidence or a correlated request trace.
• "The gateway receipt timestamp is later" follows only from the displayed wall times.
• "The records establish order within their stated uncertainty" follows when the intervals do not overlap.
• "The records cannot establish order" is the correct result when intervals overlap and no causal evidence fills the gap.

People dislike the fourth conclusion because incident reports want a neat story. Inventing precision does not improve the story. It gives the next reviewer a reason to doubt every conclusion around it.

This also changes alert design. Do not flag an agent merely because a gateway event appears a few hundred milliseconds before an agent's local start. Flag a negative elapsed time only after applying known offset bounds, or mark it for clock health investigation.

Approval and execution must keep separate timestamps

A human approval proves that someone permitted a capability at a particular time. It does not prove that the agent sent a request at that same instant, and it certainly does not prove that a remote system finished the requested work then.

Keep approval events distinct from calls. An approval record should include the actor, the scope granted, the process or session it applied to, the observed time, and the authorization system's event ID. A call record should reference that authorization where applicable and retain its own receive and release times.

This distinction matters most with session authorization. One approval can cover several actions over the life of a process. If a later action causes harm, an investigator needs to answer two separate questions: when did the authorization occur, and did the process remain within the approved session when it made this call? A single approved_at field cannot answer both.

Per call approval creates a tighter sequence but still has a gap. The user may approve at 14:07:14, the gateway may dispatch at 14:07:14.1, and the remote service may commit at 14:08:02. A remote timeout after dispatch does not erase the possibility that the service completed the action.

For a gateway controlled workflow, log denial events too. A denied call establishes that an agent attempted an action, even though no remote request should have occurred. If the agent retries after permission changes, the timeline needs separate attempts with separate evidence. Do not overwrite a denial with a later success.

Work through a disputed deployment without flattening the evidence

Assume an agent requested a deployment after a developer approved a session. The developer later says the approval happened after business hours, while the remote service record says the deployment began before the approval. The apparent contradiction often comes from comparing local display time to UTC server time.

The evidence contains these records:

Convert the first two records, but preserve their received forms. The approval occurred at 00:58:40Z, and the agent began at 00:59:02Z. The gateway and service records follow in a plausible causal sequence. Nothing occurred before approval; someone simply read 17:58 next to 00:59 as though both values used the same clock display.

Now add the uncomfortable part. Suppose the agent host has an estimated uncertainty of 90 seconds because it had been asleep. You can still say that the gateway received a call after the authorization system recorded approval, because those records use the gateway and authorization trail. You cannot use the agent's local start time to establish a second level ordering claim within that interval.

The investigation should preserve both conclusions. One is strong and one is limited. That is better than making the whole timeline look equally precise.

Verify audit integrity before trusting its placement in time

An audit log can tell you whether someone altered, reordered, or removed evidence only if you verify its integrity mechanism. Perform that verification before you use audit entries to support a sequence. Then ask the separate question of what clock stamped each entry.

Sallyport projects session and call journals from one encrypted, hash chained audit log, and sp audit verify can verify that chain offline over ciphertext. That check gives an investigator a way to test log continuity without exposing the stored secrets.

Integrity does not turn an audit timestamp into a globally perfect clock. A valid entry proves that the log contains the recorded event in its chain. It still needs a documented clock source, offset format, and uncertainty policy if you want to compare it closely against an external server.

Run verification against the evidence copy and save the command result with the case material. A useful record includes the command, the input artifact identifier, the verification outcome, and the person or automated job that performed it. Do not paste only a green status line into a ticket. The next investigator needs to reproduce the same check.

Hash chaining also changes how you handle gaps. If the chain says entries are missing or altered, do not quietly continue with a normalized timeline. Mark the affected interval as incomplete and look for independent server records. A missing audit segment may say more about the incident than the entries around it.

Build the investigation view from declared assumptions

A good investigation view shows raw timestamps, UTC conversions, source identity, and uncertainty. It does not hide conversion behind a dashboard label such as "event time." The reader should be able to see why the display orders two events.

Use this sequence when you assemble an action timeline:

1. Collect immutable source records and preserve their original timestamp strings, identifiers, and offsets.
2. Identify the event each timestamp marks: decision, approval, gateway receipt, server acceptance, completion, or result release.
3. Convert offset timestamps to UTC in a separate field, recording the parser or method used.
4. Estimate clock uncertainty for every source that can affect the disputed order.
5. Sort on normalized instants, then inspect overlapping uncertainty intervals and request IDs before writing causal claims.

Do not convert a local timestamp by attaching the investigator's own current offset. That mistake shifts historic events whenever the investigator works in another zone or when daylight saving rules differ. Parse the offset supplied with the record. If the record has no offset, mark it unresolved until you can establish a source zone and its applicable rules.

Teams should test this before an incident. Create an action near a daylight saving transition in a nonproduction environment. Change a client clock within a safe test boundary. Force a retry and a delayed server response. Then ask someone who did not build the logging to reconstruct the order. If they need verbal context to explain the evidence, the record format is incomplete.

Time is rarely the only evidence in an agent incident. Request identifiers, authorization scope, process identity, response bodies, and tamper checks often establish facts that clocks cannot. Keep those facts attached to their own events. The next disputed action will be easier to explain because you recorded what each system knew, rather than forcing every system to agree on one fictional timestamp.