# Can agent audit timestamps survive clock changes? Wall clock time is evidence, not a sorting guarantee. An agent can make two perfectly legitimate calls while the host clock moves backward, repeat an hour during daylight saving time, or gets corrected after waking from a long sleep. If your audit viewer sorts those calls only by displayed time, it can tell a convincing but false story. Use a sequence number to answer "which record did this journal accept first?" Use a wall clock timestamp to answer "what calendar time did the recorder report?" Those are different questions. Teams get into trouble when they pretend one field answers both. For autonomous agents, that distinction has teeth. A reviewer may need to establish whether an SSH command followed an approval, whether an HTTP call was retried, or whether a revoke happened before the next action. The answer must survive a bad laptop clock and an awkward time change, not merely look tidy in a table. ## Timestamps cannot establish a reliable event order A wall clock timestamp cannot prove that one event happened before another when the clock can change. It can only report the clock reading observed when software wrote the record. Consider this sequence from a machine that was ten minutes fast until time synchronization corrected it: ```text seq 841 2026-11-03T14:10:12.481Z agent requested deploy status seq 842 2026-11-03T14:00:13.107Z agent requested deploy status seq 843 2026-11-03T14:00:14.052Z approval recorded ``` A timestamp sort places 842 and 843 before 841. The journal accepted 841 first. Neither view is a typo. They answer different questions. This matters even when timestamps increase. A clock can run slowly, jump forward, or be corrected by a user. Two records with increasing calendar time may still be farther apart, or closer together, than their values imply. A timestamp gives an observed coordinate on a civil time scale. It does not supply an unbroken duration measurement. RFC 3339 makes the storage side clear: timestamps need an offset when local time is involved, and UTC expressed with `Z` removes offset ambiguity for interchange. That is good practice, but RFC 3339 does not turn the host's clock into a witness that cannot be mistaken. It defines notation, not truth. Give every append-only audit stream a monotonically increasing `seq`. Assign it in the component that serializes writes, not in each agent process. If two agent processes can both allocate numbers locally and later upload their records, you have created two orders and called them one. The sequence number has a narrow but useful claim: - Within one journal, lower `seq` entered the journal first. - Gaps mean an investigator must account for missing, withheld, or intentionally excluded records. - A duplicate means the writer, importer, or storage layer failed. - The value says nothing by itself about an event on another machine. That last point gets ignored because a global-looking integer feels authoritative. It is not. A sequence number needs a namespace. `journal_id=macbook-17, seq=841` is a statement with boundaries. `seq=841` in a spreadsheet is an invitation to overreach. ## Daylight saving time repeats local clock readings Daylight saving time makes a local clock repeat an hour in many jurisdictions. During the autumn transition, 01:15 occurs once with one UTC offset and again with another. A log that stores only `2026-11-01 01:15:00` has discarded the information needed to distinguish them. Do not repair this later by guessing which occurrence an operator meant. Capture enough context when the event is written: ```json { "journal_id": "build-mac-07", "seq": 842, "event_id": "01JXYZ...", "recorded_at_utc": "2026-11-01T08:15:00.000Z", "local_offset": "-07:00", "time_zone": "America/Los_Angeles", "event_type": "agent.http.requested" } ``` At the later occurrence, the same local clock display might carry `local_offset: "-08:00"` and a UTC value one hour later. The numeric offset preserves the fact you observed. The time zone name helps a person explain why the offset existed, but it should not replace the offset in the record. Time zone rules change. Governments have changed start dates, end dates, and the existence of daylight saving time with little concern for your log parser. If you store a local date and a zone name, then recalculate an offset years later using a newer time zone database, you can render historical evidence differently than the machine did at the time. Preserve the original UTC value and captured offset. Render with current rules only when the viewer clearly labels the result as a present-day conversion. The spring transition has a different failure. Some local times never occur. A system that accepts a human-entered local deadline of 02:30 on a skipped hour must reject it or ask for an explicit resolution. Silently moving it to 03:30 is a product decision disguised as calendar arithmetic. For audit records, UTC should be canonical. Display local time when it helps the reader, but show the offset in the same field. `2026-11-01 01:15:00 -08:00` is clumsier than `01:15`, and it is also evidence. ## A manual correction must create a new record Manual edits are where an audit trail either keeps its value or turns into a polished activity feed. If an administrator fixes a timestamp in place, the original observation disappears. An investigator can no longer tell whether the old clock was wrong, the event payload was wrong, or someone wanted the history to look different. Keep the original record immutable. Add a correction record that identifies the earlier event, states the field under dispute, records the proposed corrected value, and explains the basis for it. The correction does not erase the first record. It adds a later fact: someone made and justified a claim about the first one. A correction payload can be small and still do the job: ```json { "seq": 913, "event_type": "audit.timestamp_corrected", "corrects_event_id": "01JXYZ...", "original_recorded_at_utc": "2026-11-03T14:10:12.481Z", "asserted_occurred_at_utc": "2026-11-03T14:00:12.481Z", "basis": "host time service report and neighboring journal entries", "actor": "admin-identifier" } ``` Use `asserted_occurred_at_utc` only when you have evidence for the revised time. Do not relabel the original `recorded_at_utc`; it describes the recorder's clock reading and remains historically correct even when that reading was inaccurate. Separate these three ideas in both schema and language: 1. `occurred_at` is the time an action took place, if the actor can establish it. 2. `observed_at` is the time a particular collector saw the action. 3. `recorded_at` is the time the audit writer committed its entry. They can be equal. They often are. They do not deserve the same field name. The popular recommendation to "just correct bad data" comes from reporting systems, where a wrong number should disappear from the dashboard. Audit systems have another job. They preserve the path from observation to conclusion. A visible correction is less convenient for casual readers, but it stops the system from laundering uncertainty into certainty. ## Network time corrections can move the wall clock Network time synchronization improves clock accuracy, but the correction itself can make timestamps surprising. A client may gradually adjust its rate, or it may step the clock when the offset is large enough or policy allows it. A user setting the clock by hand, a virtual machine resuming, and a firmware clock with a bad value can produce the same visible result: calendar time changes between two audit entries. The POSIX `clock_gettime` documentation distinguishes a real-time clock from a monotonic clock. `CLOCK_REALTIME` tracks calendar time and can be set. `CLOCK_MONOTONIC` has no useful calendar origin but is not set by `clock_settime`; it is the right kind of source for measuring an interval inside one running system. That distinction gives you a practical data rule. Record wall clock time for the investigator. Record a monotonic sample when you need to reason about elapsed time, timeout behavior, or ordering around a correction. Do not serialize a monotonic value as though it were a date. Its absolute value has meaning only relative to its boot and clock domain. RFC 8633, the IETF guidance for NTP operations, explicitly discusses large time shifts and says operators should not blindly bypass the NTP panic threshold on cold starts. The usual temptation is to treat every correction as harmless housekeeping. It is not harmless when time controls token validity, retention, incident reconstruction, or replay detection. Your audit system should report time anomalies as audit facts. It should not attempt to hide them with a sort rule. A useful event might contain the last wall clock value, the new value, the estimated delta, the source of the change if known, and the process that detected it. If the operating system does not expose a cause, say that. A false explanation is worse than a missing one. Keep a small tolerance for ordinary jitter. Do not create a dramatic clock-change event because adjacent values differ by a few milliseconds in an unexpected direction across concurrent writers. The serialized `seq` supplies order. Flag a discontinuity when the wall clock moves backward beyond the precision you claim, or when a forward gap conflicts with expected activity and needs review. ## Sequence numbers need a defined scope A sequence number works only inside the log that assigned it. Treating it as global order after records leave that log produces bad conclusions in distributed agent systems. Suppose a coding agent on one Mac requests an API action, and a remote build host writes an SSH action. Each host has its own journal: ```text build-mac-07 seq 842 14:00:13Z HTTP action requested build-host-2 seq 91 14:00:14Z SSH action accepted build-mac-07 seq 843 14:00:15Z HTTP result received ``` You can say that 842 precedes 843 in `build-mac-07`. You can say that 91 was recorded by `build-host-2` at its reported time. You cannot prove, from those fields alone, whether the remote host accepted the SSH action before or after the first HTTP request in real time. To establish cross-system relationships, add an explicit link. A request identifier propagated from caller to receiver can connect the request event to the receiving event. A signed receipt from the receiver can add stronger evidence. A central collector can allocate a collector sequence when it receives records, but that sequence proves arrival order at the collector, not occurrence order at the sources. Do not overstate any of these: - A request ID proves correlation when both sides preserve it. It does not prove delivery time. - A central ingest sequence proves ingest order. Network delay can reorder arrival. - A synchronized clock narrows uncertainty. It does not remove it. - A distributed logical clock can express causality if every participant carries it correctly. It does not give wall clock time. For many agent audit trails, you do not need a grand distributed ordering system. You need honest boundaries. Keep a local sequence for each trusted journal, include correlation IDs at action boundaries, and expose the source journal in every export. That lets a reviewer see where the evidence is strong and where it becomes an inference. ## Store enough time evidence to explain a disagreement A bare `timestamp` field is not an audit schema. It is a display preference that escaped into storage. Use a record shape that keeps order, calendar time, source identity, and integrity material separate. The exact field names are yours, but the concepts should survive export and retention: ```json { "journal_id": "build-mac-07", "seq": 842, "event_id": "01JXYZ...", "recorded_at_utc": "2026-11-03T14:00:13.107Z", "recorded_offset": "-08:00", "time_zone": "America/Los_Angeles", "monotonic_ns": 3982188001123, "boot_id": "boot-identifier", "event_type": "agent.http.requested", "correlation_id": "request-identifier", "actor_process": "process-identifier", "previous_hash": "hex-value", "record_hash": "hex-value" } ``` `boot_id` prevents a common mistake with monotonic values. A monotonic counter may restart after reboot, so `3982188001123` from one boot cannot be compared to the same value from another without more context. Store it only if you will use it, and document its unit. A field named `monotonic_time` that leaves readers guessing whether it means nanoseconds, milliseconds, or a platform tick count wastes the field. `recorded_offset` is the offset in force when the writer recorded the event. It is not a substitute for UTC. It lets a human see the source's local-time context and lets a formatter preserve the original civil interpretation. Include a named time zone only if the platform can provide it reliably. A fixed offset is enough for ordering and reconstruction. The hash fields need equally clear rules. Calculate a record hash over a canonical representation of every field whose alteration would change the meaning of the record, including `seq`, timestamps, event type, actor identity, and payload digest. Do not hash a pretty-printed JSON blob if different serializers may change whitespace, property order, or number formatting. Canonicalize first. A hash chain detects a changed record when verification starts from a trusted anchor and each record commits to the prior record. It does not prove that the source clock was accurate. It does not prove an event happened outside the machine. It proves a narrower and still important claim: the verified history has retained the cryptographic relationships the writer produced. That is why a chain and a sequence belong together. The sequence gives local order. The chain makes a later rewrite visible. The timestamp adds calendar context. None can do the others' job. ## Investigate time reversals without inventing a story When a later sequence has an earlier timestamp, start with the evidence you already have. Do not begin by calling it a replay, an agent bug, or tampering. Use this short investigation sequence: 1. Verify the journal's sequence continuity and integrity result before interpreting time values. 2. Compare the earlier and later UTC values, local offsets, boot identifiers, and any monotonic samples. 3. Check whether the host crossed a daylight saving boundary, restarted, resumed, or recorded a time-service correction. 4. Follow correlation IDs into adjacent journals, but label cross-host timing as an estimate unless a receipt or shared ordering mechanism proves it. 5. Add an annotation record if evidence supports a correction or a time anomaly finding. Here is a failure that turns up in real reviews. An agent requests an API action at `09:02:04`, the machine wakes, network time pulls the wall clock back four minutes, and the action result is recorded at `08:58:07`. A dashboard sorts chronologically and shows the result before the request. An operator concludes the result was replayed, blocks the agent, and starts rotating credentials. The sequence and monotonic values tell a plainer story. The request is sequence 117. The result is sequence 118. Both have the same boot identifier. The monotonic counter advances by about three seconds. The wall clock changed between entries. The result followed the request in that journal, and the calendar display is inaccurate across the correction. That conclusion still leaves useful questions. Why did the clock differ by four minutes? Did the action depend on a credential with a time-bound expiry? Did another system receive the request before the correction? The investigation should answer those questions separately. It should not try to force a clean timestamp order because a viewer prefers one. If an integrity check fails, stop treating the journal as a settled timeline. Preserve the export, record the verification failure outside the affected journal if possible, and obtain a fresh copy through an independent path. A failed chain does not identify the person or process that changed data. It tells you the evidence no longer supports an unmodified-history claim. ## A tamper-evident trail needs offline verification An audit log has little forensic value if its only proof of integrity depends on the live service that produced it. A reviewer needs to export the records, carry them to another machine, and verify the chain without using the vault or asking the agent that performed the actions. Sallyport projects its Sessions and Activity journals from one encrypted, hash-chained audit log, and `sp audit verify` can verify the chain offline over ciphertext without a vault key. That design addresses a different problem from clock accuracy. Offline verification tells you whether the encrypted history remains internally consistent under the audit design. It does not certify the host's wall clock. Keep the two claims separate in reports: "the audit chain verified" and "the event time was corroborated by an independent source" are both useful, and neither implies the other. Export procedures should preserve journal identity, sequence range, verification result, and the software version that performed verification. A CSV with just timestamp and action text is a report, not audit evidence. It loses the fields that let someone challenge or confirm the report later. ## Build the viewer around disagreement, not just happy-path sorting A good audit viewer does not conceal clock anomalies. It sorts by journal sequence by default within one journal, shows UTC alongside local time when the user expands a row, and marks a timestamp reversal as a time discontinuity rather than rearranging history. Give readers a choice of views, but name each one precisely. "Journal order" means sequence order. "Reported calendar time" means timestamp order and may place causally later records first. "Collector arrival order" means the order another service received data. Avoid a generic "time" sort because it makes a material forensic choice invisible. The interface should also show the scope of its certainty. When events come from different journals, group them by source or show a clear source label beside every sequence number. If a request ID connects records, show that relationship as a link in the data model and the interface. Do not draw a continuous timeline line across hosts unless your system can defend it. The first action is simple: inspect one of your existing exports for a local timestamp with no offset, a sequence number with no journal identity, or an in-place edit path. Any one of those is enough to create a misleading incident timeline. Fixing it now is cheaper than explaining it after an agent action has become evidence.