How to diagnose the first break in a hash chained audit log

A hash-chained audit log gives you a hard boundary when it fails: there is a prefix you can verify, followed by material you cannot authenticate through that chain. That boundary is useful only if you report it precisely. Saying "the audit log is corrupted" throws away the part investigators, incident responders, and counsel need most.

Your report should identify the last valid record, the first mismatch, the exact byte offset in the acquired file, and the sessions that cross or follow the boundary. It should also separate what the chain proves from what you still need to investigate. A failed link is evidence of inconsistency. It is not a verdict on intent.

A failed link marks the end of proof, not the start of blame

A chained record normally includes its own content and a reference to the preceding record's hash. Verification recalculates the expected value from the record bytes and compares it with the value the record claims to follow. When that comparison fails, the verifier can still trust the fully verified prefix before the failure. It cannot extend that trust through the mismatch.

That sounds obvious until an incident report says, "record 842 was altered." Record 842 may be the first record that exposed the problem, but record 841 might have been modified. A record between them might be missing. A storage error may have damaged one byte in either record. The log might have been copied incompletely, leaving a damaged tail. The chain tells you that the relationship cannot be verified. It does not identify the mechanism.

Keep these terms separate:

• Last valid record: the latest complete record whose content, stored hash, and predecessor relationship verify against the trusted prefix.
• First mismatch: the first complete record whose claimed predecessor or calculated hash does not verify.
• Break boundary: the point between those two records, expressed as record identifiers and byte offsets.
• Unverified suffix: every later record that depends on the broken sequence, unless another independent checkpoint authenticates it.

This distinction is not paperwork. It determines how you scope access. The verified prefix can support a statement such as, "This agent session made this approved API call before the break." The suffix can support a lead such as, "This session identifier appears to have attempted an SSH action," but you should not present it as cryptographically authenticated by the damaged chain.

RFC 5848, the IETF specification for signed syslog messages, makes the same practical point in a different design. It treats message sequencing and missing-message detection as verification properties, and it warns that alteration or truncation can invalidate validation. The useful lesson is not to force an audit log into syslog format. It is to treat integrity verification as a property of exact serialized data, not as a vague assurance that a log "looks unchanged."

Preserve the bytes before you ask a parser to explain them

Start with acquisition. A perfect verifier cannot rescue an investigation that begins with an overwritten source file, a text editor rewrite, or a compressed export that silently changed line endings.

On macOS, create a working copy through a controlled path and record its size and digest before using any analysis tool that might rewrite metadata or content. Use commands appropriate to your evidence procedures, but the following minimal record gives a reviewer something concrete to reproduce:

mkdir -p case-2026-07-22
cp -p /path/to/audit-log.bin case-2026-07-22/audit-log.bin
stat -f '%z bytes  %N' case-2026-07-22/audit-log.bin
shasum -a 256 case-2026-07-22/audit-log.bin

The final command produces output in this shape:

9d5e...c41a  case-2026-07-22/audit-log.bin

Record the full digest, not an abbreviated display. Also record the source path, acquisition time with timezone, hostname or device identity under your evidence process, the operator, and whether you obtained the file while the application was still writing. If the log was live, acquire a second copy later. Two copies that fail at different offsets may reveal an incomplete write rather than a stable historical break.

Do not open the original in an editor. Do not run a formatter. Do not convert it from a binary export into JSON "for convenience." Do not move the only copy into a ticket attachment that may transform it. Make derivatives for each experiment and preserve the acquisition copy unchanged.

This is where teams often make a quiet but serious mistake: they call a parser failure "log corruption" after the parser has already normalized the evidence. A parser may decode escapes, reorder fields, discard unknown fields, replace invalid Unicode, or treat a missing newline as harmless. Those choices may be sensible for display. They are destructive for a chain that covers exact bytes.

For Sallyport, begin with sp audit verify against the preserved audit-log copy or the product's supported audit source. The audit chain can be verified offline over ciphertext, so the verification result does not require vault access. Capture the command exactly as run, its exit status, and its complete output in the case record. Do not invent a cleaner summary from a partial terminal screenshot.

The byte offset must point to the acquired file

A file offset is the number of bytes from the start of the acquired evidence file to a defined location. It gives another investigator a stable coordinate. A record number often does not.

Record numbers can change when a parser skips blank entries, a collector combines files, or an export drops a malformed line. Timestamps can collide, arrive out of order, or be absent. A byte offset lets a reviewer inspect the exact surrounding bytes using a different implementation.

Use two offsets when possible:

1. The offset where the last valid record begins.
2. The offset where the first mismatching record begins.

If the mismatch arises because a record points to the wrong predecessor value, report the first mismatching record's start offset. If the record itself is incomplete, report the start of the incomplete record and the end-of-file offset. Those are different cases.

On a line-oriented export, grep -b can locate a known identifier by byte position, but treat it as a convenience only after you know the identifier is encoded literally and uniquely. For binary or encrypted records, use a hex viewer that does not alter the file. A simple inspection command can establish the bytes around a known position:

xxd -g 1 -s 104832 -l 256 case-2026-07-22/audit-log.bin

The number after -s is the offset in bytes. The output begins with hexadecimal offsets, followed by byte values and an ASCII rendering when possible. Preserve that excerpt as an analysis artifact, but do not use the excerpt as a replacement for the full file.

Be explicit about what your offset means. "Offset 104832" is incomplete. Write: "First mismatching record begins at byte offset 104832 in SHA-256 digest 9d5e...c41a, measured from byte zero of the acquired file." If the log has a container header, say whether the offset includes it. It should, because another reviewer will be opening the file you acquired, not your parser's internal record stream.

A common reporting error is to give an offset after decompression. That may help a developer reproduce a parser issue, but it does not identify a location in the evidence file. Report both only if you label them clearly: one is the raw evidence offset, the other is a derivative-analysis offset.

Verify records in order and stop extending trust at the break

A verifier should process records in their stored order, calculate each record's expected chain value from the exact authenticated representation, and compare it with the stored chain reference. The first failure is the first point where the verifier cannot derive the claimed relationship from the verified prefix.

Do not scan ahead and select a later mismatch because it looks more dramatic. The earliest failure controls the scope of proof. Later failures may be downstream consequences of the first one, independent defects, or artifacts of a parser that is no longer synchronized.

The useful output shape looks like this, even if your verifier uses different field names:

verification_status: failed
last_valid_record: 841
last_valid_offset: 104576
first_mismatch_record: 842
first_mismatch_offset: 104832
failure_kind: predecessor_hash_mismatch
expected_predecessor: 6f4a...
observed_predecessor: c928...
affected_session_ids: sess-17, sess-21, sess-24

Those values are an example of a report structure, not output you should fabricate from a tool. Your actual report needs the record's stable identifier if one exists, the ordinal position in the file, and the byte offset. If identifiers are encrypted or unavailable to an offline tool, record the ordinal and offset first, then use an authorized analysis path to map the boundary to sessions.

Classify the failure accurately. These cases have different implications:

• Predecessor mismatch: the record's reference to the previous hash does not match the verified predecessor.
• Record hash mismatch: the record's own stored digest does not match the digest calculated from the authenticated bytes.
• Missing sequence: explicit sequence values or checkpoints show that one or more records are absent.
• Malformed record: the verifier cannot parse a complete record well enough to calculate the chain value.
• Truncated tail: the file ends before a final record is complete.

Do not collapse all five into "tampering." A predecessor mismatch in a complete record is different from a power loss during append. A malformed record may result from transport damage. Missing sequence evidence may prove that records are absent even when every remaining record hashes correctly.

NIST Special Publication 800-92 frames log management as more than storage: organizations need to configure sources, analyze logs, respond to events, retain data, and audit the log-management operation itself. That is the right operational frame for a chain failure. The verifier tells you where authenticity stops. You still need to examine collection, retention, endpoints, and response records to learn why.

A session can cross the boundary without appearing at it

Affected sessions are not simply the session IDs printed on the first bad record. A session may begin in the verified prefix, perform actions after the break, and never repeat its identifier in the damaged portion. Another may first appear in the unverified suffix but have a related process, approval, or credential use visible before the boundary.

Build the session scope from records on both sides of the break. For each session, classify it by its relationship to the boundary:

Do not rely on timestamps alone to decide whether a session crossed. Clocks drift, records can be buffered, and a log writer may flush in batches. Use session IDs, process identifiers where available, agent-run identifiers, action correlation IDs, and explicit start or end records. If the system does not expose all of those fields, state the limitation instead of pretending a time window solves it.

A practical worksheet keeps the reasoning reviewable:

Boundary: valid record 841 at offset 104576
          mismatch record 842 at offset 104832

Session sess-17
  first observed: record 809, verified
  last verified action: record 838
  later references: records 842-850, unverified
  classification: crossing

Session sess-21
  first observed: record 842, unverified
  supporting evidence: endpoint process journal reference
  classification: first seen at boundary

The worksheet should name the evidence source for each conclusion. "Supporting evidence" might be a session journal, endpoint process record, application event, remote API receipt, or SSH server log. It is not enough to write that a session was "probably active." Investigators need to know whether the conclusion comes from the chain, another log, or an operator statement.

Sallyport keeps agent runs in a Sessions journal and individual calls in an Activity journal, both projected from the same encrypted hash-chained audit log. That makes the chain boundary relevant to both views, but it does not let you treat a friendly journal display as a substitute for the verifier's finding. Use the journals to identify the sessions and calls that need review, then label the chain status of each claim.

The last complete record needs its own decision

A file that ends abruptly poses a different question from a record that fails a predecessor check. You need to decide whether the final bytes contain a complete record that is inconsistent, or an incomplete write that never formed a record.

Start at the end of the file and determine the record framing rules for that format. A line-delimited format may require a newline, but absence of a newline does not automatically mean the record is incomplete. A length-prefixed binary format lets you compare the declared length with the remaining bytes. An encrypted container may have an authenticated framing tag that can distinguish an invalid complete record from an unfinished append.

Report one of these findings, not a vague hybrid:

• "The final complete record verifies; the file ends with 73 bytes that do not form a complete record."
• "The final complete record begins at offset 104832 and fails predecessor verification."
• "The final record declares 512 bytes but only 301 bytes remain; content hash calculation was not possible."

That wording matters because the operational response differs. A tail truncated during a crash may require recovery of another copy, storage diagnostics, and comparison with a collector. A complete mismatching record requires the same comparison, but it also calls for immediate preservation of endpoint state and access records because the content was present and inconsistent.

Do not fill in a truncated record from memory, another export, or a similar entry. You may create a reconstructed derivative for troubleshooting, but the reconstructed file is not the acquired evidence. Keep the reconstruction name, method, and source bytes in the case notes.

A clean final record does not prove the file is complete either. A chain can verify perfectly after deletion of an entire terminal segment if the design lacks an external checkpoint, signed root, expected sequence count, or trusted retention record. Hash chaining detects edits within the inspected sequence. Completeness needs an anchor outside that sequence.

Compare independent copies before you call it tampering

The fastest way to overstate a chain break is to inspect one copy and assume it is the canonical log. Before you accuse an actor, obtain independent copies where your authority and incident process allow it.

Useful comparisons include a local application store, an exported archive, a backup snapshot, a collector copy, filesystem snapshots, and records from systems that received the audited action. Give each copy its own digest and acquisition details. Never overwrite one with another because they are "supposed to match."

Use the following decision pattern:

1. If two independently acquired copies fail at the same record and byte offset with identical preceding bytes, the issue likely existed before acquisition. That still does not establish intent.
2. If one copy verifies farther than another, compare the files byte for byte around the first divergence. The shorter or damaged copy may be incomplete.
3. If both copies verify internally but differ as whole files, investigate whether they are separate legitimate segments, exports with different scopes, or evidence of replacement.
4. If a supposedly identical acquisition changes digest between reads, stop the chain analysis and investigate the source medium, live writer, permissions, and collection process.

The popular but wrong recommendation is to "just restore the log from backup and rerun verification." A restored copy can help establish what another retained copy contains. It cannot repair the original evidence or explain whether the live file changed, disappeared, or was collected badly. Keep both. The difference is often the incident.

External action logs can narrow scope. If an agent invoked an HTTP API, request identifiers, provider-side audit entries, and resource changes may show whether the action occurred. If it used SSH, remote server authentication and command records may help. Those records do not restore cryptographic trust to the broken suffix, but they can establish facts independently and identify sessions that need containment.

Write the incident report so another investigator can reproduce it

A good report makes a narrow claim with enough detail to test it. It does not hide uncertainty under broad phrases such as "audit integrity compromised."

Use this report skeleton:

Artifact
  Evidence file: audit-log.bin
  SHA-256: <full digest>
  Size: <bytes>
  Source and acquisition reference: <case record>

Verification
  Tool and version: <verifier>
  Command: <exact command>
  Result: failed
  Last valid record: <stable ID and ordinal>
  Last valid record offset: <raw byte offset>
  First mismatch record: <stable ID and ordinal>
  First mismatch offset: <raw byte offset>
  Failure classification: <specific classification>

Scope
  Verified sessions: <identifiers>
  Crossing sessions: <identifiers>
  First seen at boundary: <identifiers>
  Suffix-only sessions: <identifiers>
  Related external evidence: <sources and references>

Limits
  The hash chain verifies the prefix through <record>.
  The chain does not establish the cause of the mismatch.
  Records after <offset> require independent corroboration.

Actions taken
  Evidence preserved: <references>
  Access or session revocations: <references>
  Copies compared: <references>
  Follow-up owner and deadline: <names or case roles>

Avoid reporting the last valid record as the "last safe action." It only means the chain can authenticate the record sequence through that point. The action itself may still have been harmful, unauthorized under a separate approval process, or later reversed. Likewise, avoid describing the unverified suffix as fake. It may be completely accurate. It is simply no longer proved by that chain.

The first action after a stable failure should be containment proportional to the affected sessions and credentials, not a debate over wording. Preserve the evidence, revoke live sessions that cross the boundary when the risk warrants it, rotate exposed credentials under your incident procedure, and compare independent records. Then write the boundary with the precision that lets the next investigator verify your work without trusting your memory.