# Can encrypted audit log verification catch deletions? A hash chain can catch a missing record in the middle of an audit file, but it cannot reliably tell you that someone chopped records off the end. That distinction is where teams overstate what their verifier proves, then discover during an incident that they preserved integrity evidence without preserving history. Encryption does not change that answer. It protects the contents of records. A chain over encrypted bytes can show that those bytes still connect as expected, even when nobody running the verifier can decrypt them. It cannot prove that the file contains every record that ever existed unless some evidence outside the file remembers a later point in its history. NIST describes a hash chain as an append-only structure where each block includes a hash of previous data, so altering a block changes the digest recorded by its successor. That is tamper evidence, not a time machine. ## Chain verification checks a relationship, not completeness A verifier checks whether each record points to the correct predecessor and whether each record digest matches its stored bytes. Given a trusted first record, it can establish continuity through every record that remains in front of it. Suppose a file contains records 1 through 100. Record 58 includes the digest of record 57, and record 59 includes the digest of record 58. If somebody deletes record 58 but leaves everything else alone, record 59 still points at a digest the verifier cannot find. Verification fails at the gap. Now remove records 91 through 100. Record 90 still points correctly to record 89. Nothing inside the shortened file says that record 90 was once followed by ten more records. A verifier that starts at record 1 and stops when the file ends can return success. It verified the history presented to it. It did not verify that the presented history was complete. People often collapse three separate claims into "the log is tamper-proof": - The surviving records have not changed. - No record was removed from the middle. - The file ends where the original history ended. Those claims need different evidence. A linear chain handles the first claim well and catches a simple version of the second. The third needs an independent later commitment, often called a checkpoint, head, seal, receipt, or witness. Certificate Transparency makes the same separation with a different data structure. RFC 9162 says a Merkle consistency proof can prove that a newer tree is append-only relative to an earlier advertised tree. The earlier advertised tree head is doing real work there. Without it, a current root tells you nothing about records that a log operator decided not to show you. ## Build a test log that treats payloads as opaque ciphertext Do not practice on a production audit file. Make a disposable fixture whose payloads look like ciphertext to the verifier, then test deletion against exact byte records. The following script writes a JSON Lines file. Each record carries a sequence number, an opaque base64 payload, the predecessor digest, and its own digest. The payload is deliberately random test data, not real encryption. That is enough for this test because the chain verifier only needs stable opaque bytes. In a real encrypted log, substitute the actual serialized ciphertext record and its authenticated metadata. ```python # make_log.py import base64 import hashlib import json import os OUT = "audit.jsonl" ZERO = "0" * 64 def canonical(record): return json.dumps(record, sort_keys=True, separators=(",", ":")).encode() def digest(record): return hashlib.sha256(canonical(record)).hexdigest() prev = ZERO with open(OUT, "w", encoding="utf-8") as f: for seq in range(1, 13): body = { "seq": seq, "ciphertext": base64.b64encode(os.urandom(24)).decode(), "prev": prev, } body["hash"] = digest(body) f.write(json.dumps(body, sort_keys=True) + "\n") prev = body["hash"] print(f"wrote 12 records to {OUT}") print(f"head seq=12 hash={prev}") ``` Use a separate verifier rather than trusting the writer to validate its own output. It should reject duplicate sequence numbers, unexpected sequence jumps, malformed records, wrong predecessor references, and wrong hashes. A verifier that only checks hashes has a blind spot because it can accept a reordered file if the attacker has also rearranged enough associated fields. ```python # verify_log.py import hashlib import json import sys ZERO = "0" * 64 def canonical(record): return json.dumps(record, sort_keys=True, separators=(",", ":")).encode() def digest_without_hash(record): copy = dict(record) supplied = copy.pop("hash", None) return supplied, hashlib.sha256(canonical(copy)).hexdigest() def fail(message): print(f"FAIL {message}") raise SystemExit(1) path = sys.argv[1] prev = ZERO expected_seq = 1 last_hash = ZERO with open(path, encoding="utf-8") as f: for line_no, line in enumerate(f, start=1): try: record = json.loads(line) except json.JSONDecodeError: fail(f"line={line_no} invalid JSON") if record.get("seq") != expected_seq: fail(f"line={line_no} expected_seq={expected_seq} got={record.get('seq')}") if record.get("prev") != prev: fail(f"line={line_no} seq={expected_seq} predecessor mismatch") supplied, calculated = digest_without_hash(record) if supplied != calculated: fail(f"line={line_no} seq={expected_seq} digest mismatch") prev = supplied last_hash = supplied expected_seq += 1 print(f"OK records={expected_seq - 1} head={last_hash}") ``` Run the fixture and retain the reported head outside the log file: ```text python3 make_log.py python3 verify_log.py audit.jsonl ``` The success shape should look like this, with a different digest on every run: ```text wrote 12 records to audit.jsonl head seq=12 hash=8f...c2 OK records=12 head=8f...c2 ``` That final `records=12` and `head=...` pair is your checkpoint. Put it in a separate test-notes file before altering copies. If you leave the checkpoint only in the file you plan to attack, you have built a convenient record of what the attacker can edit. ## Removing the tail shows the limit immediately Tail deletion is the test that should make everyone cautious about the word "complete." Copy the file, remove the final three lines, and run the same verifier. ```text cp audit.jsonl tail-cut.jsonl head -n 9 audit.jsonl > tail-cut.jsonl python3 verify_log.py tail-cut.jsonl ``` The verifier will report success for nine records. It should do that. Records 1 through 9 still form a valid chain. Calling this a verifier failure would encourage a dangerous design where valid partial histories look corrupt. Compare its result to the checkpoint instead: ```text OK records=9 head=4a...91 expected records=12 head=8f...c2 ``` Now you have evidence of truncation. The proof comes from disagreement with evidence captured when the log was longer, not from the internal links of the shortened file. A sequence number helps human diagnosis. It does not create security by itself. An attacker who can rewrite the tail can change the final sequence number from 12 to 9. The sequence number becomes useful when the expected value came from a source the attacker cannot rewrite, such as a signed checkpoint stored by a separate service, a protected release artifact, or an export that reached a different administrative domain. This is the test many audit designs skip because it appears too obvious. It is also the failure that matters after a destructive action. A malicious process does not need to make history internally inconsistent. It only needs to make the history stop early enough to hide the action. ## A middle deletion should fail, but only under stated conditions Make another copy and delete a record that has both a predecessor and successor. Record 6 is a good target. ```text cp audit.jsonl middle-cut.jsonl sed '6d' audit.jsonl > middle-cut.jsonl python3 verify_log.py middle-cut.jsonl ``` The result should have this shape: ```text FAIL line=6 expected_seq=6 got=7 ``` If you removed the sequence check, the verifier would fail one test later because record 7 carries the digest of record 6 while the verifier has record 5's digest in hand. Keep both checks. The sequence mismatch makes the problem obvious to an operator; the predecessor mismatch shows the cryptographic relationship that broke. Do not make the stronger claim that a hash chain always detects middle deletion. It detects a simple deletion when the attacker cannot rewrite the remaining chain. A public digest algorithm lets anybody calculate new hashes. If somebody can delete record 6, alter record 7's predecessor field, recalculate record 7's hash, then repeat that work through record 12, the rebuilt file can verify against its own new head. That result is not a collision attack or a break in SHA-256. It is ordinary recomputation. The system accepted a new history because it had no protected evidence that the old history existed. NIST's guidance on digital evidence makes the operational point plainly: a stored hash must live where people with access to the evidence cannot alter or overwrite it. The same report names hash chains as useful for securing hashes, but the external storage of the reference value remains necessary. A middle deletion test therefore has two versions: 1. Delete one line and leave later bytes untouched. The chain must fail. 2. Delete one line and regenerate every later digest. The rebuilt chain must fail against an independently retained checkpoint. If your test plan runs only the first version, it tests file damage and careless tampering. It does not test an attacker who can write the log storage. ## Encryption and chain integrity answer different questions Encrypted records create a useful division of labor. Operators can verify the chain over ciphertext without reading secrets, request bodies, command outputs, or other sensitive audit content. Investigators with appropriate access can later decrypt the relevant records. That is a sensible design when audit data is itself sensitive. But encrypted bytes do not certify their own origin. A ciphertext record can be deleted. A new ciphertext record can be added by someone who controls the encryption and logging path. If the encryption mode does not authenticate data, a ciphertext can sometimes be altered into garbage without the decryptor knowing why. Use authenticated encryption for record confidentiality and integrity, then use the chain to bind records into an ordered history. Keep these checks separate when you write the design and the incident report: - Record authentication asks whether one encrypted record was altered. - Chain verification asks whether each surviving record follows the claimed predecessor. - A checkpoint comparison asks whether the observed history reaches a previously observed head. - Event capture asks whether the system wrote the action to the log at all. The last question is the awkward one. No cryptographic log can prove an event was recorded if a compromised writer chose not to emit a record. You can narrow that exposure by collecting evidence from independent boundaries, such as a network service, a host audit facility, or a human approval event. You cannot make the gap disappear by calling the local log append-only. Sallyport's audit model is useful in this context because its agent-run and individual-call journals come from one encrypted, hash-chained audit log, and `sp audit verify` can verify the ciphertext chain offline without a vault key. That lets a reviewer test continuity without exposing the credentials or action data that the audit trail protects. ## A checkpoint must be hard to rewrite, not merely copied A checkpoint is a statement about a log at a point in time. At minimum, include a log identifier, final sequence number, final digest, creation time, and format or algorithm version. Treat it as a separate artifact with its own custody. This small format is enough for a test harness: ```json { "log_id": "agent-actions-test-a", "sequence": 12, "head": "8f...c2", "created_at": "2026-07-22T14:30:00Z", "format": "jsonl-chain-v1" } ``` Do not trust the `created_at` field by itself. A local clock is useful for sorting and investigation, but an attacker who controls the machine may control that clock and the file that contains it. The checkpoint gets its force from where it went and who can change it. A practical arrangement has one party write audit records and another party retain periodic heads. That second party does not need decryption access. It needs enough information to reject a later file whose final digest and sequence do not match what it saw earlier. For a developer tool, the independent party can be modest: a protected CI artifact, a release-attestation system, a separate collector account, or a daily export approved by a different administrator. For higher-risk actions, emit the checkpoint immediately after the action and retain it outside the workstation. The cadence should follow the damage window you can accept. A checkpoint every day cannot establish completeness for the hours between yesterday's head and today's missing tail. Signing a checkpoint helps when verifiers need to know who issued it. A signature binds the checkpoint data to the signing authority, but it does not make the signer honest. If the same compromised process writes the audit log and signs replacement checkpoints, you still have one trust boundary. Put the witness somewhere the original writer cannot quietly command. ## Write-blind storage changes the attacker you are testing A chain stored in a place where the writer can freely edit old records has a different threat model from a chain stored through a write-blind append path. The first design needs external witnesses to catch history rewrites. The second tries to prevent the writer from performing those rewrites in the first place. Be precise about the word "write-blind." It should mean the component that submits a new record cannot read or mutate prior encrypted log records at will. It does not mean the disk cannot fail, an administrator cannot delete a file, or the operating system cannot be compromised. It narrows one attack path: rewriting a selected part of history after seeing it. That distinction affects your test cases. For ordinary file access, test deletion plus rehashing because an attacker can read the data and calculate hashes. For a write-blind store, test whether a caller can request deletion, replacement, rollback, or a second log under the same identity. Also test how the verifier identifies the intended log. A perfect chain from the wrong file is still the wrong evidence. The most common mistake here is treating storage permissions as a cryptographic anchor. Permissions can be changed by a sufficiently privileged attacker. They still matter because they reduce casual damage and limit which processes can act, but they do not replace a checkpoint that crossed a boundary the attacker did not control. ## Test rollback separately from truncation Rollback looks like tail truncation with a plausible older file. A machine may restore yesterday's valid audit file after a crash, a backup restore, or a deliberate replacement. The restored file can pass its internal verifier because it really was valid yesterday. Your checkpoint catches this if it records a newer head. Compare the candidate file to the newest retained checkpoint, not the oldest one you happen to find. A chain verifier cannot infer which valid version should be current. Test it directly: 1. Save `audit.jsonl` and its 12-record checkpoint. 2. Generate another fixture file with more records and retain its newer checkpoint. 3. Replace the newer file with the 12-record copy. 4. Verify the restored file internally, then compare it to the newer checkpoint. Expect two different outcomes. Internal verification should succeed. The checkpoint comparison should fail because the file ends at an older sequence and digest. If both checks fail, your fixture may have changed bytes rather than reproducing rollback. If both checks succeed, you are comparing against the wrong checkpoint or you did not retain one. Do not "fix" a rollback by appending fresh records to the older file and carrying on. Preserve the state first. Once you append, you have mixed potentially missing history with post-incident activity. Start a new log segment with a recorded incident boundary, or follow the retention and recovery procedure your organization has already approved. ## Make verifier results useful during an investigation A verifier that returns only `invalid` creates unnecessary work. It should identify the first failed condition without printing sensitive record payloads. Sequence number, byte offset or line number, expected predecessor digest, observed predecessor digest, and calculated record digest usually give enough detail. Avoid logging decrypted content in an error message. A verification tool often runs in CI, a support bundle, or a terminal capture. If an encrypted audit design leaks cleartext whenever verification fails, it turns an integrity incident into a data exposure. A disciplined response to a failed verification has a short order of operations: - Preserve the original file and record a digest of that preserved copy. - Collect the newest checkpoint, prior checkpoints, and any external exports. - Run verification against copies and save the exact command output. - Compare the reported head to every retained checkpoint. - Determine whether the failure is corruption, a middle gap, tail loss, rollback, or a rewritten chain that conflicts with a witness. The classification matters. A middle-gap failure says the file contains an internal contradiction. A valid but shorter chain says the file may be complete only up to its final record. A valid chain that disagrees with a later checkpoint says you have evidence of rollback, truncation, or substitution. A rewritten chain can look internally clean, but it will disagree with an older independent head. Do not promise more than the evidence supports. "The audit chain verified through sequence 90 and does not match the retained sequence 100 checkpoint" is strong and specific. "Nobody changed the log" is not. ## The test that counts is the one against your actual boundary Run the disposable line-deletion tests first, because they teach the mechanics. Then run the same cases against the real audit export and its real verifier, using copies and the format's documented manipulation boundaries. Do not edit a production encrypted log in place just to see what happens. For Sallyport, preserve the original encrypted log and use `sp audit verify` on a copy before and after each controlled alteration. Record whether the verifier detects an internal gap, whether a shortened copy still verifies, and whether your retained head exposes the shorter history. The answer to all three is more useful than a vague claim that the log is tamper-evident. A valid chain means the records you checked agree with one another. A valid chain plus a trusted later checkpoint means the records reach a known point in history. Build and test for the second statement when missing actions would matter.