# Audit format upgrades that preserve old evidence An audit format upgrade is safe only when an investigator can verify yesterday's exported evidence with the rules that applied yesterday. A new parser that produces a plausible screen, a successful database migration, and a green deployment are not proof of that. They often conceal the exact break that matters: a record whose bytes remain intact but whose meaning, hash input, or verifier behavior has changed. Treat the audit format as a protocol with a long tail. Your application can change every week. Evidence cannot. Once someone relies on a record to explain who approved an action, which credential was used, or what an agent sent to an external service, that record needs a stable interpretation long after the code that wrote it has disappeared. ## Preserve bytes before you preserve convenience The authoritative artifact is the original record byte sequence plus the context needed to verify it. A parsed row in a current database is a working copy. A JSON object rendered in a dashboard is a view. Neither is a substitute for the evidence bytes that participated in a signature, hash chain, or authenticated envelope. This distinction sounds fussy until an upgrade rewrites a field. Suppose version 1 stored an SSH target as a user supplied string: ```json {"schema_version":1,"event":"ssh.execute","target":"build@prod.example:22","command":"uptime"} ``` Version 2 wants separate fields so it can filter by host and port: ```json {"schema_version":2,"event":"ssh.execute","user":"build","host":"prod.example","port":22,"command":"uptime"} ``` Those two records can describe the same action, but they are not interchangeable evidence. A v2 encoder may normalize a hostname, insert a default port, or reject a target that v1 accepted. If you overwrite the first record with the second, you have made a new claim about the old event. You may have made a correct claim, but you cannot prove that merely by pointing at the rewritten data. Keep three things separate: - **Original evidence**: immutable bytes exactly as accepted into the audit sequence. - **Derived representation**: an indexed, decoded, or migrated form used for search and display. - **Interpretation notes**: documented rules that explain fields, defaults, and version-specific behavior. A derived representation can be rebuilt. Original evidence cannot. Store originals in an append-only evidence bundle or object store, address them by digest, and make every derived item point back to that digest. If retention rules require deletion, record that deletion as its own event. Do not quietly compact history and call the result an upgrade. This applies even when the log uses encrypted records. Encryption protects content from unauthorized readers; it does not make a lossy migration harmless. The verifier must still identify which ciphertext, header, and chain rule produced the result. ## Give every record an explicit version Put an explicit `schema_version` inside every record before hashing or signing it. Do not infer version from a file extension, a database migration number, an application release, or the presence of a recently added field. Inference works right up to the first forensic export. An investigator gets a folder of records copied from a backup, a support ticket, or a machine that no longer runs the current app. The surrounding context is incomplete, while the record is still present. A record that carries its own version tells the verifier which decoder and rules it needs. Use a small integer for a format whose rules you own. Reserve version 0 as invalid so a missing field cannot quietly become a legacy format. The version needs a narrow job: select the record grammar and the verification recipe. Do not turn it into a general product release identifier. A durable envelope can look like this: ```json { "schema_version": 3, "record_id": "01J8X7K5W3H0Q9M6P2R4A1C8ZD", "recorded_at": "2026-07-22T14:08:31.482Z", "kind": "http.request.completed", "previous_digest": "sha256:4a4d...", "payload": { "method": "POST", "authority": "api.example.test", "status": 201 } } ``` The version belongs in the authenticated input. If a record says version 3 but the digest was computed without that field, an attacker who can edit stored bytes may be able to redirect the verifier into a different interpretation path. Include every field that chooses a parser, hash algorithm, canonicalization rule, or signature algorithm in the protected bytes. Also separate **record schema version** from **event meaning version**. The first answers, "How do I parse and verify these bytes?" The second answers, "What did this event mean when it was emitted?" For example, changing `actor` from a free form display name to a stable process identity may preserve the JSON shape while changing the claim. That is not merely schema version 4. It is a semantic change, and an investigator needs language in the evidence specification that says when the new meaning begins. The same warning applies when a field changes units, a timestamp changes from local time to UTC, or a status changes from an observed response to a policy decision. ## Freeze the verification recipe, not only the fields A versioned schema is incomplete if the verifier cannot reconstruct the exact byte recipe used to authenticate a record. The field layout is only one part of that recipe. Write down, per version: - the accepted record grammar and required fields; - the text or binary encoding and canonicalization rules; - the digest and signature algorithms; - the domain separator, if you use one; - the chain linking rule and genesis rule; - the failure behavior for malformed or unknown input. RFC 8785 explains why JSON needs a deterministic representation before cryptographic operations: ordinary JSON permits several serializations of the same logical value, while hashing and signing require invariant bytes. Its JSON Canonicalization Scheme constrains input and sorts object properties deterministically. It also warns, through its format restrictions, that duplicate property names and numbers outside the supported representation are not harmless details. That standard is useful only if you name the exact profile. Saying "we hash JSON" is not a recipe. Saying "we call the serializer from the current runtime" is worse, because a runtime upgrade can change escaping, numeric formatting, or ordering without any deliberate audit change. The same issue appears with binary formats. RFC 8949 defines deterministic encoding requirements for CBOR and notes that a prior ordering convention needs an explicitly named compatibility mode. A verifier cannot safely assume that every historical producer used the same meaning of "canonical." Do not build a new verifier that parses any JSON, reserializes it using today's library, then hashes the result. That pattern breaks old evidence in two ways. It can reject records that were valid under the older recipe, and it can accept a record under a new recipe that never produced the original digest. Instead, keep version dispatch close to the byte boundary: ```text read envelope bytes -> identify protected schema_version -> select verifier V1, V2, or V3 -> validate that version's grammar -> reproduce that version's authenticated bytes -> verify digest, signature, and chain link -> decode a display model only after verification ``` The display model comes last on purpose. A renderer is allowed to be pleasant. A verifier is not allowed to be imaginative. ## Unknown versions must fail closed A verifier facing a version it does not support must return an explicit `unsupported_version` result. It must not treat unknown fields as ignorable, assume the latest layout, or run a generic fallback decoder. Engineers often resist this because they want forward compatibility. Forward compatibility is appropriate for an application reading optional presentation fields. It is dangerous for evidence verification, where a field that appears optional may later control the authenticated interpretation. Use a result shape that separates evidence failure from tool limitations: ```json { "record_id": "01J8X7K5W3H0Q9M6P2R4A1C8ZD", "schema_version": 4, "status": "unsupported_version", "verified": false, "supported_versions": [1, 2, 3], "reason": "Verifier 2.7.0 has no verification recipe for schema version 4" } ``` That result says something precise: the tool has not established authenticity. It does not accuse the record of tampering, and it does not pretend the record is valid. Keep `invalid`, `incomplete`, `unsupported_version`, and `verified` distinct in both command output and APIs. A hash chain adds another compatibility requirement. The chain's history cannot be assumed from a single final digest. A verifier needs the version-specific rules for the first record, record ordering, parent digest encoding, and any checkpoint format. Certificate Transparency offers the right mental model here: RFC 9162 defines consistency proofs that show an earlier tree is the same prefix of a later tree, rather than asking auditors to trust a newly reported root hash. Your audit sequence may not use a Merkle tree, but the lesson holds. When chain rules change, prove continuity at the boundary. Create a terminal v1 checkpoint containing its final verified digest, record count, and version. Make the first v2 record authenticate that checkpoint in a defined field. The v2 verifier should verify both sides with their own rules before it reports one continuous history. Never join two histories by storing an old digest as a comment or a display field. The join needs to be part of the protected input. ## Migrations should produce derivatives, never replacements A good migration creates a new, reproducible derivative beside the source evidence. It records enough provenance that another person can recreate the same output and compare it against the source. For every migrated record or batch, capture: ```json { "source_digest": "sha256:4a4d...", "source_schema_version": 1, "migration_id": "audit-v1-to-v2", "migration_build": "2.7.0+e31c9f4", "output_digest": "sha256:77c8...", "migrated_at": "2026-07-22T14:12:09Z" } ``` The `migrated_at` time describes the derivative, not the original event. Do not overwrite `recorded_at` or present a generated v2 record as if it were emitted by the old system. That mistake has caused more confusion in internal investigations than an obvious parser crash ever will. Some migrations cannot be lossless. A v1 record might have a single `target` string, while v2 requires a structured URI. If parsing fails or contains ambiguity, preserve the source string and record an explicit migration status. Do not invent a structured value just because your new index wants one. For instance: ```json { "source_digest": "sha256:4a4d...", "migration_status": "partial", "derived": { "target_raw": "build@prod.example:22", "host": "prod.example", "port": 22 }, "unresolved": ["user"] } ``` That may look less tidy than a fully populated row. It is more honest. A future investigator can see both what the older record said and what the migration inferred. Avoid the popular recommendation to run every old entry through the current writer and call that an upgrade. It is popular because it simplifies one code path and makes reports uniform. It is wrong for evidence because writers usually apply current defaults, omit deprecated fields, and normalize values. The resulting output can be useful for search, but it is a translation, not original testimony. ## A permanent fixture corpus catches the quiet breaks Compatibility is a test asset, not a promise in a release note. Build an evidence corpus for every released schema version and run it against every verifier build that claims support for that version. The corpus needs more than a handful of happy path records. Keep byte exact fixtures for these cases: - a normal valid record and a valid multi-record chain; - boundary timestamps, Unicode text, empty optional values, and numeric limits accepted by that version; - a record with a changed payload byte; - a record with a changed parent digest or reordered sequence; - malformed, duplicate, truncated, and unknown-version input. Store expected outcomes, not only expected decoded objects. The test should assert the evidence result and the diagnostic category. A verifier that correctly marks a valid record as invalid has still failed. A verifier that turns a malformed record into a generic parser exception has failed less dramatically, but it has made investigations harder. Use a manifest that pins fixture digests and expected verifier behavior: ```yaml fixture: v1/0007-http-request.json sha256: 4a4d5f0c... expect: status: verified schema_version: 1 chain_position: 7 fixture: v1/0007-http-request-tampered.json sha256: 91af2a7d... expect: status: invalid error_code: payload_digest_mismatch ``` Then test more than one direction. 1. The oldest retained verifier must still verify its original corpus. 2. The current verifier must verify every retained historical corpus. 3. A candidate writer must create records that the current verifier accepts under the new version. 4. A candidate migration must preserve its stated source digest and produce the expected derivative. 5. Every verifier must reject fixtures designed for unsupported future versions. Do not rewrite expected fixtures whenever a test fails. First inspect the bytes, the verifier version selected, and the failure code. Fixture updates should be rare, reviewed like a protocol change, and accompanied by a reason that distinguishes an intended new fixture from altered evidence. Add property testing around parsers, but do not confuse it with the permanent corpus. Random input finds crashes and strange edge cases. Named fixtures preserve the cases you already learned the hard way, including records from real releases after you have removed sensitive content. ## Test the upgrade boundary as an investigation would The highest risk is usually the boundary between versions, not either version in isolation. Write a scenario that starts before the upgrade and ends after it, then ask whether an independent person can explain the whole sequence. Consider a chain where v1 records an agent session approval, several HTTP calls, and a session revoke. Version 2 introduces a more detailed request result field and a new checksum encoding. The test should begin with a v1 genesis record, append valid v1 entries, create the documented boundary checkpoint, append v2 entries, and export the entire bundle. The expected verification report needs to show the transition plainly: ```text $ audit verify evidence-bundle verified v1 records: 18 verified v1 terminal digest: sha256:6c12...e98a verified v1-to-v2 continuity checkpoint verified v2 records: 6 chain status: verified ``` Now run the failures that production upgrades create: - remove the final v1 record but leave the v2 records; - alter the checkpoint's v1 digest; - use a v2 record with a v1 version label; - export only the v2 segment and ask for a full-history verdict; - run a pre-v2 verifier against the mixed bundle. A correct system gives different answers. The first three are invalid. The fourth may verify as a partial segment if the bundle declares its starting checkpoint, but it cannot claim to have verified the full history. The fifth returns `unsupported_version` after reporting any v1 evidence it could verify, if its command design allows partial reports. It must not report the whole bundle as verified. This is where teams discover that their journals and dashboards hide source boundaries. A UI that merges records into one timeline may be fine, provided it labels a schema transition and lets a reviewer inspect the original envelope. Do not make an investigator infer a format change from a sudden field appearance. ## Keep the verifier small enough to outlive the app The audit verifier should have fewer dependencies and fewer privileges than the application that produces records. If reading old evidence requires starting a graphical application, connecting to an account, opening a credential vault, or downloading a compatibility package, your evidence plan depends on conditions that disappear at the worst possible moment. Split the responsibilities: - The application writes records and presents live activity. - A compact verifier reads an exported bundle, selects versioned recipes, and emits a machine readable report. - A renderer can turn verified records into tables and timelines without participating in the authenticity decision. Make the verifier deterministic. Given the same bundle and command options, it should return the same status codes and report structure. Include the verifier release identifier in its report, but do not allow that identifier to change the evidence result. For Sallyport, `sp audit verify` is the right check to preserve in an upgrade runbook because it verifies the encrypted hash chain offline over ciphertext and does not need the vault key. Save the command's report next to an untouched export before changing the app, then verify that same export again afterward. The word "offline" needs discipline. It means the verifier can establish the chain result from the available evidence and its built-in verification recipes. It does not mean the verifier can reconstruct missing records, decide who operated a machine, or prove that a user understood an approval card. A well designed report says exactly which claim it has checked. Publish the evidence format specification with the verifier source and fixtures. A source release without the fixture corpus leaves future maintainers guessing about compatibility. A corpus without a written recipe leaves them guessing whether a passing test reflects a deliberate rule or an accident in one implementation. ## Set a retirement policy before the first emergency You can stop supporting a historical format only after you decide what happens to the evidence that uses it. That decision belongs to security, legal, operations, and the people who investigate incidents. It should not arrive as an incidental consequence of deleting an old package. Write a support table that states the retained versions, verifier releases capable of reading them, the expected evidence retention period, and the process for an exceptional archive. If you intend to retire a reader, provide a standalone archive verifier and freeze its fixture corpus first. Keep its build instructions and expected checksums with the evidence documentation. Do not promise perpetual support casually. Algorithms age, operating systems change, and old parsers can carry security defects. But do preserve a route to verify retained evidence. Sometimes that means a sandboxed archive tool that accepts only local files. Sometimes it means retaining a container or virtual machine image with recorded hashes. The choice depends on your environment; the obligation is to avoid asking a future investigator to reproduce a vanished toolchain from memory. The first action is concrete: export a small real evidence bundle, run the verifier, then write down every versioned rule the command relied on. If you cannot describe that recipe and reproduce the result after a test upgrade, you do not yet have an upgrade plan. You have a hope that old evidence will remain readable.