Hash-chained audit logs: what they prove and miss

A hash chain can give you strong evidence that a set of audit records has not been edited, reordered, or quietly spliced in the middle after the chain was created. It does not prove that the application logged every relevant event, that a timestamp matches real time, or that a named human took an action. Treating it as proof of all those things is how teams build impressive-looking evidence that collapses under the first serious question.

I have seen incident reviews stall because people asked a log to answer questions it was never designed to answer. A chain verified cleanly, yet nobody could establish whether the service had logged the denied request, whether the operator identity was genuine, or whether the system clock had drifted. The cryptography was fine. The evidence package was incomplete.

For developers running agents with access to APIs and SSH, this distinction matters more than usual. An agent can perform many actions quickly, and the useful question later is often precise: which process requested this call, under which authority, what exact request did the executor send, what came back, and did anyone approve it? A hash chain protects part of that story. It cannot write the rest for you.

A valid chain proves record continuity, not reality

A valid hash chain proves that each available record commits to the preceding record and that the sequence has remained internally consistent since the chosen starting point. If someone changes an old event, swaps two events, or inserts a record between two existing records, verification fails unless that person can recompute every later link and replace every trusted checkpoint.

That is meaningful evidence. It lets an investigator say, "These records form the same sequence that produced this known chain head." The wording matters. The claim depends on a known chain head or another trusted reference. If the only copy of the chain and its final digest sit on the same machine an attacker controlled, the attacker may rewrite both.

A hash chain does not prove these claims:

• The application observed every event that should have been recorded.
• The event payload accurately describes what happened outside the logger.
• The event time matches a reliable clock.
• A person, rather than a compromised process using their credential, caused the action.
• The chain began before an attacker gained control of the system.

Those are different propositions with different evidence. Do not call a log "tamper proof." Software does not receive that status because it uses SHA-256. A chain makes changes detectable under stated assumptions. That is the useful, defensible property.

The distinction between integrity and completeness deserves special care. Integrity asks whether records you have were altered. Completeness asks whether records are missing. A chain addresses the first question well. It only addresses the second when outside evidence fixes both expected checkpoints and the scope of events the logger was required to emit.

The record format decides what the hash actually covers

A chain only protects bytes that enter the digest. Before debating algorithms, define a canonical event format and include all fields that an investigator will need to interpret the action.

A minimal record might look like this:

{
  "sequence": 1842,
  "event_id": "7b2ea6de-9c3f-4bb4-b1d7-8b13fbb1c5b9",
  "recorded_at": "2025-03-08T17:14:22.481Z",
  "actor": {"kind": "agent_process", "process_id": "p-91f"},
  "action": "http.request",
  "target": "api.example.internal/v1/releases",
  "request_digest": "sha256:...",
  "result": {"status": 201, "response_digest": "sha256:..."},
  "previous_hash": "sha256:..."
}

The writer serializes the record in one defined form, hashes those bytes, and stores the resulting digest as the predecessor reference in the next record. Conceptually:

record_hash[n] = SHA-256(canonical_record[n])
canonical_record[n+1].previous_hash = record_hash[n]

The previous_hash field must itself sit inside the bytes that record_hash[n] covers. Omitting it is an embarrassing but real implementation mistake. In that design, records carry digest-shaped decorations without binding the sequence.

Canonicalization is not a detail. Two JSON serializers may order object fields differently, escape Unicode differently, or format numbers differently. A verifier that rebuilds JSON rather than verifying the exact stored bytes can reject honest records or, worse, create disagreement about what the record means. Store the original byte representation, specify the encoding, and test verification across independent implementations.

The chain should cover context, not only the request body. A record that says "deployed" is thin evidence. A record that binds the executor version, action type, target identifier, authenticated process identity, authorization decision, request digest, response digest, sequence number, and recorded time gives a reviewer something to assess. It still does not prove each field is true, but it prevents a later editor from changing the story one field at a time.

NIST Special Publication 800-92, Guide to Computer Security Log Management, makes the same practical point in plainer operational terms: log records need enough event, source, user, status, and timing information to support analysis, and organizations need protection for log data. A perfect chain around vague records preserves vague records perfectly. That is not an audit design.

The first record and the missing tail remain exposed

Every chain has a first record, often called a genesis record. Its predecessor value is fixed by convention, such as a digest of an empty byte string, or it references an earlier checkpoint. The chain can establish continuity after that point. It cannot establish why that point is the beginning of history.

Consider a service that writes records 1 through 10,000 to local storage. An attacker obtains full control, deletes records 1 through 7,000, changes the sequence field in the retained records, and constructs a fresh chain starting at what used to be record 7,001. The forged chain verifies. A reviewer who has no earlier checkpoint sees no cryptographic defect.

The same problem exists at the tail. A crash, power loss, or attacker can prevent the last few events from reaching durable storage. The last retained record may verify correctly even though an action occurred moments later. A chain proves that the retained end was not modified after the fact. It does not prove that it is the true final event.

You reduce both gaps by issuing checkpoints outside the writer's control. A checkpoint contains at least the chain identifier, sequence number, record hash, and checkpoint time. Send it to a separate account, a write-once store, an external timestamping service, or an independently administered collector. A signed checkpoint is better than an unsigned one because it binds the statement to a signing identity.

RFC 3161 describes a time-stamp protocol where a time-stamp authority signs evidence that it observed a message imprint at a stated time. That can support a narrow and useful claim: the authority had this digest by that time. It cannot tell you whether the event payload was honest, whether prior events were omitted before the submitted checkpoint, or whether the actor was authorized. Use it for timing and existence evidence, not as a substitute for operational records.

Checkpoint frequency is a risk decision. Frequent checkpoints shrink the interval in which someone can remove an unanchored tail. They also create more external records to retain and reconcile. Do not pretend that hourly checkpoints protect minute-level incident reconstruction. State the maximum unanchored interval in your operating procedure.

A hash does not turn a timestamp into trusted time

A log timestamp records what the logger's clock said when it created the record. That can be enough for ordinary debugging. It is weak evidence for questions involving deadlines, trading windows, access after termination, or the order of events across machines.

An administrator can change a local clock. A virtual machine can resume with stale time. Network time synchronization can fail. Even a correctly synchronized host records when its application wrote the event, which may differ from when a remote service received the request or committed the change.

Keep these times separate when the distinction matters:

• observed_at: when the originating component observed the event.
• recorded_at: when the audit writer created its record.
• remote_at: a time returned by the remote system, if that system provides one.
• checkpoint_at: when an independent witness accepted a chain head.

Do not overwrite them into one cheerful timestamp field. Each has a different source and failure mode. An investigator can then reason about a range instead of relying on false precision.

If you need time evidence, record how the host synchronizes time, retain synchronization health alerts, and preserve signed checkpoint receipts. For high-consequence operations, compare the local record with the remote service's own audit entry. A request logged at 10:02:01 and a remote change logged at 10:02:05 may describe the same action. The chain protects your record from editing; corroboration connects it to the external system.

A common recommendation says, "Use an append-only database and timestamps, then the audit problem is solved." It is popular because it sounds operationally simple. It is wrong because append behavior inside one service says little about clock trust, missing events, identity, or outside effects. Use append-only storage if it fits, but name the proof you still lack.

Attribution requires an identity trail outside the digest

A hash chain can preserve an attribution claim such as actor = [email protected]. It cannot prove that Alice supplied the credential, that the identity provider authenticated her correctly, or that an attacker did not use her active session. The digest protects the sentence, not its truth.

For agent actions, process identity is often more useful than a vague user field. Record the agent process identifier, parent process where available, executable signing authority, launch time, session identifier, and the human or service account that authorized the session. A process name alone is poor evidence. Any program can choose a familiar name.

Code signing offers a bounded statement: the operating system can identify that an executable bears a signature associated with a signing authority and that the signature validates under the platform's rules. It does not prove that the executable's operator had good intentions. It does help distinguish a known build from an arbitrary binary with the same filename.

Authentication and authorization need separate records too. Authentication says which credential or principal the system accepted. Authorization says why the system permitted this action at that time. An approval dialog, role assignment, token scope, or change ticket may be the authorization evidence. Put a stable reference or digest of that decision in the action record, then retain the underlying decision record under access controls.

Digital signatures improve this layer when they are used correctly. If an audit writer signs periodic chain heads, a verifier can check that the holder of a private key produced those signatures. That adds a source claim beyond an unkeyed hash. It still depends on private-key custody, certificate status, rotation records, and the mapping between the certificate subject and a real operational identity.

Do not bury all of this in one user string. During an incident, people need to separate the human who approved a run, the process that requested the action, the service that executed it, and the credential authority that accepted it. Those may be four different actors.

Authorization evidence answers a different question than activity

An activity record answers, "What did the executor do?" An authorization record answers, "Why was it allowed to do that?" Teams often merge them because both appear during the same request. The merger makes investigations harder.

Suppose an agent requests an SSH command. The executor records the requested host, command digest, result, process identity, and time. Separately, the authorization layer records that a new process received approval, who approved it, what the approval covered, and when that approval expired. If the SSH command runs later, the activity event should reference the applicable authorization record.

That design lets a reviewer ask the right questions. Was the command issued? Check the activity record. Did the session have permission? Check the authorization record. Did an approval screen show an accurate identity? Check the UI and process identity records. Did the remote host execute the command? Check its server logs or the resulting state.

One approval for a session and approval for every sensitive credential use carry different evidence. Session approval establishes that a person allowed a particular process to operate for its lifetime. Per-action approval establishes a narrower decision close to a specific operation. Neither is automatically superior. The choice depends on action frequency, consequence, and whether a human can realistically evaluate repeated prompts.

Approval fatigue is a design failure, not a reason to stop recording approvals. If a person sees hundreds of indistinguishable requests, the resulting clicks provide little evidence of considered authorization. Group low-risk work under a bounded session decision, reserve repeated confirmation for credentials or actions where the human can judge the target and consequence, and record the scope in plain terms.

An approval event also cannot prove informed consent by itself. It proves that the approval mechanism recorded a decision. The process details displayed to the user, the binding between that display and the executing process, and the audit record all determine whether the decision can carry weight later.

Completeness depends on where actions can escape the logger

You cannot obtain a complete record of agent actions by adding a hash chain to a log file after the agent already holds raw credentials. Once an agent receives an API token or SSH private key, it can call another client, copy the secret, or make a request without using your audited path. The chain may faithfully preserve the requests it saw while missing the ones that matter.

Completeness begins with control of the action boundary. The component that holds the credential should execute the network request or SSH operation itself and log the decision and outcome before it returns a result. The agent should receive the result, not the secret. That changes the claim from "we asked the agent to log its work" to "the credential holder observed each use through this channel."

Even then, scope remains explicit. If a developer can also use the same credential in a terminal, the audit trail covers agent-mediated use but not all credential use. If an agent can reach a second network path that bypasses the executor, the trail does not cover that path. A completeness claim must name the channel, principal, and period it covers.

Sallyport follows this boundary for its supported HTTP and SSH channels: the encrypted vault keeps API and SSH credentials out of the agent, while the app executes the action and records it. That gives its audit trail a stronger scope than an agent-side activity log, but it does not make claims about actions taken through an unrelated credential path.

A useful failure test is to draw every route from an agent to an external effect. Include direct HTTP clients, shell tools, local files that contain tokens, browser sessions, cloud metadata services, CI variables, and SSH configuration. Every route outside the logged executor is an exception to any completeness claim. Close it, isolate it, or record the exception honestly.

Confidentiality, access, and retention need their own controls

Hash chains disclose nothing by themselves, but audit records often contain information you would not put in an ordinary application log. Request URLs may reveal customer identifiers. Command lines can contain secrets. Responses may contain personal data. A chain that makes these records hard to alter can also make careless disclosure hard to undo.

Encrypt audit storage and restrict who can read it. Keep the ability to verify integrity separate from the ability to decrypt content when possible. A verifier can recompute hashes over encrypted record bytes and compare the predecessor references without seeing the protected payload. That is useful when a reviewer needs to establish that a sealed archive remains intact before an authorized investigator opens it.

Redaction must happen before the record enters the chain. Replacing a secret after writing it changes the bytes and breaks the chain. Instead, record a digest of a sensitive request body, a stable token reference, length information where appropriate, or a structured statement that a field was withheld. Document the redaction rule. Otherwise two records that look similar may have been redacted by different logic.

Retention also affects evidentiary value. If a retention job deletes old records, it should produce its own record identifying the deleted range, the governing retention rule, the deletion time, and the final checkpoint before deletion. Preserve the checkpoint in a separate archive. A chain cannot make deleted material reappear, but the retained evidence can show that the deletion occurred under a stated process rather than silently.

Access logging matters too. Record who exported an audit archive, who decrypted it, and who changed retention configuration. Audit logs attract attention during an incident. The chain protects past records from quiet edits, but a broad group of readers can still copy sensitive contents or pressure an operator to suppress future logging.

Verify the archive before you need it

Verification should be a routine operation with a saved result, not a command somebody tries for the first time during an outage. Your procedure should identify the exact archive range, verifier version, expected checkpoint source, verification time, operator, and outcome.

A verifier needs to detect at least four failures: malformed records, an incorrect predecessor reference, a record whose digest does not match its stored bytes, and a mismatch between the computed head and an expected checkpoint. Its output should make the result easy to preserve. For example:

$ audit verify --archive agent-actions-2025-03-08.log --checkpoint checkpoints/2025-03-08.json
chain_id: agent-actions-prod-a
records_checked: 1842
first_sequence: 1
last_sequence: 1842
computed_head: sha256:4e7c...a912
checkpoint_head: sha256:4e7c...a912
result: VALID

A result of VALID means the verifier checked the supplied bytes against the supplied checkpoint. It does not mean "every production action is present" or "the events are true." Write that limitation directly into the procedure. People reading a green result under pressure will otherwise assign it more meaning than it has.

Test destructive cases in a nonproduction archive. Change one byte in a record body. Move two records. Delete a middle record. Replace the checkpoint head. Truncate the tail. The verifier should reject each case except ordinary tail truncation without an expected final checkpoint, where it may report a valid prefix. That report is correct and should worry you for the right reason.

Sallyport exposes offline chain verification through sp audit verify, and its verifier can check the encrypted audit chain without a vault key. Run that check after exports and practice reconciling it with the separate session and activity views before an investigation forces the work.

An evidence package needs records that disagree productively

A defensible investigation uses several records with different failure modes. Agreement between independent sources carries more weight than a single log that repeats its own assumptions.

For a sensitive agent action, retain the hash-chained activity record, the session or approval record, the authenticated process identity, the relevant configuration version, the external checkpoint receipt, and the target system's own record or observable result. You do not need every source for every ordinary request. You do need enough for the consequence of being wrong.

Expect sources to disagree sometimes. The local activity record may precede the remote service's timestamp because network delivery took time. A request may have a success response while the remote system later rolls back the operation. A session record may show approval, while the action record shows no use of the granted authority. These differences are evidence to investigate, not defects to smooth away.

Write down the exact claims your system can support. For example: "This archive contains an unmodified sequence of executor records through checkpoint X." Then write the claims it cannot support without other records: "This archive alone does not prove that no direct credential use occurred." Clear limits make the strong claims credible.

The first practical action is to take one high-consequence agent operation and reconstruct it from start to finish. Identify the action boundary, fields written before execution, fields written after execution, approval evidence, checkpoint destination, remote corroboration, and every bypass route. If any part depends on somebody remembering what happened, you do not yet have an audit trail. You have a story that may not survive contact with an incident.