Should audit evidence backups include credentials?

An incident archive should help you answer who did what, when, through which approved path, and whether someone changed the record afterward. It should not give the person opening that archive a working API token, an SSH private key, or the ability to mint either one.

Teams regularly get this wrong because backup software makes everything look alike. A database dump, a vault export, and an audit journal can all be files encrypted at rest. Their security jobs are different. A vault backup restores authority. An evidence backup preserves accountability. Put them in the same archive and the archive inherits the most dangerous property of the vault.

That distinction matters more with autonomous coding agents. An agent can create a large number of outward actions in a short run: repository changes, deployment calls, ticket updates, cloud administration, and remote commands. After an incident, people need a durable record of those actions. They do not need a portable bundle of the credentials that made them possible.

An evidence backup must not restore authority

A credential-free archive is an evidence package that cannot authenticate as a user, service, or machine after restoration. Its contents may be confidential, but none of them can independently cause an external action.

That means the package can hold a record such as this:

2026-07-22T02:14:09Z
session=ses_8f0d...
process_authority=Developer ID Application: Example Engineering LLC
channel=http
method=POST
host=api.internal.example
path=/deployments
credential_ref=prod-deploy-token
authorization=approved
result=201 Created
entry_hash=2ebc6f...
previous_hash=78dd91...

It must not hold the bearer token represented by credential_ref. It must not hold an SSH private key, a session cookie, an OAuth refresh token, a recovery code, a decrypted vault database, or an export password stored beside the archive. If any of those artifacts appear, you no longer have a pure evidence backup.

People sometimes object that a vault copy is necessary for complete reconstruction. It is necessary only when the purpose is disaster recovery of authority. That is a different operation with a different threat model. Treating it as ordinary incident evidence creates an archive that must be guarded like production access, often for longer than the original credential should remain valid.

Use two explicit classifications in your backup inventory:

• Authority recovery material can restore the ability to act. It belongs in tightly controlled vault recovery procedures.
• Evidence preservation material can explain and verify completed actions. It belongs in an archive workflow designed for retention and review.

Do not soften the boundary with labels such as "sanitized vault export." A vault export that includes encrypted secret blobs, wrapped data keys, or the means to unwrap them is still vault material. A reviewer may not be able to use it today, but you have preserved another target that can become usable after a separate compromise.

The practical test is simple: restore the archive onto a clean machine with no access to your credential system. If a user can turn any artifact in that archive into a live authenticated request, fix the design.

Encryption and evidence integrity answer different questions

Encryption answers "who can read this archive?" Integrity evidence answers "was this record changed, reordered, or cut short?" You need both answers, and neither substitutes for the other.

A password-protected ZIP file can keep an incident timeline private while offering almost no meaningful evidence that its contents stayed intact. An attacker who can decrypt, edit, and re-encrypt it can leave you with a neat archive and no reliable way to distinguish the original from the rewrite.

A hash chain takes a different approach. Each event includes the digest of the preceding event and a digest of its own canonical contents. Change a past entry and later hashes stop matching. Remove a middle entry and the successor no longer points to the expected predecessor. The archive can then prove that the exported sequence is internally consistent, or show exactly where consistency breaks.

That does not make a hash chain magic. If an attacker controls the writer before it emits a record, the chain cannot prove the missing event happened. If the attacker controls every copy of the log and can replace the whole chain, a chain alone cannot identify which chain is genuine. You still need protected checkpoints, controlled export, and independent storage.

RFC 5848, "Signed Syslog Messages," is useful because it separates the claims people casually blend together. It describes origin authentication, message integrity, replay resistance, sequencing, and missing-message detection. It also makes clear that an authenticated log record does not prove the originating host itself was honest. That limit is healthy. Logs are evidence about what a recording system observed and emitted, not a supernatural view of reality.

For an incident archive, write down the integrity claim you can actually support:

• The archive bytes match the manifest created at export time.
• The journal sequence is complete up to a signed or externally retained checkpoint.
• An entry changed after capture causes verification to fail.
• The archive does not prove that an unlogged action never occurred.

That last sentence prevents bad investigations. A clean audit result supports a claim about the journal. It does not absolve an administrator who used a separate credential path outside the recorded gateway.

Keep journals and vault recovery in separate protection domains

Separate files are a start. Separate protection domains are the actual requirement. Different access groups, different recovery procedures, different retention clocks, and different deletion rules keep a convenient backup from becoming a sideways path into production.

A sensible arrangement has three stores:

1. The live vault contains credentials and the material needed to use them. Its recovery process is rare, tightly authorized, and logged.
2. The live journal contains recent activity records and integrity state. Operators need it for investigation and ongoing review.
3. The evidence archive holds exported journal segments, manifests, verifier instructions, and retained checkpoints. Investigators can access it under a different approval path without receiving live credential authority.

Do not use the same archive encryption passphrase for the vault recovery copy and the evidence copy. Do not place both copies in the same cloud bucket because "the bucket already has encryption." Do not put their recovery contacts in the same small group by default. Those shortcuts collapse the separation when a single administrator account, backup role, or storage integration gets compromised.

Retention should differ as well. Credentials should rotate, expire, or be revoked when their operational purpose ends. Evidence may need to survive much longer because the incident is discovered later, a customer requests an explanation, or an internal review takes months. When an old evidence archive contains an old secret, that longer retention quietly extends the secret's useful lifetime.

There is another failure that shows up during offboarding. A departing engineer loses access to the current vault, but they still have an old encrypted backup on a personal recovery drive. If that backup contains credentials that remained valid, the offboarding process has a hole. If the drive holds only a credential-free journal archive, it may still contain sensitive evidence, but it cannot reanimate a production identity.

The separation also makes destruction easier. You can destroy evidence under a retention schedule without wondering whether you just erased the only usable recovery copy. You can rotate or retire credentials without running a mass archive purge. Those jobs should never be coupled.

Export a verification bundle, not a pretty report

A PDF timeline is useful for a meeting. It is poor primary evidence. It loses record structure, usually hides the exact field representation that was hashed, and leaves a reviewer unable to verify whether the source sequence was complete.

Export a package with machine-verifiable contents and a human-readable index. A concrete archive layout might look like this:

agent-evidence-2026-07-22/
  manifest.json
  entries.ndjson
  checkpoints.ndjson
  activity-summary.txt
  VERIFYING.md
  hashes.sha256

entries.ndjson contains one canonical event object per line. Keep fields stable. If you must redact a request body, say so in the entry and preserve a digest of the original protected field when that digest can be retained safely. Do not silently replace a value with [REDACTED] and leave the reviewer guessing whether it was absent, suppressed, or altered after the fact.

checkpoints.ndjson holds the information that anchors journal segments. Depending on your design, that could include a signed digest, an externally stored root value, a witness receipt, or a record of a completed archival transfer. A chain without a retained checkpoint proves only that its internal links agree.

manifest.json should describe the archive as an object rather than rely on a directory name. Include the export identifier, time window, earliest and latest sequence identifiers, record count, schema version, canonicalization version, hash algorithm, redaction policy identifier, and the expected final chain value. Add the exporter version and host identity when that information is meaningful to your investigation.

The activity summary is deliberately secondary. It can state that a run made seven HTTP calls and two SSH calls, received one session approval, and included a failed command. A human can read it quickly. The detailed entries remain the authoritative record.

A manifest shape can be this small:

{
  "archive_id": "evd_2026_07_22_001",
  "window_start": "2026-07-22T00:00:00Z",
  "window_end": "2026-07-22T23:59:59Z",
  "entry_count": 184,
  "canonicalization": "journal-json-v1",
  "hash_algorithm": "SHA-256",
  "first_sequence": 9012,
  "last_sequence": 9195,
  "final_entry_hash": "2ebc6f...",
  "redaction_policy": "evidence-redaction-v3"
}

A report can be regenerated from this bundle. The reverse is rarely true. Keep the structured evidence, then make reports as needed.

Backups fail when exports happen after the incident

The familiar failure sequence looks harmless until someone needs the record. An agent performs an unusual deployment at 1:40 a.m. A maintainer notices customer impact later that morning. The team starts examining the host, restarts the application, rotates credentials, and copies whatever logs happen to be nearby. By lunchtime, the local journal has rolled over, the original process state is gone, and the copied file has no manifest or checkpoint.

Nobody necessarily acted maliciously. The team still cannot establish a defensible history. They have an incomplete excerpt, not an evidence archive.

NIST SP 800-92 describes log management as more than collection. It includes generating, transmitting, storing, accessing, and disposing of log data. That broader view is right. If your plan ends at "the app writes logs," you have planned generation, not preservation. NIST also warns that indiscriminate logging can create operational problems and even cause loss of log data. Record enough to reconstruct agent actions, but do not spray raw secrets, full response bodies, and arbitrary file contents into every event.

Build exports into normal operations before there is an incident. The schedule depends on action volume and your tolerated loss window, but it needs an owner and a test. Trigger an export after a high-consequence agent run, before system maintenance that could affect local state, and at a routine cadence for ordinary activity.

Test the ugly case. Start with a restored archive on a separate machine. Do not give the tester vault access. Ask them to answer these questions using only the package:

• Which agent process initiated the action?
• Which authorization decision allowed or denied it?
• What destination and operation did it target?
• Did the journal sequence verify through the archived checkpoint?
• Which entries were redacted, and under which stated rule?

If the test requires someone to reopen the original application, find a missing teammate, remember a passphrase, or copy a secret from the live system, the archive is incomplete for incident use.

Redaction must preserve the shape of the action

Teams often choose between two bad extremes: dump every request and response into the journal, or remove so much that the record cannot explain the event. The first approach turns the journal into another secret store. The second turns it into a compliance prop.

Preserve the facts needed to identify and assess the action. For an HTTP call, that usually means the session identity, code-signing authority or equivalent process identity, destination host, method, path, credential reference, approval outcome, timestamp, result status, and an appropriate representation of the request and response. For SSH, record the destination, account identity or credential reference, command or a carefully designed command representation, approval result, exit status, and relevant output classification.

The phrase "appropriate representation" requires judgment. A POST /deployments request with a release identifier and environment name may be safe to retain in clear text. A request body that carries an access token, customer record, or private key is not. You can store a fixed redaction marker plus a digest of the original field if later comparison matters. The digest lets an investigator see whether two protected values were identical without revealing them.

Be careful with identifiers. An authorization header is obviously secret. A URL query parameter can be just as bad. A command line can contain a password, a cloud access token, or a customer data path. Response bodies frequently contain temporary download URLs, personal data, or service configuration. Logging tools do not know your business semantics well enough to make every redaction decision for you.

Write the rule in the archive. For example:

redaction_policy=evidence-redaction-v3
request_body=retained only for allowlisted JSON fields
authorization_header=omitted
query_parameters=names retained, values digested
response_body=classified and omitted unless allowlisted
ssh_command=stored after secret argument filtering

That statement lets a reviewer understand why information is missing. It also gives your engineers something concrete to test when an API changes.

Do not redact authorization outcomes, destination identities, or failure details merely because they are uncomfortable. A denied request, a failed authentication attempt, and an unexpected host are often the parts that explain the incident.

A clean chain still needs independent retention

Hash chaining prevents quiet edits inside a preserved sequence. It does not prevent someone with control of the source system from deleting the entire sequence and starting another one. A serious archive plan gives the chain an outside reference point.

The simplest method is periodic checkpoint export. At a defined interval, write the final journal digest, sequence number, and timestamp into a separately controlled archive. Keep multiple copies with separate write authority. An attacker who changes the local journal later must now also alter every retained checkpoint without leaving a discrepancy.

You can make this stronger with signed checkpoints, a trusted timestamping process, or an append-only storage design. The correct choice depends on your environment, but do not jump straight to elaborate infrastructure if you cannot run restore and verification tests. A modest, repeated checkpoint that someone actually checks beats an impressive design nobody has exercised.

Keep the verification process offline-capable. This is where Sallyport's audit design is useful: its write-blind encrypted, hash-chained log can be checked over ciphertext with sp audit verify, without a vault key. A credential-free verifier avoids the absurd situation where the person checking whether an agent abused a secret must first obtain access to that secret.

Your evidence procedure should preserve the verifier itself or a documented way to obtain the exact version used for the archive format. Record a cryptographic digest of the verifier binary or source revision in the manifest. A future reviewer does not need an antique laptop, but they do need enough information to avoid silently verifying old data with changed rules.

An expected verification result should be unambiguous:

$ sp audit verify evidence/journal.enc
verified: 184 entries
chain: intact
range: sequence 9012 through 9195
checkpoint: matched

When verification fails, preserve the failure output and stop treating the archive as a clean sequence. Failure does not automatically prove malice. It can reveal transfer damage, a faulty exporter, a format mismatch, or deliberate alteration. The point is that the archive made the disagreement visible instead of quietly accepting a revised history.

Access to evidence should be broad enough to investigate and narrow enough to protect people

Evidence is safer to share than credentials, but it is not public by default. Audit records can disclose repository names, internal service topology, employee actions, customer identifiers, and operational mistakes. Remove authority from the archive, then control evidence access according to the sensitivity of what remains.

Give investigators read access to the package and the verifier, not write access to the authoritative archive location. Give system operators a documented export role, but make each export visible in the same audit trail where possible. Give outside counsel, customers, or assessors a redacted derivative package when they do not need raw internal details.

Avoid a single "security archive" group that can read vault recovery material, alter retention settings, and download evidence. That group will accumulate access because it makes emergencies easier. It also makes one compromised account capable of erasing both your ability to recover and your ability to explain what happened.

Define an archive custody record. It does not need courtroom theater. It needs basic facts: archive ID, creator, export time, source range, storage location class, access grants, transfer events, verification results, and destruction approval. If you hand a package to an investigator through a file-sharing system, record the package hash before and after transfer.

The payoff appears when the incident gets awkward. An engineer can inspect whether an agent called a production endpoint without being handed the production token. A security reviewer can verify the chain without Touch ID access to the vault. A manager can receive a readable account of events without receiving the raw request bodies. Each person gets the smallest amount of information and authority required for their job.

Make the archive boring before you need it

The best evidence backup is one your team can create and verify while nothing is on fire. It has a fixed package layout, an explicit redaction rule, an independent checkpoint, and a restore test that does not depend on the live vault.

Run one drill with a deliberately damaged copy. Change one byte in an entry, remove a middle record, and alter the manifest count. Your verifier should fail each case in a way that a tired responder can understand. Then run the clean copy through the same process and confirm that the evidence package answers who authorized the action, what happened, and whether the sequence stayed intact.

Keep credential recovery elsewhere. When an incident forces you to preserve history, the archive should make it easier to investigate without quietly creating one more place from which production can be accessed.