# Agent audit logs must fail closed under disk pressure An agent gateway that keeps acting after it can no longer durably record those actions has broken its own security boundary. Disk pressure is not an observability nuisance. It decides whether the next API request or SSH command leaves behind evidence that operators can inspect and verify. The safe rule is plain: deny new credentialed calls before the audit writer runs out of room. Alert earlier. Let work that already crossed the admission boundary finish only if the gateway can still record its completion or failure. When appends fail, preserve the ciphertext already on disk and stop treating the log as a convenient file that can be cleaned up later. ## Audit logging must be part of authorization A gateway cannot call audit logging "best effort" if its purpose is to put a human-controlled boundary around agent actions. Once an agent can ask for a credentialed HTTP request or an SSH command, the record of that request is part of the authorization decision. If the gateway performs the action but loses the record, an operator cannot later distinguish a legitimate run from an unaccountable one. This is where teams often blur two different failures: - The dashboard or activity view cannot update because a derived index is unavailable. - The encrypted source log cannot accept a durable event append. The first failure is inconvenient. The second failure changes whether the gateway may take a new external action. Do not make them share a severity label or a recovery path. Sallyport's model gives this distinction a useful shape: its Sessions and Activity journals project from one encrypted, hash-chained audit log, rather than each being a separate source of truth. A projection can lag or need rebuilding; the write-blind encrypted chain is the record that must remain intact. That arrangement also makes the admission rule easier to state. Before dispatching an action, the gateway must know it has enough storage capacity and writer health to record at least the attempt. After dispatch, it must record the result. If the action itself has an irreversible effect, such as changing a remote production setting, the gateway should record a durable intent event before dispatch and a result event afterward. Do not promise exact chronology that the storage layer cannot support. Promise something stronger and more useful: every admitted action has a durable, ordered record of its request identity, authorization context, and final known outcome. If the gateway cannot sustain that promise, it should not admit another action. ## Alerting and denial are different states Alerting at the same instant that you deny calls guarantees a bad operational choice. The operator finds out when the agent is already blocked, then must decide under pressure whether to delete data, stop work, or weaken the audit rule. A storage policy needs separate states with explicit transitions. Use four states: 1. **Normal**: free capacity exceeds the operating reserve, and a recent durability check succeeded. 2. **Warning**: capacity fell below the warning threshold, but the gateway still has its full reserve. New calls continue, and operators receive one alert. 3. **Restricted**: capacity fell below the reserve, or the writer saw a transient failure that has not yet crossed the hard-failure boundary. Do not admit actions that can create large or multi-event records. Allow only work already admitted, and keep testing the writer. 4. **Denied**: the gateway cannot durably append an audit event, or remaining capacity cannot cover the minimum record budget. Deny every new credentialed call. A percentage alone does not define any of those states. Five percent free on a 4 TB volume can be ample. Five percent free on a small volume can disappear during one agent run, one system update, or one burst of verbose remote command output. Storage policies fail when they borrow their numbers from a monitoring template instead of the gateway's actual write behavior. The warning threshold buys operator time. The reserve protects the evidence trail. The denied state protects the claim that every admitted action is recorded. They have different jobs, so they should not use the same number. A good state transition also avoids flapping. Enter Warning when free capacity crosses below its threshold. Leave it only after capacity rises above a higher recovery threshold and the writer completes a durable test append. The same hysteresis applies to Restricted. Without it, an agent run near the boundary can alternate between permitted and denied calls based on a few filesystem blocks, which is both confusing and hard to investigate. ## A reserve is a record budget, not free space optimism Set the hard reserve by calculating the records that may still need to exist after you stop admitting new work. The calculation does not need fake precision. It needs conservative inputs that you can explain during an incident. Start with these questions: - What is the largest admitted action record, including encrypted payload, headers that your policy retains, result metadata, and hash-chain fields? - How many actions can be in flight at the moment the gateway enters Restricted? - Can one action produce a separate completion record, a timeout record, or a retry record? - What local files grow beside the source log, such as segment metadata, indexes, temporary files, or crash reports? - How much room does the operating system and the app need to report the condition cleanly? Suppose a gateway permits 12 in-flight actions, each could require an intent record and a result record, and a conservative encrypted record budget is 128 KiB. The action records alone consume 3 MiB. That number is not the reserve. Add the largest expected journal projection batch, file rollover overhead, a failure marker, and a material safety margin for allocation behavior. Then round upward to a number that is easy to monitor. The useful output is a written policy, not a magic number: ```text warning threshold: 2 GiB free on the audit volume hard reserve: 512 MiB reserved for audit completion and recovery records admission budget: 256 KiB minimum available per new action recovery threshold: 3 GiB free plus one durable test append ``` These values are examples, not defaults for every Mac. A single developer machine with short API calls can justify a smaller reserve than a shared build machine with autonomous agents executing long SSH jobs. The policy must reflect the worst credible output that the gateway records, not the average request size from a quiet week. Apple's current APFS documentation makes a related point that people miss when they build a percentage alarm: check whether the space needed for a particular operation is available, rather than trying to derive one reliable total from available free space across a partition. APFS also uses space sharing, clones, and sparse files, so apparent room and immediately usable room do not always behave like an old fixed-slice disk. For an audit writer, that means the admission check should ask, "Can I safely afford this record budget now?" It should not ask, "Does the menu bar still show a nonzero amount of free space?" ## A failed append must change the gateway's behavior immediately Treat an append failure as a state transition, not as a line in a debug log followed by another attempt. A retry can make sense for a temporary interruption. It does not make sense as permission to keep sending unrecorded calls. Picture a plausible failure sequence. The audit volume has fallen below its reserve because a local development tool created a large cache. An agent asks the gateway to perform an HTTP deployment action. The gateway writes the intent event, dispatches the request, receives a successful response, then cannot append the completion event because the filesystem returns an out-of-space error. At that point, the gateway knows three things: it admitted the action, the outside system may have changed, and its normal completion record is missing. The right behavior is to preserve the durable intent event, record a failure marker if any safe channel remains, stop new calls, and present the operator with the action identifier and the storage failure. It must not quietly re-run the request. A retry could duplicate the remote side effect. Now consider a worse sequence. The intent append itself fails before dispatch. The gateway must deny the action. It has no durable basis for claiming the action was requested, approved, or executed. A user may find this frustrating when a build is waiting, but the alternative is an audit gap that nobody can reconstruct with confidence. The same rule applies to failures surfaced during synchronization. Apple's fsync manual says that fsync moves modified data and attributes toward permanent storage, and that a queued I/O operation can cause fsync to fail with errors associated with read or write. The manual also warns that an ordinary flush does not itself guarantee physical media ordering across power loss, which is why software must define its own durability boundary rather than equate an in-memory append with a committed event. Use a small failure classifier: ```text append or sync succeeds -> action may proceed or complete normally append fails before dispatch -> deny the action append fails after dispatch -> deny new actions, preserve intent, raise incident sync reports I/O failure -> deny new actions, preserve files, investigate storage health space check below admission budget -> deny this new action before dispatch ``` Do not delete old segments to make the current write succeed. That turns a capacity incident into evidence destruction. ## The writer needs a durable commit boundary A writer that simply appends bytes to an open file has not finished the audit problem. It needs a boundary that tells the rest of the gateway when an event became part of the durable chain. Keep that boundary small and explicit. One workable pattern uses append-only segment files, each event carrying the previous event hash, and a small segment manifest. The exact encoding can vary. The ordering of operations should not. ```text 1. Serialize the next encrypted event with sequence number N and previous hash H(N-1). 2. Append the complete event frame to the active segment. 3. Sync the segment file and check the result. 4. Update the manifest with the new high-water sequence and segment hash. 5. Sync the manifest and check the result. 6. Only now mark event N as committed to the dispatcher. ``` The manifest matters because recovery needs an answer to "which complete event frames count?" A trailing partial frame after a crash or a full disk is not a committed event merely because some of its bytes reached the segment. Frame length, authentication data, sequence number, and the manifest's committed high-water mark let recovery reject ambiguity instead of guessing. Do not solve this with a rewrite-in-place JSON file. It looks easy until the filesystem has room for the new file but not the rename, or until a crash leaves an old index beside a newer segment. Append-only segments plus a tiny manifest make failure behavior easier to reason about and easier to verify offline. There is a second distinction that matters here: encryption protects event contents, while a hash chain protects ordered continuity. Neither property means an append was durable. The writer must establish durability first, then the verifier can establish that the retained sequence has not been altered. Sallyport's `sp audit verify` can verify its encrypted hash chain offline over ciphertext without a vault key. That verification is useful after a storage incident because the operator can check preserved evidence before unlocking the vault or restarting agent work. ## Run the drill on a disposable audit volume A disk pressure drill should prove behavior at each state transition, not merely prove that a write eventually errors. Use a disk image or isolated test volume, configure only a nonproduction gateway instance to store its audit data there, and remove it after the exercise. Do not fill your normal Mac volume. The test should inconvenience a disposable environment, not gamble with the machine that holds your source code and credentials. On macOS, create and mount a small APFS disk image for the drill: ```sh hdiutil create -size 2g -fs APFS -volname AuditDrill /tmp/audit-drill.dmg hdiutil attach /tmp/audit-drill.dmg df -h /Volumes/AuditDrill ``` The exact mount path can differ if a volume with that name already exists. Confirm it with `df` before changing any test configuration. Your expected output shape should identify a mounted filesystem and show roughly 2 GiB of capacity: ```text Filesystem Size Used Avail Capacity Mounted on /dev/diskXsY 2.0G ... ... ...% /Volumes/AuditDrill ``` Point the test instance's audit storage to that mounted path. Generate a few known-good actions first. Record their action identifiers and run the chain verifier. This baseline matters because a later failed verification may come from your setup, not from disk pressure. Then consume space only inside the mounted disk image: ```sh dd if=/dev/zero of=/Volumes/AuditDrill/fill.bin bs=1048576 ``` `dd` stops when the volume cannot allocate more blocks. It is intentionally blunt. Do not use a sparse-file command for this test, because a sparse file can appear large without consuming the blocks that trigger the condition you need to test. Run the drill in stages rather than racing directly to zero space: 1. Fill until the warning threshold triggers. Confirm that new small actions still work and that the operator alert appears once. 2. Fill until the hard reserve triggers. Confirm that the gateway denies new actions according to its admission budget. 3. If your test design allows it, admit one controlled action immediately before Restricted and observe whether its final event records successfully. 4. Fill until an append or sync actually fails. Confirm that the gateway enters Denied and exposes the failure reason. 5. Stop the process, remount the image if needed, and verify the encrypted chain before deleting the filler file. The point is not to admire an ENOSPC error. The point is to answer whether the gateway fails at the correct moment, whether its journals tell the truth afterward, and whether an operator has enough evidence to act without guessing. ## Test the ugly timing windows, not only a full volume A shallow drill fills a volume, observes a denial, frees space, and declares success. That misses the timing windows that create audit gaps. Test an action whose external side effect finishes quickly while its audit completion record is delayed. The controlled target can be a test HTTP endpoint that returns a unique request identifier. Start the action while room remains for the intent event, then consume the remaining test-volume space before the gateway commits the result event. The expected outcome is not necessarily a neat success or failure status. The gateway may need to report that the external action's final state requires reconciliation. That is an honest outcome. The gateway should show the durable intent record, the remote request identifier if it received one, and the local storage failure. It should deny subsequent calls. An operator can inspect the test target and decide whether the remote action succeeded. What it must not do is convert uncertainty into a second request. Also test a failure while the gateway rotates an audit segment. Segment rollover consumes metadata and can involve a new file, a directory update, and a manifest update. A design that survives a regular append but fails during rollover has not solved storage pressure. Test restart behavior as well. After an append failure, force-close only the disposable test instance, then start it again with the volume still full. It should remain in a safe denied condition rather than assuming a fresh process means healthy storage. Free a measured amount of space, start it again, and confirm that it verifies the last committed sequence before opening a new segment. Finally, inject a permission or I/O failure if your test harness can do so. Capacity exhaustion and an I/O error demand the same immediate admission response when they block durable audit writes, but the investigation differs. Freeing space may cure ENOSPC. It does not cure a failing volume, a filesystem error, or an interrupted storage device. ## Preserve ciphertext before you attempt a repair When an audit write fails, people reach for cleanup because they want the agent unblocked. That reflex is how a recoverable incident becomes a contested one. Preserve first, repair second. Freeze the affected audit directory. Do not compact segments, regenerate a manifest, truncate the last file, rotate encryption material, or rerun an action just to make the activity view look complete. If the storage device appears unhealthy, copy the directory with a method that reports read errors, then work from the copy. The original may be the only evidence of which records reached storage. Your preservation checklist should answer five questions: - Which audit segment and manifest were active when the first write failed? - What was the last sequence that the verifier accepts? - Which admitted actions have an intent record without a result record? - Which external systems can confirm those actions independently? - Did any process modify the audit directory after the failure? A hash-chain verification result is evidence, not a repair instruction. If verification stops at sequence 8,412, preserve that fact. Do not automatically truncate a later partial frame unless your documented recovery procedure defines it and the original artifact has been retained. A partial encrypted frame may be expected after a crash; it may also reveal a writer bug. You need the original bytes to tell the difference. This is another reason to keep the source log distinct from the user-facing journal. A journal may show incomplete rows until projection resumes. That is acceptable if it tells the operator it is catching up. Rebuilding or hiding derived views must never rewrite the source evidence to make a screen look tidy. ## Alerts need to tell operators what decision the gateway made "Disk almost full" is a weak alert for an action gateway. It tells the operator about a machine condition but not whether agent activity remains permitted, whether audit evidence is at risk, or what changed since the previous notification. Emit an alert when the gateway enters Warning, Restricted, Denied, and verified recovery. Include the fields that let an operator act: ```text state: denied reason: audit append failed with ENOSPC audit path: /configured/audit/path free bytes observed: 41,943,040 hard reserve: 536,870,912 last committed sequence: 8412 unresolved admitted actions: 1 new credentialed calls: denied existing action handling: completion record could not be committed first failure time: 2026-07-22T14:37:18Z ``` Do not put secrets, request bodies, or credentials in this alert. The alert needs identifiers that join to the encrypted evidence, not a second copy of sensitive activity data scattered through notification systems. Avoid paging on every low-space sample. Page on entry to Restricted or Denied, then send a reminder only if the condition persists or the state worsens. A noisy warning at Normal-to-Warning transition can be a ticket or a visible local notification. A transition to Denied is an operational incident because the gateway has intentionally stopped new external actions. The alert should also state whether the gateway is protecting records already admitted. That detail changes the response. If it has room to finish in-flight records, operators may let those runs settle while they free space. If an append already failed after dispatch, operators need to investigate the unresolved actions before they resume automation. ## Recovery must prove the writer is healthy again Freeing space is necessary, but it does not prove that the audit writer can safely return to Normal. A recovery procedure should make the gateway earn that transition. First, preserve the incident artifacts and record what freed the space. Deleting an unrelated build cache is a different event from deleting audit files, and the incident record should say which one happened. Then verify the retained chain offline. If verification fails, keep the gateway in Denied and investigate the evidence before permitting any new actions. Next, run a bounded recovery append. The gateway should write a recovery event that identifies the prior denied state, append it to a new or documented recovery segment, synchronize it, update its manifest, and verify the resulting sequence. It should not silently resume in the middle of the segment that failed. Only after that append succeeds should the gateway rebuild or catch up its derived journals. The journal work can fail independently. If it does, keep the source evidence and report that the view is incomplete. Do not deny that an action occurred merely because an activity row has not appeared yet. Then apply the recovery threshold. If you configured Warning at 2 GiB, a hard reserve at 512 MiB, and recovery at 3 GiB, do not reopen calls at 600 MiB just because the writer managed one append. The higher threshold prevents immediate relapse and gives the operator time to find the source of the pressure. A storage pressure drill earns its keep when it changes one concrete decision: the exact point where your gateway stops authorizing new work. Write that point down, test it against a disposable volume, and treat every failed append as evidence to preserve rather than clutter to remove.