API audit log investigation when agent records conflict

An API provider says a request changed production data. The agent record says it never got that far. Both statements can be true, and treating either log as a verdict is how teams turn a contained discrepancy into a bad incident response.

Investigate the action as a chain of observations. Establish who initiated the run, what the agent attempted, what crossed the credential boundary, what the provider accepted, and what changed afterward. Timestamps help order that chain. Request identifiers join it. Results and state tell you whether it mattered. Missing events are evidence too, but only after you rule out the ordinary ways records disappear.

I have seen people start with a spreadsheet and immediately sort by time. That is backwards. A timestamp is often the weakest join field in the room. Start with stable identifiers and immutable exports, then use time to test whether the proposed sequence makes sense.

Preserve the records before anyone refreshes a dashboard

Capture original evidence before you filter, retry, revoke access, or ask a provider's support team to investigate. Interactive dashboards change, retention jobs run, and a retry can create the second event that confuses the first.

Open a case folder with a case ID and collect raw exports, not screenshots alone. Include the agent session record, individual action records, provider audit export, application logs for the affected system, and any egress record available to the team. Record the collection time in UTC, the person who collected it, the account or role used, and the filter used to generate each export.

Hash each file after collection. A shell command is enough if your operating system provides a standard SHA-256 utility:

$ shasum -a 256 provider-events.json agent-activity.json
81b5777b8416320fe26cb8a8dddb6a9e736fab4f5e7aa5812bf6afeffc5f4e82  provider-events.json
a1e98c01992b51104fbc8c5fcbaa78e65db31f1edb3e546f4c14d0e6d3673ba  agent-activity.json

Put the hashes in a plain case note. The hash does not prove that a provider export was complete. It proves that your working copy has not silently changed since you collected it. Those are different claims, and incident reports often blur them.

Do not “clean” JSON into a spreadsheet as the first copy. Normalization can discard duplicate fields, array order, fractional seconds, empty values, and the exact request body that later explains a mismatch. Keep an untouched export and make a separate parsed working file.

If the discrepancy could involve a credential leak or unauthorized use, contain access in a way that preserves the sequence. Revoke a live agent session or lock the action path if you can. Avoid rotating the provider credential before you have collected the provider's recent audit records, unless active abuse requires immediate rotation. A rotation is sometimes necessary, but it can erase the one remaining route to attribution.

A request identifier outranks a timestamp

Join records on identifiers that survive boundaries: a provider request ID, a client-supplied correlation ID, an idempotency key, an object ID returned by the write, and a trace ID if the provider documents one. Keep every identifier, because a provider may expose different ones in headers, audit events, support exports, and error bodies.

The best case looks simple. The action record says the agent called POST /v1/invoices; the response headers contain x-request-id: req_72M...; the provider export includes req_72M...; and the created invoice has inv_4P.... You now have a join across intent, delivery, provider handling, and durable state.

The harder cases are more common. A provider may assign a request ID only after it parses the request. A TLS failure then has no provider request ID because the request never reached the application. A gateway may generate one ID and the downstream service another. An asynchronous API may return a job ID, then write the requested object minutes later. Record which boundary issued each ID instead of flattening them into a single request_id column.

Use a reconciliation table that makes uncertainty visible:

This table exposes a familiar failure: the caller timed out, but the provider accepted the write and processed it after the caller gave up. Calling the action “failed” because of the caller's result would be wrong. Calling the provider log “proof that the agent intended it” would also be wrong. The evidence says the agent sent a request that the provider accepted, then the caller did not receive a timely response.

If the provider allows an idempotency key for writes, use it. The IETF Idempotency-Key Internet-Draft describes the practical goal well: a client retries an unsafe HTTP operation without accidentally creating the same effect twice. Provider-specific behavior differs, so read the provider's documentation on retention and matching rules. Do not assume that matching only the endpoint is enough.

For APIs that accept custom headers, generate a correlation ID before the call and send it in a documented header such as X-Client-Request-ID. Store it with the local event. Never put secrets, prompts, user data, or raw tokens in this ID. A safe value has no meaning outside the case, for example case-2025-041-run7-call18.

Time can disprove a story, but it rarely proves one

Use timestamps to bound events and detect impossible orderings. Do not use them as a primary identity field unless every source lacks a better identifier.

RFC 3339 defines a common Internet timestamp profile and recommends the uppercase UTC form ending in Z, such as 2025-03-08T10:04:03.219Z. Keep the original string even after parsing it. The distinction between 10:04:03Z and 10:04:03.219Z matters when one source rounds to seconds and another reports milliseconds.

Create four time fields for every relevant event:

• the source timestamp exactly as exported
• the normalized UTC timestamp
• the event type, such as sent, accepted, completed, or logged
• the clock owner, such as local Mac, provider edge, provider worker, or database

A provider edge timestamp can precede a local “response received” timestamp without contradiction. A provider worker completion timestamp can follow the agent process exit. A local clock that drifted can make an action appear to occur before the session began. These are ordinary mechanics, not proof of tampering.

Build a window around a known anchor, usually a request ID or a session start. Start narrowly enough to prevent accidental joins. Widen it only when you can name the reason: the provider records seconds only, the operation is asynchronous, or you measured clock drift against a trusted reference. Write the chosen window into the case note. “We searched roughly around the time” is not a method.

Be careful with log ingestion time. Many systems expose both event_time and created_at. The first describes when the event happened according to the emitting system. The second may describe when an aggregator received or indexed it. Late arrival does not mean late execution. If an event appears after an incident began, inspect both fields before you build a narrative.

A useful ordering test asks only whether the proposed story is possible. A provider event at 10:04:03, paired with a local send at 10:04:03.219, may be possible if clocks differ or the provider rounds down. A claimed completion at 10:02 when the provider says it accepted the job at 10:04 is not possible unless you have mixed two events or misunderstood the field.

Separate attempted, delivered, accepted, and completed

Teams often compress four different states into the word “called.” That shortcut causes most log disputes.

An agent can attempt an action by constructing a request. A local component can deliver bytes to a remote endpoint. The provider can accept the request. A downstream worker can complete the effect. Each stage has a different record and a different failure mode.

The HTTP Semantics specification, RFC 9110, makes the status code a statement about the server's response, not a complete history of the caller's experience. A 202 Accepted explicitly says processing has been accepted but not completed. A 204 No Content says the server completed the request successfully, but does not by itself explain every downstream effect. A network timeout may yield no HTTP response at all while the server still processed the request.

Classify each disputed event with a status such as these:

• Attempted only: a local action record exists, but no evidence shows network delivery.
• Delivered, outcome unknown: the request left the local boundary, but the caller received no reliable response and the provider has no searchable record yet.
• Accepted, effect pending: the provider returned an acceptance or job reference, with no completed state yet.
• Completed: a provider result and an observed state change agree.
• Contradicted: sources make claims that cannot both be true after you account for their field meanings.

“Outcome unknown” is a legitimate conclusion. Do not relabel it as a failure simply because the agent received an exception. For a write operation, that exception should stop automatic retries unless an idempotency mechanism or a read-back check makes retry safe.

The inverse mistake is just as bad: a 200 response does not mean the intended business result happened. An endpoint can return success for a syntactically valid request while a later validation, async job, or downstream dependency rejects the intended change. Inspect the returned object, job status, or target-system event that the API contract says represents completion.

Missing events need a bounded explanation

A missing record can mean the request never occurred, but it can also mean you asked the wrong service, used the wrong account scope, searched the wrong retention tier, or expected a record that the provider never promises to emit.

Work through missing events in a fixed order. First, confirm the exact account, project, region, environment, and API product. Providers frequently segregate audit views by one or more of these fields. Second, search by every identifier, then by a documented time window and endpoint. Third, inspect whether the provider records accepted requests, rejected requests, data-plane calls, control-plane calls, or only administrative actions. Fourth, check retention and export delay. Fifth, ask whether a proxy, SDK, or asynchronous queue creates a different provider event than the one you expected.

A concrete failure is worth remembering. An agent submits POST /exports and gets a connection timeout. The team searches provider audit logs for the local client ID and finds nothing. They retry, then receive two export completion notifications.

The first request went to a regional ingestion endpoint. The audit screen they checked showed only control-plane events. The provider recorded the job under a generated export ID, not the client header, and the job service completed it after the timeout. Nothing in this sequence required malicious activity. The duplicate came from retrying a write before checking for an idempotency key, a job query endpoint, or a business-level marker.

That failure also shows why absence has to be phrased carefully. Say, “The export we collected contains no matching data-plane event for this window,” rather than, “The provider has no record.” The first statement identifies the evidence and its limit. The second makes a claim you often cannot support.

If a log is expected but absent, preserve the query parameters and capture the provider documentation that states the expected event coverage. A support request without the exact request ID, account scope, UTC window, endpoint, and evidence hashes will waste days.

Results need inspection beyond status codes

Compare the request's stated intent with the response payload and the observable effect. Status codes tell you about a protocol exchange. They do not tell you whether the request was scoped correctly, whether the provider applied a default, or whether an agent sent a stale identifier.

For each action, capture these fields where the API exposes them: HTTP method, normalized path, request ID, idempotency key, actor or credential identity, status code, response body hash, returned object ID, and any asynchronous job ID. Redact credentials and sensitive payload data before broader sharing, but retain a protected original if policy permits.

A response body hash helps distinguish two superficially identical 200 records. Compute it over the raw response bytes before pretty-printing. If the API returns JSON and field order changes between layers, preserve both the raw bytes and a canonical parsed copy. Do not claim that equal status codes mean equal responses.

Then query the resource that should exist or change. For a create operation, retrieve the returned object ID and compare its creator, creation time, and attributes. For an update, retrieve a version, revision, or an audit entry if the service provides one. For a delete, check whether the object is absent and whether a provider audit record attributes the deletion to the same credential.

This is where broad credentials hurt investigations. If many tools share one API token, the provider can often tell you that the token acted, but not which local process or human initiated the action. Treat credential identity as a boundary marker, not actor identity.

A gateway record is useful only if it records the boundary

An action gateway gives you a clean observation point between an agent and the credentialed operation. It should record the invoking process or run, the approved authorization state, the requested operation, the result returned to the agent, and enough identifiers to join provider records. It should not hand the credential to the agent and then call the resulting local telemetry an audit trail.

Sallyport keeps API and SSH credentials in its encrypted vault, executes the action itself, and returns the result to the agent rather than the secret. Its Sessions and Activity journals project from a write-blind encrypted, hash-chained audit log, which gives an investigator both run-level and call-level records. sp audit verify can verify that chain offline over ciphertext without needing the vault key.

That design addresses a specific gap. A provider log can identify a credential and an API request. It does not tell you which agent process received permission to use that credential, nor does it prove that the agent never saw the secret. A local audit record can answer part of that question only when the credential boundary actually sits inside the component generating the record.

Do not oversell a gateway record. It cannot report a request that bypassed it, and it cannot turn an ambiguous provider API into a precise one. It gives you a stronger place to compare evidence and a place to revoke a known agent run while the investigation proceeds.

Write the finding as claims with evidence and limits

A good finding lets another engineer reproduce your reasoning without inheriting your assumptions. Write separate claims for invocation, permission, request delivery, provider handling, and observed effect. Attach the identifiers, timestamps, source files, and field meanings that support each claim.

Use language that matches confidence. “The action journal records that process X requested POST /v1/invoices at this time.” “The provider export contains a request with the same provider request ID.” “The invoice exists and its attributes match the recorded response.” Those are testable statements. “The agent definitely caused the invoice” may be justified only if the joins and the credential boundary support it.

When records disagree, leave the disagreement visible in the final report. Do not average timestamps or discard the inconvenient source. State the most likely explanation, the alternatives you ruled out, and the evidence you still lack. If you cannot establish whether a write completed, record it as unknown and fix the API path before allowing automatic retries.

The practical change after an incident is usually small and unglamorous: require a correlation ID, retain the right provider event class, preserve fractional UTC timestamps, and use idempotency for writes. Those controls turn the next disagreement from a forensic argument into a short reconciliation.