Webhook audit trail: trace agent workflows end to end

An agent action that starts an asynchronous workflow has two histories: the request it sent and the event that came back. Teams often log the first one well enough to answer “did the agent call the API?” and log the second one well enough to answer “did our receiver get a webhook?” They then discover, during an incident, that nobody can prove those records describe the same piece of work.

A webhook audit trail must connect intent, authorization, outbound delivery, remote acknowledgement, incoming receipt, verification, and the business state you chose to accept. Treating a 200 response as the end of the record is how a payment, deployment, ticket, or access change becomes impossible to explain three days later.

A webhook audit trail records two different facts

A webhook audit trail should preserve the difference between an action request and an event notification because they answer different questions. The outbound side says what an agent asked a remote service to do. The inbound side says what some sender later claimed happened.

Those facts may refer to one workflow, but they have separate failure modes. An agent may submit a command, lose its connection, and submit it again. The remote service may accept the command, execute it minutes later, and send two identical callbacks. Your receiver may verify the first callback but fail before it commits the resulting state. A tidy line in an application log cannot explain that chain.

I use six record types when I review these systems:

• An action record identifies the agent run, human authorization, requested operation, and intended target.
• An outbound attempt record identifies each HTTP transmission, including a request digest and the response received.
• A remote reference record captures any identifier the provider returns, such as a job or operation ID.
• A receipt record captures each inbound HTTP delivery before business processing changes anything.
• A verification record says exactly why the receiver accepted, rejected, or quarantined that delivery.
• A state transition record says what the workflow changed after it processed a verified event.

Do not merge an attempt with an action. One action can produce several attempts. Do not merge a receipt with an event either. One provider event can reach your endpoint several times. This distinction seems fussy until an engineer has to explain whether a second deployment came from the agent retrying, the HTTP client retrying, or the provider redelivering one event.

RFC 9110 defines POST in deliberately broad terms: the target resource processes the representation according to its own semantics. That is why a 202 response commonly means “accepted for later work,” and even a 200 only means the endpoint has completed its request handling. It does not certify a remote business outcome. If the provider exposes a separate operation status endpoint or callback, that later evidence owns the outcome.

A good record lets a reviewer read the story in order without inferring facts from timestamps:

agent session sess_7c1e authorized action act_01
act_01 created outbound attempt out_01 with idempotency ref idem_44
remote service accepted out_01 and returned operation op_903
receiver accepted delivery rcp_01 for provider event evt_775
rcp_01 verified its signature and linked evt_775 to op_903
workflow wf_18 moved from pending to completed

That is a chain of assertions, not a single status field. Each assertion needs its own source and time.

Correlation needs more than one identifier

A single correlation ID does not solve webhook tracing because different parties create IDs for different scopes. Use a small set of identifiers with explicit ownership, then record their relationships.

Start with an internal action ID. Create it before any network call, and attach it to the agent session, the requested operation, the authorization decision, and the immutable audit entry. This ID answers, “Which agent instruction caused this work?” It should not change when the client retries.

Create an outbound attempt ID every time your HTTP client transmits. This ID answers, “Which wire attempt got this response or error?” Include an idempotency reference when the remote API supports one. An idempotency reference says repeated submissions should map to one logical remote operation. It does not tell you whether a particular HTTP attempt reached the server.

When the remote service returns an operation ID, persist it immediately alongside the attempt that received it. If your request allows a client reference or metadata field, put your action ID in that field after confirming that the provider will return it in callbacks or status responses. Never put a secret, employee name, or full prompt into a reference field. Those fields tend to surface in vendor consoles, support tickets, and event payloads.

Incoming callbacks add two more IDs: the provider's event ID and your receipt ID. The provider event ID supports deduplication for that sender. Your receipt ID identifies the exact HTTP delivery received by your infrastructure, including its headers, source address if you keep it, raw-body digest, and verification result.

The relationship table should look like this:

CloudEvents is useful here, even when a provider does not send CloudEvents. Its specification separates id, source, type, subject, and time. That separation prevents a recurring mistake: treating an event ID as a workflow ID. An event ID identifies one event from one source. A workflow ID identifies the work you are tracing. They may point at the same remote object, but they do not mean the same thing.

If a provider gives you only a callback payload with an object ID, link it cautiously. Mark the link as exact only when the object ID came from your recorded outbound response or an authenticated status lookup. A match on email address, title text, amount, or timestamp is a guess dressed up as correlation. Keep it out of audit conclusions.

A 2xx response and a callback settle different questions

A 2xx response settles the HTTP exchange. A verified callback may settle a remote state change. Your workflow needs both, and it must describe the gap between them honestly.

Consider an agent that asks a hosted build service to publish an artifact. The service returns 202 and an operation ID. Your service records the request as accepted, then waits. Ten minutes later, a callback says the publication failed because a downstream repository rejected a required manifest. If the audit record switched to “success” at 202, the record now contradicts the provider's own evidence.

Use states that name the evidence you have. For example:

1. requested means the agent action passed authorization and created a work item.
2. submitted means at least one outbound attempt received an acceptance response or a recoverable ambiguous result awaits checking.
3. confirmed means a verified callback or authenticated status response established the intended outcome.
4. failed means authoritative evidence established failure.
5. unknown means you cannot yet establish whether the remote side acted.

The unknown state is necessary. Teams dislike it because it makes dashboards less pretty. I dislike silent duplication more. A timeout after sending a POST produces ambiguous delivery: the remote system may have received and processed it, or it may never have seen it. Retrying without an idempotency mechanism can create two remote operations. Calling the first attempt “failed” encourages exactly that mistake.

A late callback also does not automatically win. Suppose an agent requests cancellation after the initial command, and your internal workflow records a valid cancellation. A completion callback that arrives later may report what happened remotely before cancellation took effect. Retain it, verify it, link it, and record the conflict. Do not let a generic handler overwrite a terminal cancellation state merely because “completed” ranks higher in somebody's enum.

Write a transition rule for every callback type. A payment approval, build completion, user-provisioned event, and deletion confirmation do not deserve the same transitions. The rule should state which prior states permit the transition, what evidence the handler requires, and whether an operator must resolve a conflict.

The receiver must preserve evidence before parsing it

Your receiver should capture the raw delivery, verify it, and deduplicate it before it performs a side effect. Parsing JSON first and storing only selected fields destroys evidence when the parser, schema, or application code later turns out to be wrong.

At receipt time, record the following in a protected event store:

• The receipt ID and server receipt timestamp.
• Request method, route, selected headers, and a cryptographic digest of the exact raw body.
• The sender identity you expected and the verification scheme you applied.
• The provider event ID, if the payload supplies one, plus the parsed event type.
• The decision: accepted, duplicate, rejected, or quarantined, with a reason code.

Keep raw payloads only for the retention period your investigation and compliance needs justify. A digest is usually enough to prove two payloads match. If you retain a body, encrypt it, restrict access, and avoid copying it into ordinary application logs. Webhooks routinely contain personal data, repository metadata, addresses, and internal notes. An audit store that leaks the payload is a liability, not evidence.

Signature verification must operate on the body exactly as the sender signed it. A middleware layer that parses JSON, reformats it, and then verifies the reformatted bytes will reject legitimate deliveries or, worse, leave room for inconsistent handling. Read the provider's verification document closely. Some schemes sign timestamp + "." + raw_body; others sign just the raw body; others use asymmetric signatures and rotating public keys.

For a generic HMAC scheme that signs only the raw body, this command shows the digest shape you should expect from the unmodified bytes:

printf '%s' "$RAW_BODY" | openssl dgst -sha256 -hmac "$WEBHOOK_SECRET"
# SHA2-256(stdin)= 4d3c...hex digest...

This is a diagnostic, not a substitute for the provider's exact canonical string. If the provider includes a timestamp or version prefix, copying the generic command will give the wrong result. That error appears often because engineers verify a convenient approximation rather than the sender's documented algorithm.

Signature validity does not stop replay. If a sender provides a signed timestamp, reject deliveries outside a narrow window after allowing for measured clock skew. Then record provider event IDs in a durable deduplication store before you invoke downstream work. If you cannot rely on an event ID, deduplicate using a sender-scoped digest and an appropriate retention period, knowing that two legitimate identical events may then need special handling.

Return an HTTP response only after you have made the receipt decision durable. If you return success first and crash before the deduplication write, the sender can retry and your handler can process the same event twice. That bug hides in low-volume testing and appears under exactly the outage conditions where webhook traffic spikes.

Retries reveal where your records are too vague

Retries are normal behavior, not an edge case, and each layer can retry independently. Agents retry after a timeout. HTTP libraries retry a connection failure. API providers retry callbacks. Queue consumers retry a failed handler. An audit record that collapses these into “retry count: 3” does not help anyone.

Walk through a failure I have seen in several forms. An agent requests creation of a remote access record. The client sends a POST and times out after the bytes leave the machine. The remote service creates the record and queues a callback. The agent framework retries because it sees a timeout. The second request creates another record because the action layer generated a new idempotency reference on every attempt. Both callbacks arrive. The receiver uses only an email address to match them, decides they are duplicates, and suppresses the second one. The audit page shows one completed request. The remote service now has two access records.

Every component behaved in a plausible way. The system failed because it did not preserve a logical command across retry boundaries.

Fix the sequence instead:

1. Generate the action ID and idempotency reference once, before the first outbound attempt.
2. Record each attempt separately, including timeout and transport errors.
3. On ambiguity, query the provider by idempotency reference or client reference before issuing another command.
4. Accept every authenticated callback as a receipt, then deduplicate only the provider event ID, not the remote object itself.
5. Reconcile the expected number of remote objects against the recorded action before declaring the workflow complete.

The first idempotency check belongs at the sender, and the second belongs at the receiver. They solve different problems. Sender idempotency prevents duplicate remote commands. Receiver deduplication prevents repeated processing of one remote event. Teams often install one and assume they bought the other.

Do not use arrival time as your sort order for business truth. Providers can deliver events late or out of order, and your own queue can delay processing. Store at least three times: when your action service created the action, when your HTTP client sent the attempt or received its response, and when your receiver accepted the callback. Preserve the sender's claimed event time separately. A sender clock is evidence from that sender, not your clock.

Authorization must survive the asynchronous boundary

Human approval for an agent action should attach to the action itself, not to whatever callback happens later. A callback carries information about remote work. It should never quietly acquire the authority to trigger a new privileged operation because it shares a correlation field with an approved request.

This matters when callbacks can contain URLs, object names, user-controlled metadata, or instructions that an internal handler follows. A common bad design receives a “job completed” event and lets a generic automation worker fetch a result URL or run a follow-up command using broad credentials. The original agent approval covered submission of a job, not an open-ended set of actions embedded in an event.

Record the authorized action in concrete terms: actor session, requested endpoint or SSH command template, target scope, credential identity, approval result, and approval time. For every outbound call, point back to that authorization record. For every callback, point back to the action only after verification and correlation. That direction matters. An incoming request should not search your database for any convenient prior approval and borrow it.

Sallyport keeps agent credentials out of the agent process and records both agent runs and individual calls, which makes the outbound half of this evidence easier to preserve. The callback receiver still needs its own receipt and workflow records because an HTTP action journal cannot know whether a remote system later sent a valid event.

Use separate credentials for the two directions. The credential that authorizes your outbound API call should not normally verify inbound signatures, and the inbound verification secret should not authorize a callback handler to call arbitrary external APIs. Separate custody limits the damage when a receiver route, dependency, or log sink goes wrong.

Tamper evidence should cover the joins, not only the calls

An append-only record of outbound calls helps, but it does not prove the correlation decisions made afterward. An operator or application bug can attach the wrong callback to the wrong action without changing either original HTTP record.

Make correlation a first-class audit event. The event should include the action ID, receipt ID, basis for the link, actor or process that made the decision, and a digest of the fields used. Use explicit bases such as remote_operation_id_exact, client_reference_exact, authenticated_status_lookup, or manual_review. Do not write “matched” and leave the investigator to guess.

A hash-chained log can show that records have not changed after their creation, provided you protect the log's append path and retain checkpoints. It cannot prove the application made a correct decision at the time. That limitation is healthy to state plainly. Tamper evidence gives you a stable account of what your system recorded; it does not turn weak correlation into fact.

Sallyport's encrypted, hash-chained audit log can be checked offline with sp audit verify, even without a vault key. Use that kind of verification for the action records, then retain a comparable immutable reference from your workflow store to the relevant action and call identifiers.

For high-consequence workflows, add a reconciliation job that compares three populations: actions submitted, remote operations known to the provider, and callbacks accepted by your receiver. The job should create an exception record for an unpaired item instead of auto-closing it. A missing callback might mean a provider outage, a bad endpoint, a signature rotation failure, or a workflow bug. You need evidence before you need optimism.

Observability must let an investigator replay the decision

An investigator should be able to start from any identifier and reconstruct the workflow without privileged access to agent prompts or API secrets. Design the search paths before you ship the integration.

Starting from an action ID, the record should show the agent session, authorization, credential label, request shape after redaction, all attempts, remote references, linked receipts, and terminal workflow state. Starting from a provider event ID, it should show every delivery of that event, verification results, deduplication outcome, linked operation, and state changes. Starting from an internal business object, it should show the exact evidence that associated it with an agent action.

Use structured fields, not a single narrative string. A useful event contract can be copied into a schema review or log pipeline:

{
  "record_type": "callback_receipt",
  "receipt_id": "rcp_01J...",
  "received_at": "2025-03-08T22:14:31Z",
  "sender": "build-service",
  "provider_event_id": "evt_775",
  "event_type": "publication.finished",
  "raw_body_sha256": "4d3c...",
  "signature": {"scheme": "hmac-sha256", "result": "valid"},
  "correlation": {
    "action_id": "act_01J...",
    "remote_operation_id": "op_903",
    "basis": "remote_operation_id_exact"
  },
  "processing": {"deduplication": "new", "result": "completed"}
}

The body digest, verification result, and correlation basis do more work than a vague status: success. They let you test claims. If a provider disputes a callback, compare the preserved digest. If an engineer disputes a match, inspect the basis. If a duplicate caused side effects, inspect whether the receiver wrote its deduplication record before dispatching work.

Avoid logging authorization headers, bearer tokens, private keys, signature secrets, or full credential-bearing URLs. Redact query values when they contain sensitive data, but retain enough request identity to distinguish two targets. I have seen teams redact a URL into uselessness and then have no way to tell whether an agent contacted production or a test endpoint. Store a normalized host, route template, method, and a carefully scoped target identifier.

Build the trace before agents make asynchronous calls

You should define the identifiers, receipt rules, and state transitions before giving an agent an action that starts asynchronous work. Retrofitting them after a dispute is expensive because the missing evidence never existed.

Run one deliberate failure drill. Submit a harmless test action, force the client to time out after transmission if your test environment permits it, resend the same callback, send a callback with an invalid signature, and deliver a valid callback after the workflow enters a terminal state. Check that the audit record explains each outcome without a human filling gaps from memory.

If your system cannot answer “which authorized action caused this callback, by what exact evidence, and what did we do with it?” then it does not yet have an audit story for webhooks. It has two sets of logs that happen to share a clock.