# How AI incident acknowledgement should preserve open incidents

An agent that acknowledges a page has made one useful claim: someone or something has accepted responsibility for investigating it. The agent has not proved that customers recovered, the alert condition cleared, or the next responder can stand down. Any integration that turns acknowledgement into closure destroys that distinction and writes fiction into the incident record.

Treat incident actions as separate state transitions with separate permissions. Acknowledge, resolve, suppress, and route may sit beside one another in a pager UI, but they answer different operational questions. An agent should call the narrowest action, a human should approve the consequential transition when policy requires it, and the audit trail should name both actors.

## Acknowledgement records ownership, not recovery

Acknowledgement means that a responder has taken the page and started investigating. It changes who is expected to act and often changes escalation behavior. It says nothing about whether the fault still exists.

PagerDuty's Incidents documentation makes the distinction explicit: an acknowledged incident is being worked on but is not yet resolved. Acknowledgement claims ownership and halts escalation until an acknowledgement timeout; resolution means the issue has been fixed. That timeout matters. If nobody resolves the incident before it expires, PagerDuty can return the incident to triggered status and resume escalation. An agent that closes the incident on acknowledgement silently removes that safety mechanism.

Google Cloud Monitoring uses slightly different terms but preserves the same boundary. Its documentation defines Acknowledged as an open incident that somebody has manually marked while investigating. It also says acknowledgement does not stop repeated notifications. A snooze or policy change controls those notifications, while closure follows recovery observations, a manual close, or an automatic close condition. The product behavior is a useful warning against assuming that all pager verbs have the same side effects.

The practical contract should be plain: `acknowledge` records an assignee, an acknowledgement time, and the actor that accepted the work. It may pause an escalation path if the pager supports that behavior. It must leave the incident's recovery state open, preserve active alerts, and avoid changing notification or routing rules unless the caller requests another action through another control.

This boundary also protects incident metrics. If acknowledgement closes a record, time to acknowledge and time to recover collapse into one timestamp. Teams then reward fast button presses while losing the measure that tells them how long users actually suffered. Postmortems inherit the same error because the timeline claims recovery before mitigation happened.

## One agent verb must map to one transition

Each agent tool should expose one operational verb and one primary effect. Bundled actions sound convenient during a calm design review, then become impossible to reason about during a failure.

Acknowledge answers who accepted the work. Its primary effect is to record ownership and acknowledgement time, and it must not imply service recovery or alert clearance. Resolve answers whether the incident ended. It closes the record after recovery evidence and must not mute future pages or change ownership as a side effect.

Suppress answers whether a selected signal should create or notify an incident. It prevents selected notifications or incident creation for a defined scope, but it does not recover an incident that is already open. Route answers who should receive or own the work. It changes a service, team, escalation path, or assignee without claiming that the destination acknowledged or resolved anything.

These definitions form an authorization boundary. A human may be comfortable letting an agent acknowledge any page assigned to its team, yet require a person to approve resolution. The same person may allow suppression only during a declared maintenance window and routing only among two services. One broad `manage_incident` capability cannot express those choices without hidden branches.

Avoid a tool such as `handle_page` with flags for `ack`, `mute`, `assign`, and `close`. It lets the model choose a bundle based on prose, and a single misunderstood phrase can select several mutations. Separate tools force the caller to state intent and let the gateway authorize each transition on its own.

The separation should survive vendor adapters. If one provider calls the quieting action a snooze and another calls it downtime, the adapter can map those names to a common `suppress_notifications` intent. It should not pretend that suppression and resolution are interchangeable merely because both can make a pager quiet.

## A safe API makes illegal combinations unrepresentable

A narrow request schema prevents more incidents than a clever prompt. The request below can acknowledge one existing incident and cannot smuggle in a close, mute, or reassignment operation.

```json
{
  "operation": "incident.acknowledge",
  "incident_id": "inc_01JQ8K4Q6M",
  "expected_version": 17,
  "actor": {
    "type": "agent",
    "session_id": "ses_01JQ8JY2P3"
  },
  "reason": "Accepted investigation after the database latency page"
}
```

The server, not the agent, should add identity facts it can verify. A model can supply a reason, but it should not be trusted to declare its own executable hash, code-signing identity, user account, or approval result. The gateway derives those values from the authenticated process and the approval channel.

A successful response should describe the exact transition and the state that remains:

```json
{
  "operation_id": "op_01JQ8K5D7A",
  "incident_id": "inc_01JQ8K4Q6M",
  "transition": "triggered_to_acknowledged",
  "incident_status": "acknowledged",
  "recovery_status": "open",
  "version": 18,
  "approved_by": "usr_2048",
  "approved_at": "2026-07-24T02:14:31Z"
}
```

Returning `recovery_status: open` may look redundant. Keep it. Agents often reason from the last response they received, and explicit state is cheaper than asking a model to infer vendor semantics. The response also gives an orchestrator a stable assertion for a test: acknowledgement succeeded and recovery stayed open.

Do not accept contradictory convenience fields such as `resolve_if_healthy`, `mute_for`, or `route_to` on the acknowledgement request. Those fields turn one endpoint back into a bundle. If the workflow needs a second action, it should issue a second request, receive a separate decision, and leave a separate audit event.

## Approval belongs to the transition

Approval should authorize one described state change, not bless an agent in the abstract. A session approval can establish that a known process may run, but it should not automatically authorize every incident mutation that process discovers later.

For each transition, record the proposed operation, target incident, state observed before the decision, requested scope, requesting agent session, approving human, decision time, and approval method. Add the state version or provider event identifier that the approver saw. Without that binding, an approval card displayed for incident A can be replayed against incident B, or an approval granted while an incident was triggered can execute after somebody else resolved it.

The approval screen should use operational language. `Acknowledge database latency incident inc_01JQ8K4Q6M and assign investigation to Payments Duty` is reviewable. `Allow agent action` is not. For resolution, show the recovery evidence the agent cited and state what remains active. For suppression, show the exact signal scope, duration, and expiry. For routing, show both the current and proposed destination.

Capture three identities instead of flattening them into a vague `actor` field:

- The requester is the agent session that proposed the change.
- The approver is the person who authorized it, when approval is required.
- The executor is the gateway or integration identity that called the provider.
- The subject is the incident, alert, service, or route that changed.

These identities answer different questions during review. The requester explains why automation started. The approver establishes human authority. The executor ties the event to credentials and provider logs. The subject prevents a valid decision from drifting to a different resource.

Sallyport can put HTTP API and SSH actions behind a per-call approval when a stored key requires it, while the credential stays in its encrypted vault and never reaches the agent. That mechanism fits incident transitions because the approval can bind to the concrete provider request rather than a free-form promise from the model.

## Races turn plausible automation into a false timeline

Incident state changes while an agent reasons, waits for approval, or retries a network call. A design that ignores that delay will eventually acknowledge a resolved incident, route work away from an active responder, or apply an old suppression after maintenance has ended.

Use optimistic concurrency. The agent reads version 17, proposes an acknowledgement against version 17, and the gateway executes only if the provider record still matches the relevant state. If another responder changes the incident first, return a conflict that names the mismatch instead of silently applying the operation to the new state.

```json
{
  "error": "state_conflict",
  "incident_id": "inc_01JQ8K4Q6M",
  "expected_version": 17,
  "actual_version": 19,
  "actual_status": "resolved",
  "retryable": false
}
```

Do not teach the agent to retry every conflict. A transport timeout may justify an idempotent retry with the same operation identifier. A state conflict demands a new read and usually a new decision. If the incident is already resolved, an acknowledgement is obsolete. If it was routed to another team, the original approval may no longer cover the destination.

A common failure starts with a slow approval. An agent reads a triggered database page and asks Alice to approve acknowledgement. While the card waits, Bob mitigates the issue and resolves the incident. Alice then approves the stale card. A naive adapter sends `acknowledge`, receives a provider-specific success or coerces the incident back into an active state, and records Alice as the owner of work that has ended. Version binding stops the action before the timeline becomes nonsense.

Idempotency solves a different problem. Give every proposed transition an immutable operation ID and persist its result. If the gateway loses the HTTP response after the provider applied the acknowledgement, a retry returns the stored result instead of adding a second timeline entry or sending another notification. Concurrency guards intent against changed state; idempotency guards one intent against duplicate execution.

## Routing and suppression need their own permissions

Routing changes who bears responsibility, while suppression changes which signals create noise. Neither action proves recovery, and neither should hide inside acknowledgement.

Route before acknowledgement when the current service is plainly wrong and nobody has accepted the work. Route after acknowledgement only with care because a responder may already be investigating. The transition should state whether ownership moves, whether the original responder remains subscribed, and whether the destination's escalation policy starts. An agent should never infer those effects from a team name.

Suppression needs an explicit scope and expiry. PagerDuty's Event Management documentation says suppressed alerts remain available for forensics but do not create incidents. Its Event Orchestration documentation also describes routing unmatched events to a service or suppressing them through a catch-all path. Those are ingestion decisions, not synonyms for closing an existing incident.

Datadog's Downtimes documentation draws another useful line: downtime silences alerts and notifications but does not prevent monitor state transitions. When downtime ends while the monitor still reports an alert state, notification can resume. That is the behavior operators usually need during maintenance. The system remembers the unhealthy state instead of rewriting it as healthy merely to stop a notification.

A suppression request should therefore include the signal selector, start and end times, reason, creator, and behavior at expiry. Broad selectors such as an entire service deserve stronger approval than one monitor group. An indefinite suppression should be rejected or require an explicit exception path. The agent must not create a quiet failure that nobody owns.

Do not automatically suppress after acknowledgement just because the responder has accepted the page. Some systems pause escalation on acknowledgement, while others keep repeated notifications running. Google Cloud Monitoring explicitly documents that acknowledgement does not stop repeated notifications. Preserve the provider's behavior or request suppression separately so reviewers can see why the noise stopped.

## Resolution must follow evidence, not agent confidence

Resolve only when recovery evidence meets a declared condition. A successful remediation command is evidence that an action ran; it is not proof that the service recovered.

This distinction catches a familiar automation failure. An agent restarts a process, receives exit code zero, and closes the incident. The process starts, fails its readiness check, and crashes again thirty seconds later. The command succeeded while the service never recovered. Closing on command success creates two incidents, two pages, and a misleading recovery time instead of one continuous event.

Define resolution evidence near the incident type. An availability incident might require the alert condition to clear and remain clear for an evaluation window. A queue incident might require backlog and oldest message age to fall below their thresholds. A certificate page may resolve only after the deployed endpoint presents the expected certificate, not when a file was written to one host.

The agent can gather evidence and propose resolution. The gateway should attach the observations, their timestamps, their source, and any gaps to the approval. If the monitor still reports an active condition, reject the close. Google Cloud Monitoring takes this stance for metric incidents: its documentation reports an `Unable to close incident with active conditions` error when recent data still violates the policy.

Manual resolution still has a place. Monitors can lag, telemetry can fail, and a responder may know that the affected service was intentionally retired. Make the override explicit with a reason and an approver, and preserve the last unhealthy observation. An override is an accountable exception, not a reason to weaken the normal transition.

Reopening behavior also belongs in the contract. State whether a recurring signal reopens the same incident or creates a new one, and preserve correlation identifiers either way. An agent must not assume that `resolve` mutes the next event. PagerDuty says a resolved incident can be reopened if more work is needed, while Datadog documents that a manual monitor resolve only sets status to OK until the next evaluation. The next unhealthy evaluation can alert again.

## The audit trail must preserve the decision

Provider activity logs show that an API credential called an endpoint. They rarely capture enough context to explain what an agent proposed, what a person approved, and what state the person saw. Keep a separate decision record and link it to the provider event.

A complete transition event contains the immutable operation ID, request payload hash, normalized action, target, before and after state, requester identity, approver identity, executor identity, approval method, decision timestamps, provider response identifier, and result. Record rejected and expired proposals too. A denied suppression tells a future reviewer why notifications continued, and an expired approval explains why a proposed acknowledgement never ran.

Protect the sequence against quiet editing. Append-only storage, restricted writers, retention controls, and cryptographic chaining address different threats. Sallyport projects its Sessions and Activity journals from one encrypted hash-chained audit log, and `sp audit verify` can check the ciphertext chain offline without the decryption key. That gives an incident integration evidence about both the agent run and each external call without exposing the credential to the agent.

Do not store only the agent's prose explanation. Preserve structured facts beside it because models can produce confident but inaccurate summaries. A reason such as `latency page accepted` helps a human scan the timeline; the incident ID, state version, HTTP request hash, and approval identity let an investigator verify it.

Audit readers need stable semantics as the integration evolves. Version the event schema and normalized action names. If an adapter changes a provider mapping, record the adapter version that executed each call. Otherwise an old `acknowledge` event can become ambiguous after a new release changes its side effects.

Access to the audit trail should not imply access to incident credentials. Separate the ability to verify event order from the ability to decrypt sensitive payload fields. Redact secrets before they enter the record rather than hoping every reader handles them safely later.

## Test transitions as hostile workflows

Happy-path tests prove that an endpoint works when nothing interesting happens. Incident automation needs tests that interrupt the workflow between every read, approval, call, and response.

Start with five invariants:

1. Acknowledgement never changes recovery from open to closed.
2. Resolution never creates or extends a suppression.
3. Routing never implies that the destination acknowledged the incident.
4. Every approved mutation names the requester, approver, executor, and subject.
5. A stale approval cannot execute against a different state version or target.

Then run the adapter against a fake provider that can change state between calls. Resolve the incident while an acknowledgement waits for approval. Change the route before a suppression executes. Return a timeout after accepting a request, then send the same operation ID again. Revoke the agent session after approval but before execution. Each case should produce a deterministic result and an audit event.

Test provider differences instead of erasing them. One pager may halt escalation on acknowledgement; another may continue repeated notifications. Your normalized response can expose `escalation_paused` and `notifications_suppressed` as separate facts. Do not promise a universal side effect that the adapter cannot verify.

Review the tool descriptions with the same suspicion as the code. `Take care of this incident` invites the model to choose an outcome. `Record that the Payments Duty team accepted investigation; keep recovery open` describes a bounded transition. Tool text is part of the control surface because it shapes which request the agent attempts.

Finally, test the absence of authority. An agent with acknowledge permission should receive a clear denial when it calls resolve, route, or suppress. The denial should not offer an automatic fallback that performs a different mutation. Safe failure leaves the incident visible and unchanged.

## Agent authority should shrink as consequences grow

Permission design should follow the effect of each action, not the convenience of the workflow that calls it. Reading an incident, acknowledging assigned work, moving responsibility, silencing a signal, and declaring recovery deserve progressively tighter grants when their consequences differ.

Start with capabilities that name both verb and scope. `incident.acknowledge` for the Payments service is narrower than `incident.write` across production. A route grant can limit destinations to services owned by the same group. A suppression grant can limit selectors and duration. A resolve grant can require an approved evidence profile. The gateway should evaluate these restrictions from structured request fields, never from the agent's explanation.

Time and process identity belong in the grant. An agent session that started for one coding task should not retain incident authority after the process exits. If a person revokes the session while an approval waits, execution should fail even though the earlier decision was valid at the time. Approval confirms a proposed transition; it does not revive a requester whose authority has ended.

Separate permission from approval. Permission answers whether this requester may attempt this class of action. Approval answers whether a specific attempt may proceed now. Requiring approval cannot repair an overbroad permission because reviewers eventually develop approval fatigue, especially when cards hide scope. Likewise, a narrow permission does not replace approval for a consequential transition if the organization requires a human decision.

Use stricter treatment for actions that reduce visibility. Acknowledgement adds an owner and leaves the fault visible. Routing can move the page outside the current team's view. Suppression can stop people from hearing about continuing failure. Resolution can remove the incident from active queues and alter performance reports. That ordering is not universal, but writing it down exposes disagreements before an agent encounters them at night.

Credentials should match the same boundaries. If the provider offers distinct roles or tokens, do not give the executor an account that can administer schedules and services merely to acknowledge an incident. When the provider exposes only broad credentials, enforce the narrow operation at the gateway and make that limitation visible in the threat model. The provider audit will show what the credential could do; the gateway record must show what it was allowed to do in this request.

Revocation needs a predictable result. Revoke the agent session to block every new operation from that process. Revoke an approval to block the bound operation if execution has not started. Revoke a route or suppression grant to stop later requests, while handling existing suppressions through an explicit cancellation transition. Do not silently delete the evidence that the earlier decision existed.

Teams often propose one human approval at the start of an incident so the agent can handle the rest. The idea is popular because repeated prompts interrupt responders. It is still wrong for mixed consequences. Approve the agent session once for routine access, then reserve transition approval for actions such as broad suppression, cross-team routing, and resolution. That keeps low-risk work moving without turning the first hurried click into authority to close the incident an hour later.

Before deployment, write an authority matrix with actions as rows and scopes as columns. For every cell, decide whether the agent may read, propose, execute without a new decision, or execute only after approval. Include the maximum suppression duration, allowed route destinations, acceptable resolution evidence, and what happens when the approver does nothing. An empty cell should deny the action rather than inherit permission from a broader label.

Exercise that matrix with the actual provider credential. A policy that denies resolution in the gateway is incomplete if another exposed HTTP tool lets the agent call the provider endpoint directly. Inventory every route to the pager, including generic request tools, command runners, webhook relays, and stored scripts. Credentials that bypass the gateway should not be available to the agent process.

Approval prompts also need rate controls, but rate limiting must fail safely. If an agent floods the reviewer with repeated acknowledgement requests, coalesce identical proposals or expire the extras. Do not respond to approval overload by auto-approving, auto-resolving, or hiding the incident. Record the excess requests so the team can fix the loop that produced them.

## A quiet pager can still describe an open incident

Pager noise, responder ownership, service health, and team routing are independent dimensions. Compressing them into a single status makes the interface look simpler while moving ambiguity into automation, metrics, and postmortems.

Keep acknowledgement narrow even when an agent performs it perfectly. Let it claim the work, record who approved that claim, and preserve the open incident. If the workflow also needs a new route, a temporary suppression, or a resolution after verified recovery, make each one visible as its own request and decision.

That design costs a few more tool calls. It buys a timeline an incident commander can trust at 2 a.m.: who took the page, who allowed the transition, what the system showed at that moment, and why the incident finally closed.