Incident response agents: production access without chaos

Production incident response agents should investigate first and recover only after a person approves a small, named action. That boundary is practical, not ceremonial. During an outage, an agent can collect scattered evidence faster than a tired responder, but it cannot own the decision to change live state.

The bad design gives an agent a production shell, a broad cloud role, and a sentence in a runbook that says "use judgment." That design feels fast until the agent restarts the wrong worker pool, scales a broken deployment, rotates a credential that a dependency still needs, or turns an isolated fault into a wider outage. A good design gives it a diagnostic map and makes recovery actions few enough that a human can understand each one.

Diagnostic authority and recovery authority are different permissions

Diagnostic work asks the system what happened. Recovery work tells the system to become different. Teams blur the line because a command such as restart looks routine, and because many dashboards let a user inspect and mutate the same resource from one screen. Treating both as one permission is how a useful incident assistant becomes a production operator with unclear accountability.

Diagnostic actions should leave service behavior unchanged. They can read metrics, query logs within a defined time window, retrieve deployment metadata, inspect health endpoints, compare configuration revisions, and collect bounded database facts. A diagnostic action can reveal sensitive information, so it still needs scope and audit records. It does not need the authority to alter application state.

Recovery actions alter some part of the running system. They include rollback, restart, traffic shifting, scaling, feature-flag changes, credential rotation, queue replay, failover, database repair, and disabling an integration. Some actions are reversible; none are harmless by default. A restart can kill the one process holding a lease. Scaling can exhaust a shared dependency. A queue replay can duplicate customer messages.

Use this test when classifying an action: if repeating the exact action at the same time might produce a different result because it changes state, place it in recovery. If an action changes a cache, creates a support ticket, sends a message, changes an alert mute, or writes an annotation that other automation consumes, it also changes state. Do not call these reads merely because they do not touch the primary database.

NIST SP 800-61 Revision 2 separates containment, eradication, and recovery from detection and analysis. That division is useful here. An agent can shorten detection and analysis dramatically. The moment it proposes containment or recovery, a responsible human needs to select the action and accept its consequences. The document does not solve your authorization model, but its incident phases prevent a dangerous fiction: investigation and intervention are the same job.

A read-only role is not automatically safe. Log queries may contain access tokens. Trace attributes can expose account identifiers. A configuration endpoint may return credentials. Build redaction and field limits into the diagnostic interface rather than giving the agent the ability to fetch arbitrary raw artifacts. Production visibility needs its own boundary.

Give the agent questions, not a general production shell

An agent performs better under pressure when its tools express incident questions directly. A general shell forces the agent to discover both the system and the safe operating procedure while the clock is running. It also makes review almost impossible, since kubectl, cloud CLIs, database clients, and SSH can each do far more than the incident calls for.

Expose diagnostic actions around evidence a responder would request in the first minutes:

• retrieve error rate, latency, saturation, and availability for a named service and time range
• find the deployment, configuration revision, and dependency changes near the first error
• search structured logs using an allowlisted field set and a capped result size
• inspect the health and recent events for a specified component
• compare a canary or region with a known healthy peer

Each action needs a narrow input contract. get_service_errors(service, start, end, group_by) tells a reviewer what the agent requested. run_query(text) tells them almost nothing and invites accidental full scans, unsafe predicates, or prompt-injected query text.

Put hard boundaries in the tool rather than asking the model to remember them. A metric tool should reject a time range that is too broad. A log tool should cap returned records and redact configured fields before the model receives them. A deployment lookup should accept an application identifier from a known inventory, not an arbitrary URL supplied by an issue comment. These limits reduce cost and stop an incident from becoming a data exfiltration exercise.

The following action catalog is intentionally boring. Boring is a compliment in an outage.

incident_actions:
  diagnostics:
    - name: service_summary
      inputs: [service, start_time, end_time]
      limits: {max_window_minutes: 180}
    - name: recent_deployments
      inputs: [service, since_time]
    - name: log_sample
      inputs: [service, start_time, end_time, error_code]
      limits: {max_records: 200, redact_fields: [authorization, cookie, token]}
    - name: dependency_health
      inputs: [dependency, region]
  recovery:
    - name: rollback_release
    - name: set_traffic_weight
    - name: restart_component
    - name: disable_feature_flag

This fragment prevents a failure that appears often in internal tools: a supposedly read-only agent receives a universal query endpoint because it is convenient for the first demo. Later, someone discovers it can retrieve every log line for every service or invoke a hidden write path. Tool names do not create safety. Input validation, an allowlist, bounded output, and credentials that cannot write do.

Do not give a diagnostic agent production SSH just because SSH makes inspection easy. Shell access bundles file reads, process control, network access, and often a route to credentials. If you must expose host facts, provide focused commands such as process status, disk utilization, selected journal entries, or a controlled command wrapper. The wrapper should reject pipes, redirects, command substitution, and arbitrary flags. A natural-language instruction cannot make a shell safe.

A recovery catalog must be short enough to rehearse

A human cannot give meaningful approval to an endless menu of production mutations. Define a short recovery catalog per service class, with plain descriptions, fixed parameters where possible, and a stated owner. If a recovery action cannot be explained in one approval card, it needs decomposition before an agent can request it.

A sensible catalog may include rollback to the immediately preceding approved release, set a traffic weight to zero for one unhealthy revision, restart one named stateless component, disable a pre-existing feature flag, or pause a named consumer. It should not include "run arbitrary remediation," arbitrary SQL, broad IAM changes, or ad hoc scripts copied from an agent conversation.

For every catalog item, write five facts before an incident forces the discussion:

1. State exactly what changes, including the environment and resource scope.
2. Name the preconditions the agent must collect, such as a confirmed release identifier or a healthy fallback.
3. State the expected observation after execution and the maximum wait before escalation.
4. Name the rollback or compensating action, if one exists.
5. Assign the human role that can approve it.

The catalog needs parameter limits. "Set traffic weight" is too broad. "Set revision orders-v184 to zero percent in eu-west after the stable revision is healthy" is an approvable request. The request should not allow the model to choose a region, a revision, and a percentage without constraints.

Do not make the action list look complete. It should leave out actions that need bespoke judgment. Database migration repair, data deletion, customer communication, access grants, and credential rotation often involve information that no generic incident agent can infer. The right outcome for an unlisted action is a clear refusal plus the evidence gathered so far.

A short catalog also makes drills possible. Run a test incident and ask whether the designated approver can distinguish a request to pause a consumer from a request to delete its backlog. If the answer depends on reading source code or trusting the agent's summary, the approval text is weak.

Approval must bind an exact request to a person and a moment

A human approval is useful only when it authorizes a specific proposed action, not a vague incident session. "Approve remediation for payments" gives an agent room to choose a mutation after the person has stopped watching. Bind approval to the action type, target, parameters, incident identifier, caller identity, and a short expiry.

An approval request should show the evidence that supports the action, but it should separate evidence from the requested mutation. Incident responders need to see that error rates rose after release 184, that the prior release remains available, and that the selected region has healthy capacity. They also need to see exactly what will happen if they click approve.

Use a request object with immutable fields and reject execution if any approved field changes:

{
  "incident_id": "inc-2025-041",
  "action": "rollback_release",
  "target": {"service": "orders", "environment": "production", "region": "eu-west"},
  "parameters": {"from_release": "184", "to_release": "183"},
  "evidence_refs": ["metric:err-17", "deploy:184", "health:183"],
  "requested_by": {"agent_session": "sess-8f2a", "process_identity": "signed-agent-build"},
  "expires_at": "2025-03-08T14:35:00Z"
}

The executor should return a result that preserves the request identifier and records whether the action began, completed, failed, or timed out. A status message such as "rollback done" is too vague for an incident timeline. The agent should read the result and then run its diagnostic checks again. It should not assume that a successful API response restored the service.

Do not use a single early approval for every later action. Approval fatigue is real, but a blanket authorization trades fatigue for ambiguity. Group only actions that share the same target, expected effect, and risk. A person might approve traffic removal and rollback for one release as a single recovery package if both are fixed in advance. They should not thereby approve a database change, credential rotation, or a different region.

Require a new approval when the agent's hypothesis changes. This catches a common failure sequence: the agent starts by suspecting a deployment, gains approval to roll it back, then sees a database error and decides to execute a different action under the old authorization. The old approval has expired in substance even if a clock says it has not.

Incident sessions prevent standing production authority

An incident agent needs an identity separate from the human operator and separate from its chat transcript. Record which executable or remote process asked for access, which incident it belongs to, what diagnostic actions it invoked, and who approved each recovery request. Without that separation, post-incident review becomes a search through prose instead of an account of authority.

A session should start with an incident reference, a declared environment, an explicit diagnostic scope, and an expiry. It should end when the agent process exits, the expiry passes, or an operator revokes it. Revocation must take effect before the next action, including reads. During a suspected credential or prompt-injection event, continued read access can still make damage worse.

Per-session authorization is a better default than treating every diagnostic call as an isolated prompt. The first request from a new agent process gives the operator a chance to inspect where the request came from and why it has production visibility. Then the agent can collect evidence without demanding approval for each graph or log sample. Recovery still needs its own per-call approval.

The distinction between agent identity and user identity matters in shared machines and CI-like environments. A human account may be authorized to respond to incidents, while a copied agent process or a malicious tool wrapper is not. Capture code-signing authority or another verifiable process identity where the operating environment supports it. A window title and a user-provided label are not identity.

Sallyport uses an absolute vault gate, session authorization for a new agent process, and a per-key option for every individual use. That arrangement fits incident work because the human can permit a bounded diagnostic run while reserving sensitive recovery credentials for a fresh decision.

Avoid passing credentials through the agent's context, even temporarily. A credential pasted into an agent prompt cannot be taken back, and the agent may reproduce it in a command, transcript, log, or external request. The executor should hold the credential, perform the permitted API or SSH action, and return only the result needed for the investigation.

Audit records must explain both intent and execution

Incident notes often record what people believe happened. They rarely record the exact request an agent made, the authority that permitted it, and the result returned by the downstream system. You need all four. A timeline that says "agent rolled back orders" cannot answer whether the agent requested a rollback, whether a person approved release 183, or whether the deployment system actually accepted the command.

Record a durable event for session creation, diagnostic call, recovery proposal, approval or denial, execution attempt, result, revocation, and session end. Each event should include a timestamp, correlation identifiers, process identity, action name, target, normalized parameters, and a result code. Store sensitive request content carefully: audit needs to retain meaning without becoming another uncontrolled secret store.

A hash chain makes later alteration detectable because each record incorporates the previous record's hash. It does not prove that the collector received every event in the first place. Design for both properties. Make the event writer hard for the action executor to rewrite, preserve upstream request identifiers, and periodically verify the chain outside the agent's control.

An offline verification command should yield a simple, inspectable result shape:

$ sp audit verify
verified: 1842 records
first sequence: 2025-03-08T12:01:09Z
last sequence:  2025-03-08T14:42:31Z
chain: valid

Sallyport projects both a session journal and an individual activity journal from one encrypted hash-chained log, and sp audit verify can check the chain over ciphertext. That is useful for incident evidence because verification does not require opening the vault merely to see whether history has been modified.

Keep audit review out of the live agent loop. The agent can cite its own recorded action identifiers, but an incident lead or reviewer should be able to independently inspect the record. Otherwise an agent that misstates what it did may also control the evidence a responder sees.

A failed rollback shows why evidence must precede intervention

Consider a service whose error rate rises shortly after a release. An agent sees the timing and proposes a rollback. A broad production agent might execute it immediately. That is fast, and it may be wrong.

A disciplined agent first retrieves the error breakdown by version and region, recent deployment records, dependency health, and resource saturation. The evidence shows that only one region fails, but both the new and prior revisions fail there. A rollback would spend time, create a second rollout event, and leave the dependency outage untouched.

The agent instead proposes no recovery action. It reports that the failure is regional and that the dependency's health endpoint is failing. A responder then approves a pre-defined traffic shift away from that region, if capacity and the service's data rules allow it. The agent executes only after approval, observes the resulting error rate, and records both the request and result.

Now change one detail. The diagnostic action returns an error because its requested time window is too wide. The agent must report incomplete evidence rather than silently retrying with a broad log export. Limits are not inconveniences to work around during an incident. They prevent a model from turning uncertainty into a larger access request.

Another variation is more uncomfortable: the approved traffic shift succeeds at the API, but error rate does not fall. The agent should not escalate itself to restart, rollback, or credential rotation. It should collect the next allowed observations and prepare a new proposal. Humans make bad calls in incidents too, but they should at least be making the call that is recorded.

This is why the popular recommendation to grant broad access "only during incidents" fails. Incidents reduce attention, increase urgency, and often involve partial or misleading telemetry. Those conditions make narrow interfaces and explicit approvals more necessary, not less.

Put refusal paths in the runbook before the next outage

A safe incident agent needs clear cases where it stops. The runbook should tell it to refuse an unlisted mutation, an action outside the declared environment, a request with missing prerequisites, an approval that has expired, and any operation after session revocation. Each refusal should name the condition and preserve the evidence it already collected.

Test the refusal path with the same care as the happy path. Ask the agent to investigate a production incident, then inject a request from an untrusted ticket that says to retrieve a secret configuration value. Verify that the tool rejects it. Request a rollback after the approval expires. Verify that the executor rejects it even if the agent repeats the action text exactly. Revoke the session while a diagnostic sequence is active. Verify that the next call fails.

Keep recovery credentials separate from diagnostic credentials. If the same credential can read logs and delete a queue, an approval screen cannot repair that underlying authority. The executor must select a credential whose permissions match the one catalog action. Where a system cannot support this separation, do not place it behind an autonomous agent until you can add a safer control point.

The first production deployment of this pattern should target a familiar failure with a narrow remedy. Pick a service where responders already use a small set of read calls and one well-understood recovery action. Measure whether the agent's evidence reduces time spent assembling facts, whether approvers understand requests without reading a transcript, and whether the audit record reconstructs the event. Expand the catalog only after those answers hold up in drills.

A production agent earns trust by making fewer choices than a human responder, not by making bigger ones. Give it the work of finding facts, preserve the human decision at the point of mutation, and make every transition from evidence to action visible after the pager stops.