AI agent access pilot: metrics that earn expansion

A pilot should prove that people can see, judge, and stop an AI agent's external actions before they give that agent broader authority. It should not merely prove that an agent can call an API, open an SSH connection, or finish a ticket faster.

Teams often collect a pile of event counts, announce that nothing caught fire, and expand access. That is not evidence. The useful measurements answer four harder questions: did humans approve requests they understood, did controls stop the right requests, did the agent keep failing in ways that signaled a design flaw, and how quickly could someone cut off a live run?

An AI agent access pilot needs a small set of measurements tied to decisions. Track approval rate, blocked requests, repeated failures, and time to revoke, but refuse to treat any of them as a score by itself. Each metric can mislead you when you remove the surrounding request, target, identity, and outcome.

A pilot earns expansion through control evidence

An AI agent access pilot is ready to expand only when the team can explain a sample of allowed, denied, and failed actions, then contain a live session on demand. Output quality belongs in the evaluation, but it cannot substitute for control evidence.

Define the proposed expansion before the pilot begins. Write down the current authority, the next authority under consideration, and the external consequences of that change. "Let the coding agent use our issue tracker" is not a scope. "Allow this signed local agent process to create issues in one test project during developer initiated runs" is a scope. The difference gives reviewers something they can judge.

Keep the first pilot deliberately narrow. Limit the agent processes, the human owners, the target systems, the credential paths, and the action types. Narrow scope is not bureaucratic caution. It lets you tell whether a surprising request comes from an unclear task, a broken integration, a confused agent, or access that was too broad from the start.

A useful pilot record answers these questions for every action:

• Which agent process made the request, and who started it?
• What action did it ask to perform, against which named target?
• Which access path or credential class authorized it?
• Did a person allow or deny it, and why?
• What result came back, including an external error if one occurred?

Do not reduce that record to "agent used tool." A request to read an issue and a request to change deployment settings may both use HTTP, but they create different risks and demand different review.

NIST SP 800-53 Rev. 5 places audit review, analysis, and reporting in control AU-6. The important word is "review." Retaining records does not meet the practical purpose if nobody can reconstruct why a request happened, who approved it, and whether it succeeded. For an agent pilot, audit data must support a decision about access expansion. If it cannot, it is storage, not evidence.

Set a review cadence that the owners will actually keep. A short review after each active day often works better than a large end of month session, because the task and surprises remain fresh. The person who approved a strange request can still explain the context. The engineer who owns the integration can still reproduce the failure.

Do not set a target number for "safe enough" before you have any baseline. Set decision rules instead. For example, no expansion occurs while reviewers cannot attribute requests to an initiating process, while high consequence actions lack a tested revocation path, or while repeated failures remain unexplained. Those rules force the conversation toward control quality rather than a flattering dashboard.

Approval rate measures review quality, not trust

Approval rate tells you whether requests receive a yes, but it does not tell you whether those yeses were informed. Calculate it as approved requests divided by requests that required a human decision. Keep expired prompts, abandoned requests, and automatic denials in separate fields rather than quietly dropping them.

A high rate can mean the agent requests sensible work in a well bounded environment. It can also mean that reviewers click through prompts because they are busy, the prompts omit the target, or the same request appears so often that people stop reading. I have seen approval systems become ceremonial within days when every harmless call asks the same question. The team then treats the one dangerous prompt exactly like the previous twenty harmless ones.

Read approval rate in slices. At minimum, separate action type, target class, session, and reviewer. If you combine read operations with account changes, or a test environment with production, the average hides the part that needs attention. A 95 percent approval rate for reads tells you almost nothing about whether people properly inspect the remaining five percent of requests that can alter state.

Use a small weekly sample review. Choose several approvals, including one that occurred late in a session and one involving a sensitive target. Ask the approving person to answer four questions from the preserved record:

1. What did the agent intend to do?
2. Which process issued the request?
3. What external system would receive it?
4. Why was approval appropriate at that moment?

If the person needs to consult chat history, terminal scrollback, or a second system to answer basic questions, the approval display is missing context. If they cannot answer even with that material, the access design is asking them to certify work they do not understand.

Separate session authorization from approval of a sensitive action. Session authorization answers whether a newly started process may act within an agreed boundary. A per action decision answers whether this particular operation deserves attention. Teams often blur those two decisions and then claim they have "human in the loop." They may have only approved a process once, without any usable way to notice that its work changed character halfway through a run.

The inverse error is just as common: requesting approval for every low consequence action. That design produces a healthy looking count of human decisions and a useless review habit. Reduce repetitive prompts by narrowing or restructuring the access path, not by teaching people to approve them faster.

Record the denominator honestly. If an agent obtains authority outside the approval path, those calls belong in the pilot record as an exception, even if the call succeeded. A metric that excludes bypasses makes the controls look better by definition.

A denial needs a cause and a verified outcome

Blocked requests show whether the boundary catches work that falls outside the pilot, but only if you classify the reason for the block. Treating all denials as wins is one of the easiest ways to learn nothing.

Use a cause taxonomy that matches an operator's next action. Four categories cover most pilot cases:

• The session was not authorized or had ended.
• The requested action required explicit approval and the person denied it.
• The access path was unavailable because its local vault or gate was locked.
• The request did not match the access that the pilot deliberately granted.

A denial after a locked local vault means someone preserved a hard boundary. A denial because the user rejected a request may mean the agent acted beyond its task, or it may mean the request description was too vague. A denial because the target was outside scope may reveal a good boundary, but it can also reveal an incomplete design. Do not merge these causes into a single "blocked" bar.

Then verify the external result. A control decision matters only if the external action did not occur through that route. For an API call, preserve the attempted method, endpoint category, and returned error. For an SSH command, preserve the requested host and command context appropriate to the audit design. Do not assume a local refusal means the remote side saw nothing if your architecture can issue work before the decision point.

This is where teams discover the distinction between prevented and failed. A prevented request never reached the service because the access boundary refused it. A failed request reached the service and received an error. Both deserve review, but they tell different stories. Calling every 403 a successful block can conceal an agent repeatedly trying to change a resource that your access design accidentally exposed to the network.

Walk through a real review sequence when a denial occurs:

1. Locate the agent run and identify the process that issued the request.
2. Read the requested operation and target, then compare it with the original task.
3. Confirm whether the control stopped the request before an external action occurred.
4. Assign a cause: task ambiguity, prompt behavior, missing capability, incorrect access scope, or an attempted boundary violation.
5. Decide whether to change the task, the agent setup, the access grant, or nothing.

That last option matters. Some blocked requests should remain blocked. A pilot often tempts a team to grant access after every failed attempt because the agent looked productive right until it hit the boundary. That replaces design with drift.

Watch for denial clusters. Ten denials against one endpoint after a new agent release usually point to an integration assumption. Ten denials across unrelated targets can point to an agent task that gives the model too much freedom. Repeated denied requests after a reviewer has already rejected the same intent deserve a sharper response: stop the session, preserve the record, and inspect the task and tool instructions before retrying.

Repeated failures expose unsafe pressure to broaden access

Repeated failed requests are an early warning that the agent, its instructions, or the access route has a mismatch. Count them by fingerprint, not as one raw total.

A useful fingerprint combines the operation category, target, error class, agent version or configuration, and a short time window. This lets you distinguish one transient service outage from an agent that retries the same unauthorized command twenty times. It also shows whether a new prompt or tool update created a behavioral change.

Do not use retries as evidence that the agent "really needs" an access grant. Agents repeat because their planning loop finds the same apparent next move. The loop does not understand your risk decision merely because it encountered the decision five times. Granting a credential to end retries trains the team to widen access whenever the agent applies pressure.

Inspect the first occurrence and the final occurrence of each repeated failure. The first tells you the original assumption. The final tells you whether the agent changed its strategy, stopped, or escalated attempts. That difference matters. An agent that notices an error and takes a permitted alternative action may need a better tool description. An agent that continues to probe a prohibited host may need a narrower task and a session stop.

Track four fields for every repeated failure group: attempt count, elapsed time, result class, and human response. The human response field is routinely omitted, which leaves a false impression that the error ended on its own. Someone may have altered a prompt, granted access, restarted the agent, or revoked the session. Those interventions explain the result.

A failure that comes back from the external service is not necessarily an access problem. Authentication errors can indicate a bad credential route. Validation errors often indicate malformed agent input. Rate limits can indicate uncontrolled retry behavior. Network errors can indicate service conditions. Treat each category differently. A team that grants broader permissions to solve validation errors will create more exposure and still have broken requests.

Set a practical escalation threshold based on pattern rather than a universal number. For example, review immediately when the same agent session repeats an access denial against a sensitive target after a human rejected it. Review when a new configuration produces a burst of errors against an external service. Review when retries continue after the service has clearly told the caller to stop. These are behavior rules, not vanity metrics.

Keep a short decision log beside the raw data. Write "corrected endpoint construction," "kept access denied," or "terminated run and rewrote task boundary." That log makes the pilot cumulative. Without it, each weekly review rediscovers the same error pattern and calls it a new finding.

Revocation time must include discovery and proof

Time to revoke measures containment, and containment begins when someone recognizes that a run needs to stop. Measure the interval from the first actionable signal to confirmed inability of that run to make another authorized action.

A button labeled revoke may operate quickly. That is useful, but it is only one segment of the interval. In a real incident, an engineer notices a strange request, identifies which session owns it, finds the authority to stop it, performs the action, and checks that the agent cannot continue. If the session identity is unclear, the time lost finding it belongs in the metric.

Record these timestamps during a drill:

• The suspicious action appeared in the review record.
• A named person decided to revoke the run.
• That person issued the revocation.
• The system recorded the revocation.
• A controlled follow up request from the same run was denied.

The final check is the proof. Do not test revocation by stopping the agent process and assuming the access ended. Process termination can hide stale authorizations, a second process, queued work, or an operator who revoked the wrong session. Use a harmless operation within the test scope and verify the boundary rejects it after revocation.

Run this drill during normal work, not only when the person who built the controls is available. Give the task to someone who did not configure the pilot. If that person cannot identify the live run from the record, the process is too dependent on tribal knowledge. A fast revocation path that only one engineer can use does not give a team meaningful containment.

Distinguish revoking a session from rotating a credential. Session revocation stops one active agent run. Credential rotation changes the secret or authority itself and can affect legitimate work that shares it. You may need both during a serious event, but they solve different problems. Measuring them as one action makes a targeted response look slow and a broad response look efficient.

Sallyport keeps a Sessions journal for agent runs with instant revoke, alongside an Activity journal for individual calls. In a pilot, that separation makes it possible to practice finding the run, revoking it, and confirming the later call record without confusing session control with credential replacement.

Keep the drill result even when it goes badly. A long interval exposes a fixable weakness: unclear ownership, inadequate session labeling, missing notifications, or a revocation action that people cannot reach. Hiding failed drills turns the first real urgent revocation into the test you should have run earlier.

Audit records must survive disagreement

An audit trail helps only when a skeptical reviewer can verify what it says without trusting the person who exported it. Agent access pilots need that property because authorization disputes arrive after the fact, when memories and terminal history have changed.

For each sampled event, a reviewer should be able to connect the initiating agent run, authorization decision, action request, external result, and later revocation if one happened. Preserve ordering. A timestamp alone can mislead when clocks differ or events arrive late. A sequence within a signed or protected record gives the reviewer a basis for reconstruction.

The hard distinction is between an activity log and a tamper evident audit record. An activity log tells you what the application currently displays. A tamper evident record lets you detect a changed, deleted, or reordered history. You need the latter when pilot evidence decides whether to grant wider access.

Sallyport projects its Sessions and Activity journals from an encrypted hash chained audit log, and its sp audit verify command checks the chain offline without requiring a vault key. Run verification as part of the review routine, not only after an argument. A check that no one performs has no bearing on confidence.

sp audit verify

Keep the verification result with the review record and note the time range it covered. If verification reports a problem, stop using that log as proof of normal operation until the team investigates. Do not quietly export the readable entries and continue with a spreadsheet. That preserves appearance while discarding the evidence property you needed.

Audit records can also expose review fatigue. Compare the time between a prompt and its approval, the request context, and later outcome. An extremely fast approval does not automatically mean carelessness, especially for familiar low consequence work. A pattern of instant approvals across unfamiliar targets and sensitive actions deserves a conversation with the reviewer and a redesign of the prompts.

Do not give every person unrestricted access to raw audit content merely because the record helps with oversight. Agent activity can contain repository names, hostnames, request metadata, and task details. Define who reviews it, who can export it, and who resolves disputes. Control over audit visibility is part of access design.

A weekly review should produce decisions, not charts

A weekly pilot review should end with explicit decisions about scope, fixes, and ownership. If it ends with a dashboard tour, the team has measured activity without changing its risk posture.

Bring a compact packet: a small sample of approvals, every denial involving a sensitive action, the top repeated failure groups, the latest revocation drill, and the audit verification result. This is enough material for a serious review without forcing people through every routine read request.

Ask each owner for a decision. The task owner decides whether the agent's requested work matched its assigned task. The system owner decides whether the target access remains appropriate. The person responsible for the access mechanism decides whether the control behaved as expected. One person can hold several roles in a small team, but name them anyway. Unnamed responsibility becomes collective inattention.

Write decisions in plain language. "Permit issue creation in the pilot project for approved sessions" is testable. "Improve guardrails" is not. Pair every change with a measurement that tells you whether it worked. If you reduce prompts for low consequence reads, sample the remaining approvals to ensure people now receive enough context for consequential actions. If you alter retry handling, inspect whether repeated failures actually decline without a new rise in blocked requests.

Do not expand every dimension at once. Widening targets, adding write actions, accepting more agent processes, and removing approvals in the same week destroys attribution. When something goes wrong, nobody knows which change caused it. Expand one authority dimension, observe it through a review cycle, and keep a rollback path.

A pilot can end with a decision not to expand. That is a successful result when the measurements reveal that the team cannot yet review requests, contain sessions, or explain failures. The bad outcome is pretending that those gaps will disappear after access becomes broader and the number of agent runs increases.

Broader authority should follow a specific passed test

Expand access only after a proposed new authority has passed a concrete test under the current pilot. The test should resemble the consequence you plan to permit, not a convenient substitute.

If you plan to allow a higher consequence API action, test reviewers on requests that contain the same target context and approval burden. If you plan to permit agent SSH access to more hosts, test attribution, denial handling, and revocation against a representative nonproduction host first. A read only pilot cannot prove that people will correctly handle changes to external state.

Write the expansion decision as a claim that evidence can refute: "The team can authorize this action class with enough context, detect a request outside its task, and revoke the initiating session within the agreed response window." Then examine the approvals, denials, failures, revocation drill, and audit record against that claim.

Do not promote a pilot because its approval percentage rose. It may have risen because reviewers learned the prompts, because the agent stopped asking for difficult work, or because the team silently removed denials from the denominator. Promotion needs representative evidence and a record of exceptions.

The first action worth taking is simple and unglamorous: pick one current agent run, trace one approved call and one denied or failed call, then revoke that exact run in a controlled drill. If your team cannot reconstruct those events and prove the run lost authority, wider access is premature.