# AI agents changing feature flags: control every write AI agents changing feature flags need to be treated as production writes, because that is what they are. A flag may sit outside the deployment pipeline, but it can turn code on for customers, suppress a payment path, widen an experiment, or disable a safety check within seconds. The mistake I keep seeing is granting an agent access to a flag provider because the task sounds harmless: "turn on the new flow for internal users" or "drop the rollout to zero." The change itself may be correct. The missing record is the dangerous part. After an outage, people need to know what the agent requested, who accepted the request, what the provider did, and whether the final state matched the request. A chat transcript cannot carry that burden. ## A flag mutation is a production write A feature flag update has the same operational character as changing a production database setting or editing a live routing rule. The API call may be small, but the effect can be broad and immediate. Treating flags as harmless configuration produces a familiar failure. An agent investigates an elevated error rate, finds a flag attached to the recent feature, and sets it to `false`. That may protect users. It may also disable an unrelated code path because the flag uses a targeting rule rather than a simple Boolean, or because the agent selected the wrong environment. If nobody can identify the exact requested mutation and its approving human, the team spends the incident debating history instead of restoring service. Martin Fowler's Feature Toggles article makes a distinction that teams often flatten away. Release toggles, experiment toggles, operational toggles, and permissioning toggles have different lifetimes and different levels of dynamism. That distinction should change the control around an agent action. An operational toggle that disables a failing integration may warrant fast human approval. A permissioning toggle that changes who can access regulated data deserves a far tighter path. Calling both "a flag" says very little about risk. Flag changes also bypass controls that engineers associate with code deployments. A pull request may show peer review, a build may show test results, and a release may show the artifact version. A flag provider call may show only a token name and a timestamp. If an autonomous process owns the token, even that token name may be shared across many runs. Use the same question you would ask before any production write: what exact state will this request change, under whose authority, and how will we prove the resulting state? A read is different. An agent can inspect a flag, its environment, and its current rules to prepare a recommendation. A write changes an external system. Do not bundle those permissions because an integration makes it convenient. ## Separate intent, authorization, execution, and observed state One event is not enough to describe a feature flag action. A reliable record has four different facts, and collapsing them hides the failure you will need to diagnose later. The **intent** is what the agent asked to happen. It should name a stable flag identifier, environment, requested value or rule change, reason, and scope. It must be captured before the provider call. The **authorization** says who allowed that intent. A human approval must bind to the precise proposed change, not to a vague request such as "let the agent manage flags." The approver needs enough context to make a decision: target environment, current state, proposed state, affected segment, expiry if any, and the task or incident that led to it. The **execution** records the outbound request and provider response. This is evidence that the gateway attempted the approved action. It is not proof that the desired state exists. The **observed state** comes from reading the flag back after the write. It catches malformed payloads, partial updates, defaulting behavior, and calls sent to the wrong project or environment. A read-back still has limits. Cached SDK clients may not have fetched the changed configuration, and it cannot prove that the code behind the flag behaves correctly. Those are separate observations that belong in deployment telemetry and application monitoring. This distinction matters during a rollback. Suppose an agent requests `checkout_v2=false`, a human approves it, and the provider returns success. If a second operator changes the flag before the agent reads it back, a naive log will tell a clean but false story. A proper record says the approved request succeeded at the API boundary, then reports the state it observed, including a version or revision if the provider exposes one. Do not let a free-form reason replace these fields. "Mitigate checkout errors" is useful context. It is not a target, an old value, an approved new value, or a final result. ## Capture a mutation envelope before the provider call The agent should submit a structured change request, not compose an arbitrary HTTP request to the flag provider. A fixed envelope creates something a reviewer can read and a gateway can validate. This example uses a Boolean flag, but the same shape applies to JSON configuration, percentage rollouts, and targeting rules. Keep rule changes separate from scalar value changes. A targeting rule can alter exposure far more than a single true or false value suggests. ```json { "request_id": "ffchg_01J8KQ4W6D7P", "correlation_id": "incident_INC-1842", "operation": "set_boolean_flag", "flag": { "project": "storefront", "environment": "production", "name": "checkout_v2" }, "expected": { "value": true, "revision": "481" }, "requested": { "value": false }, "reason": "Reduce checkout failures while payment timeout is investigated", "rollback": { "value": true, "expires_at": "2025-03-08T18:00:00Z" } } ``` The `expected` block prevents a quiet overwrite. It says: perform this mutation only if the current value and revision still match what the requester inspected. If another human or automation changed the flag after the agent read it, reject the request and show the new state. Retrying blindly is the wrong response. The agent needs to ask again because its basis for action is gone. The `request_id` must be idempotent. Network failures happen after a provider receives a request but before the caller receives the response. Without idempotency, an agent retry can create duplicate audit events or apply a percentage rule twice when the provider models updates as patches. Store the request ID and return the original outcome for a repeated submission. A gateway can return a result in this form: ```json { "request_id": "ffchg_01J8KQ4W6D7P", "status": "applied", "provider_request_id": "p_9d3ab", "authorization": { "approver": "ops@example.com", "approved_at": "2025-03-08T17:18:32Z" }, "observed": { "value": false, "revision": "482", "read_at": "2025-03-08T17:18:35Z" } } ``` Do not include provider tokens in any of these records. A request log that contains credentials eventually becomes a second secret store, usually with worse access controls and more copies. ## An approval must describe the blast radius A human cannot approve a safe change from a flag name alone. Names drift, flags outlive their original intent, and a Boolean can hide a large targeting rule behind a pleasant label. The approval card or review screen should show the current representation beside the requested representation. For a percentage rollout, show the exact old and new percentages, the population or segment, any prerequisite flag, and the environment. For a rule edit, show the complete before and after rule in a readable canonical form. A diff that omits a clause because it looks repetitive is how someone accidentally turns "employees in region A" into "all users." The approver also needs the reason and the expiry. Temporary operational changes have a habit of becoming permanent because the incident ends and everyone moves on. An expiry is not a magic rollback. It gives an operator a scheduled point to re-evaluate the flag, and it creates a visible commitment in the action record. Approval scope should match the mutation, not the agent. "Approve this process for the rest of the session" can make sense for repeated inspection or for a predefined set of non-production actions. It is poor scope for a production rollout where each action changes a different customer population. I would use separate approval for each production mutation when any of these conditions applies: - The change affects an operational control, payments, authentication, authorization, or data retention. - The request changes a targeting rule, segment, prerequisite, or percentage rather than one Boolean value. - The action targets production or an environment connected to real customer traffic. - The agent proposes a value that differs from an approved rollback plan. This is not approval theater. The human's job is not to retype the request. The human decides whether the stated scope and the present operational condition justify the write. If the card hides the scope, the person can only rubber-stamp it. Avoid a standing approval that says an agent may "manage feature flags." It is popular because it eliminates interruptions. It also turns every later mutation into an unreviewed production write, including the unusual one that happens while the agent is confused by stale context or a misleading tool result. ## Concurrency makes automated rollback unsafe by default An agent can roll back a flag only when it proves that it is undoing its own change. The common advice to "always have the agent revert on failure" ignores concurrent operators and is unsafe. Consider this sequence. At 10:00, the current value is `true`, revision 481. The agent receives approval to set it to `false`, and the provider records revision 482. At 10:06, an on-call engineer sees a different symptom and deliberately sets the flag to `true`, revision 483. At 10:08, the agent's monitoring condition fires and it executes its planned rollback to `true`. In that specific case the duplicate value looks harmless, but the same pattern with a targeting rule creates damage. The on-call engineer may have changed the rule to limit a path to one tenant. The agent restores the old broad rule because it remembered a pre-change snapshot. It has now overwritten a deliberate intervention without seeing it. A rollback request should include the revision created by the original action as its expected state. The gateway should apply the rollback only if the provider still reports that revision, or the exact canonical configuration the agent wrote. If the condition fails, return `needs_review` with the current configuration. The agent may explain the conflict to a human, but it should not resolve it itself. Use a rollback payload that carries its parent action: ```json { "request_id": "ffrb_01J8KR0Y8J2M", "operation": "rollback_boolean_flag", "parent_request_id": "ffchg_01J8KQ4W6D7P", "flag": { "project": "storefront", "environment": "production", "name": "checkout_v2" }, "expected": { "value": false, "revision": "482" }, "requested": { "value": true } } ``` Some providers do not expose revisions or conditional update APIs. In that case, you cannot make automatic rollback safe enough for a contested production flag. Read the state, present the difference, and require human approval for the restoring write. Accepting that limitation is better than pretending a timestamp provides concurrency control. Also distinguish rollback of a feature flag from recovery of user state. Turning a flag off may stop further exposure, but it will not reverse data migrations, queued work, or records created while the feature ran. The approval context should say that plainly when a flag controls writes. ## Restrict the verbs, targets, and credentials An agent should have permission to request a narrow vocabulary of flag actions, not administrator access to the provider. The provider token or API credential must stay outside the agent's context. Start by listing allowed verbs. `get_flag` and `list_flag_metadata` are reads. `set_boolean_flag` is a narrow write. `set_rollout_percentage`, `replace_targeting_rule`, `create_flag`, `archive_flag`, and `edit_segment` each have much larger consequences and should be distinct operations. Do not expose a generic `PATCH /flags/{id}` action and assume instructions will keep the agent careful. Generic patch endpoints invite fields that no reviewer expected. Then restrict targets. Bind a credential to a project and environment when the provider supports that scope. In the action gateway, maintain an allowlist of flag identifiers and operations for the specific job. A release agent that owns `checkout_v2` does not need access to every flag in the organization. An action request should fail before it reaches the provider if it attempts an unapproved verb, unknown environment, absent expected revision, or disallowed target. That validation has to be deterministic. Natural-language policy such as "only make safe changes" gives an agent a phrase to interpret, not a boundary to enforce. Sallyport keeps HTTP credentials in its encrypted vault and performs the API request without passing the secret to the agent. That is useful for this pattern because the agent can request an action while the credential remains on the controlling Mac. Keep the provider credential separate from the agent's own identity. A provider audit trail may only see one service account, but your action record can identify the agent session, the source task, and the human who approved the exact write. That separation also makes revocation practical: you can stop one agent run without rotating a credential used by a legitimate human workflow. Do not put an API token in an agent configuration file "just for the incident." Agents copy context into transcripts, shell history, generated patches, and tool arguments more readily than teams expect. Rotating the token later does not erase those copies. ## Test the control path with failures, not happy paths A feature flag integration is ready only after you test how it behaves when its assumptions fail. The happy path, read value, approve, update value, proves very little. Run a controlled test against a non-production environment and deliberately force these outcomes: 1. Change the flag after the agent reads it, then submit the original request. The gateway must reject the stale expected revision. 2. Submit the same request ID twice after simulating a lost response. The second call must return the first recorded outcome, not issue a new mutation. 3. Reject the approval. The provider must receive no write, and the audit record must show rejection rather than an ambiguous timeout. 4. Let a human edit the flag after the agent's action, then attempt automatic rollback. The rollback must stop for review. 5. Lock or revoke the agent session during a pending request. The action must fail before credential use. These tests expose a subtle design problem: many teams log only successful changes. Failed and denied requests matter just as much. A denied request tells you an agent attempted a scope it did not have. A stale-write rejection tells you the system prevented an overwrite. Both records explain why a provider state did not change when an operator expected it to. Test canonicalization too. Two JSON targeting rules can mean the same thing while having fields in a different order. If your compare-and-set code compares raw JSON, it will produce false conflicts. If it normalizes too aggressively, it may miss a semantic difference. Choose a canonical representation for the provider's model, record it, and test it with reordered fields, omitted defaults, and equivalent segment references. Finally, test the path when the provider returns success but the read-back fails. Record `execution=accepted` and `observed=unknown`; do not label the overall request successful. Someone must inspect the provider state before the agent makes a dependent change. ## An audit trail must survive a disputed incident A useful feature flag record must answer a skeptical question: "How do we know this account of the change has not been edited after the fact?" Ordinary application logs are often enough for debugging, but they rarely answer that question once many people can access the logging system. Write append-only action events with a sequence number, timestamp, request envelope, authorization result, execution result, and observed state. Link related events through request IDs and parent request IDs. Hash chaining makes later alteration visible: each event includes a digest of its own content and the digest of the preceding event. Verification checks the chain in order. Hash chaining does not make a log true. It cannot prove that the person who approved a request understood it, and it cannot recover records someone never wrote. It does make after-the-fact deletion or modification detectable when you preserve the chain and verify it independently. That is the correct claim, and it is already more useful than calling a log "immutable" without describing the mechanism. Keep the flag provider's native history as supporting evidence, not the only record. Match provider request IDs when the provider exposes them. Match the action `correlation_id` to an incident record or deployment change. When someone asks why a flag moved, you should be able to trace the decision without reconstructing it from chat messages and human memory. Sallyport projects session and individual-call records from an encrypted hash-chained audit log, and `sp audit verify` checks the chain offline without a vault key. That gives a team a way to verify that its controlling action record remains internally consistent even when the vault stays locked. Review denied actions and stale-write conflicts as part of normal operational work. They are not noise. A rising number of conflicts can show that several automations own the same flags. Repeated denied target requests can show that an agent's task scope is too broad or poorly specified. ## Make the agent produce a change proposal first The safest operational pattern is simple: let the agent inspect, diagnose, and draft the mutation, then require a controlled write path to execute it. The proposal should be detailed enough that another engineer can approve it without reading the entire agent transcript. For every production request, require the agent to state the current state it observed, the exact desired state, the expected revision, why the change helps, what could be affected, and the rollback condition. If it cannot supply those facts, it has not earned permission to write. Do not demand a long essay. Demand a complete request. The difference matters. Long prose often hides that the agent never checked the target environment or never fetched the current rule. A structured envelope exposes the omission immediately. Teams that adopt this discipline find that many proposed changes do not need execution. The agent may discover that the flag already has the desired value, the failing cohort does not match the rule, or a deployment rollback is the actual remedy. Reading first and recording expected state stops the agent from performing a ceremonial write just to satisfy a task. A feature flag is a fast control over live behavior. Give an AI agent the ability to use that control only when the system records its request, binds a human decision to the exact mutation, prevents stale overwrites, and verifies what the provider stored. Anything less leaves your most convenient production switch attached to an account nobody can fully explain.