Rollback endpoints for autonomous deployments without collisions

Autonomous deployment agents need permission to fail safely, not permission to guess. The dangerous rollback is rarely a command that fails. It is a command that succeeds after the world changed, restoring an old artifact over a release the agent did not create.

A safe rollback action identifies a specific deployment attempt, names a specific prior release, and refuses to act when its view of the current state has gone stale. Treat rollback as a guarded state transition, not as a shortcut to "the last good version." That distinction decides whether an agent repairs its own work or erases somebody else's.

A rollback must belong to one deployment attempt

Rollback endpoints must bind a reversal to the deployment attempt that caused the suspected failure. A version number does not do that job on its own.

Teams often store a sequence like 1.8.4, 1.8.5, and 1.8.6, then expose an operation that says "deploy 1.8.4 to production." An agent deploys 1.8.5, receives an alert, and calls that operation. Meanwhile, an engineer has deployed urgent release 1.8.6. The rollback endpoint accepts the request because 1.8.4 exists. Production now runs an artifact from before both changes. The API did exactly what it was told, which is the problem.

Keep these identities separate:

• A release is an immutable package of code, configuration references, and metadata. Give it a durable ID and an artifact digest.
• A deployment attempt is one request to put a release into a named target. It has a deployment ID, an actor, and a lifecycle.
• A target state is what the environment currently runs. It includes a revision or generation that changes on every accepted transition.
• A rollback intent says that deployment attempt D may restore target release R only if D still owns the current state.

The field that people omit is the ownership link. When deployment dep_842 promotes release rel_105, record that dep_842 created current target generation gen_913. A rollback tied to dep_842 can only proceed while gen_913 remains current. If a later deployment creates gen_914, the service must refuse the old request.

Do not infer ownership by comparing timestamps. Clock ordering breaks under retries, queued workers, manual repair, and any system that lets a user select an older release. Store the direct relationship when you accept the promotion.

This also answers an awkward operational question: can an agent roll back another agent's deployment? Usually, no. A separate authority can explicitly authorize that intervention, but the ordinary rollback capability should cover only actions the caller initiated. Broad authority looks convenient until two deployment loops react to the same incident.

Immutable provenance makes the prior release knowable

The service must capture rollback provenance before it changes the target, because after promotion the word "previous" becomes ambiguous. Querying release history after an alert is a recipe for selecting whatever happened to be adjacent in a list.

When a deployment service accepts a promotion, it should create a record that includes the prior stable release chosen at that moment. That prior release may differ from the immediately preceding event. For example, a canary can promote rel_105 after rel_103 remains the stable baseline while rel_104 was an aborted experiment. The correct recovery target may be rel_103, not the row directly before rel_105.

A minimal record can look like this:

{
  "deployment_id": "dep_842",
  "environment": "production",
  "release_id": "rel_105",
  "artifact_digest": "sha256:8b2c...",
  "source_revision": "4f1d9c7",
  "config_digest": "sha256:1a06...",
  "prior_release_id": "rel_103",
  "created_target_generation": "gen_913",
  "migration_set_id": "mig_77",
  "actor_id": "agent-run-27"
}

The prior_release_id is a decision, not a convenience field. Your promotion controller should choose it using rules that operators can inspect: the last verified stable release for that target, perhaps with compatible configuration and migration requirements. The rollback operation consumes that saved decision. It does not recalculate it because the history table has changed.

Artifact identity needs more than a human release label. Tags can move. Build labels can be reused by accident. A rollback should deploy the content address or equivalent immutable artifact reference recorded in the original attempt. If your registry allows a tag to point to different bytes later, the target release ID must resolve to a digest captured at promotion time.

Configuration deserves the same discipline. Reverting application bytes while leaving a changed feature flag, runtime secret reference, image pull policy, or resource setting can create a system that never existed in testing. You need not duplicate every value in the deployment record, but record an immutable configuration revision or digest and define whether rollback restores it.

Database state creates a separate boundary. A release that only adds nullable columns often permits an application rollback. A release that drops a column, rewrites values, or changes semantics may not. Do not promise a generic "full rollback" if the release planner cannot prove compatibility. Mark the deployment as application reversible, traffic reversible, or repair required. A refusal is less embarrassing than running old code against a schema it cannot understand.

The endpoint needs an expected current release

A rollback request must carry both the release it wants and the live state it expects to replace. Without that precondition, the endpoint has no way to distinguish a valid recovery from a stale instruction.

Use a request contract shaped like this:

POST /v1/environments/production/rollbacks
Idempotency-Key: 7e4cd1ee-62cb-4efa-985f-4ee0b77d577b
Content-Type: application/json

{
  "origin_deployment_id": "dep_842",
  "expected_current_release_id": "rel_105",
  "expected_target_generation": "gen_913",
  "restore_release_id": "rel_103",
  "reason": "error rate exceeded release threshold",
  "approval_id": "apr_551"
}

The service should derive restore_release_id from origin_deployment_id where possible, then compare the supplied value with the stored prior_release_id. Keeping both fields in the request helps auditors see the agent's claimed intent, but the server record wins. Never let a caller turn its own deployment into a license to pick any historical artifact.

On success, return the newly created deployment attempt and the new target generation. Do not return a vague accepted response if the system can synchronously reserve the transition.

{
  "rollback_deployment_id": "dep_849",
  "reverted_deployment_id": "dep_842",
  "previous_release_id": "rel_105",
  "current_release_id": "rel_103",
  "target_generation": "gen_914",
  "status": "running"
}

If the live target no longer matches, return a conflict response. The body should include enough information for an agent to report what happened, but not enough authority to improvise a new action.

HTTP/1.1 409 Conflict
Content-Type: application/json

{
  "error": "stale_rollback",
  "origin_deployment_id": "dep_842",
  "expected_target_generation": "gen_913",
  "observed_target_generation": "gen_914",
  "observed_release_id": "rel_106"
}

A 409 is a successful safety outcome. Teach the agent that it must stop on this result, attach the response to its incident record, and ask for a new decision. Do not give it a fallback instruction such as "try again without the expected generation." That fallback converts your guard into theater.

Some teams use an If-Match HTTP header carrying an ETag rather than a JSON field. That works if the ETag represents the target state and changes on every transition. The mechanism matters less than the invariant: the command must name the state it may replace.

Serialization stops two valid requests from colliding

A precondition check alone cannot protect a target if two workers can pass the check before either commits. The deployment service must serialize changes to the same environment and use an atomic compare and swap at the state store.

Suppose the target currently reads (rel_105, gen_913). An agent submits a rollback and an operator submits rel_106. Both requests read gen_913. If the service checks in application memory and later writes unconditionally, both calls can claim success. The later write wins, and your audit trail reports a state that may never have existed for users.

Put the comparison and mutation in one transaction or one conditional write. A relational implementation might issue this pattern:

UPDATE environment_targets
SET release_id = :restore_release_id,
    generation = generation + 1,
    active_deployment_id = :rollback_deployment_id,
    updated_at = CURRENT_TIMESTAMP
WHERE environment = :environment
  AND generation = :expected_generation
  AND release_id = :expected_release_id;

The service checks the affected row count. One changed row reserves the state transition. Zero rows means conflict. It should then read the current target and return the observed values in the 409 response.

A queue does not replace this condition. Queues lower the chance of concurrent work, but duplicate delivery, a manual path outside the queue, or a worker retry can still produce competing commands. Keep the conditional write where state lives.

Idempotency solves a different failure. An agent may lose the response after the service accepts the rollback. If it retries with the same idempotency key, the service should return the original rollback deployment and status. It must not reserve another generation or launch a second execution.

Scope idempotency to the caller and endpoint, record a digest of the request body, and reject a reused key with a different body. Otherwise a buggy client can attach a fresh intent to an earlier request by reusing an identifier.

A rollback can preserve a bad dependency state

Application rollback and environment recovery are separate operations. An endpoint that deploys old code cannot automatically make every dependency compatible again.

I have seen the predictable version of this failure: release rel_105 introduced code that writes a new enum value. A migration then tightened a database constraint to permit only the new values. The release failed for an unrelated reason, and the operator put rel_103 back. Old code wrote the former value, the database rejected it, and the incident grew because the rollback looked complete in the deployment dashboard.

The endpoint did not cause the schema change, but its success response made a false claim. Avoid that claim by requiring release metadata that describes compatibility in concrete terms. At minimum, capture whether the restore release can read current data, write current data, and operate with the target's configuration revision.

Traffic management has its own trap. A canary rollback should normally change only the traffic allocation owned by that canary. If an unrelated release has adjusted the stable pool or another controller has changed a routing rule, a rollback endpoint that writes an entire routing document can erase those changes. Use resource level versions or patch only the allocation fields that the deployment reserved.

The same principle applies to infrastructure. If a release created a queue, bucket, role, or firewall rule that later work adopted, deleting it during reversal can hurt a separate service. Cleanup requires a resource ownership record and a check that no later deployment claimed the resource. If you cannot establish that ownership, leave the resource in place and create a repair task.

For irreversible operations, choose forward repair. The agent can disable a feature flag, route traffic away, or deploy a corrective release. Operators dislike that answer because "rollback" sounds quicker, but a clean looking reversal that destroys later data costs more time than a repair plan.

Agents need narrow authority and a visible stop point

An autonomous agent should receive the smallest action authority that can complete its assigned deployment. It does not need raw cloud credentials, a general shell with production access, or an endpoint that accepts arbitrary release IDs.

Give the agent a deployment handle when it starts a release. That handle can authorize status reads, health checks, traffic changes within the deployment's allocation, and one rollback that names the original deployment. Expire it when the deployment reaches a terminal state, or when a human revokes the run. The deployment service must still enforce server side ownership because a handle can be copied or a client can malfunction.

Human approval should occur before the irreversible boundary, not after the agent has already assembled an irreversible command. A sensible policy asks for approval when an agent starts a production deployment, then allows it to reverse that exact deployment only while the ownership precondition holds. If the agent encounters a later release, it needs fresh approval for any intervention. That is a useful place to slow down because someone changed the situation.

Sallyport can keep HTTP and SSH credentials out of an agent process while a person approves the agent run or marks a credential for approval on every use. That control helps protect the action path, but the rollback service still needs its own release and generation checks; credential custody cannot define deployment ownership.

Avoid capability names such as production:rollback:any. They invite the caller to choose scope at runtime. Prefer a server issued capability tied to dep_842, environment production, and the specific rollback route. If the agent asks for an unrelated target, the authorization layer should reject it before the deployment controller evaluates the request.

Record the agent process or workload identity with every request. A human should be able to answer who initiated dep_842, what code signed or authenticated the caller, which approval covered it, and whether someone revoked access before execution finished. Anonymous automation accounts turn every incident into archaeology.

Verification must test the restored release, not the request

A rollback reaches completion only after the target runs the intended release and the service verifies the conditions that justified recovery. HTTP 202, a successful command exit, or a controller event that says "applied" does not prove the old release serves traffic correctly.

Define verification against the deployment's actual failure mode. If latency or errors triggered rollback, observe the restored service through the same measurement path after traffic reaches it. If a worker release consumed malformed jobs, verify the worker version and a controlled workload. If a configuration error caused startup failures, inspect ready instances and the configuration revision they loaded.

Keep a bounded observation window and record its outcome. The endpoint can report verifying, then succeeded, failed, or needs_operator. Do not wait forever for a metric that might be unavailable. A time limit should produce an explicit inconclusive result, followed by an operator decision.

The audit event should connect every stage: the alert or rule that requested reversal, the origin deployment, expected state, conditional reservation, execution events, health evidence, final state, and any revocation. Events need ordering and integrity controls because a friendly deployment timeline is not sufficient during a dispute.

Sallyport records agent runs and individual actions from an encrypted hash chained audit log, and sp audit verify checks that chain offline without a vault key. Use that sort of evidence to show that an agent requested an action, then keep the deployment service's own state transition and verification records as the authoritative account of what changed.

Familiar rollback commands need a safer wrapper

kubectl rollout undo is useful for an operator working directly on a Kubernetes Deployment, but it is not a complete contract for autonomous recovery. Kubernetes documents that kubectl rollout undo rolls back to the previous deployment revision unless the caller supplies --to-revision. That default makes sense during hands on diagnosis. It does not prove that the previous revision belongs to the failed agent run.

A Kubernetes Deployment tracks revision history in ReplicaSets, while revisionHistoryLimit controls how much history Kubernetes retains. That history is a controller artifact, not your business record of a release's approved baseline, configuration compatibility, or agent ownership. Once history is pruned, "undo" may also lack the revision an external deployment process expects.

Do not hand an agent a general kubectl credential and call the command your rollback endpoint. Place a controller or deployment service between the agent and the cluster. The service should resolve the origin deployment record, compare the target's live generation, reserve the state change, and invoke the underlying platform only after it passes those checks.

The same criticism applies to cloud provider commands that say "deploy revision X" or CI controls that say "rerun previous release." They operate on a platform resource. They do not know whether a release is related to the agent's current incident unless your control plane supplies that context.

Keep the platform command narrow too. If the service can patch a named workload revision, avoid granting it cluster wide mutation. A wrapper that preserves broad credentials has only moved the danger behind another API.

Build a release ledger before you automate recovery

You can introduce guarded rollback without replacing every deployment system. Start by making the release ledger authoritative for one production target, then force both human and agent paths through the same conditional transition.

A practical rollout sequence has five parts:

1. Assign immutable IDs to releases and deployment attempts, then record the prior approved release and target generation at promotion time.
2. Add an endpoint that requires origin_deployment_id, expected release, expected generation, and an idempotency key.
3. Make the target update conditional in the database or control store, and return 409 on every mismatch.
4. Classify each release for application, configuration, data, and traffic reversibility before promotion.
5. Require verification evidence before the controller marks a rollback complete.

Run this contract in report mode before agents can execute it. Let the service calculate what it would restore and whether it would reject the request. Compare those decisions with actual incident actions. This exposes missing provenance records and hidden manual paths without giving automation the chance to overwrite production.

Then make rejection boring. A stale rollback should create an understandable incident item with the observed release and generation, not a mysterious failure that encourages someone to bypass the endpoint. The endpoint earns trust when it refuses an unsafe request consistently, including requests from the people who built it.

The first field to add is not force. It is expected_target_generation. Once your deployments carry that fact and preserve the original baseline, an agent can reverse its own release with a clear boundary. Until then, autonomous rollback is an old deploy command pointed at a moving target.