Skip to main content
A rollout moves traffic from one deployment to another under the same endpoint, without changing the endpoint URL. Traffic shifts to the new deployment (the target) while the old one (the source) drains. For example:
The CLI examples on this page use these placeholders, which you should replace with your own endpoint/deployment IDs:
  • Endpoint: ep_abc123.
  • Source deployment: dep_source123.
  • Target deployment: dep_target456.

When to use rollouts

Use a rollout instead of editing the traffic split by hand when you’re migrating a deployment on live traffic. Rollouts are especially useful when you want to:
  • Ship a new model version or engine config: Canary rollouts shift traffic gradually from the old deployment to the new one, allowing you to monitor the new deployment’s performance before fully committing to it.
  • Validate a change under production traffic: Use metric gates to pause the rollout automatically when the new deployment is worse than the original.
  • Recover from a bad release: Canceling freezes traffic where it is, allowing you to walk the change back safely with a reverse rollout.
  • Keep a record of the migration: Each rollout records its steps, state changes, and pause/cancel reasoning in the endpoint’s rollout history.
For a one-off comparison between two deployments, use an A/B test or a shadow experiment instead.

Requirements

A rollout needs two eligible deployments under the same endpoint:
  • Source: The deployment you’re migrating away from. The source must:
    • Be in the READY deployment state, or in the SCALING state with at least one ready replica and a ready routing tier.
    • Have a positive weight in the endpoint’s traffic split. A rollout shifts only the source’s share of the traffic. Any other deployments in the split keep serving their shares, and the target takes over the source’s slot when the rollout completes.
    • Not be part of a shadow experiment or A/B test.
  • Target: The deployment you’re rolling out to. The target must:
    • Be in the READY, STOPPED, or SCALING deployment state. The rollout restarts a stopped target when it scales it up.
    • Not be receiving 100% of the traffic on the endpoint.
    • Not be part of a shadow experiment or A/B test.
    • Belong to a project in good billing standing.
Each deployment must be in one of its allowed states when you create the rollout and again when you start it. A deployment with a stop in progress doesn’t qualify, whatever its state. If either deployment is still SCALING when the rollout starts, the first traffic shift waits until that deployment has enough ready replicas. If that doesn’t happen within 30 minutes, the rollout pauses with UNDER_SERVED.
Pass --detach when starting the rollout to detach the target from its experiment automatically. --detach covers only the target, so detach a source from its experiment yourself before you start the rollout.
Make sure enough hardware is available for the rollout strategy and final target size you choose. To check, retrieve the per-region headroom value for the target’s instance type.

Choose a rollout strategy

The rollout strategy you choose determines how traffic moves from the source deployment to the target. Pick exactly one strategy per rollout:
Comparison of the canary, blue-green, and rolling rollout strategies. Canary ramps traffic to the new deployment in gated steps (10 to 50 to 100 percent); blue-green cuts over in a single gated step (0 to 100 percent); rolling swaps replicas one in, one out while keeping total capacity constant. Orange represents traffic on the new deployment.
During a rollout, the source and target deployments will run at the same time while traffic shifts, so you’ll be billed for both. The amount of extra capacity the overlap requires depends on the strategy (see Extra capacity in the table below).

Start a rollout

To start a rollout, run the rollout command, passing the target deployment (ID or name) and a strategy flag. The command creates the rollout and starts shifting traffic from the source to the target. Before you run it, check the following:
  • Source: The deployment currently serving the traffic you’re migrating. It must be referenced in the endpoint’s traffic split, which is automatic on a single-deployment endpoint. If the source isn’t referenced in the traffic split, the rollout still starts but doesn’t shift any traffic.
  • Target: The new deployment you’re rolling out to. It doesn’t need to be READY yet. The rollout scales up a STOPPED or SCALING target and waits for its replicas before it shifts traffic. A READY target starts fastest.
  • Inactivity timeouts disabled: Neither deployment can have an inactivity timeout enabled. A deployment that stops automatically when idle can’t also carry a rollout, so create rejects the pair and names the offending deployment. Set the timeout to 0 on both, and turn it back on after the rollout completes.
  • Billing standing: The target deployment’s project must be in good billing standing to create the rollout, start it, and keep scaling the target up during the run. If billing lapses mid-run, the rollout pauses with ENTITLEMENT_LAPSED before the next target scale-up. Scale-downs and recovery actions, such as canceling the rollout, aren’t blocked.
See the CLI reference for the full flag list.

Canary

Pass --canary. To customize the ladder, set the traffic percentages with --steps and the soak between steps with --interval:
If the target is already serving some traffic (for example, after a canceled rollout), your custom ladder must start above the share the target is already serving. When in doubt, omit --steps and the platform picks steps that fit the live traffic.You can also configure a canary rollout to evaluate live metrics after each step, pausing automatically instead of advancing when the target regresses. Attach a metric gate with the --metric* flags:

Blue-green

Pass --blue-green:

Rolling

Pass --rolling:

Monitor progress

Retrieve the rollout with get, passing the rollout ID (rol_...) printed under Active Rollout when you started it. You can also pass the endpoint ID to print the endpoint summary, which includes the active rollout’s ID, state, strategy, and progress.
To follow traffic-split transitions event by event, list the endpoint’s events with --subject-id set to the rollout ID. Traffic-shift rows include old_traffic_percent and new_traffic_percent.

Step status on retrieve

When you retrieve a rollout, status.steps lists one row per step with its state: started_at is set only when the step actually ran.

Manage a running rollout

You can pause, resume, promote, and cancel a rollout using the CLI or console. An endpoint has at most one active rollout, so the CLI accepts any of these identifiers: the endpoint or deployment ID or name, or the rollout ID (rol_...).

Pause and resume

To manually pause a rollout, pass --pause (and optionally a reason). You can resume it later with --resume:
The platform may pause a rollout automatically, which sets the rollout state to SYSTEM_PAUSED (for example on a failed metric gate or when capacity runs out). See Troubleshooting for the pause causes and how to respond to each.

Promote a rollout

To promote a rollout, immediately shifting all traffic to the target, pass --promote:

Cancel a rollout

Canceling a rollout freezes the current traffic split wherever it is. Both deployments keep the weights that were serving at cancel time. A target that isn’t serving traffic may be left at weight 0 or removed from the split. After the cancellation completes, both deployments keep serving at those frozen shares (rollout events describe this as traffic frozen at the current split). This leftover state, the frozen pair, keeps serving until you either edit the traffic split to put all traffic on one deployment or run the rollout in reverse.
Canceling before any traffic has shifted (a 0% split) leaves the target deployment running with no traffic. If you no longer need the target, scale it to zero or delete it once the cancellation is finished.
To cancel a rollout, pass --cancel and optionally a reason:

Roll back to the source

There is no separate rollback operation. To move traffic back, run another rollout with the roles reversed.

After a completed rollout

This works like any other rollout. The deployment now serving traffic becomes the source, and the old source is the target (any stopped deployment is an eligible target for the new rollout).

After a canceled rollout

If you canceled a rollout and the endpoint is still serving from the frozen pair, roll back by running a new rollout with the roles reversed. Any strategy works. The platform adapts the rollout to the frozen state:
  • Replica counts: --final-target-replicas defaults to the pair’s combined replica count (the capacity serving all of the endpoint’s traffic today), not only the source’s count.
  • Canary steps: For a canary, the default ladder skips the steps the new target has already passed. For example, if the cancel froze the target at 40%, the rollout runs only the 50% and 100% steps. A custom ladder must start above the frozen share, the same rule as when starting any canary.

Delete a rollout

You can delete a rollout that hasn’t started yet or has finished (completed or canceled). Deleting the rollout record doesn’t change the traffic split it left behind. To stop serving both deployments of a frozen pair, edit the traffic split so one deployment carries all of the traffic, then scale the other to zero if you no longer need it.
Use rm to delete a rollout, passing the rollout ID (rol_...):

Rollout states

A rollout can pass through the following states:

Limitations

While a rollout is in any non-terminal state (including PENDING and paused rollouts), it locks the endpoint resources it depends on:
  • Traffic split edits are blocked: Editing the endpoint’s traffic split returns a 409 until the rollout completes, is canceled, or is deleted. To check for an active rollout before editing, retrieve the endpoint: get, update, and list responses include the active rollout’s ID as activeRolloutId (omitted when there is none).
  • One active rollout per endpoint: Creating a second rollout returns a 409, and the error names the rollout that blocks it and its state. A PENDING rollout left behind by a failed start also occupies the slot.
  • The source and target can’t be stopped or deleted: Deleting either deployment returns a 409, and stopping one returns a 400. The console disables Stop and Delete on both deployments, along with the endpoint’s Stop all deployments.
  • The source and target can’t join experiments: A deployment can only serve one traffic role at a time, so adding either deployment to a shadow experiment or A/B test is rejected while the rollout is active.
  • Replica bounds stay editable, with one caveat: Lowering the target’s --max-replicas below what the current rollout step requires pauses the rollout with a POLICY_INFEASIBLE error.

Troubleshooting

When the platform pauses a rollout (SYSTEM_PAUSED), the rollout’s status.condition carries a typed category and a human-readable message.
Recoverable pauses often clear on their own. The platform retries the failed process every 15 minutes for up to three hours, then leaves the rollout paused for you to act on. Operator pauses (PAUSED) never auto-resume.
Other common issues and their fixes:
  • New replicas are slow to take on traffic: Each step brings up new replicas, each of which requires a cold start. For canary rollouts, set the --interval long enough for the replica to warm up before the gate starts sampling.
  • Create or start returns 409: The error message names the rollout or deployment that blocks it. If it names a rollout, the endpoint already has an active one, so finish or cancel it before you create another. If it names a deployment, that deployment either has a stop in progress or is in a state that doesn’t allow a rollout. Wait for it to reach one of the states listed in Requirements. A source that finishes stopping stays ineligible until you restart it.
  • Preview warns with START_WILL_REJECT: The rollout can be created, but starting it would fail while the deployments keep their current traffic shape (for example, the target already serves the pair’s whole share). Follow the fix in the warning message, then start the rollout again.
  • Create returns 400, or preview warns with FIRST_STEP_AT_SEED, for a custom canary ladder: The ladder’s first step is at or below the share of traffic the target already serves. Create rejects a first step below that share. Preview warns when the first step exactly equals it. In that case, creating may work, but starting can still fail. Raise the first step above the target’s share, or omit --steps (leave Canary steps on auto in the console) so the platform picks steps that fit.
  • Create returns 400 for a metric gate: The gate’s threshold or regression budget could never pass, or could never fail, so the gate would be meaningless. Fix the operator and value so a realistic reading can pass and a bad one can fail. See Supported metrics for each metric’s unit.
  • Preview shows a landing max above the target’s current max replicas: The rollout raises the target’s max replicas to cover the larger of the two deployments (or --final-target-replicas, if higher), so the target can hold the endpoint’s whole traffic. The raised bound stays after the rollout ends, so lower it afterward if you don’t want to keep it.
  • Target scales down after the rollout completes: --final-target-replicas sets how many replicas the target runs at cutover, not a standing floor. After the rollout completes, the target’s autoscaling policy takes over and scales it from the cutover count to what your traffic needs, so a rollout created at peak load doesn’t keep its peak replica count. The target scales no lower than its own min replicas, or the source’s min replicas if the target started stopped. Preview reports this floor as the landing minimum.
  • Preview warns with LANDING_MIN_ABOVE_CUTOVER: The target started stopped, so it inherits the source’s min replicas, and your --final-target-replicas is below that min. After the rollout completes, the autoscaler scales the target up from the cutover count to the source’s min. To land at the cutover count instead, lower the source’s min replicas before you start the rollout.
  • Updating the traffic split returns 409: The rollout controls the endpoint’s traffic split in every non-terminal state (including pending or paused). Complete, cancel, or delete the rollout before editing traffic weights. In the console, Traffic weight is also disabled on New deployment and Edit while a rollout is active.
  • Stop or delete is disabled on a deployment in the console: The deployment is the source or target of an active rollout. Complete or cancel the rollout first. Stop all deployments on the endpoint is also disabled while any rollout is active.
  • Create returns a 400 that names a shadow experiment target: The source or target is registered as a shadow experiment target. Pass --detach to detach a shadow or A/B target automatically. A shadow source must be removed by hand.
  • rollout reports Created rollout ... but failed to start it: Creation succeeded but the start call failed, so a PENDING rollout now occupies the endpoint’s single active-rollout slot. Run the cleanup command the CLI prints (tg endpoints rm rol_...), fix the underlying cause, then create the rollout again.
  • An error says the project is not in good standing: The target deployment’s project failed the billing check, which runs when you create a rollout, when you start it, and before each step scales up the target. A failure before a scale-up pauses the rollout with ENTITLEMENT_LAPSED. Resolve the billing issue on the project, then retry or resume.
  • An error says only internal error or Failed Precondition: Some platform failures surface as a generic message with no further detail. Contact support with the rollout ID and the time of the failure.

Next steps

Gate rollouts with metrics

Pause a canary automatically when the target regresses on live metrics.

Create a deployment

Create the source and target deployments a rollout shifts between.