Skip to main content
A shadow experiment copies a fraction of live traffic on an endpoint to one or more target deployments for observation, without changing what your application gets back.
Request mirroring. A client sends a request to the endpoint, which acts as a sampler. Every request goes to the live deployment, which serves every user and returns the response to the client. A sampled copy is sent to the shadow deployment as fire-and-forget: it is measured, then discarded, and never reaches the client.
Use a shadow experiment to warm up a new deployment under real load, stress-test configuration changes, or gather latency data for a candidate before routing live requests to it. Unlike an A/B test, a shadow experiment never affects the client-visible response.

​
How it works

A shadow experiment has two parts:
  • Source: The endpoint where mirrored traffic comes from and how much of it gets sampled. The source samples traffic at the API gateway using one of four sampling strategies.
  • Targets: The deployments that receive the mirrored traffic. Each target names a deployment under the same endpoint that’s excluded from the endpoint’s traffic split (weight 0 or unset), so it serves only mirrored traffic. Size it for the mirrored volume you expect.
When a request is sampled, a copy is sent to every target in the experiment. In an experiment with two targets that sample at 10% each, each target receives 10% of endpoint traffic. Adding a target multiplies the mirrored volume rather than dividing it. For a single target, copies are spread across its replicas through normal load balancing. All mirroring stops when you delete the experiment.

​
Hosting multiple experiments on an endpoint

An endpoint can host several experiments at once. Each experiment samples the endpoint’s traffic independently, with its own sampling strategy and rate, so a single request can be mirrored by more than one experiment. Mirrored volumes add up across experiments, so watch the total load on your targets. Use the structure that matches your use case:
  • More targets in one experiment: Compare how candidates perform for the same sampled requests. Each sampling decision applies to every target, so each target receives the same traffic. Use this for apples-to-apples comparisons, such as comparing fp8, fp4, and baseline builds on the same prompts.
  • Multiple experiments on the endpoint: Compare how candidates perform under different rules. Use this method when candidates require different sampling rates (a large candidate can absorb 50% vs. a small one that can only take 10%), different strategies (sticky key-based sampling vs. random uniform), or different owners and lifetimes (a long-lived 1% safety net vs. a short-lived high-rate test).

​
Sampling strategies

The source can be configured with one of four sampling strategies:
The four sampling strategies. uniform samples a fixed fraction of all requests at random (rate 0.1). key_based samples all requests from a fraction of distinct key values, so one user is either always or never mirrored (key: user_id). adaptive_uniform auto-adjusts the uniform rate to hold mirrored traffic at or below a target QPS as production arrivals rise and fall. adaptive_key_based combines sticky key sampling with the same target-QPS throttle.
The key for key-based strategies names a top-level field in the request body (for example body.user or body.prompt_cache_key), not a nested path. Requests with no value in that field are sampled at random. Setting --rate to 0 mirrors nothing, which is a way to pause an experiment without deleting it. For the adaptive strategies, target_qps is an approximate throttle, not a precise cap on total mirrored volume. The volume your targets actually receive may run higher than the value you set, and it grows as your endpoint scales up under load. Treat target_qps as a way to keep mirroring loosely bounded rather than pinned to an exact rate. Set it conservatively, check the volume your targets actually receive, and adjust. If you need a predictable, exact fraction of traffic, use a uniform strategy instead.

​
Requirements

Before you start a shadow experiment, you need a running endpoint under load and a candidate deployment to shadow to. The candidate must be excluded from the endpoint’s traffic split (weight 0 or unset) so it receives only mirrored traffic, never live requests. The CLI’s shadow command provisions a new shadow deployment from a model ID and wires up the experiment in one command, so you only need the endpoint and a model. The SDK mirrors to a deployment you’ve already created. The examples below use these example IDs, which you should replace with your own:
  • Endpoint: ep_abc123.
  • Model to shadow-deploy: ml_CbJNwQC2ZqCU2iFT3mrCh.
  • Target deployment (already created, for the SDK path): dep_target456.

​
Create a shadow experiment

Create an experiment that samples 10% of gateway traffic uniformly and mirrors it to one target. The CLI’s shadow command creates a new shadow deployment from a model, then starts mirroring sampled traffic to it. The SDK mirrors to an existing target deployment, with the sampling strategy in the source.endpoint block:
Note the experiment ID (exp_...) from the response. You use it to inspect, update, or delete the experiment.
name is immutable after creation and must be unique within the endpoint. description can’t be set on create. Set it by updating the experiment.
After creating the experiment, send requests to the endpoint as you normally would, using the endpoint string as the model field. The request path doesn’t change, but a sampled fraction of requests is mirrored to each target in the background.

​
Sampling strategy examples

Use key_based sampling to make mirroring decisions sticky on a request field (for example body.user), so all requests from the same user are either always mirrored or never:
Use adaptive_uniform to throttle mirroring toward a target request rate (QPS) rather than a fixed fraction. The sampler adjusts the rate automatically:

​
Response fields

​
Observe results

Once the experiment is ACTIVE, keep calling the endpoint as you normally would. A sampled fraction of your requests is mirrored to each target in the background, and the response you get back is always the baseline’s.
A new or updated experiment takes effect within about 30 to 60 seconds. Deleting an experiment stops mirroring within roughly the same window.
Because targets never return their responses to your caller, evaluate them the same way you would evaluate any deployment:
  • Compare each target’s own latency, throughput, and error rate against the baseline.
  • If you log outputs, compare each target’s outputs against the baseline’s for the mirrored requests.
  • Confirm each target is receiving the expected mirrored volume. As a rough check, target volume should equal the sampled requests times the number of targets: about (endpoint traffic times rate) for a uniform strategy, or roughly tracking your configured target_qps (which can run higher and rises as the endpoint scales) for an adaptive one.

​
Pause, retune, or stop an experiment

  • Pause without deleting: Update the experiment’s source to set the sampling rate to 0. The experiment stays ACTIVE but samples nothing.
  • Retune sampling: Update the experiment’s source, for example to raise uniform.rate or switch strategy. Fetch the experiment first to get its current etag and pass it back on the update; a stale value returns 409 ABORTED.
  • Fan out to more targets: Add a target to the experiment. One sampling decision fans out to every target, so adding a target roughly multiplies mirrored volume.
  • Stop mirroring: Delete the experiment (which cascade-deletes its targets), or remove all its targets so the experiment goes INACTIVE.
Mirrored requests are never sampled and mirrored again; the system prevents shadow loops automatically.
Before deleting a deployment that an experiment targets, remove it from the experiment first (or delete the experiment). That keeps the experiment’s configuration consistent and avoids leaving a target that points at a deployment that no longer exists.

​
Delete an experiment

Deletion cascade-deletes all targets, and mirroring stops shortly after. The CLI’s smart-delete rm accepts the experiment ID:

​
Troubleshooting

  • A target receives no mirrored traffic: The experiment may be INACTIVE (no targets), the sampling rate may be 0, or the change may not have propagated yet. Confirm the experiment is ACTIVE with at least one target and a non-zero rate, then re-check after the 30-to-60-second propagation window. Mirroring also needs live traffic on the endpoint: a sampled fraction of zero requests is zero.
  • A target’s shadow metrics show dropped copies (build_error or 5xx): Mirroring is fire-and-forget. If a target is still provisioning, scaled to zero, or unhealthy, copies are still sent and then silently dropped, and the failures show up only in that target’s shadow metrics. Your callers are never affected, and mirroring recovers on its own once the target is serving again.
  • Mirrored volume is lower than expected: With key_based sampling, whole keys are sampled sticky, so a few high-volume keys can pull the effective rate away from the nominal rate. With an adaptive strategy, the sampler throttles toward target_qps rather than a fixed fraction, and the actual volume can drift as the endpoint scales. Switch to uniform if you need a predictable fraction.
  • An update or delete returns 409 ABORTED: The etag you passed is stale. Re-read the experiment (or target) to get the current etag, then retry.
  • Create returns 400: The request shape is invalid. Check for a missing required field, a rate outside [0.0, 1.0], more than 100 inline targets, or a name longer than 256 characters.
  • Create or add target returns 400 with the deployment is serving live traffic on this endpoint and cannot also be a shadow target; remove it from the traffic split first: The target deployment has a non-zero weight in the endpoint’s traffic split. Remove it from the split (set its weight to 0 or omit it) before adding it as a shadow target.
  • Create or a lookup returns 404: The experiment, target, or parent endpoint doesn’t exist in the project named in the path. The API returns the same 404 for a missing ID, a wrong-endpoint ID, and an ID that belongs to a different project, so you can’t use it to probe for cross-tenant resources.

​
Limits

  • Up to 100 inline targets when creating an experiment. Add more with the target methods after creation.
  • name (on both experiments and targets) is at most 256 characters.
  • limit on list calls defaults to 50, up to a maximum of 500.

​
Next steps

A/B tests

Split live traffic and compare user-visible responses across deployments.

Split traffic

Route live traffic to a deployment by weight once you’ve validated it.

Observability

Monitor per-deployment metrics during experimentation.

Manage deployments

Create and manage the deployments used in shadow experiments.