Split traffic across deployments

Running more than one deployment on an endpoint lets you serve traffic across them for high availability, providing redundancy in case one of them fails. To split traffic between multiple deployments, list each with its relative weight. A weight sets a deployment’s share of traffic relative to its capacity: the actual share a deployment receives is proportional to its weight times its number of ready replicas. You set each deployment’s weight individually, and the split preserves the other deployments’ weights. When each deployment has the same number of ready replicas, the weights behave like a direct ratio. For example, with equal replica counts, these weights send roughly 70% of traffic to one deployment and 30% to the other.

CLI
Python
Console

Set each weight with endpoints update --traffic-weight. Pass the deployment ID (dep_...); the CLI resolves its parent endpoint and preserves the other deployments’ weights, so run it once per deployment:

Shell

tg beta endpoints update dep_abc123 --traffic-weight 70
tg beta endpoints update dep_def456 --traffic-weight 30

The Python SDK replaces the entire traffic split, so include every deployment that should continue receiving traffic:

Python

from together import Together

client = Together()
project_id = client.whoami().project_id

client.beta.endpoints.update(
    "ep_abc123",
    project_id=project_id,
    traffic_split=[
        {"deployment_id": "dep_abc123", "weight": 70},
        {"deployment_id": "dep_def456", "weight": 30},
    ],
    update_mask="trafficSplit",
)

The SDK uses snake case for Python arguments, such as traffic_split and update_mask, but field-mask values use the API field path. Pass trafficSplit, not traffic_split, as the update mask.

Weights are relative ratios, not percentages, so they don’t have to sum to any particular number: a split of 7 and 3 is equivalent to a split of 70 and 30. But weight sets relative capacity, not a fixed percentage. Each weight must be non-negative and finite. Two deployments with the same weight but different replica counts do not receive equal traffic. A deployment with weight 1 and two ready replicas draws the same traffic as one with weight 2 and a single ready replica. A deployment receives no traffic if it has a weight of 0, is absent from the split, or has zero ready replicas. Scaling a deployment to zero replicas takes it out of rotation even when it keeps a non-zero weight. Requests to an endpoint with no routable deployment return HTTP 400 with the error code endpoint_not_configured.

Shift traffic with replica counts

Treat weights as a stable definition of each deployment’s relative capacity, and shift traffic between deployments by changing their replica counts rather than editing weights. Because a deployment’s share tracks its ready replicas, scaling one deployment up (or another down) moves traffic without you having to recompute a set of weights. A deployment’s weight is remembered when it scales to zero and reapplies automatically when it scales back up, so you don’t need to re-add it to the split after a scale-down.

Troubleshooting

Update traffic split returns 400 with the deployment is a shadow experiment target and cannot serve live traffic; remove the shadow target first: The deployment is registered as a target in an active shadow experiment. Remove it from the experiment (or delete the experiment) before giving it a non-zero weight in the traffic split.

Next steps

Route traffic

Understand how an endpoint resolves each request to a deployment.

Configure autoscaling

Set the replica bounds that shift traffic between deployments.

Run an A/B test

Compare a candidate deployment against a baseline on live traffic.

Manage deployments

Create, poll, scale, stop, and delete deployments.

GET STARTED

SERVERLESS

INFERENCE APIS

DEDICATED MODEL INFERENCE

DEDICATED CONTAINER INFERENCE

GPU CLUSTERS

FINE-TUNING

CODE EXECUTION

ADMINISTRATION

Split traffic across deployments

Shift traffic with replica counts

Troubleshooting

Next steps

Route traffic

Configure autoscaling

Run an A/B test

Manage deployments

​Shift traffic with replica counts

​Troubleshooting

​Next steps

Route traffic

Configure autoscaling

Run an A/B test

Manage deployments

Shift traffic with replica counts

Troubleshooting

Next steps