Manage endpoints and deployments

This page covers the lifecycle operations for dedicated model inference (DMI): creating endpoints and deployments, scaling, stopping, and deleting resources. The CLI’s tg beta endpoints deploy command bundles several API/SDK operations into one step for convenience: it creates the endpoint (when you pass a new endpoint name), attaches a deployment to it, and routes 100% of traffic to that deployment. This page shows the individual operations underneath it. To create your first deployment end-to-end, follow the quickstart. You can run every operation on this page from the Together CLI and SDK or from the web console. Each section below shows both: the CLI or SDK command, and the equivalent steps in the console.

Create an endpoint

When you deploy a model to a new endpoint, Together creates the endpoint, attaches the deployment, and routes all traffic to it. (To create an endpoint resource with no deployments, use the SDK or API.) Before you deploy, choose a supported model and a deployment profile.

CLI
Console

Pass a model and a new endpoint name to tg beta endpoints deploy. It creates the endpoint, attaches a deployment on the model’s default hardware, and routes 100% of traffic to it:

CLI

tg beta endpoints deploy google/gemma-4-E4B-it \
  --endpoint my-endpoint

Add --config <cr_...> when the model has more than one deployment profile, and --min-replicas / --max-replicas to set the replica bounds.

Open the create form

On the Endpoints page, select New endpoint.

Name the endpoint and deployment

Enter an Endpoint name and a Deployment name.

Choose the model and hardware

Select the Model and its Quantization, then pick a Hardware configuration and a Region. The console lists one hardware card per deployment profile, so the model and quantization you choose determine the config.

Set autoscaling

Set Min replicas and Max replicas (both default to 1).

Create the endpoint

Select Create endpoint. Together creates the endpoint and its first deployment and routes all traffic to it.

The endpoint serves as a logical grouping of deployments, and the entry point for routing traffic to your models.

Create a deployment

Add more deployments to an endpoint to run several models or hardware configs behind it, for traffic splitting, A/B tests, or shadow experiments. It works like creating an endpoint, except you target an existing endpoint and give the deployment a traffic weight so it takes a share of the traffic split.

CLI
Console

Pass an existing endpoint ID (or a new endpoint name) to tg beta endpoints deploy --endpoint to add a deployment. The model is the positional argument, and the config is --config:

CLI

tg beta endpoints deploy ml_CbJNwQC2ZqCU2iFT3mrCh \
  --endpoint ep_abc123 \
  --deployment-name my-deployment \
  --config cr_CbzGdmn14t3HYrXXitmKa \
  --min-replicas 1 --max-replicas 2

When a model has more than one deployment profile, deploy returns an error that lists the available profiles, for example:

Model has multiple deployment profiles. Re-run with --config <config_id>:
  cr_CbzGdmn14t3HYrXXitmKa  NVIDIA-H100 x1  BF16  TP1
  cr_CciJqTB35QmpMupbQNPPW  NVIDIA-H100 x1  FP8   TP1

Re-run with --config <cr_...> to choose one. When a model has a single profile, the CLI selects it automatically. List a model’s profiles anytime with tg beta models configs <model_id>.The CLI defaults --min-replicas and --max-replicas to 1, so a bare deploy creates a single-replica deployment.For the full flag list, including placement, the autoscaling windows, and the scaling percentile, see the CLI reference.

Open the endpoint from the Endpoints page and select New deployment. The dialog has the same fields as the create form (Create an endpoint), plus a Traffic weight: leave it at 0 to add the deployment without serving live traffic (for example, as an A/B variant). Fill in the fields and select Create deployment.

The New deployment dialog in the Together AI console, with fields for deployment name, model, quantization, hardware, region, autoscaling replica bounds, and traffic weight.

After you’ve created a deployment, you’ll need to route traffic to it before it can serve requests.

Poll deployment status

CLI / SDK
Console

To check a deployment’s status, run tg beta endpoints get on its endpoint. The output lists up to the 10 newest deployments’ state and ready/desired replica counts, so re-run it to watch a specific deployment come up:

CLI

# Show the endpoint with each deployment's state and replica counts
tg beta endpoints get ep_abc123

For the full set of status fields (scheduled replicas, status message), retrieve the deployment from the SDK or API and read status:

Python

from together import Together

client = Together()
project_id = client.whoami().project_id

deployment = client.beta.endpoints.deployments.retrieve(
    "dep_abc123",
    project_id=project_id,
    endpoint_id="ep_abc123",
)
print(deployment.status.state)

The status object exposes these fields:

Field	Description
`desiredReplicas`	Target replica count from autoscaling.
`status.scheduledReplicas`	Replicas the scheduler has placed on clusters. May trail `desiredReplicas` while capacity is still being found, and exceed `status.readyReplicas` while placed replicas start.
`status.readyReplicas`	Replicas actively serving traffic.
`status.message`	Human-readable explanation of the current stage or cause. Replica progress lives in the counts above, not in this string.
`status.state`	See below.

Deployment states

A deployment reports its lifecycle in status.state. The API returns the fully-qualified enum (for example DEPLOYMENT_STATE_READY). This page uses the short name for readability.

State	Description
`PROVISIONING`	The scheduler is placing replicas on clusters. `status.message` is `Scheduling replicas`.
`SCALING`	Replicas are starting or draining to reach the desired count. `status.message` is `Starting replicas` or `Scaling down`.
`READY`	All replicas are healthy and serving. `status.message` is `All replicas ready`. A deployment must also be in the endpoint’s traffic split to receive requests.
`DEGRADED`	The deployment is below the requested capacity or blocked by a transient issue. `status.message` explains the cause. It usually resolves on its own.
`STOPPING`	A transient teardown state. Replicas are draining after a stop was requested. It settles to `STOPPED` once cleanup completes, or to `FAILED` if teardown ends with a failure.
`STOPPED`	Scaled to zero replicas. The deployment isn’t billing and isn’t serving.
`FAILED`	Terminal state. The `status.message` field explains why.

A deployment that never reaches READY within six hours after starting will be marked as FAILED.

Scale a deployment

Deployment scale is controlled by the deployment’s replica bounds, and optionally autoscaled using scaling metrics. Set the initial bounds when you create the deployment, then change them on a running deployment.

CLI
Console

CLI

tg beta endpoints update dep_abc123 --min-replicas 2 --max-replicas 4

Stop a deployment

A deployment runs until you stop it. Stopping scales it to zero replicas and releases its hardware.

CLI
Console

Set both replica bounds to 0:

CLI

tg beta endpoints update dep_abc123 --min-replicas 0 --max-replicas 0

The replicas keep serving until they finish draining, then the deployment moves to DEPLOYMENT_STATE_STOPPED and billing stops.

Restart a deployment

A stopped deployment doesn’t restart on its own. Only deployments in DEPLOYMENT_STATE_STOPPED can be restarted. A deployment in FAILED is terminal and can’t be brought back this way; deploy a new deployment instead. To restart a stopped deployment, raise both bounds to 1 or more.

CLI
Console

CLI

tg beta endpoints update dep_abc123 --min-replicas 1 --max-replicas 2

List resources

CLI
Console

List and get endpoints with the CLI:

CLI

# All endpoints in the project
tg beta endpoints ls

# One endpoint (includes up to the 10 newest deployments' state and replica counts)
tg beta endpoints get ep_abc123

Endpoint get and list responses embed lightweight deployment summaries in each endpoint’s deployments array. The array includes at most the 10 newest deployments per endpoint (ordered by createdAt, descending). To list every deployment on an endpoint, use the SDK or API.

List flags

tg beta endpoints ls accepts these flags:

Flag	Description
`--limit`	Maximum number of endpoints to return.
`--after`	Pagination cursor to start from.
`--org`	List org-scoped endpoints instead of project-scoped ones.
`--public`	List public endpoints.

List responses are paginated: when more results are available, the response includes next_cursor, which you pass as --after on the next request.

Delete resources

Deletion is permanent. A deployment must be stopped before it can be deleted. Follow this order:

Scale the deployment to zero and wait for DEPLOYMENT_STATE_STOPPED.
Delete the deployment. If you use the SDK or API (not the CLI), set the deployment’s traffic split weight to 0 on the endpoint first.
Delete the endpoint once it has no deployments.

CLI
Console

The CLI’s rm command is a smart-delete: it resolves the resource by its ID prefix, so the same command deletes an endpoint (ep_), a deployment (dep_), an A/B experiment (abx_), or a shadow experiment (exp_). When you run tg beta endpoints rm dep_..., the CLI automatically detaches the deployment from the traffic split and from any experiments it belongs to:

CLI

# Delete the deployment (must be stopped first; auto-detaches from the traffic split)
tg beta endpoints rm dep_abc123

# Delete the endpoint once it has no deployments
tg beta endpoints rm ep_abc123

To delete an endpoint that still has deployments, pass --force to rm. If the endpoint has other deployments you want to keep, rebalance the remaining weights instead of clearing the split. See Route traffic.

First stop every deployment under the endpoint. Then open the endpoint, select Endpoint actions, and select Delete endpoint. The console keeps Delete endpoint disabled until every deployment is stopped or deleted, so there’s no console equivalent of the CLI’s rm --force on a running endpoint.

The Endpoint actions menu open in the Together AI console, with a Delete endpoint item that is disabled while a deployment is still running.

Troubleshooting

endpoint_not_configured (HTTP 400) though the deployment is READY: Confirm the deployment is in the endpoint’s traffic split with a non-zero weight.
Deployment DEGRADED with Cannot place replicas: insufficient GPU capacity: Hardware for the config is constrained, so the scheduler couldn’t place all replicas yet. Compare status.scheduledReplicas to desiredReplicas. The scheduler keeps retrying and the deployment starts once capacity frees up. To improve the chance of placement, request fewer replicas or choose a config with a smaller hardware footprint.
Deployment DEGRADED with Startup stalled or Not ready: A placed replica is still booting or hit a startup failure. Read the detail after the colon in status.message. The deployment stays DEGRADED rather than FAILED once any replica has been successfully started.
Deployment FAILED with Timed out waiting for readiness: No replica could be provisioned within six hours of the current run’s start. Read the stall cause at the end of status.message. Deploy a new deployment to try again with a fresh readiness budget.
Restart fails with the deployment is in a terminal FAILED state and cannot be restarted; create a new deployment (HTTP 400): A FAILED deployment can’t be brought back by raising replica bounds. Deploy a new deployment on the endpoint instead.
Restart fails with the deployment must be stopped before it can be restarted (HTTP 400): Wait for the deployment to reach DEPLOYMENT_STATE_STOPPED after you stop it, or confirm both replica bounds are 0, before raising them again.
Deployment FAILED for another reason: Read status.message. Common causes include deterministic placement rejection (Cannot place replicas: …), manifest generation failure, or remediation exhaustion.
Model not supported: Not every model can be deployed. See the model catalog. A fine-tuned model deploys only if its base model is supported.
Deploy fails with the model has no revisions to deploy: The model record exists but has no uploaded weights yet. Finish uploading the model and wait for the upload to succeed before you deploy it.
Deploy fails with a revision validation error: When you pin a specific model or speculator revision, that revision must have passed validation first. Check validationStatus on the revision (custom models, adapters). Deploy the latest validated revision, or wait for the pinned revision to finish validating.
Deployment delete fails with the deployment is referenced by an endpoint's traffic split and cannot be deleted; please drop traffic split weight to 0 before deleting the deployment (HTTP 400): The deployment still has weight in the endpoint’s traffic split. Set its weight to 0 (or remove it from the split) before deleting. The CLI’s tg beta endpoints rm dep_... detaches it automatically.

Next steps

Configure autoscaling

Autoscale a deployment on the right metric.

Route traffic

Split traffic across deployments behind one endpoint.

Observability

Monitor metrics and scrape the Prometheus-compatible endpoint.

Pricing

Understand per-minute and reserved pricing.

GET STARTED

SERVERLESS

INFERENCE APIS

DEDICATED MODEL INFERENCE

DEDICATED CONTAINER INFERENCE

GPU CLUSTERS

FINE-TUNING

CODE EXECUTION

ADMINISTRATION

Manage endpoints and deployments

Create an endpoint

Create a deployment

Poll deployment status

Deployment states

Scale a deployment

Stop a deployment

Restart a deployment

List resources

List flags

Delete resources

Troubleshooting

Next steps

Configure autoscaling

Route traffic

Observability

Pricing

​Create an endpoint

​Create a deployment

​Poll deployment status

​Deployment states

​Scale a deployment

​Stop a deployment

​Restart a deployment

​List resources

​List flags

​Delete resources

​Troubleshooting

​Next steps

Configure autoscaling

Route traffic

Observability

Pricing

Create an endpoint

Create a deployment

Poll deployment status

Deployment states

Scale a deployment

Stop a deployment

Restart a deployment

List resources

List flags

Delete resources

Troubleshooting

Next steps