Manage endpoints and deployments

This page covers the lifecycle operations for dedicated model inference (DMI): creating endpoints and deployments, scaling, stopping, and deleting resources. The CLI’s tg beta endpoints deploy command bundles several API/SDK operations into one step for convenience: it creates the endpoint (when you pass a new endpoint name), attaches a deployment to it, and routes 100% of traffic to that deployment. This page shows the individual operations underneath it. To create your first deployment end-to-end, follow the quickstart.

Create an endpoint

Before you can create a deployment, you must select a model, choose a config, and create an endpoint. The CLI has no standalone create-endpoint command: tg beta endpoints deploy creates the endpoint and its first deployment together (see Create a deployment below). To create an empty endpoint on its own, use the SDK or API:

Python

from together import Together

client = Together()
project_id = client.whoami().project_id

endpoint = client.beta.endpoints.create(
    project_id=project_id,
    name="my-endpoint",
)
print(endpoint)

The endpoint serves as a logical grouping of deployments, and the entry point for routing traffic to your models.

Create a deployment

A deployment binds a model and a hardware config to an endpoint, and sets the autoscaling policy for spinning replicas up and down based on demand. --min-replicas and --max-replicas set the replica bounds. Pass an existing endpoint ID (or a new endpoint name) to tg beta endpoints deploy --endpoint to add a deployment. The model is the positional argument, and the config is --config:

CLI

tg beta endpoints deploy ml_CbJNwQC2ZqCU2iFT3mrCh \
  --endpoint ep_abc123 \
  --deployment-name my-deployment \
  --config cr_CbzGdmn14t3HYrXXitmKa \
  --min-replicas 1 --max-replicas 2

When a model has more than one deployment profile, deploy returns an error that lists the available profiles, for example:

Model has multiple deployment profiles. Re-run with --config <config_id>:
  cr_CbzGdmn14t3HYrXXitmKa  NVIDIA-H100 x1  BF16  TP1
  cr_CciJqTB35QmpMupbQNPPW  NVIDIA-H100 x1  FP8   TP1

Re-run with --config <cr_...> to choose one. When a model has a single profile, the CLI selects it automatically. List a model’s profiles anytime with tg beta models configs <model_id>. The CLI defaults --min-replicas and --max-replicas to 1, so a bare deploy creates a single-replica deployment. List and retrieve responses return a flat deployment object whose model and config resource names you can copy straight into a new create request:

{
  "id": "dep_abc123",
  "projectId": "proj_abc123",
  "endpointId": "ep_abc123",
  "name": "your-project-slug/my-endpoint/my-deployment",
  "createdAt": "2026-07-13T18:52:00Z",
  "updatedAt": "2026-07-13T18:52:00Z",
  "model": "projects/proj_abc123/models/ml_CbJNwQC2ZqCU2iFT3mrCh/revisions/rv_CbJ9yNrws93VQrZum1fTE",
  "config": "projects/proj_abc123/configs/cr_CbzGdmn14t3HYrXXitmKa",
  "autoscaling": {"minReplicas": 1, "maxReplicas": 2},
  "hardware": "1xnvidia-h100-80gb",
  "trafficMode": "TRAFFIC_MODE_LIVE",
  "desiredReplicas": 1,
  "status": {"state": "DEPLOYMENT_STATE_PROVISIONING", "scheduledReplicas": 0, "readyReplicas": 0, "message": "Scheduling replicas"}
}

After you’ve created a deployment, you’ll need to route traffic to it before it can serve requests.

Create request flags

Set these flags on tg beta endpoints deploy when you create a deployment:

Flag	Required	Description
`MODEL`	Yes	Positional argument. The model to deploy: a public model name, a private model name, a private model ID (`ml_...`), or a fully resolved model path.
`--endpoint`	Yes	The endpoint to deploy to. Pass an existing endpoint name or ID, or a new name to create the endpoint first.
`--config`	No	Config ID (`cr_...`). Required when the model has more than one deployment profile; auto-selected when it has a single one.
`--deployment-name`	No	Name for the deployment. Defaults to a combination of the endpoint and model names.
`--min-replicas`	No	Minimum replicas. Default `1`.
`--max-replicas`	No	Maximum replicas. Default `1`.
`--scaling-metric`	No	Metric to autoscale on, set together with `--scaling-target`. Omit to use the default metric. See scaling metrics.
`--scaling-target`	No	Target value for `--scaling-metric`.
`--enable-lora`	No	Run the multi-LoRA kernel so adapters hot-load after deploy. Set at create time only: it can’t be changed on a running deployment, so redeploy to toggle it.

For the full flag list, including placement, the autoscaling windows, and the scaling percentile, see the CLI reference. Decoding optimizations, including speculative decoding, come from the config you select.

Poll deployment status

To check a deployment’s status, run tg beta endpoints get on its endpoint. The output lists each deployment’s state and ready/desired replica counts, so re-run it to watch a specific deployment come up:

CLI

# Show the endpoint with each deployment's state and replica counts
tg beta endpoints get ep_abc123

For the full set of status fields (scheduled replicas, status message), retrieve the deployment from the SDK or API and read status:

Python

from together import Together

client = Together()
project_id = client.whoami().project_id

deployment = client.beta.endpoints.deployments.retrieve(
    "dep_abc123",
    project_id=project_id,
    endpoint_id="ep_abc123",
)
print(deployment.status.state)

From here you can track its progress scaling up:

Field	Description
`desiredReplicas`	Target replica count from autoscaling.
`status.scheduledReplicas`	Replicas the scheduler has placed on clusters. May trail `desiredReplicas` while capacity is still being found, and exceed `status.readyReplicas` while placed replicas start.
`status.readyReplicas`	Replicas actively serving traffic.
`status.message`	Human-readable explanation of the current stage or cause. Replica progress lives in the counts above, not in this string.
`status.state`	See below.

Deployment states

A deployment reports its lifecycle in status.state. The API returns the fully-qualified enum (for example DEPLOYMENT_STATE_READY). This page uses the short name for readability.

State	Description
`PROVISIONING`	The scheduler is placing replicas on clusters. `status.message` is `Scheduling replicas`.
`SCALING`	Replicas are starting or draining to reach the desired count. `status.message` is `Starting replicas` or `Scaling down`.
`READY`	All replicas are healthy and serving. `status.message` is `All replicas ready`. A deployment must also be in the endpoint’s traffic split to receive requests.
`DEGRADED`	The deployment is below the requested capacity or blocked by a transient issue. `status.message` explains the cause. It usually resolves on its own.
`STOPPING`	A transient teardown state. Replicas are draining after a stop was requested. It settles to `STOPPED` once cleanup completes, or to `FAILED` if teardown ends with a failure.
`STOPPED`	Scaled to zero replicas. The deployment isn’t billing and isn’t serving.
`FAILED`	Terminal state. The `status.message` field explains why.

A deployment that never reaches READY within six hours after starting will be marked as FAILED.

Scale a deployment

Deployment scale is controlled by the deployment’s replica bounds, and optionally autoscaled using scaling metrics. Set the initial bounds at deploy time (--min-replicas/--max-replicas), then change them on a running deployment:

CLI

tg beta endpoints update dep_abc123 --min-replicas 2 --max-replicas 4

Stop a deployment

A deployment runs until you stop it. To stop a deployment and release its hardware, set both its replica bounds to 0:

CLI

tg beta endpoints update dep_abc123 --min-replicas 0 --max-replicas 0

The replicas keep serving until they finish draining, then the deployment moves to DEPLOYMENT_STATE_STOPPED and billing stops.

Restart a deployment

A stopped deployment doesn’t restart on its own. To bring it back, raise both bounds to 1 or more:

CLI

tg beta endpoints update dep_abc123 --min-replicas 1 --max-replicas 2

List resources

List and get endpoints with the CLI:

CLI

# All endpoints in the project
tg beta endpoints ls

# One endpoint (includes each deployment's state and replica counts)
tg beta endpoints get ep_abc123

List flags

tg beta endpoints ls accepts these flags:

Flag	Description
`--limit`	Maximum number of endpoints to return.
`--after`	Pagination cursor to start from.
`--org`	List org-scoped endpoints instead of project-scoped ones.
`--public`	List public endpoints.

List responses are paginated: when more results are available, the response includes next_cursor, which you pass as --after on the next request.

Delete resources

Deletion is permanent. A deployment must be stopped before it can be deleted. Follow this order:

Scale the deployment to zero and wait for DEPLOYMENT_STATE_STOPPED.
Delete the deployment. If you use the SDK or API (not the CLI), set the deployment’s traffic split weight to 0 on the endpoint first.
Delete the endpoint once it has no deployments.

The CLI’s rm command is a smart-delete: it resolves the resource by its ID prefix, so the same command deletes an endpoint (ep_), a deployment (dep_), an A/B experiment (abx_), or a shadow experiment (exp_). When you run tg beta endpoints rm dep_..., the CLI automatically detaches the deployment from the traffic split and from any experiments it belongs to:

CLI

# Delete the deployment (must be stopped first; auto-detaches from the traffic split)
tg beta endpoints rm dep_abc123

# Delete the endpoint once it has no deployments
tg beta endpoints rm ep_abc123

To delete an endpoint that still has deployments, pass --force to rm. If the endpoint has other deployments you want to keep, rebalance the remaining weights instead of clearing the split. See Route traffic.

Troubleshooting

endpoint_not_configured (HTTP 400) though the deployment is READY: Confirm the deployment is in the endpoint’s traffic split with a non-zero weight.
Deployment DEGRADED with Cannot place replicas: insufficient GPU capacity: Hardware for the config is constrained, so the scheduler couldn’t place all replicas yet. Compare status.scheduledReplicas to desiredReplicas. The scheduler keeps retrying and the deployment starts once capacity frees up. To improve the chance of placement, request fewer replicas or choose a config with a smaller hardware footprint.
Deployment DEGRADED with Startup stalled or Not ready: A placed replica is still booting or hit a startup failure. Read the detail after the colon in status.message. The deployment stays DEGRADED rather than FAILED once any replica has been successfully started.
Deployment FAILED with Timed out waiting for readiness: No replica could be provisioned within six hours of the current run’s start. Read the stall cause at the end of status.message. Restart the deployment to begin a fresh budget.
Deployment FAILED for another reason: Read status.message. Common causes include deterministic placement rejection (Cannot place replicas: …), manifest generation failure, or remediation exhaustion.
Model not supported: Not every model can be deployed. See the model catalog. A fine-tuned model deploys only if its base model is supported.
Deploy fails with the model has no revisions to deploy: The model record exists but has no uploaded weights yet. Finish uploading the model and wait for the upload to succeed before you deploy it.
Deploy fails with a revision validation error: When you pin a specific model or speculator revision, that revision must have passed validation first. Deploy the latest validated revision, or wait for the pinned revision to finish validating.
Deployment delete fails with the deployment is referenced by an endpoint's traffic split and cannot be deleted; please drop traffic split weight to 0 before deleting the deployment (HTTP 400): The deployment still has weight in the endpoint’s traffic split. Set its weight to 0 (or remove it from the split) before deleting. The CLI’s tg beta endpoints rm dep_... detaches it automatically.

Next steps

Configure autoscaling

Autoscale a deployment on the right metric.

Route traffic

Split traffic across deployments behind one endpoint.

Observability

Monitor metrics and scrape the Prometheus-compatible endpoint.

Pricing

Understand per-minute and reserved pricing.

GET STARTED

SERVERLESS

INFERENCE APIS

DEDICATED MODEL INFERENCE

DEDICATED CONTAINER INFERENCE

GPU CLUSTERS

FINE-TUNING

CODE EXECUTION

ADMINISTRATION

Manage endpoints and deployments

Create an endpoint

Create a deployment

Create request flags

Poll deployment status

Deployment states

Scale a deployment

Stop a deployment

Restart a deployment

List resources

List flags

Delete resources

Troubleshooting

Next steps

Configure autoscaling

Route traffic

Observability

Pricing

​Create an endpoint

​Create a deployment

​Create request flags

​Poll deployment status

​Deployment states

​Scale a deployment

​Stop a deployment

​Restart a deployment

​List resources

​List flags

​Delete resources

​Troubleshooting

​Next steps

Configure autoscaling

Route traffic

Observability

Pricing

Create an endpoint

Create a deployment

Create request flags

Poll deployment status

Deployment states

Scale a deployment

Stop a deployment

Restart a deployment

List resources

List flags

Delete resources

Troubleshooting

Next steps