Overview

Dedicated model inference (DMI) lets you serve a model on reserved hardware, providing several advantages over serverless models:

Better performance: Dedicated GPUs provide higher throughput, lower latency, and more predictable performance.
No hard rate limits: You’re only limited by the capacity of your selected hardware, plus the bounds of your autoscaling configuration.
Fine-tuned models: Deploy a model you fine-tuned from a supported base model.
Cost-efficient at scale: DMI bills per-GPU-minute, which is cheaper at high utilization than serverless models (which bill per-token).

Dedicated model inference uses the same inference APIs as serverless models, so you can prototype on serverless, then deploy on DMI without changing your application code.

If you’re running a stock model in production and want a defined SLA without managing hardware, contact sales for provisioned throughput.

Get started

Quickstart

Deploy and call your first endpoint in a few minutes.

Concepts

The DMI resource model and development workflow.

Manage deployments

Create, scale, stop, and delete your deployments.

Supported models

Browse the list of Together-hosted models you can deploy.

Upload a fine-tuned model

Deploy a model you fine-tuned from a supported base model.

Migrate from v1

Migrate a dedicated endpoint to the new DMI resource model.

Together CLI

The easiest way to manage dedicated model inference is by using the Together CLI. Each command creates and wires up the underlying resources for you:

Command	Description
`tg beta endpoints deploy`	Deploys a model: creates an endpoint, attaches a deployment, and routes all traffic to it.
`tg beta endpoints ab`	Starts an A/B test: adds a variant deployment and splits traffic between it and a control.
`tg beta endpoints shadow`	Starts a shadow experiment: mirrors a fraction of live traffic to a new deployment without serving its responses.
`tg beta endpoints rm`	Deletes any endpoint, deployment, or experiment by its ID.

To learn more about the underlying resources, see Concepts.

Project scope

The tg beta commands, the management API, and the Python SDK’s client.beta.* methods operate within a Together AI project. The CLI reads the project from the TOGETHER_PROJECT_ID environment variable, or you can pass --project on any command. If neither is set, it uses the project associated with your API key. In the Python SDK, pass project_id to Together() or set TOGETHER_PROJECT_ID. Otherwise, call client.whoami().project_id before project-scoped API calls.

export TOGETHER_PROJECT_ID=your_project_id

Development workflow

To create a deployment, run the tg beta endpoints deploy command, passing the model and endpoint name. This creates an endpoint, attaches a deployment, and routes all traffic to it:

tg beta endpoints deploy google/gemma-4-E4B-it \
  --endpoint my-endpoint

The command prints the new endpoint’s inference name (<project_slug>/<endpoint_name>): this is the endpoint string you pass as the model parameter on inference requests. Once the deployment is ready, send inference requests to the endpoint using the same inference API as serverless models. Pass the endpoint string as the model parameter:

from together import Together

client = Together(base_url="https://api-inference.together.ai/v1")

response = client.chat.completions.create(
    model="your-project-slug/my-endpoint",
    messages=[{"role": "user", "content": "What is 2+2?"}],
    max_tokens=512,
)

print(response.choices[0].message.content)

import Together from 'together-ai';

const client = new Together({
  baseURL: 'https://api-inference.together.ai/v1',
});

const response = await client.chat.completions.create({
  model: 'your-project-slug/my-endpoint',
  messages: [{ role: 'user', content: 'What is 2+2?' }],
  max_tokens: 512,
});

console.log(response.choices[0].message.content);

curl -s -X POST https://api-inference.together.ai/v1/chat/completions \
  -H "Authorization: Bearer $TOGETHER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "your-project-slug/my-endpoint",
    "messages": [{"role": "user", "content": "What is 2+2?"}],
    "max_tokens": 512
  }'

deploy bundles the endpoint, deployment, and traffic-routing steps for you. To run those steps individually, or to drive them from the Python and TypeScript SDKs, see Manage deployments. For a step-by-step walkthrough, follow the quickstart. For more details on the DMI resource model, see Concepts.

Key features

Deploy any supported model: Run a Together-hosted model, a model you fine-tuned on Together, or a fine-tuned model you upload.
Autoscale on demand: Scale your deployments with replicas to meet demand, and stop them when you don’t need them to reduce costs.
Split traffic across deployments: Host multiple deployments behind one endpoint URL and route requests between them by weight.
Compare deployments on live traffic: Run an A/B test with control and variant splits to measure a candidate against a baseline before you promote it.
Monitor endpoints: Track latency, throughput, and utilization in built-in dashboards, and trace lifecycle changes through the events feed.

Pricing

Dedicated model inference bills per minute by hardware while a deployment runs, regardless of model or request volume. Each running replica bills independently and stops billing as soon as it scales down. For more details, see Pricing.

GET STARTED

SERVERLESS

INFERENCE APIS

DEDICATED MODEL INFERENCE

DEDICATED CONTAINER INFERENCE

GPU CLUSTERS

FINE-TUNING

CODE EXECUTION

ADMINISTRATION

Get started

Quickstart

Concepts

Manage deployments

Supported models

Upload a fine-tuned model

Migrate from v1

Together CLI

Project scope

Development workflow

Key features

Pricing

​Get started

Quickstart

Concepts

Manage deployments

Supported models

Upload a fine-tuned model

Migrate from v1

​Together CLI

​Project scope

​Development workflow

​Key features

​Pricing

Get started

Together CLI

Project scope

Development workflow

Key features

Pricing