Send requests

After deploying model and routing traffic to it, you can send requests to the endpoint using the same inference APIs as serverless models. Features available on the underlying model work the same way on DMI, including:

Function calling for tool use.
Structured outputs for JSON-shaped responses.
Streaming responses.

Dedicated model inference is served at https://api-inference.together.ai. There’s no CLI for inference, so send requests with curl or the SDK, pointing the base URL at https://api-inference.together.ai/v1. Pass the endpoint string as the model parameter, and use the same request shape you’d use against a serverless model. The endpoint string has the form your-project-slug/endpoint-name:

from together import Together

client = Together(base_url="https://api-inference.together.ai/v1")

response = client.chat.completions.create(
    model="your-project-slug/endpoint-name",
    messages=[{"role": "user", "content": "What is 2+2?"}],
    max_tokens=512,
)

print(response.choices[0].message.content)

import Together from 'together-ai';

const client = new Together({
  baseURL: 'https://api-inference.together.ai/v1',
});

const response = await client.chat.completions.create({
  model: 'your-project-slug/endpoint-name',
  messages: [{ role: 'user', content: 'What is 2+2?' }],
  max_tokens: 512,
});

console.log(response.choices[0].message.content);

curl -s -X POST https://api-inference.together.ai/v1/chat/completions \
  -H "Authorization: Bearer $TOGETHER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "your-project-slug/endpoint-name",
    "messages": [{"role": "user", "content": "What is 2+2?"}],
    "max_tokens": 512
  }' | jq .

Prompt caching

Prompt caching stores the result of previously processed prompt prefixes so the model can reuse them instead of recomputing. It reduces redundant compute for repeated prefixes, such as a system prompt that’s shared across many requests. Prompt caching is enabled by default for dedicated model inference. No configuration is required.

Decoding optimizations

Decoding optimizations such as speculative decoding are set by the config your deployment runs on. To change them, deploy a different config.

Prompt caching

Decoding optimizations

Next steps

Create a deployment

Route traffic

​Prompt caching

​Decoding optimizations

​Next steps

Create a deployment

Route traffic

Prompt caching

Decoding optimizations

Next steps