---
name: together
description: Use when running inference on open-source LLMs, fine-tuning models,
  deploying dedicated endpoints, building agents with function calling, or
  managing GPU clusters. Agents should reach for this skill when users need to
  call models via API, customize models with training data, set up production
  inference infrastructure, or build AI applications with tool use.
metadata:
  mintlify-proj: together
  version: "1.0"
---

# Together AI Skill

## Product summary

Together AI is a platform for running, training, and serving open-source AI models. Agents use it to call 100+ models via serverless inference (pay-per-token), deploy dedicated endpoints (pay-per-GPU-minute), fine-tune models on custom data, or manage GPU clusters for large-scale workloads.

**Key files and commands:**
- API endpoint: `https://api.together.ai/v1/chat/completions` (serverless) or `https://api-inference.together.ai` (dedicated)
- CLI: `tg` (Together CLI 3.0+) — authenticate with `tg login`, deploy with `tg endpoints deploy`, manage training with `tg fine-tuning`
- SDKs: Python (`together`) and TypeScript (`together-ai`) — both read `TOGETHER_API_KEY` from environment
- Config: API keys are project-scoped; create at https://api.together.ai/settings/projects/~current/api-keys
- Primary docs: https://docs.together.ai

## When to use

Reach for this skill when:
- **Running inference:** User asks to call a model (chat, vision, image generation, video, audio, reasoning)
- **Choosing deployment:** User needs serverless (variable traffic, prototyping), provisioned throughput (production SLA), or dedicated endpoints (fine-tuned models, steady traffic, custom hardware)
- **Fine-tuning:** User has training data and wants to adapt a model with SFT, DPO, or full fine-tuning
- **Building agents:** User needs function calling, multi-turn loops, or tool use workflows
- **Production setup:** User needs monitoring, traffic splitting, A/B tests, rollouts, or autoscaling
- **GPU clusters:** User needs to run large training jobs or batch workloads on reserved hardware

## Quick reference

### Inference modes

| Mode | Billing | Best for | Setup |
|------|---------|----------|-------|
| **Serverless** | Per token | Prototyping, variable traffic | No setup; call `model="moonshotai/Kimi-K3"` |
| **Provisioned throughput** | Per PTU (fixed term) | Production, defined SLA | Contact sales |
| **Dedicated endpoints** | Per GPU-minute | Fine-tuned models, steady traffic, custom hardware | `tg endpoints deploy <model> --endpoint <name>` |

### Common models

| Use case | Recommended model | API string |
|----------|-------------------|-----------|
| **Fast chat** | MiniMax M3.1 Flash | `MiniMaxAI/MiniMax-M3.1-Flash` |
| **Reasoning** | DeepSeek V4 Pro | `deepseek-ai/DeepSeek-V4-Pro-0813` |
| **Vision** | Qwen3.5 9B | `Qwen/Qwen3.5-9B` |
| **Image generation** | FLUX.2 dev | `black-forest-labs/FLUX.2-dev` |
| **Video generation** | Seedance 2.5 | `ByteDance/Seedance-2.5` |
| **Speech-to-text** | Whisper large-v3 | `openai/whisper-large-v3` |

### CLI essentials

```bash
# Authentication
tg login                              # Interactive login
tg logout                             # Clear credentials
export TOGETHER_API_KEY="..."         # Use API key instead

# Inference
tg models list                        # Browse serverless models
tg models configs <model_id>          # Show hardware configs for dedicated

# Endpoints (dedicated inference)
tg endpoints deploy <model> --endpoint <name>    # Create and deploy
tg endpoints ls                       # List endpoints
tg endpoints get <endpoint_id>        # Check status
tg endpoints update <deployment_id> --min-replicas 0 --max-replicas 0  # Stop
tg endpoints rm <endpoint_id> --force # Delete

# Fine-tuning
tg files upload <file.jsonl>          # Upload training data
tg fine-tuning create --training-file <FILE_ID> --model <base_model> --lora  # Start job
tg fine-tuning retrieve <JOB_ID>      # Check progress
tg fine-tuning list                   # List all jobs

# Training (post-training with custom loop)
tg training list                      # List resources
tg training model-limits <model>      # Check constraints
```

### SDK patterns

**Python:**
```python
from together import Together
client = Together()  # reads TOGETHER_API_KEY

# Serverless inference
response = client.chat.completions.create(
    model="MiniMaxAI/MiniMax-M3.1-Flash",
    messages=[{"role": "user", "content": "Hello"}],
    stream=False,
)

# Dedicated endpoint
response = client.chat.completions.create(
    model="your-project-slug/my-endpoint",
    messages=[{"role": "user", "content": "Hello"}],
)

# With function calling
response = client.chat.completions.create(
    model="MiniMaxAI/MiniMax-M3.1-Flash",
    messages=[{"role": "user", "content": "What's the weather?"}],
    tools=[{"type": "function", "function": {...}}],
)
```

**TypeScript:**
```typescript
import Together from 'together-ai';
const client = new Together();

const response = await client.chat.completions.create({
  model: 'MiniMaxAI/MiniMax-M3.1-Flash',
  messages: [{ role: 'user', content: 'Hello' }],
});
```

## Decision guidance

### Serverless vs. dedicated endpoints

| Condition | Use serverless | Use dedicated |
|-----------|---|---|
| Traffic is variable or bursty | ✓ | |
| Traffic is steady and predictable | | ✓ |
| Model is fine-tuned | | ✓ |
| Need custom hardware (H100, B200) | | ✓ |
| Need latency SLA | | ✓ |
| Prototyping or testing | ✓ | |
| High volume (>1M tokens/day) | | ✓ |

### LoRA vs. full fine-tuning

| Factor | LoRA | Full |
|--------|------|------|
| Cost | Lower | Higher |
| Speed | Faster | Slower |
| Quality on in-domain data | Good | Better |
| Quality on out-of-domain data | May degrade | Maintains better |
| Deployment size | Smaller | Larger |
| When to use | Most cases | Major domain shift |

### Training methods

| Method | Use when | Billing |
|--------|----------|---------|
| **Fine-tuning (SFT)** | You have labeled examples of desired behavior | Per token processed |
| **Preference tuning (DPO)** | You have preferred/dispreferred response pairs | Per token processed |
| **Post-training (SFT/DPO/GRPO)** | You need step-level control, custom loss, or RL | Per GPU-minute |

## Workflow

### 1. Make your first inference request

1. Create an API key at https://api.together.ai/settings/projects/~current/api-keys
2. Set `export TOGETHER_API_KEY="your_key"`
3. Install SDK: `pip install together` (Python) or `npm install together-ai` (TypeScript)
4. Run a request:
   ```python
   from together import Together
   client = Together()
   response = client.chat.completions.create(
       model="MiniMaxAI/MiniMax-M3.1-Flash",
       messages=[{"role": "user", "content": "What is 2+2?"}],
   )
   print(response.choices[0].message.content)
   ```
5. Verify the response contains `choices[0].message.content`

### 2. Deploy a dedicated endpoint

1. Choose a model from `/docs/dedicated-endpoints/models`
2. Run: `tg endpoints deploy <model_id> --endpoint <name>`
3. Wait for status to show `DEPLOYMENT_STATE_READY`: `tg endpoints get <endpoint_id>`
4. Call it like serverless, but use `model="<project_slug>/<endpoint_name>"`
5. Stop when done: `tg endpoints update <deployment_id> --min-replicas 0 --max-replicas 0`
6. Delete: `tg endpoints rm <endpoint_id> --force`

### 3. Fine-tune a model

1. Prepare training data in JSONL format (see `/docs/fine-tuning/data-preparation`)
2. Upload: `tg files upload train.jsonl`
3. Start job: `tg fine-tuning create --training-file <FILE_ID> --model <base_model> --lora`
4. Monitor: `tg fine-tuning retrieve <JOB_ID>` (check `status` field)
5. Deploy: Once complete, `tg endpoints deploy <fine_tuned_model_id> --endpoint <name>`
6. Call the endpoint like any other model

### 4. Build an agent with function calling

1. Define tools as JSON schema objects with `type: "function"` and `function: {name, description, parameters}`
2. Send request with `tools` parameter
3. Check response for `tool_calls` in `choices[0].message`
4. Execute each tool call in your code
5. Append tool results to messages as `role: "tool"` with `tool_call_id`
6. Send messages back to model (loop until no more `tool_calls`)
7. Cap loops at 20–50 iterations to prevent spirals

### 5. Stream responses

1. Set `stream=True` (Python) or `stream: true` (TypeScript)
2. Iterate over chunks; each has `choices[0].delta`
3. For reasoning models, check `delta.reasoning_content` or `delta.reasoning` first, then `delta.content`
4. Accumulate deltas until `finish_reason` is not `null`

## Common gotchas

- **API key not found:** Ensure `TOGETHER_API_KEY` is set in environment or passed to `Together(api_key=...)`
- **Model not available:** Serverless and dedicated endpoints have different model catalogs; check `/docs/serverless/models` vs. `/docs/dedicated-endpoints/models`
- **Endpoint still provisioning:** New deployments take 1–5 minutes to reach `DEPLOYMENT_STATE_READY`; poll with `tg endpoints get` before sending requests
- **Rate limit (429):** Reduce request rate, use exponential backoff, or upgrade to dedicated endpoints for higher throughput
- **Context length exceeded:** Set `context_length_exceeded_behavior="truncate"` to auto-truncate, or lower `max_tokens`
- **Function calling not working:** Verify model supports it (check `/docs/serverless/models` for "Function calling" column); not all models support tool use
- **Fine-tuning data format:** JSONL must match schema exactly; validate with `tg fine-tuning preview --training-file <FILE_ID> --model <base_model>`
- **Streaming with function calls:** Streaming works but tool calls arrive in chunks; accumulate `delta.tool_calls` until `finish_reason` is set
- **Dedicated endpoint costs:** Billing starts immediately when deployment reaches `READY`, even if idle; scale to 0 replicas to pause
- **Project scope:** CLI and SDK operations are scoped to a project; set `TOGETHER_PROJECT_ID` or pass `--project` to CLI commands

## Verification checklist

Before submitting work:

- [ ] API key is set and valid (test with `tg whoami`)
- [ ] Model name is correct for the deployment mode (serverless vs. dedicated)
- [ ] For dedicated endpoints: deployment status is `DEPLOYMENT_STATE_READY` before sending requests
- [ ] For fine-tuning: training data is valid JSONL/Parquet and matches the schema
- [ ] For function calling: tools are defined with required fields (`name`, `description`, `parameters`)
- [ ] Streaming is enabled if low latency is needed
- [ ] Error handling includes retry logic for 429, 503, 504 responses
- [ ] For production: monitoring is set up (check `/docs/dedicated-endpoints/monitoring`)
- [ ] Project ID is correct if using multiple projects

## Resources

- **Comprehensive navigation:** https://docs.together.ai/llms.txt
- **Quickstart:** https://docs.together.ai/docs/quickstart
- **Serverless models catalog:** https://docs.together.ai/docs/serverless/models
- **Dedicated endpoints:** https://docs.together.ai/docs/dedicated-endpoints/overview
- **Function calling patterns:** https://docs.together.ai/docs/inference/function-calling/overview
- **Fine-tuning guide:** https://docs.together.ai/docs/fine-tuning/overview
- **Error codes:** https://docs.together.ai/docs/error-codes
- **CLI reference:** https://docs.together.ai/reference/cli/getting-started

---

> For additional documentation and navigation, see: https://docs.together.ai/llms.txt