Run a batch job

This tutorial walks through a complete batch inference job from start to finish. By the end you’ll have uploaded a JSONL file of chat completion requests, run them as a single job at up to 50% off serverless rates, and reconciled the responses with your original inputs.

Prerequisites

Before you begin, make sure you have:

Created an account and generated an API key.
Set TOGETHER_API_KEY as an environment variable: export TOGETHER_API_KEY=<your-key>. See API keys and authentication for details.
Installed the Python or TypeScript SDK. Python examples require together>=2.0.0.

Step 1: Prepare a JSONL input file

Each request lives on its own line in a JSONL file. A request has two fields: a custom_id you choose, and a body matching the schema of the endpoint you’re calling. The Batch API runs every line independently and stamps each output with the same custom_id, so this is how you’ll map results back to inputs at the end. Save the following as batch_input.jsonl:

batch_input.jsonl

{"custom_id": "request-1", "body": {"model": "meta-llama/Llama-3.3-70B-Instruct-Turbo", "messages": [{"role": "user", "content": "Hello, world!"}], "max_tokens": 200}}
{"custom_id": "request-2", "body": {"model": "meta-llama/Llama-3.3-70B-Instruct-Turbo", "messages": [{"role": "user", "content": "Explain quantum computing."}], "max_tokens": 200}}

Field	Type	Required	Description
`custom_id`	string	Yes	Unique identifier for tracking (max 64 chars).
`method`	string	Conditional	Set to `"FILE"` when batching `/v1/audio/transcriptions` or `/v1/audio/translations` so the worker dispatches each request as `multipart/form-data`. Omit for chat completion batches. See Run an audio transcription batch.
`body`	object	Yes	Request body matching the endpoint’s schema.

Step 2: Upload the file

Upload the JSONL file with purpose="batch-api". The upload returns a file object whose id you’ll pass to the batch job in the next step. Pass check=False to skip client-side validation. The server still validates the file during the VALIDATING phase, and skipping the client check is faster for large files without changing the error surface. With check=True (default), the SDK parses each JSONL line locally and raises TogetherException before uploading if a line is malformed.

from together import Together

client = Together()

file_resp = client.files.upload(
    file="batch_input.jsonl",
    purpose="batch-api",
    check=False,
)

print(file_resp.id)

import Together from "together-ai";

const client = new Together();

const fileResp = await client.files.upload(
  "batch_input.jsonl",
  "batch-api",
  false,
);

console.log(fileResp.id);

curl -X POST "https://api.together.ai/v1/files/upload" \
  -H "Authorization: Bearer $TOGETHER_API_KEY" \
  -F "purpose=batch-api" \
  -F "file_name=batch_input.jsonl" \
  -F "file=@batch_input.jsonl"

The SDK examples infer the file name from the local path. When calling the REST API directly, include file_name in the multipart form. See the file upload reference for the full request shape.

Step 3: Create the batch

Now hand the uploaded file’s id to the batch endpoint, along with the API endpoint each request should run against. For chat completion requests, that’s /v1/chat/completions. Audio batches use /v1/audio/transcriptions or /v1/audio/translations — see Run an audio transcription batch.

response = client.batches.create(
    input_file_id=file_resp.id,
    endpoint="/v1/chat/completions",
)

batch = response.job
print(batch.id)

const response = await client.batches.create({
  input_file_id: fileResp.id,
  endpoint: "/v1/chat/completions",
});

const batchId = response.job?.id;
console.log(batchId);

curl -X POST "https://api.together.ai/v1/batches" \
  -H "Authorization: Bearer $TOGETHER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"input_file_id": "file-abc123", "endpoint": "/v1/chat/completions"}'

batches.create() returns a wrapper; the batch object lives at .job. batches.retrieve() (used in the next step) returns the batch object directly.

Step 4: Poll for completion

The job moves through VALIDATING, then IN_PROGRESS, then a terminal status: COMPLETED, FAILED, EXPIRED, or CANCELLED. Poll every 30 to 60 seconds until you hit a terminal status. Tighter loops will hit rate limits without giving the server time to make progress.

import time

while True:
    batch = client.batches.retrieve(batch.id)
    print(f"{batch.status}: {batch.progress:.0f}%")

    if batch.status == "COMPLETED":
        break
    if batch.status in ("FAILED", "EXPIRED", "CANCELLED"):
        raise SystemExit(f"Batch ended: {batch.status}")

    time.sleep(30)

let batch = await client.batches.retrieve(batchId);

while (true) {
  batch = await client.batches.retrieve(batchId);
  console.log(`${batch.status}: ${(batch.progress ?? 0).toFixed(0)}%`);

  if (batch.status === "COMPLETED") break;
  if (["FAILED", "EXPIRED", "CANCELLED"].includes(batch.status)) {
    throw new Error(`Batch ended: ${batch.status}`);
  }

  await new Promise((r) => setTimeout(r, 30_000));
}

progress is a float from 0 to 100 representing the percentage of requests completed. It is present on all batch objects but may remain 0 while the job is in VALIDATING.

Most batches under 1,000 requests finish in minutes. The 24-hour completion window is a maximum, not a typical wait.

Step 5: Retrieve the results

When the job reaches COMPLETED, the batch object carries an output_file_id. Download that file and you’ll get one JSON object per line, each keyed by the custom_id from your input. Output line order does not match input line order, so use custom_id to reconcile.

with client.files.with_streaming_response.content(
    id=batch.output_file_id,
) as response:
    with open("batch_output.jsonl", "wb") as f:
        for chunk in response.iter_bytes():
            f.write(chunk)

import * as fs from "fs";

const resp = await client.files.content(batch.output_file_id);
fs.writeFileSync("batch_output.jsonl", await resp.text());

curl -X GET "https://api.together.ai/v1/files/file-output456/content" \
  -H "Authorization: Bearer $TOGETHER_API_KEY" \
  -o batch_output.jsonl

A successful output line looks like:

{
  "custom_id": "request-1",
  "response": {
    "status_code": 200,
    "body": {
      "choices": [
        {
          "index": 0,
          "message": { "role": "assistant", "content": "Hello!" },
          "finish_reason": "stop"
        }
      ],
      "usage": { "prompt_tokens": 12, "completion_tokens": 3, "total_tokens": 15 }
    }
  }
}

Per-request failures land in a separate file referenced by error_file_id. Always check it: a batch can be COMPLETED and still contain individual request failures. See retrieve results and error files on the manage page.

Run an audio transcription batch

The Batch API also supports /v1/audio/transcriptions and /v1/audio/translations for audio workloads (for example, openai/whisper-large-v3). The upload, poll, and retrieve steps above are identical. Two things change: 1. Each JSONL line must include "method": "FILE". The audio endpoints expect multipart/form-data requests, so the worker uses the method field to choose its dispatch mode. Omitting it causes every line to fail with Content-Type must be multipart/form-data in the error file.

audio_batch.jsonl

{"custom_id": "transcription-1", "method": "FILE", "body": {"file": "https://example.com/clip-1.wav", "model": "openai/whisper-large-v3"}}
{"custom_id": "transcription-2", "method": "FILE", "body": {"file": "https://example.com/clip-2.wav", "model": "openai/whisper-large-v3"}}

body.file is the publicly-reachable URL of the audio clip; the worker fetches the audio at execution time. Optional fields such as response_format, language, and prompt pass through to the underlying API — see the audio transcriptions reference for the full schema. 2. Pass the audio endpoint when creating the batch.

response = client.batches.create(
    input_file_id=file_resp.id,
    endpoint="/v1/audio/transcriptions",
)

batch = response.job
print(batch.id)

const response = await client.batches.create({
  input_file_id: fileResp.id,
  endpoint: "/v1/audio/transcriptions",
});

const batchId = response.job?.id;
console.log(batchId);

curl -X POST "https://api.together.ai/v1/batches" \
  -H "Authorization: Bearer $TOGETHER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"input_file_id": "file-abc123", "endpoint": "/v1/audio/transcriptions"}'

A successful output line looks like:

{
  "custom_id": "transcription-1",
  "response": {
    "status_code": 200,
    "body": {
      "duration": 4.825,
      "language": "en",
      "text": "Yet these thoughts affected Hester Prynne less with hope than apprehension."
    }
  }
}

For /v1/audio/translations, swap the endpoint and use a translation-capable model — the JSONL line shape is the same.

Complete script

The full Python program combining all steps above:

Python

import time
from together import Together

client = Together()

file_resp = client.files.upload(
    file="batch_input.jsonl",
    purpose="batch-api",
    check=False,
)
print(f"Uploaded file: {file_resp.id}")

response = client.batches.create(
    input_file_id=file_resp.id,
    endpoint="/v1/chat/completions",
)
batch = response.job
print(f"Created batch: {batch.id}")

while True:
    batch = client.batches.retrieve(batch.id)
    print(f"{batch.status}: {batch.progress:.0f}%")

    if batch.status == "COMPLETED":
        break
    if batch.status in ("FAILED", "EXPIRED", "CANCELLED"):
        raise SystemExit(f"Batch ended: {batch.status}")

    time.sleep(30)

with client.files.with_streaming_response.content(
    id=batch.output_file_id,
) as response:
    with open("batch_output.jsonl", "wb") as f:
        for chunk in response.iter_bytes():
            f.write(chunk)

print("Results saved to batch_output.jsonl")

Next steps

Manage batch jobs: cancel, list, and download error files.
Batch processing overview: rate limits, discounted models, best practices, and FAQ.

GET STARTED

SERVERLESS

INFERENCE APIS

DEDICATED MODEL INFERENCE

DEDICATED CONTAINER INFERENCE

GPU CLUSTERS

FINE-TUNING

CODE EXECUTION

ADMINISTRATION

Prerequisites

Step 1: Prepare a JSONL input file

Step 2: Upload the file

Step 3: Create the batch

Step 4: Poll for completion

Step 5: Retrieve the results

Run an audio transcription batch

Complete script

Next steps

​Prerequisites

​Step 1: Prepare a JSONL input file

​Step 2: Upload the file

​Step 3: Create the batch

​Step 4: Poll for completion

​Step 5: Retrieve the results

​Run an audio transcription batch

​Complete script

​Next steps

Prerequisites

Step 1: Prepare a JSONL input file

Step 2: Upload the file

Step 3: Create the batch

Step 4: Poll for completion

Step 5: Retrieve the results

Run an audio transcription batch

Complete script

Next steps