Queue API

The Queue API provides asynchronous job processing for Dedicated Containers. Submit jobs to a managed queue, and workers automatically claim and process them. This model supports long-running inference, batch workloads, and explicit priority control.

New to Dedicated Containers? Start with the Overview to understand the platform, or jump to the Quickstart to deploy your first container.

Core Concepts

Jobs

A job is a single unit of work submitted to your deployment. Jobs can run for seconds or hours, making them ideal for:

Video generation
Batch image processing
Long-running inference tasks
Any workload that doesn’t fit the request-response pattern

Job Lifecycle

Status	Description
`pending`	Job is queued, waiting for a worker to claim it
`running`	Job has been claimed and is being processed
`done`	Job completed successfully
`failed`	Job failed with an error
`canceled`	Job was canceled before processing started

Priority

Jobs are processed in strict order of priority first, then submission time. Priority is an integer where higher values are processed first.

# High priority job (processed first)
client.beta.queue.submit(model="my-model", payload={...}, priority=10)

# Normal priority job
client.beta.queue.submit(model="my-model", payload={...}, priority=1)

# Low priority job (processed last)
client.beta.queue.submit(model="my-model", payload={...}, priority=0)

By default, priority is not considered for autoscaling metrics—the autoscaler scales based on total queue depth regardless of priority. Contact [email protected] for advanced scaling policies that account for priority tiers.

Job State with `info`

The info field provides persistent state that survives across the job lifecycle. You can:

Set initial state when submitting a job via the info parameter
Update state during processing using emit() in your Sprocket worker
Preserve state across retries—info accumulates rather than resets

This is useful for tracking progress, storing metadata, or passing context between retries.

job = client.beta.queue.submit(
    model="my-model",
    payload={"prompt": "A cat playing piano"},
    info={"user_id": "user_123", "tier": "premium"},
)

from sprocket import emit


def predict(self, args: dict) -> dict:
    emit({"progress": 0.5, "stage": "encoding"})
    # ... more processing
    emit({"progress": 1.0, "stage": "complete"})
    return {"output": result}

For full endpoint documentation — request parameters, response schemas, and error codes — see the Queue REST API Reference: submit, status, cancel, metrics.

Polling for Job Completion

For jobs that take time to complete, poll the status endpoint until the job reaches a terminal state (done, failed, or canceled).

import time
from together import Together

client = Together()

# Submit job
job = client.beta.queue.submit(
    model="my-deployment", payload={"prompt": "Generate a video of a sunset"}
)

print(f"Submitted job: {job.request_id}")

# Poll for completion
while True:
    status = client.beta.queue.retrieve(
        request_id=job.request_id, model="my-deployment"
    )

    if status.status == "done":
        print(f"Success! Result: {status.outputs}")
        break
    elif status.status == "failed":
        print(f"Failed: {status.error}")
        break
    elif status.status == "canceled":
        print("Job was canceled")
        break
    else:
        # Show progress if available
        if status.info and "progress" in status.info:
            print(f"Progress: {status.info['progress']:.0%}")
        time.sleep(2)  # Poll every 2 seconds

#!/bin/bash
REQUEST_ID="019ba379-92da-71e4-ac40-d98059fd67c7"
MODEL="my-deployment"

while true; do
  RESPONSE=$(curl -s "https://api.together.ai/v1/queue/status?request_id=$REQUEST_ID&model=$MODEL" \
    -H "Authorization: Bearer $TOGETHER_API_KEY")

  STATUS=$(echo $RESPONSE | jq -r '.status')

  case $STATUS in
    "done")
      echo "Success!"
      echo $RESPONSE | jq '.outputs'
      break
      ;;
    "failed")
      echo "Failed:"
      echo $RESPONSE | jq '.error'
      break
      ;;
    "canceled")
      echo "Cancelled"
      break
      ;;
    *)
      echo "Status: $STATUS"
      sleep 2
      ;;
  esac
done

Best Practices

Use Priority for Tiered Service

Implement different service tiers by assigning priority based on customer type:

def submit_job(user, payload):
    priority = 10 if user.tier == "premium" else 1
    return client.beta.queue.submit(
        model="my-deployment",
        payload=payload,
        priority=priority,
        info={"user_id": user.id, "tier": user.tier},
    )

Track Progress for Long-Running Jobs

For jobs that take more than a few seconds, emit progress updates so clients can show status:

class VideoGenerator(Sprocket):
    def predict(self, args: dict) -> dict:
        total_frames = args.get("num_frames", 60)

        for i, frame in enumerate(self.generate_frames(args)):
            emit(
                {
                    "progress": (i + 1) / total_frames,
                    "current_frame": i + 1,
                    "total_frames": total_frames,
                }
            )

        return {"video": FileOutput("output.mp4")}

Handle All Terminal States

Always check for done, failed, and canceled when polling:

terminal_states = {"done", "failed", "canceled"}

while status.status not in terminal_states:
    time.sleep(2)
    status = client.beta.queue.retrieve(...)

Store Metadata in `info`

Use info to store job metadata that you’ll need when the job completes:

job = client.beta.queue.submit(
    model="my-deployment",
    payload={"prompt": "..."},
    info={
        "user_id": "user_123",
        "callback_url": "https://myapp.com/webhook",
        "requested_at": datetime.now().isoformat(),
    },
)

Error Codes

Code	Description
`400`	Invalid request (missing required fields, malformed payload)
`401`	Unauthorized (invalid or missing API key)
`404`	Job or deployment not found
`409`	Cannot cancel job (already running or completed)
`500`	Internal server error

Dedicated Containers Overview – Architecture and concepts
Quickstart – Deploy your first container
Sprocket SDK – Build queue-integrated workers
Jig CLI – Deploy and manage containers

GET STARTED

SERVERLESS

INFERENCE APIS

DEDICATED MODEL INFERENCE

DEDICATED CONTAINER INFERENCE

GPU CLUSTERS

FINE-TUNING

CODE EXECUTION

ADMINISTRATION

Core Concepts

Jobs

Job Lifecycle

Priority

Job State with `info`

Polling for Job Completion

Best Practices

Use Priority for Tiered Service

Track Progress for Long-Running Jobs

Handle All Terminal States

Store Metadata in `info`

Error Codes

​Core Concepts

​Jobs

​Job Lifecycle

​Priority

​Job State with info

​Polling for Job Completion

​Best Practices

​Use Priority for Tiered Service

​Track Progress for Long-Running Jobs

​Handle All Terminal States

​Store Metadata in info

​Error Codes

​Related Resources

Core Concepts

Jobs

Job Lifecycle

Priority

Job State with `info`

Polling for Job Completion

Best Practices

Use Priority for Tiered Service

Track Progress for Long-Running Jobs

Handle All Terminal States

Store Metadata in `info`

Error Codes

Related Resources