Deploy a prebuilt vLLM image

This guide uses dedicated container inference to serve Qwen2.5-VL-32B-Instruct from the public vllm/vllm-openai image. vLLM already serves an OpenAI-compatible API, so you don’t need a Sprocket worker or a Docker build. Jig deploys the image as-is, using two settings in [tool.jig.deploy]:

image: Deploy an existing image instead of building one. Jig skips docker build and docker push.
command: Override the image’s startup command. Here it runs vllm serve with the model and serving flags.

Requests go straight to vLLM’s HTTP server instead of through the managed queue, so you call the deployment with any OpenAI client or curl, not jig submit. The full example is in the Sprocket repo at examples/qwen_vllm.

Requirements

Together API key: Get one from together.ai.
Dedicated container inference access: Contact your account representative or the sales team to enable it for your organization.
Together CLI: Install with uv tool install "together[cli]".
OpenAI Python SDK: The Python request sample uses it. Add it to your Python project with uv add openai.

Export your API key. The deploy step and the request samples read it from TOGETHER_API_KEY:

export TOGETHER_API_KEY=your_key_here

Configure the deployment

Create a directory with a single pyproject.toml:

pyproject.toml

[project]
name = "qwen25-vl"
version = "1.0.0"

[tool.jig.deploy]
# Deploy an existing image instead of building one
image = "vllm/vllm-openai:v0.17.1"

description = "Qwen2.5-VL-32B on stock vLLM"
gpu_type = "h100-80gb"
gpu_count = 2
cpu = 4
memory = 64
port = 8000
min_replicas = 1
max_replicas = 10
command = [
    "vllm",
    "serve",
    "Qwen/Qwen2.5-VL-32B-Instruct",
    "--port",
    "8000",
    "--host",
    "0.0.0.0",
    "--dtype",
    "bfloat16",
    "--limit-mm-per-prompt.image",
    "5",
    "--limit-mm-per-prompt.video",
    "5",
    "--tensor-parallel-size",
    "2",
]

[tool.jig.deploy.autoscaling]
metric = "CustomMetric"
custom_metric_name = "vllm:num_requests_running"
target = 4

Before you deploy, check these settings:

Pin the image tag: This guide is tested with v0.17.1. Pin a specific tag, not latest, so a replica that restarts doesn’t pick up a new vLLM release.
Match GPUs to tensor parallelism: Set --tensor-parallel-size to gpu_count so vLLM uses every GPU in the replica. Here, vLLM splits the 32B model in BF16 across two H100s.
Match port to --port: Together routes traffic to the port in port, so it must match the port vLLM listens on.
Keep the default health check: The default health_check_path is /health, which vLLM serves. The deployment reports ready once vLLM has loaded the model.
Autoscale on vLLM’s own metric: vLLM exports Prometheus metrics on /metrics, so the CustomMetric autoscaler scales on vllm:num_requests_running with no extra code.
Use a unique project name: The name under [project] is the deployment name and the last segment of the endpoint URL. Deployment names must be unique, so if qwen25-vl is taken, change name and use the new name in the endpoint URL.

Serve a different model

Change the model ID in command and in your requests, then size gpu_count, --tensor-parallel-size, and memory to fit the new model. Any vllm serve flag can go in command. Qwen2.5-VL-32B-Instruct isn’t gated, so this guide needs no Hugging Face token. For a gated model, store your token as a secret before you deploy. vLLM reads HF_TOKEN from the environment when it downloads the weights. Run the command from the directory that contains pyproject.toml, so the secret attaches to this deployment:

tg beta jig secrets set --name HF_TOKEN --value your_hf_token

Deploy

If you’re logged in with tg login, pass your API key explicitly on the first deploy: tg beta jig deploy --detach --api-key "$TOGETHER_API_KEY". The first deploy stores an API key in the deployment, and it can’t use a tg login session for that. While you’re logged in, the CLI ignores TOGETHER_API_KEY.

Run these from the directory that contains pyproject.toml:

# Create the deployment from the configured image
tg beta jig deploy --detach

# Watch vLLM start up (Ctrl+C to stop)
tg beta jig logs --follow

# Check readiness
tg beta jig status

--detach returns as soon as the deployment is created. Without it, jig deploy waits for the replica but gives up two minutes after the container starts, while vLLM is still loading the model, and exits with Running but not ready after 120s. The deployment keeps starting either way. The first deploy takes 5 to 15 minutes while Together schedules two H100s, pulls the vLLM image, and vLLM downloads and loads about 65 GB of weights. The deployment is ready when jig status shows Ready for Status and 1/1 ready for Replicas. When the deployment is ready, get the endpoint URL:

tg beta jig endpoint

The URL has the form https://api.together.ai/v1/deployment-request/<DEPLOYMENT_NAME>. Requests to any path under it are forwarded to the container, authenticated with your Together API key.

Call the endpoint

Point any OpenAI client at the endpoint URL plus /v1, using your Together API key. The model field must match the model ID you pass to vllm serve in command:

import os

from openai import OpenAI

client = OpenAI(
    base_url="https://api.together.ai/v1/deployment-request/qwen25-vl/v1",
    api_key=os.environ["TOGETHER_API_KEY"],
)

response = client.chat.completions.create(
    model="Qwen/Qwen2.5-VL-32B-Instruct",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://huggingface.co/datasets/patrickvonplaten/random_img/resolve/main/yosemite.png"
                    },
                },
                {"type": "text", "text": "Describe this image."},
            ],
        }
    ],
)

print(response.choices[0].message.content)

curl https://api.together.ai/v1/deployment-request/qwen25-vl/v1/chat/completions \
  -H "Authorization: Bearer $TOGETHER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-VL-32B-Instruct",
    "messages": [
      {
        "role": "user",
        "content": [
          {"type": "image_url", "image_url": {"url": "https://huggingface.co/datasets/patrickvonplaten/random_img/resolve/main/yosemite.png"}},
          {"type": "text", "text": "Describe this image."}
        ]
      }
    ]
  }'

The cURL request returns a standard OpenAI chat completion, and the Python sample prints only its content. A shortened response looks like this:

{
  "id": "chatcmpl-b15bc3b243a529ed",
  "object": "chat.completion",
  "model": "Qwen/Qwen2.5-VL-32B-Instruct",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "This image depicts a serene and picturesque natural landscape, likely from a national park or wilderness area..."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": { "prompt_tokens": 1759, "completion_tokens": 471, "total_tokens": 2230 }
}

Other routes vLLM serves, such as /v1/completions and /v1/models, are available under the same endpoint URL.

Delete the deployment

The deployment keeps at least min_replicas replicas running, two H100s each, until you delete it:

tg beta jig destroy

jig destroy doesn’t delete the TOGETHER_API_KEY secret that the first deploy stored. To delete it too, run this from the same directory:

tg beta jig secrets delete --name TOGETHER_API_KEY

Next steps

OpenAI-compatible endpoint with Sprocket

Write your own worker when you need custom pre- or post-processing.

Autoscaling

Choose scaling metrics and targets, and see how replicas scale.

Jig CLI reference

Look up every [tool.jig.deploy] setting and deploy command.

Sprocket examples

Browse more deployable examples, from hello-world to speech-to-speech.

GET STARTED

SERVERLESS

INFERENCE APIS

DEDICATED MODEL INFERENCE

DEDICATED CONTAINER INFERENCE

POST-TRAINING

FINE-TUNING

GPU CLUSTERS

CODE EXECUTION

ADMINISTRATION

Requirements

Configure the deployment

Serve a different model

Deploy

Call the endpoint

Delete the deployment

Next steps

OpenAI-compatible endpoint with Sprocket

Autoscaling

Jig CLI reference

Sprocket examples

​Requirements

​Configure the deployment

​Serve a different model

​Deploy

​Call the endpoint

​Delete the deployment

​Next steps

OpenAI-compatible endpoint with Sprocket

Autoscaling

Jig CLI reference

Sprocket examples

Requirements

Configure the deployment

Serve a different model

Deploy

Call the endpoint

Delete the deployment

Next steps