> ## Documentation Index
> Fetch the complete documentation index at: https://docs.together.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Deploy a prebuilt vLLM image

> Serve a model from the stock vLLM Docker image with dedicated container inference.

This guide uses [dedicated container inference](https://docs.together.ai/docs/dedicated-container-inference.md) to serve [Qwen2.5-VL-32B-Instruct](https://huggingface.co/Qwen/Qwen2.5-VL-32B-Instruct) from the public [`vllm/vllm-openai`](https://hub.docker.com/r/vllm/vllm-openai) image. vLLM already serves an OpenAI-compatible API, so you don't need a Sprocket worker or a Docker build. Jig deploys the image as-is, using two settings in `[tool.jig.deploy]`:

* **`image`:** Deploy an existing image instead of building one. Jig skips `docker build` and `docker push`.
* **`command`:** Override the image's startup command. Here it runs `vllm serve` with the model and serving flags.

Requests go straight to vLLM's HTTP server instead of through the managed queue, so you call the deployment with any OpenAI client or `curl`, not `jig submit`.

The full example is in the Sprocket repo at [`examples/qwen_vllm`](https://github.com/togethercomputer/sprocket/tree/main/examples/qwen_vllm).

## Requirements

* **Together API key:** Get one from [together.ai](https://together.ai).
* **Dedicated container inference access:** Contact your account representative or the [sales team](https://www.together.ai/contact-sales) to enable it for your organization.
* **Together CLI:** Install with `uv tool install "together[cli]"`.
* **OpenAI Python SDK:** The Python request sample uses it. Add it to your Python project with `uv add openai`.

Export your API key. The deploy step and the request samples read it from `TOGETHER_API_KEY`:

```shell theme={null}
export TOGETHER_API_KEY=your_key_here
```

## Configure the deployment

Create a directory with a single `pyproject.toml`:

```toml pyproject.toml theme={null}
[project]
name = "qwen25-vl"
version = "1.0.0"

[tool.jig.deploy]
# Deploy an existing image instead of building one
image = "vllm/vllm-openai:v0.17.1"

description = "Qwen2.5-VL-32B on stock vLLM"
gpu_type = "h100-80gb"
gpu_count = 2
cpu = 4
memory = 64
port = 8000
min_replicas = 1
max_replicas = 10
command = [
    "vllm",
    "serve",
    "Qwen/Qwen2.5-VL-32B-Instruct",
    "--port",
    "8000",
    "--host",
    "0.0.0.0",
    "--dtype",
    "bfloat16",
    "--limit-mm-per-prompt.image",
    "5",
    "--limit-mm-per-prompt.video",
    "5",
    "--tensor-parallel-size",
    "2",
]

[tool.jig.deploy.autoscaling]
metric = "CustomMetric"
custom_metric_name = "vllm:num_requests_running"
target = 4
```

Before you deploy, check these settings:

* **Pin the image tag:** This guide is tested with `v0.17.1`. Pin a specific tag, not `latest`, so a replica that restarts doesn't pick up a new vLLM release.
* **Match GPUs to tensor parallelism:** Set `--tensor-parallel-size` to `gpu_count` so vLLM uses every GPU in the replica. Here, vLLM splits the 32B model in BF16 across two H100s.
* **Match `port` to `--port`:** Together routes traffic to the port in `port`, so it must match the port vLLM listens on.
* **Keep the default health check:** The default `health_check_path` is `/health`, which vLLM serves. The deployment reports ready once vLLM has loaded the model.
* **Autoscale on vLLM's own metric:** vLLM exports Prometheus metrics on `/metrics`, so the [`CustomMetric` autoscaler](https://docs.together.ai/docs/together-deployments#autoscaling) scales on `vllm:num_requests_running` with no extra code.
* **Use a unique project name:** The `name` under `[project]` is the deployment name and the last segment of the endpoint URL. Deployment names must be unique, so if `qwen25-vl` is taken, change `name` and use the new name in the endpoint URL.

### Serve a different model

Change the model ID in `command` and in your requests, then size `gpu_count`, `--tensor-parallel-size`, and `memory` to fit the new model. Any `vllm serve` flag can go in `command`.

Qwen2.5-VL-32B-Instruct isn't gated, so this guide needs no Hugging Face token. For a gated model, store your token as a [secret](https://docs.together.ai/reference/cli/jig.md#jig-secrets-set) before you deploy. vLLM reads `HF_TOKEN` from the environment when it downloads the weights. Run the command from the directory that contains `pyproject.toml`, so the secret attaches to this deployment:

```shell theme={null}
tg beta jig secrets set --name HF_TOKEN --value your_hf_token
```

## Deploy

> [!NOTE]
>
> If you're logged in with `tg login`, pass your API key explicitly on the first deploy: `tg beta jig deploy --detach --api-key "$TOGETHER_API_KEY"`. The first deploy stores an API key in the deployment, and it can't use a `tg login` session for that. While you're logged in, the CLI ignores `TOGETHER_API_KEY`.

Run these from the directory that contains `pyproject.toml`:

```shell theme={null}
# Create the deployment from the configured image
tg beta jig deploy --detach

# Watch vLLM start up (Ctrl+C to stop)
tg beta jig logs --follow

# Check readiness
tg beta jig status
```

`--detach` returns as soon as the deployment is created. Without it, `jig deploy` waits for the replica but gives up two minutes after the container starts, while vLLM is still loading the model, and exits with `Running but not ready after 120s`. The deployment keeps starting either way.

The first deploy takes 5 to 15 minutes while Together schedules two H100s, pulls the vLLM image, and vLLM downloads and loads about 65 GB of weights. The deployment is ready when `jig status` shows `Ready` for `Status` and `1/1 ready` for `Replicas`.

When the deployment is ready, get the endpoint URL:

```shell theme={null}
tg beta jig endpoint
```

The URL has the form `https://api.together.ai/v1/deployment-request/<DEPLOYMENT_NAME>`. Requests to any path under it are forwarded to the container, authenticated with your Together API key.

## Call the endpoint

Point any OpenAI client at the endpoint URL plus `/v1`, using your Together API key. The `model` field must match the model ID you pass to `vllm serve` in `command`:

```python OpenAI Python SDK theme={null}
import os

from openai import OpenAI

client = OpenAI(
    base_url="https://api.together.ai/v1/deployment-request/qwen25-vl/v1",
    api_key=os.environ["TOGETHER_API_KEY"],
)

response = client.chat.completions.create(
    model="Qwen/Qwen2.5-VL-32B-Instruct",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://huggingface.co/datasets/patrickvonplaten/random_img/resolve/main/yosemite.png"
                    },
                },
                {"type": "text", "text": "Describe this image."},
            ],
        }
    ],
)

print(response.choices[0].message.content)
```

```bash cURL theme={null}
curl https://api.together.ai/v1/deployment-request/qwen25-vl/v1/chat/completions \
  -H "Authorization: Bearer $TOGETHER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-VL-32B-Instruct",
    "messages": [
      {
        "role": "user",
        "content": [
          {"type": "image_url", "image_url": {"url": "https://huggingface.co/datasets/patrickvonplaten/random_img/resolve/main/yosemite.png"}},
          {"type": "text", "text": "Describe this image."}
        ]
      }
    ]
  }'
```

The cURL request returns a standard OpenAI chat completion, and the Python sample prints only its `content`. A shortened response looks like this:

```json theme={null}
{
  "id": "chatcmpl-b15bc3b243a529ed",
  "object": "chat.completion",
  "model": "Qwen/Qwen2.5-VL-32B-Instruct",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "This image depicts a serene and picturesque natural landscape, likely from a national park or wilderness area..."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": { "prompt_tokens": 1759, "completion_tokens": 471, "total_tokens": 2230 }
}
```

Other routes vLLM serves, such as `/v1/completions` and `/v1/models`, are available under the same endpoint URL.

## Delete the deployment

The deployment keeps at least `min_replicas` replicas running, two H100s each, until you delete it:

```shell theme={null}
tg beta jig destroy
```

`jig destroy` doesn't delete the `TOGETHER_API_KEY` secret that the first deploy stored. To delete it too, run this from the same directory:

```shell theme={null}
tg beta jig secrets delete --name TOGETHER_API_KEY
```

## Next steps

> [card] [OpenAI-compatible endpoint with Sprocket](https://docs.together.ai/docs/dedicated-containers-openai.md)
>
> Write your own worker when you need custom pre- or post-processing.

> [card] [Autoscaling](https://docs.together.ai/docs/together-deployments.md#autoscaling)
>
> Choose scaling metrics and targets, and see how replicas scale.

> [card] [Jig CLI reference](https://docs.together.ai/reference/cli/jig.md)
>
> Look up every `[tool.jig.deploy]` setting and deploy command.

> [card] [Sprocket examples](https://github.com/togethercomputer/sprocket/tree/main/examples)
>
> Browse more deployable examples, from hello-world to speech-to-speech.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.