> ## Documentation Index
> Fetch the complete documentation index at: https://docs.together.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Deploy a prebuilt vLLM image

> Serve a model from the stock vLLM Docker image with dedicated container inference.

This guide uses [dedicated container inference](https://docs.together.ai/docs/dedicated-container-inference.md) to serve [Qwen2.5-VL-32B-Instruct](https://huggingface.co/Qwen/Qwen2.5-VL-32B-Instruct) from the public [`vllm/vllm-openai`](https://hub.docker.com/r/vllm/vllm-openai) image. vLLM already serves an OpenAI-compatible API, so you don't need a Sprocket worker or a Docker build. Jig deploys the image as-is, using two settings in `[tool.jig.deploy]`:

* **`image`:** Deploy an existing image instead of building one. Jig skips `docker build` and `docker push`.
* **`command`:** Override the image's startup command. Here it runs `vllm serve` with the model and serving flags.

Requests go straight to vLLM's HTTP server instead of through the managed queue, so you call the deployment with any OpenAI client or `curl`, not `jig submit`.

The full example is in the Sprocket repo at [`examples/qwen_vllm`](https://github.com/togethercomputer/sprocket/tree/main/examples/qwen_vllm).

## Requirements

* **Together API key:** Get one from [together.ai](https://together.ai).
* **Dedicated container inference access:** Contact your account representative or the [sales team](https://www.together.ai/contact-sales) to enable it for your organization.
* **Together CLI:** Install with `uv tool install "together[cli]"`.
* **OpenAI Python SDK:** The Python request sample uses it. Add it to your Python project with `uv add openai`.

Export your API key. The deploy step and the request samples read it from `TOGETHER_API_KEY`:

```shell theme={null}
export TOGETHER_API_KEY=your_key_here
```

## Configure the deployment

Create a directory with a single `pyproject.toml`:

```toml pyproject.toml theme={null}
[project]
name = "qwen25-vl"
version = "1.0.0"

[tool.jig.deploy]
# Deploy an existing image instead of building one
image = "vllm/vllm-openai:v0.17.1"

description = "Qwen2.5-VL-32B on stock vLLM"
gpu_type = "h100-80gb"
gpu_count = 2
cpu = 4
memory = 64
port = 8000
min_replicas = 1
max_replicas = 10
command = [
    "vllm",
    "serve",
    "Qwen/Qwen2.5-VL-32B-Instruct",
    "--port",
    "8000",
    "--host",
    "0.0.0.0",
    "--dtype",
    "bfloat16",
    "--limit-mm-per-prompt.image",
    "5",
    "--limit-mm-per-prompt.video",
    "5",
    "--tensor-parallel-size",
    "2",
]

[tool.jig.deploy.autoscaling]
metric = "CustomMetric"
custom_metric_name = "vllm:num_requests_running"
target = 4
```

Before you deploy, check these settings:

* **Pin the image tag:** This guide is tested with `v0.17.1`. Pin a specific tag, not `latest`, so a replica that restarts doesn't pick up a new vLLM release.
* **Match GPUs to tensor parallelism:** Set `--tensor-parallel-size` to `gpu_count` so vLLM uses every GPU in the replica. Here, vLLM splits the 32B model in BF16 across two H100s.
* **Match `port` to `--port`:** Together routes traffic to the port in `port`, so it must match the port vLLM listens on.
* **Keep the default health check:** The default `health_check_path` is `/health`, which vLLM serves. The deployment reports ready once vLLM has loaded the model.
* **Autoscale on vLLM's own metric:** vLLM exports Prometheus metrics on `/metrics`, so the [`CustomMetric` autoscaler](https://docs.together.ai/docs/together-deployments#autoscaling) scales on `vllm:num_requests_running` with no extra code.
* **Use a unique project name:** The `name` under `[project]` is the deployment name and the last segment of the endpoint URL. Deployment names must be unique, so if `qwen25-vl` is taken, change `name` and use the new name in the endpoint URL.

### Serve a different model

Change the model ID in `command` and in your requests, then size `gpu_count`, `--tensor-parallel-size`, and `memory` to fit the new model. Any `vllm serve` flag can go in `command`.

Qwen2.5-VL-32B-Instruct isn't gated, so this guide needs no Hugging Face token. For a gated model, store your token as a [secret](https://docs.together.ai/reference/cli/jig.md#jig-secrets-set) before you deploy. vLLM reads `HF_TOKEN` from the environment when it downloads the weights. Run the command from the directory that contains `pyproject.toml`, so the secret attaches to this deployment:

```shell theme={null}
tg beta jig secrets set --name HF_TOKEN --value your_hf_token
```

## Deploy

From the directory that contains `pyproject.toml`, create the deployment and watch it start:

```shell theme={null}
# Create the deployment from the configured image
tg beta jig deploy --detach

# Watch vLLM start up (Ctrl+C to stop)
tg beta jig logs --follow

# Check readiness
tg beta jig status
```

With `--detach`, `jig deploy` returns as soon as the deployment is created. Without it, the command waits for the replica to become ready, but it gives up two minutes after the container starts. vLLM takes longer than that to load the model, so the command exits with `Running but not ready after 120s` even though the deployment keeps starting.

The first deploy takes 5 to 15 minutes while Together schedules two H100s, pulls the vLLM image, and vLLM downloads and loads about 65 GB of weights. The deployment is ready when `jig status` shows `Ready` for `Status` and `1/1 ready` for `Replicas`.

When the deployment is ready, get the endpoint URL:

```shell theme={null}
tg beta jig endpoint
```

The URL has the form `https://api.together.ai/v1/deployment-request/<DEPLOYMENT_NAME>`. Together authenticates requests to any path under it with your Together API key, then forwards them to the container.

## Call the endpoint

Point any OpenAI client at the endpoint URL plus `/v1`, using your Together API key. The `model` field must match the model ID you pass to `vllm serve` in `command`:

```python OpenAI Python SDK theme={null}
import os

from openai import OpenAI

client = OpenAI(
    base_url="https://api.together.ai/v1/deployment-request/qwen25-vl/v1",
    api_key=os.environ["TOGETHER_API_KEY"],
)

response = client.chat.completions.create(
    model="Qwen/Qwen2.5-VL-32B-Instruct",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://huggingface.co/datasets/patrickvonplaten/random_img/resolve/main/yosemite.png"
                    },
                },
                {"type": "text", "text": "Describe this image."},
            ],
        }
    ],
)

print(response.choices[0].message.content)
```

```bash cURL theme={null}
curl https://api.together.ai/v1/deployment-request/qwen25-vl/v1/chat/completions \
  -H "Authorization: Bearer $TOGETHER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-VL-32B-Instruct",
    "messages": [
      {
        "role": "user",
        "content": [
          {"type": "image_url", "image_url": {"url": "https://huggingface.co/datasets/patrickvonplaten/random_img/resolve/main/yosemite.png"}},
          {"type": "text", "text": "Describe this image."}
        ]
      }
    ]
  }'
```

The cURL request returns a standard OpenAI chat completion, and the Python sample prints only its `content`. A shortened response looks like this:

```json theme={null}
{
  "id": "chatcmpl-b15bc3b243a529ed",
  "object": "chat.completion",
  "model": "Qwen/Qwen2.5-VL-32B-Instruct",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "This image depicts a serene and picturesque natural landscape, likely from a national park or wilderness area..."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": { "prompt_tokens": 1759, "completion_tokens": 471, "total_tokens": 2230 }
}
```

Other routes vLLM serves, such as `/v1/completions` and `/v1/models`, are available under the same endpoint URL.

## Delete the deployment

The deployment keeps at least `min_replicas` replicas running, two H100s each, until you delete it. To delete it, run this from the directory that contains `pyproject.toml`:

```shell theme={null}
tg beta jig destroy
```

## Next steps

> [card] [OpenAI-compatible endpoint with Sprocket](https://docs.together.ai/docs/dedicated-containers-openai.md)
>
> Write your own worker when you need custom pre- or post-processing.

> [card] [Autoscaling](https://docs.together.ai/docs/together-deployments.md#autoscaling)
>
> Choose scaling metrics and targets, and see how replicas scale.

> [card] [Jig CLI reference](https://docs.together.ai/reference/cli/jig.md)
>
> Look up every `[tool.jig.deploy]` setting and deploy command.

> [card] [Sprocket examples](https://github.com/togethercomputer/sprocket/tree/main/examples)
>
> Browse more deployable examples, from hello-world to speech-to-speech.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.