Deploy a prebuilt vLLM image

Serve a model from the stock vLLM Docker image with dedicated container inference.

This guide uses dedicated container inference to serve Qwen2.5-VL-32B-Instruct from the public vllm/vllm-openai image. vLLM already serves an OpenAI-compatible API, so you don’t need a Sprocket worker or a Docker build. Jig deploys the image as-is, using two settings in [tool.jig.deploy]:

• image: Deploy an existing image instead of building one. Jig skips docker build and docker push.
• command: Override the image’s startup command. Here it runs vllm serve with the model and serving flags.

Requests go straight to vLLM’s HTTP server instead of through the managed queue, so you call the deployment with any OpenAI client or curl, not jig submit.

The full example is in the Sprocket repo at examples/qwen_vllm.

​

Requirements

• Together API key: Get one from together.ai.
• Dedicated container inference access: Contact your account representative or the sales team to enable it for your organization.
• Together CLI: Install with uv tool install "together[cli]".
• OpenAI Python SDK: The Python request sample uses it. Add it to your Python project with uv add openai.

Export your API key. The deploy step and the request samples read it from TOGETHER_API_KEY:

export TOGETHER_API_KEY=your_key_here

​

Configure the deployment

Create a directory with a single pyproject.toml:

pyproject.toml

[project]
name = "qwen25-vl"
version = "1.0.0"

[tool.jig.deploy]
# Deploy an existing image instead of building one
image = "vllm/vllm-openai:v0.17.1"

description = "Qwen2.5-VL-32B on stock vLLM"
gpu_type = "h100-80gb"
gpu_count = 2
cpu = 4
memory = 64
port = 8000
min_replicas = 1
max_replicas = 10
command = [
    "vllm",
    "serve",
    "Qwen/Qwen2.5-VL-32B-Instruct",
    "--port",
    "8000",
    "--host",
    "0.0.0.0",
    "--dtype",
    "bfloat16",
    "--limit-mm-per-prompt.image",
    "5",
    "--limit-mm-per-prompt.video",
    "5",
    "--tensor-parallel-size",
    "2",
]

[tool.jig.deploy.autoscaling]
metric = "CustomMetric"
custom_metric_name = "vllm:num_requests_running"
target = 4

Before you deploy, check these settings:

• Pin the image tag: This guide is tested with v0.17.1. Pin a specific tag, not latest, so a replica that restarts doesn’t pick up a new vLLM release.
• Match GPUs to tensor parallelism: Set --tensor-parallel-size to gpu_count so vLLM uses every GPU in the replica. Here, vLLM splits the 32B model in BF16 across two H100s.
• Match port to --port: Together routes traffic to the port in port, so it must match the port vLLM listens on.
• Keep the default health check: The default health_check_path is /health, which vLLM serves. The deployment reports ready once vLLM has loaded the model.
• Autoscale on vLLM’s own metric: vLLM exports Prometheus metrics on /metrics, so the CustomMetric autoscaler scales on vllm:num_requests_running with no extra code.
• Use a unique project name: The name under [project] is the deployment name and the last segment of the endpoint URL. Deployment names must be unique, so if qwen25-vl is taken, change name and use the new name in the endpoint URL.

​

Serve a different model

Change the model ID in command and in your requests, then size gpu_count, --tensor-parallel-size, and memory to fit the new model. Any vllm serve flag can go in command.

Qwen2.5-VL-32B-Instruct isn’t gated, so this guide needs no Hugging Face token. For a gated model, store your token as a secret before you deploy. vLLM reads HF_TOKEN from the environment when it downloads the weights. Run the command from the directory that contains pyproject.toml, so the secret attaches to this deployment:

tg beta jig secrets set --name HF_TOKEN --value your_hf_token

​

Deploy

From the directory that contains pyproject.toml, create the deployment and watch it start:

# Create the deployment from the configured image
tg beta jig deploy --detach

# Watch vLLM start up (Ctrl+C to stop)
tg beta jig logs --follow

# Check readiness
tg beta jig status

With --detach, jig deploy returns as soon as the deployment is created. Without it, the command waits for the replica to become ready, but it gives up two minutes after the container starts. vLLM takes longer than that to load the model, so the command exits with Running but not ready after 120s even though the deployment keeps starting.

The first deploy takes 5 to 15 minutes while Together schedules two H100s, pulls the vLLM image, and vLLM downloads and loads about 65 GB of weights. The deployment is ready when jig status shows Ready for Status and 1/1 ready for Replicas.

When the deployment is ready, get the endpoint URL:

tg beta jig endpoint

The URL has the form https://api.together.ai/v1/deployment-request/<DEPLOYMENT_NAME>. Together authenticates requests to any path under it with your Together API key, then forwards them to the container.

​

Call the endpoint

Point any OpenAI client at the endpoint URL plus /v1, using your Together API key. The model field must match the model ID you pass to vllm serve in command:

OpenAI Python SDK

import os

from openai import OpenAI

client = OpenAI(
    base_url="https://api.together.ai/v1/deployment-request/qwen25-vl/v1",
    api_key=os.environ["TOGETHER_API_KEY"],
)

response = client.chat.completions.create(
    model="Qwen/Qwen2.5-VL-32B-Instruct",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://huggingface.co/datasets/patrickvonplaten/random_img/resolve/main/yosemite.png"
                    },
                },
                {"type": "text", "text": "Describe this image."},
            ],
        }
    ],
)

print(response.choices[0].message.content)

The cURL request returns a standard OpenAI chat completion, and the Python sample prints only its content. A shortened response looks like this:

{
  "id": "chatcmpl-b15bc3b243a529ed",
  "object": "chat.completion",
  "model": "Qwen/Qwen2.5-VL-32B-Instruct",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "This image depicts a serene and picturesque natural landscape, likely from a national park or wilderness area..."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": { "prompt_tokens": 1759, "completion_tokens": 471, "total_tokens": 2230 }
}

Other routes vLLM serves, such as /v1/completions and /v1/models, are available under the same endpoint URL.

​

Delete the deployment

The deployment keeps at least min_replicas replicas running, two H100s each, until you delete it. To delete it, run this from the directory that contains pyproject.toml:

tg beta jig destroy

​

Next steps

OpenAI-compatible endpoint with Sprocket

Write your own worker when you need custom pre- or post-processing.

Autoscaling

Choose scaling metrics and targets, and see how replicas scale.

Jig CLI reference

Look up every [tool.jig.deploy] setting and deploy command.

Sprocket examples

Browse more deployable examples, from hello-world to speech-to-speech.