Gang-schedule GPU jobs with Volcano

Install the Volcano scheduler and run gang-scheduled GPU jobs on a Together Kubernetes cluster.

Using a coding agent? Load the together-volcano skill to teach your agent to write correct Volcano and GPU cluster code for Together AI. Learn more.

Running third-party schedulers on Together GPU clusters is in beta. The steps and pinned version here (Volcano v1.15.0) are validated on current clusters, but the workflow and defaults may change. Report issues to [e-mail].

Volcano is a Kubernetes-native batch scheduler for AI, machine learning, and high-performance computing workloads. Its defining feature is gang scheduling: a group of pods is scheduled all-at-once or not at all, so a distributed training job never starts with half its workers running and the rest stuck pending. This guide installs Volcano on a Together GPU cluster and runs a gang-scheduled job across the cluster’s GPUs.

For background on every field used here, see the Volcano documentation.

Prerequisites

• A Together Kubernetes GPU cluster in the Ready state.
• kubectl configured against the cluster. Download credentials with tg beta clusters get-credentials <cluster_id> --set-default-context.
• Cluster-admin access. The kubeconfig from the Together CLI grants it.

Confirm you can reach the cluster and see its GPU nodes:

kubectl get nodes -o custom-columns='NODE:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu'

GPU nodes report an nvidia.com/gpu count. The NVIDIA device plugin that exposes this resource is preinstalled on Together clusters.

Install Volcano

Install the scheduler, controllers, and custom resource definitions from the versioned release manifest. For other install methods and version compatibility, see the Volcano installation guide:

kubectl apply -f https://raw.githubusercontent.com/volcano-sh/volcano/v1.15.0/installer/volcano-development.yaml

Wait for the control plane to become available:

kubectl -n volcano-system wait --for=condition=Available deployment --all --timeout=180s
kubectl -n volcano-system get pods

All pods should be Running or Completed:

NAME                                  READY   STATUS      RESTARTS   AGE
volcano-admission-5f4fcdc56f-6n2j4    1/1     Running     0          3m
volcano-admission-init-bbjdf          0/1     Completed   0          3m
volcano-controllers-f79b44f67-nxbg7   1/1     Running     0          3m
volcano-scheduler-d8b968cbb-kc885     1/1     Running     0          3m

Volcano creates a default queue automatically:

kubectl get queue

Prefer Helm? Add the chart repo and install into the same namespace: helm repo add volcano-sh https://volcano-sh.github.io/helm-charts && helm install volcano volcano-sh/volcano -n volcano-system --create-namespace.

Create a queue

A queue is the unit Volcano uses to divide cluster capacity between teams or workloads. Create one with a GPU capability cap so jobs in it can request up to eight GPUs:

research-queue.yaml

apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
  name: research
spec:
  reclaimable: true      # let other queues borrow this queue's idle capacity
  weight: 1              # relative share of contended resources vs. other queues
  capability:           # hard ceiling on what jobs in this queue can use at once
    nvidia.com/gpu: 8

kubectl apply -f research-queue.yaml

• weight: the queue’s share of cluster resources relative to other queues when capacity is contended.
• capability: the hard upper bound on resources the queue can consume at once.
• reclaimable: whether the queue gives resources back to other queues when it is over its fair share.

See the queue reference for the full field list, including hierarchical queues and guaranteed resources.

Attach shared storage

Together provisions a static PersistentVolume named after your cluster’s shared volume. Bind a PersistentVolumeClaim to it so every worker reads and writes the same dataset and checkpoints:

shared-pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: shared-pvc
spec:
  accessModes:
    - ReadWriteMany                 # many pods across nodes mount it at the same time
  storageClassName: shared-wekafs   # Together's default shared storage class
  volumeName: <SHARED_VOLUME_NAME>  # the static PV named after your shared volume
  resources:
    requests:
      storage: 100Gi

kubectl apply -f shared-pvc.yaml
kubectl get pvc shared-pvc   # STATUS should be Bound

The next section mounts this claim into the job.

Run a gang-scheduled job

A Volcano Job (vcjob) wraps a set of tasks and enforces gang scheduling through minAvailable. The scheduler admits the job only once it can place at least minAvailable pods together. The following job runs four workers, each requesting two GPUs, for eight GPUs total, with the shared volume mounted at /mnt/shared:

gpu-gang-job.yaml

apiVersion: batch.volcano.sh/v1alpha1
kind: Job                          # a Volcano Job (vcjob), not a core batch/v1 Job
metadata:
  name: gpu-gang
spec:
  minAvailable: 4                  # gang size: schedule all 4 pods together or none
  schedulerName: volcano           # route pods to Volcano instead of the default scheduler
  queue: research                  # draw resources from the "research" queue
  policies:
    - event: PodEvicted            # if any pod in the gang is evicted...
      action: RestartJob           # ...restart the whole job so workers stay in sync
  tasks:
    - replicas: 4                  # number of worker pods in the gang
      name: worker
      template:
        spec:
          restartPolicy: OnFailure
          containers:
            - name: worker
              image: nvidia/cuda:12.4.0-base-ubuntu22.04
              command: ["bash", "-c", "nvidia-smi -L; sleep 300"]
              resources:
                limits:
                  nvidia.com/gpu: 2  # GPUs per worker; 4 x 2 = 8 GPUs for the gang
              volumeMounts:
                - name: shared
                  mountPath: /mnt/shared   # shared volume visible to every worker
          volumes:
            - name: shared
              persistentVolumeClaim:
                claimName: shared-pvc      # the PVC bound above

kubectl apply -f gpu-gang-job.yaml

schedulerName: volcano routes the pods to Volcano instead of the default Kubernetes scheduler. minAvailable: 4 means all four workers start together or none do.

Watch the job reach Running:

kubectl get vcjob gpu-gang

NAME       STATUS    MINAVAILABLE   RUNNINGS   AGE
gpu-gang   Running   4              4          30s

Volcano tracks the group through a PodGroup. Inspect it and the pods:

kubectl get podgroup
kubectl get pods -l volcano.sh/job-name=gpu-gang -o wide

Confirm the GPUs are visible inside a worker:

kubectl logs gpu-gang-worker-0

GPU 0: NVIDIA H100 80GB HBM3 (UUID: GPU-0b416d98-...)
GPU 1: NVIDIA H100 80GB HBM3 (UUID: GPU-149c362a-...)

Confirm the shared volume is mounted and writable from every worker. All workers read and write the same ReadWriteMany volume, so a file written by one is visible to the others:

kubectl exec gpu-gang-worker-0 -- touch /mnt/shared/hello
kubectl exec gpu-gang-worker-1 -- ls /mnt/shared   # shows hello

Verify all-or-nothing behavior

Gang scheduling matters most when a job cannot fit. Delete the running job, then submit one that requests more GPUs than the cluster has (eight workers at two GPUs each is 16 GPUs, past this cluster’s 8):

kubectl delete vcjob gpu-gang

Change minAvailable to 8 and replicas to 8 in the manifest and reapply. The job stays Pending and, critically, no pods run. The default scheduler would start as many as fit and strand the rest; Volcano keeps the whole group pending:

kubectl get pods -l volcano.sh/job-name=gpu-gang --no-headers | awk '{print $3}' | sort | uniq -c
#    8 Pending

kubectl get events --field-selector involvedObject.name=gpu-gang-worker-0 | tail -1
# Warning  FailedScheduling  ... once resource is released and minAvailable is satisfied

The PodGroup sits in the Inqueue phase until enough resources free up to admit the entire gang.

Route other workloads to Volcano

Any pod, Deployment, or third-party job type can use Volcano by setting schedulerName: volcano in its pod template. This is how you schedule multi-node training launched through the MPI Operator, which is preinstalled on Together clusters. For gang guarantees on those workloads, create a PodGroup and reference it from the pods; see the Volcano documentation.

Troubleshooting

• Pods stay Pending and the PodGroup is Inqueue: the queue cannot fit minAvailable pods at once. Lower minAvailable, raise the queue’s capability, or wait for other jobs to release GPUs.
• Pods schedule on the default scheduler instead of Volcano: schedulerName: volcano is missing from the pod template. On a vcjob it belongs under spec.
• Job rejected on creation: the Volcano admission webhook may still be starting. Confirm volcano-admission is Running in the volcano-system namespace and reapply.
• Queue rejects a job: the job’s total request exceeds the queue’s capability. Raise the cap or split the work.

Next steps

Queue GPU jobs with Kueue

Gate GPU jobs on quota with a Kubernetes-native queueing controller.

Manage GPU clusters

Deploy workloads, attach storage, and access the Kubernetes dashboard.