Gang-schedule GPU jobs with Volcano

Using a coding agent? Load the together-ai skill to teach your agent to write correct Volcano and GPU cluster code for Together AI. Learn more.

Running third-party schedulers on Together GPU clusters is in beta. The steps and pinned version here (Volcano v1.15.0) are validated on current clusters, but the workflow and defaults may change. Report issues through the support form.

Volcano is a Kubernetes-native batch scheduler for AI, machine learning, and high-performance computing workloads. Its defining feature is gang scheduling: a group of pods is scheduled all-at-once or not at all, so a distributed training job never starts with half its workers running and the rest stuck pending. This guide installs Volcano on a Together GPU cluster and runs a gang-scheduled job across the cluster’s GPUs. For background on every field used here, see the Volcano documentation.

Requirements

A Together Kubernetes GPU cluster in the Ready state.
Cluster-admin access, which the cluster’s admin kubeconfig grants. Only project admins can download the admin kubeconfig.
kubectl configured against the cluster. Download the admin kubeconfig with tg clusters get-credentials <CLUSTER_ID> --set-default-context --no-json.

Confirm you can reach the cluster and see its GPU nodes:

kubectl get nodes -o custom-columns='NODE:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu'

Each GPU node shows its nvidia.com/gpu count in the GPU column. The NVIDIA device plugin that exposes this resource is preinstalled on Together clusters.

Install Volcano

Install the scheduler, controllers, and custom resource definitions from the versioned release manifest:

kubectl apply -f https://raw.githubusercontent.com/volcano-sh/volcano/v1.15.0/installer/volcano-development.yaml

To install with Helm instead, add the chart repo and install the same version into volcano-system, the namespace the rest of this guide uses:

helm repo add volcano-sh https://volcano-sh.github.io/helm-charts && helm install volcano volcano-sh/volcano --version 1.15.0 -n volcano-system --create-namespace

For other install methods and version compatibility, see the Volcano installation guide.

Wait for the Volcano deployments to become available, then list their pods:

kubectl -n volcano-system wait --for=condition=Available deployment --all --timeout=180s
kubectl -n volcano-system get pods

All pods should be Running or Completed:

NAME                                  READY   STATUS      RESTARTS   AGE
volcano-admission-5f4fcdc56f-6n2j4    1/1     Running     0          3m
volcano-admission-init-bbjdf          0/1     Completed   0          3m
volcano-controllers-f79b44f67-nxbg7   1/1     Running     0          3m
volcano-scheduler-d8b968cbb-kc885     1/1     Running     0          3m

Volcano creates a default queue automatically. List the queues to confirm it exists:

kubectl get queue

Create a queue

A queue is the unit Volcano uses to divide cluster capacity between teams or workloads. Create a queue named research whose jobs can use at most eight GPUs at once:

research-queue.yaml

apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
  name: research
spec:
  reclaimable: true      # let other queues reclaim what this queue uses beyond its share
  weight: 1              # relative share of contended resources vs. other queues
  capability:           # hard ceiling on what jobs in this queue can use at once
    nvidia.com/gpu: 8

kubectl apply -f research-queue.yaml

The spec fields set how much the queue can use and how it shares capacity with other queues:

capability is the hard upper bound on resources the queue’s jobs can use at once.
weight sets the queue’s share of cluster resources relative to other queues when capacity is contended.
reclaimable: true lets other queues reclaim resources the queue uses beyond its fair share.

The queue reference lists every field, including the ones for hierarchical queues and guaranteed resources.

Attach shared storage

To give every worker the same dataset and checkpoints, bind a PersistentVolumeClaim to your cluster’s shared volume. Together provisions a static PersistentVolume named after the shared volume, so set volumeName to your shared volume’s name:

shared-pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: shared-pvc
spec:
  accessModes:
    - ReadWriteMany                 # many pods across nodes mount it at the same time
  volumeName: <SHARED_VOLUME_NAME>  # the static PV named after your shared volume
  resources:
    requests:
      storage: 100Gi

kubectl apply -f shared-pvc.yaml
kubectl get pvc shared-pvc   # STATUS should be Bound

Run a gang-scheduled job

A Volcano Job (vcjob) wraps a set of tasks and enforces gang scheduling through minAvailable. The scheduler admits the job only once it can place at least minAvailable pods together. The following job runs four workers with two GPUs each, for eight GPUs in total. Every worker mounts the shared volume at /mnt/shared:

gpu-gang-job.yaml

apiVersion: batch.volcano.sh/v1alpha1
kind: Job                          # a Volcano Job (vcjob), not a core batch/v1 Job
metadata:
  name: gpu-gang
spec:
  minAvailable: 4                  # gang size: schedule all 4 pods together or none
  schedulerName: volcano           # route pods to Volcano instead of the default scheduler
  queue: research                  # draw resources from the "research" queue
  policies:
    - event: PodEvicted            # if any pod in the gang is evicted...
      action: RestartJob           # ...restart the whole job so workers stay in sync
  tasks:
    - replicas: 4                  # number of worker pods in the gang
      name: worker
      template:
        spec:
          restartPolicy: OnFailure
          containers:
            - name: worker
              image: nvidia/cuda:12.4.0-base-ubuntu22.04
              command: ["bash", "-c", "nvidia-smi -L; sleep 300"]
              resources:
                limits:
                  nvidia.com/gpu: 2  # GPUs per worker; 4 x 2 = 8 GPUs for the gang
              volumeMounts:
                - name: shared
                  mountPath: /mnt/shared   # shared volume visible to every worker
          volumes:
            - name: shared
              persistentVolumeClaim:
                claimName: shared-pvc      # the PVC bound above

kubectl apply -f gpu-gang-job.yaml

The key fields tie the job to Volcano and to the resources you created earlier:

schedulerName: volcano routes the pods to Volcano instead of the default Kubernetes scheduler.
minAvailable: 4 means all four workers start together or none do.
queue: research draws resources from the queue you created in Create a queue.
claimName: shared-pvc mounts the claim you bound in Attach shared storage.

Check that the job reaches Running:

kubectl get vcjob gpu-gang

NAME       STATUS    MINAVAILABLE   RUNNINGS   AGE
gpu-gang   Running   4              4          30s

Volcano tracks the gang through a PodGroup. Inspect it and the job’s pods:

kubectl get podgroup
kubectl get pods -l volcano.sh/job-name=gpu-gang -o wide

Each worker runs nvidia-smi -L on startup. Check a worker’s log to confirm it sees its two GPUs:

kubectl logs gpu-gang-worker-0

GPU 0: NVIDIA H100 80GB HBM3 (UUID: GPU-0b416d98-...)
GPU 1: NVIDIA H100 80GB HBM3 (UUID: GPU-149c362a-...)

All workers read and write the same ReadWriteMany volume, so a file that one worker writes is visible to the others. To confirm, create a file from one worker and list it from another:

kubectl exec gpu-gang-worker-0 -- touch /mnt/shared/hello
kubectl exec gpu-gang-worker-1 -- ls /mnt/shared   # shows hello

Verify all-or-nothing behavior

Gang scheduling matters most when a job cannot fit. Delete the running job, then submit one that requests more GPUs than the cluster has (eight workers at two GPUs each is 16 GPUs, past this cluster’s 8):

kubectl delete vcjob gpu-gang

Change minAvailable to 8 and replicas to 8 in the manifest and reapply. The job stays Pending and, critically, no pods run. The default scheduler would start as many as fit and strand the rest. Volcano keeps the whole group pending:

kubectl get pods -l volcano.sh/job-name=gpu-gang --no-headers | awk '{print $3}' | sort | uniq -c
#    8 Pending

kubectl get events --field-selector involvedObject.name=gpu-gang-worker-0 | tail -1
# Warning  FailedScheduling  ... once resource is released and minAvailable is satisfied

The PodGroup sits in the Inqueue phase until enough resources free up to admit the entire gang.

Route other workloads to Volcano

Any pod, Deployment, or third-party job type can use Volcano by setting schedulerName: volcano in its pod template. This is how you schedule multi-node training launched through the MPI Operator. The MPI Operator isn’t installed on every cluster, so check for it with kubectl get crd mpijobs.kubeflow.org and install it if it’s missing. For gang guarantees on those workloads, create a PodGroup and reference it from the pods. See the Volcano documentation.

Troubleshooting

Pods stay Pending and the PodGroup is Inqueue: The queue can’t fit minAvailable pods at once. Lower minAvailable, raise the queue’s capability, or wait for other jobs to release GPUs.
Pods schedule on the default scheduler instead of Volcano: schedulerName: volcano is missing from the pod template. On a vcjob, set it under spec.
Job rejected right after you install Volcano: The Volcano admission webhook may still be starting. Confirm that the volcano-admission pod is Running in the volcano-system namespace, then reapply.
Job rejected with a queue error: The queue the job names doesn’t exist, isn’t Open, or isn’t a leaf queue. Check it with kubectl get queue.

GET STARTED

SERVERLESS

INFERENCE APIS

DEDICATED MODEL INFERENCE

DEDICATED CONTAINER INFERENCE

POST-TRAINING

FINE-TUNING

GPU CLUSTERS

CODE EXECUTION

ADMINISTRATION

Gang-schedule GPU jobs with Volcano

Requirements

Install Volcano

Create a queue

Attach shared storage

Run a gang-scheduled job

Verify all-or-nothing behavior

Route other workloads to Volcano

Troubleshooting

Next steps

Queue GPU jobs with Kueue

Run workloads on Kubernetes