> ## Documentation Index
> Fetch the complete documentation index at: https://docs.together.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Gang-schedule GPU jobs with Volcano

> Install the Volcano scheduler and run gang-scheduled GPU jobs on a Together Kubernetes cluster.

> [!TIP]
>
> Using a coding agent? Load the [together-ai](https://github.com/togethercomputer/skills/tree/main/skills/together-ai) skill to teach your agent to write correct Volcano and GPU cluster code for Together AI. [Learn more](https://docs.together.ai/docs/agent-skills.md).

> [!NOTE]
>
> Running third-party schedulers on Together GPU clusters is in beta. The steps and pinned version here (Volcano v1.15.0) are validated on current clusters, but the workflow and defaults may change. Report issues through the [support form](https://portal.usepylon.com/together-ai/forms/support-request).

[Volcano](https://volcano.sh/) is a Kubernetes-native batch scheduler for AI, machine learning, and high-performance computing workloads. Its defining feature is [gang scheduling](https://volcano.sh/en/docs/gang_scheduling/): a group of pods is scheduled all-at-once or not at all, so a distributed training job never starts with half its workers running and the rest stuck pending. This guide installs Volcano on a Together GPU cluster and runs a gang-scheduled job across the cluster's GPUs.

For background on every field used here, see the [Volcano documentation](https://volcano.sh/en/docs/).

## Requirements

* A Together [Kubernetes GPU cluster](https://docs.together.ai/docs/gpu-clusters-quickstart.md) in the **Ready** state.
* Cluster-admin access, which the cluster's [admin kubeconfig](https://docs.together.ai/docs/gpu-clusters-management.md#download-cluster-kubeconfig) grants. Only project admins can download the admin kubeconfig.
* `kubectl` configured against the cluster. Download the admin kubeconfig with `tg clusters get-credentials <CLUSTER_ID> --set-default-context --no-json`.

Confirm you can reach the cluster and see its GPU nodes:

```bash theme={null}
kubectl get nodes -o custom-columns='NODE:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu'
```

Each GPU node shows its `nvidia.com/gpu` count in the `GPU` column. The NVIDIA device plugin that exposes this resource is preinstalled on Together clusters.

## Install Volcano

Install the scheduler, controllers, and custom resource definitions from the versioned release manifest:

```bash theme={null}
kubectl apply -f https://raw.githubusercontent.com/volcano-sh/volcano/v1.15.0/installer/volcano-development.yaml
```

> [!TIP]
>
> To install with Helm instead, add the chart repo and install the same version into `volcano-system`, the namespace the rest of this guide uses:
>
> ```bash theme={null}
> helm repo add volcano-sh https://volcano-sh.github.io/helm-charts && helm install volcano volcano-sh/volcano --version 1.15.0 -n volcano-system --create-namespace
> ```
>
> For other install methods and version compatibility, see the [Volcano installation guide](https://volcano.sh/en/docs/installation/).

Wait for the Volcano deployments to become available, then list their pods:

```bash theme={null}
kubectl -n volcano-system wait --for=condition=Available deployment --all --timeout=180s
kubectl -n volcano-system get pods
```

All pods should be `Running` or `Completed`:

```text theme={null}
NAME                                  READY   STATUS      RESTARTS   AGE
volcano-admission-5f4fcdc56f-6n2j4    1/1     Running     0          3m
volcano-admission-init-bbjdf          0/1     Completed   0          3m
volcano-controllers-f79b44f67-nxbg7   1/1     Running     0          3m
volcano-scheduler-d8b968cbb-kc885     1/1     Running     0          3m
```

Volcano creates a `default` queue automatically. List the queues to confirm it exists:

```bash theme={null}
kubectl get queue
```

## Create a queue

A [queue](https://volcano.sh/en/docs/queue/) is the unit Volcano uses to divide cluster capacity between teams or workloads. Create a queue named `research` whose jobs can use at most eight GPUs at once:

```yaml research-queue.yaml theme={null}
apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
  name: research
spec:
  reclaimable: true      # let other queues reclaim what this queue uses beyond its share
  weight: 1              # relative share of contended resources vs. other queues
  capability:           # hard ceiling on what jobs in this queue can use at once
    nvidia.com/gpu: 8
```

```bash theme={null}
kubectl apply -f research-queue.yaml
```

The spec fields set how much the queue can use and how it shares capacity with other queues:

* `capability` is the hard upper bound on resources the queue's jobs can use at once.
* `weight` sets the queue's share of cluster resources relative to other queues when capacity is contended.
* `reclaimable: true` lets other queues reclaim resources the queue uses beyond its fair share.

The [queue reference](https://volcano.sh/en/docs/queue/) lists every field, including the ones for hierarchical queues and guaranteed resources.

## Attach shared storage

To give every worker the same dataset and checkpoints, bind a PersistentVolumeClaim to your cluster's shared volume. Together provisions a static [PersistentVolume](https://docs.together.ai/docs/kubernetes-workloads.md#deploy-pods-with-storage) named after the shared volume, so set `volumeName` to your shared volume's name:

```yaml shared-pvc.yaml theme={null}
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: shared-pvc
spec:
  accessModes:
    - ReadWriteMany                 # many pods across nodes mount it at the same time
  volumeName: <SHARED_VOLUME_NAME>  # the static PV named after your shared volume
  resources:
    requests:
      storage: 100Gi
```

```bash theme={null}
kubectl apply -f shared-pvc.yaml
kubectl get pvc shared-pvc   # STATUS should be Bound
```

## Run a gang-scheduled job

A Volcano [`Job`](https://volcano.sh/en/docs/vcjob/) (`vcjob`) wraps a set of tasks and enforces gang scheduling through `minAvailable`. The scheduler admits the job only once it can place at least `minAvailable` pods together.

The following job runs four workers with two GPUs each, for eight GPUs in total. Every worker mounts the shared volume at `/mnt/shared`:

```yaml gpu-gang-job.yaml theme={null}
apiVersion: batch.volcano.sh/v1alpha1
kind: Job                          # a Volcano Job (vcjob), not a core batch/v1 Job
metadata:
  name: gpu-gang
spec:
  minAvailable: 4                  # gang size: schedule all 4 pods together or none
  schedulerName: volcano           # route pods to Volcano instead of the default scheduler
  queue: research                  # draw resources from the "research" queue
  policies:
    - event: PodEvicted            # if any pod in the gang is evicted...
      action: RestartJob           # ...restart the whole job so workers stay in sync
  tasks:
    - replicas: 4                  # number of worker pods in the gang
      name: worker
      template:
        spec:
          restartPolicy: OnFailure
          containers:
            - name: worker
              image: nvidia/cuda:12.4.0-base-ubuntu22.04
              command: ["bash", "-c", "nvidia-smi -L; sleep 300"]
              resources:
                limits:
                  nvidia.com/gpu: 2  # GPUs per worker; 4 x 2 = 8 GPUs for the gang
              volumeMounts:
                - name: shared
                  mountPath: /mnt/shared   # shared volume visible to every worker
          volumes:
            - name: shared
              persistentVolumeClaim:
                claimName: shared-pvc      # the PVC bound above
```

```bash theme={null}
kubectl apply -f gpu-gang-job.yaml
```

The key fields tie the job to Volcano and to the resources you created earlier:

* `schedulerName: volcano` routes the pods to Volcano instead of the default Kubernetes scheduler.
* `minAvailable: 4` means all four workers start together or none do.
* `queue: research` draws resources from the queue you created in [Create a queue](#create-a-queue).
* `claimName: shared-pvc` mounts the claim you bound in [Attach shared storage](#attach-shared-storage).

Check that the job reaches `Running`:

```bash theme={null}
kubectl get vcjob gpu-gang
```

```text theme={null}
NAME       STATUS    MINAVAILABLE   RUNNINGS   AGE
gpu-gang   Running   4              4          30s
```

Volcano tracks the gang through a `PodGroup`. Inspect it and the job's pods:

```bash theme={null}
kubectl get podgroup
kubectl get pods -l volcano.sh/job-name=gpu-gang -o wide
```

Each worker runs `nvidia-smi -L` on startup. Check a worker's log to confirm it sees its two GPUs:

```bash theme={null}
kubectl logs gpu-gang-worker-0
```

```text theme={null}
GPU 0: NVIDIA H100 80GB HBM3 (UUID: GPU-0b416d98-...)
GPU 1: NVIDIA H100 80GB HBM3 (UUID: GPU-149c362a-...)
```

All workers read and write the same `ReadWriteMany` volume, so a file that one worker writes is visible to the others. To confirm, create a file from one worker and list it from another:

```bash theme={null}
kubectl exec gpu-gang-worker-0 -- touch /mnt/shared/hello
kubectl exec gpu-gang-worker-1 -- ls /mnt/shared   # shows hello
```

### Verify all-or-nothing behavior

Gang scheduling matters most when a job cannot fit. Delete the running job, then submit one that requests more GPUs than the cluster has (eight workers at two GPUs each is 16 GPUs, past this cluster's 8):

```bash theme={null}
kubectl delete vcjob gpu-gang
```

Change `minAvailable` to `8` and `replicas` to `8` in the manifest and reapply. The job stays `Pending` and, critically, **no** pods run. The default scheduler would start as many as fit and strand the rest. Volcano keeps the whole group pending:

```bash theme={null}
kubectl get pods -l volcano.sh/job-name=gpu-gang --no-headers | awk '{print $3}' | sort | uniq -c
#    8 Pending

kubectl get events --field-selector involvedObject.name=gpu-gang-worker-0 | tail -1
# Warning  FailedScheduling  ... once resource is released and minAvailable is satisfied
```

The `PodGroup` sits in the `Inqueue` phase until enough resources free up to admit the entire gang.

## Route other workloads to Volcano

Any pod, Deployment, or third-party job type can use Volcano by setting `schedulerName: volcano` in its pod template. This is how you schedule multi-node training launched through the [MPI Operator](https://github.com/kubeflow/mpi-operator). The MPI Operator isn't installed on every cluster, so check for it with `kubectl get crd mpijobs.kubeflow.org` and install it if it's missing. For gang guarantees on those workloads, create a `PodGroup` and reference it from the pods. See the [Volcano documentation](https://volcano.sh/en/docs/podgroup/).

## Troubleshooting

* **Pods stay `Pending` and the `PodGroup` is `Inqueue`:** The queue can't fit `minAvailable` pods at once. Lower `minAvailable`, raise the queue's `capability`, or wait for other jobs to release GPUs.
* **Pods schedule on the default scheduler instead of Volcano:** `schedulerName: volcano` is missing from the pod template. On a `vcjob`, set it under `spec`.
* **Job rejected right after you install Volcano:** The Volcano admission webhook may still be starting. Confirm that the `volcano-admission` pod is `Running` in the `volcano-system` namespace, then reapply.
* **Job rejected with a queue error:** The queue the job names doesn't exist, isn't `Open`, or isn't a leaf queue. Check it with `kubectl get queue`.

## Next steps

> [card] [Queue GPU jobs with Kueue](https://docs.together.ai/docs/kueue-on-gpu-clusters.md)
>
> Gate GPU jobs on quota with a Kubernetes-native queueing controller.

> [card] [Run workloads on Kubernetes](https://docs.together.ai/docs/kubernetes-workloads.md)
>
> Deploy pods with persistent storage and monitor them with Grafana.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.