Queue GPU jobs with Kueue

Using a coding agent? Load the together-ai skill to teach your agent to write correct Kueue and GPU cluster code for Together AI. Learn more.

Running third-party schedulers on Together GPU clusters is in beta. The steps and pinned version here (Kueue v0.18.3) are validated on current clusters, but the workflow and defaults may change. Report issues through the support form.

Kueue is a Kubernetes-native job queueing controller. Instead of letting every job compete for GPUs the moment it is created, Kueue holds jobs in a queue and admits them only when their quota is available, suspending the rest. This is how you share a fixed GPU pool across teams without overcommitting it. This guide installs Kueue on a Together GPU cluster and gates jobs on GPU quota. For the concepts behind each object below, see the Kueue documentation.

Requirements

A Together Kubernetes GPU cluster in the Ready state.
Cluster-admin access, which the cluster’s admin kubeconfig grants. Only project admins can download the admin kubeconfig.
kubectl configured against the cluster. Download the admin kubeconfig with tg clusters get-credentials <CLUSTER_ID> --set-default-context --no-json.

Install Kueue

Apply the versioned release manifest. The bundled custom resource definitions are large, so pass --server-side:

kubectl apply --server-side -f https://github.com/kubernetes-sigs/kueue/releases/download/v0.18.3/manifests.yaml

Wait for the controller to become available:

kubectl -n kueue-system wait --for=condition=Available deployment/kueue-controller-manager --timeout=240s
kubectl -n kueue-system get pods

The controller pod should be Running:

NAME                                        READY   STATUS    RESTARTS   AGE
kueue-controller-manager-6c8685b746-tlc47   1/1     Running   0          20s

To install with Helm instead, install the chart from the registry: helm install kueue oci://registry.k8s.io/kueue/charts/kueue --version 0.18.3 --namespace kueue-system --create-namespace. For other install methods and configuration options, see the Kueue installation guide.

Define quotas

Kueue admits jobs against a quota that you define with three resources:

ResourceFlavor: Represents a class of nodes. One flavor is enough when your GPU nodes are homogeneous.
ClusterQueue: Defines the cluster-wide quota pool. It must cover every resource your jobs request, including CPU and memory. Kueue never admits a job that requests a resource the ClusterQueue doesn’t cover.
LocalQueue: Accepts the jobs users submit in its namespace and points to the ClusterQueue whose quota they draw on.

Create all three, with an eight-GPU quota on the ClusterQueue:

kueue-quota.yaml

apiVersion: kueue.x-k8s.io/v1beta2
kind: ResourceFlavor               # a class of nodes; one is enough for homogeneous GPUs
metadata:
  name: gpu-flavor
---
apiVersion: kueue.x-k8s.io/v1beta2
kind: ClusterQueue                 # the cluster-wide quota pool
metadata:
  name: gpu-cluster-queue
spec:
  namespaceSelector: {}            # {} accepts jobs from every namespace
  resourceGroups:
    - coveredResources: ["cpu", "memory", "nvidia.com/gpu"]  # every resource jobs request must be listed here
      flavors:
        - name: gpu-flavor         # must match the ResourceFlavor name above
          resources:
            - name: "cpu"
              nominalQuota: 64
            - name: "memory"
              nominalQuota: 512Gi
            - name: "nvidia.com/gpu"
              nominalQuota: 8       # admit at most 8 GPUs worth of jobs at once
---
apiVersion: kueue.x-k8s.io/v1beta2
kind: LocalQueue                   # namespaced entry point users submit jobs to
metadata:
  namespace: default
  name: gpu-queue
spec:
  clusterQueue: gpu-cluster-queue  # points at the ClusterQueue above

kubectl apply -f kueue-quota.yaml

Each resource’s nominalQuota sets how much of it the ClusterQueue can admit at once. The jobs in the following sections assume this eight-GPU quota. For your own workloads, set the GPU quota to the number of GPUs you want this queue to control.

Attach shared storage

To let queued jobs read and write the same dataset and checkpoints, bind a PersistentVolumeClaim to your cluster’s shared volume. Together provisions a static PersistentVolume named after the shared volume, so set volumeName to your shared volume’s name:

shared-pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: shared-pvc
spec:
  accessModes:
    - ReadWriteMany                 # many pods across nodes mount it at the same time
  volumeName: <SHARED_VOLUME_NAME>  # the static PV named after your shared volume
  resources:
    requests:
      storage: 100Gi

kubectl apply -f shared-pvc.yaml
kubectl get pvc shared-pvc   # STATUS should be Bound

Storage isn’t a quota resource, so the claim doesn’t affect admission. The next section mounts it into the job.

Submit a job to a queue

Point a standard batch/v1 Job at the LocalQueue with the kueue.x-k8s.io/queue-name label, and set suspend: true. Kueue takes over the suspend flag and sets it to false when it admits the job:

gpu-job.yaml

apiVersion: batch/v1
kind: Job
metadata:
  name: gpu-job-a
  namespace: default
  labels:
    kueue.x-k8s.io/queue-name: gpu-queue   # send this job to the "gpu-queue" LocalQueue
spec:
  parallelism: 1
  completions: 1
  suspend: true                            # start suspended; Kueue unsuspends it on admission
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: worker
          image: nvidia/cuda:12.4.0-base-ubuntu22.04
          command: ["bash", "-c", "nvidia-smi -L; sleep 180"]
          resources:
            requests:
              cpu: "2"                       # counted against the ClusterQueue cpu quota
              memory: "8Gi"                  # counted against the memory quota
            limits:
              nvidia.com/gpu: 6              # counted against the nvidia.com/gpu quota
          volumeMounts:
            - name: shared
              mountPath: /mnt/shared         # shared volume mounted into the job
      volumes:
        - name: shared
          persistentVolumeClaim:
            claimName: shared-pvc            # the PVC bound above

kubectl apply -f gpu-job.yaml

Kueue creates a Workload object for the job. Six GPUs fit within the eight-GPU quota, so Kueue admits the job and its pod starts. Confirm that the Workload is admitted:

kubectl get workloads

NAME                  QUEUE       ADMITTED
job-gpu-job-a-272f3   gpu-queue   True

For other supported workload types, such as JobSet, RayJob, and MPIJob, see running jobs with Kueue.

Queue a second job until quota frees up

Submit a second job that also requests six GPUs. The two jobs need 12 GPUs combined, which exceeds the eight-GPU quota. To create it, copy gpu-job.yaml, change metadata.name in the copy to gpu-job-b, and apply the copy. Compare the two jobs:

kubectl get jobs -o custom-columns='NAME:.metadata.name,SUSPEND:.spec.suspend,ACTIVE:.status.active'

NAME        SUSPEND   ACTIVE
gpu-job-a   false     1
gpu-job-b   true      <none>

The first job runs, and the second stays suspended. The second job’s Workload reports why in its QuotaReserved condition:

WORKLOAD=$(kubectl get workloads -o name | grep gpu-job-b)
kubectl get "$WORKLOAD" -o jsonpath='{.status.conditions[?(@.type=="QuotaReserved")].message}'
# couldn't assign flavors to pod set main: insufficient unused quota for nvidia.com/gpu in flavor gpu-flavor, 4 more needed

When the first job finishes or you delete it, Kueue admits the queued job and its pod starts. You don’t need to resubmit it. Delete the first job, then check the second job’s suspend flag:

kubectl delete job gpu-job-a
kubectl get job gpu-job-b -o jsonpath='{.spec.suspend}{"\n"}'
# false

Troubleshooting

A job stays suspended and has no Workload: The kueue.x-k8s.io/queue-name label is missing, so Kueue ignores the job and never unsuspends it.
A job’s Workload says the LocalQueue doesn’t exist: The kueue.x-k8s.io/queue-name label names a LocalQueue that isn’t in the job’s namespace.
A job’s Workload is never admitted: Check the Workload’s QuotaReserved condition message. Either the job’s combined request exceeds the ClusterQueue quota, or the ClusterQueue doesn’t cover every resource the job requests. A ClusterQueue that covers only GPUs never admits a job whose pods also request CPU and memory, so add them to coveredResources.
The ClusterQueue shows no active quota: Its ResourceFlavor is missing or misnamed. Each flavors[].name must match the name of a ResourceFlavor.

GET STARTED

SERVERLESS

INFERENCE APIS

DEDICATED MODEL INFERENCE

DEDICATED CONTAINER INFERENCE

POST-TRAINING

FINE-TUNING

GPU CLUSTERS

CODE EXECUTION

ADMINISTRATION

Requirements

Install Kueue

Define quotas

Attach shared storage

Submit a job to a queue

Queue a second job until quota frees up

Troubleshooting

Next steps

Gang-schedule GPU jobs with Volcano

Run workloads on Kubernetes