Run workloads on Kubernetes

This page describes how to run workloads on a Together GPU cluster with Kubernetes: deploying pods with persistent storage, running GPU workloads in containers, and monitoring them with Grafana. Before you start, connect kubectl to your cluster. For background on Kubernetes and how it works on Together, see Concepts.

Deploy pods with storage

Together provides a static PersistentVolume (PV) with the same name as your shared volume. Data you write to the static PV persists across pod restarts, cluster operations, and, by default, cluster deletion. Kubernetes connects storage to your containers through three objects:

PersistentVolume (PV): The storage resource itself, managed by Together.
PersistentVolumeClaim (PVC): A request for storage that your pods can use. You create this.
Pod with volumeMounts: Mounts the PVC into your container at a specific path. You create this.

Create PVCs

For shared storage that multiple pods can read and write at the same time, create a PVC with ReadWriteMany access. Set volumeName to your shared volume’s name so the PVC binds to the static PV:

shared-pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: shared-pvc                    # Name you'll reference in pods
spec:
  accessModes:
    - ReadWriteMany                   # Multiple pods can read/write simultaneously
  resources:
    requests:
      storage: 10Gi                   # Requested size (can be adjusted)
  volumeName: <shared volume name>    # Replace with your shared volume name from the console

The static PV uses your cluster’s shared storage class, which is also the cluster’s default storage class, so the PVC doesn’t need a storageClassName. If the shared PVC stays Pending, check the PV’s class with kubectl get pv <shared volume name> and add a matching storageClassName to the PVC.

For fast local NVMe storage that only one node can mount at a time, create a PVC with ReadWriteOnce access and the local storage class:

local-pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: local-pvc                     # Name you'll reference in pods
spec:
  accessModes:
    - ReadWriteOnce                   # Only one pod/node can mount at a time
  resources:
    requests:
      storage: 50Gi                   # Requested size
  storageClassName: local

Data on local NVMe is ephemeral, so use the local PVC only for scratch files you can regenerate.Apply both PVCs and check their status:

kubectl apply -f shared-pvc.yaml -n default # change to your namespace
kubectl apply -f local-pvc.yaml -n default # change to your namespace

# Check PVC status
kubectl get pvc -A # across all namespaces

The shared PVC shows Bound right away. The local PVC stays Pending until a pod that mounts it is scheduled, which is expected for the local storage class.

Define a pod that mounts the PVCs

Define a pod that mounts both PVCs as volumes. Each volumes entry references a PVC by its claimName, and each volumeMounts entry uses a volume’s name to mount it at a path inside the container:

pod-with-storage.yaml

apiVersion: v1
kind: Pod
metadata:
  name: test-pod
spec:
  restartPolicy: Never
  containers:
    - name: debian
      image: debian:stable-slim
      command: ["/bin/sh", "-c", "sleep infinity"]  # Keeps pod running
      volumeMounts:                                   # Where to mount volumes inside container
        - name: shared-storage                        # References volume defined below
          mountPath: /mnt/shared                      # Path inside container
        - name: local-storage
          mountPath: /mnt/local
  volumes:                                            # Defines volumes from PVCs
    - name: shared-storage                            # Internal name for this volume
      persistentVolumeClaim:
        claimName: shared-pvc                         # Must match PVC name from Step 1
    - name: local-storage
      persistentVolumeClaim:
        claimName: local-pvc

Deploy and access the pod

Deploy the pod in the same namespace as its PVCs, then open a shell in it:

# Deploy the pod
kubectl apply -f pod-with-storage.yaml -n default # should be same as the namespace in which PVC is deployed

# Wait for pod to be running
kubectl get pods -w

# Once STATUS shows "Running", access the pod
kubectl exec -it test-pod -- bash

Verify mounted volumes

Inside the pod, check that both volumes are mounted and that you can write to the shared volume:

# Check mounted filesystems
df -h | grep /mnt

# List mounted directories
ls -la /mnt/shared
ls -la /mnt/local

# Test write access
echo "Hello from pod" > /mnt/shared/test.txt
cat /mnt/shared/test.txt

Access the shared volume from multiple pods

Because the shared PVC uses ReadWriteMany access, multiple pods can mount it at the same time. To read the file you wrote from test-pod, create a second pod that mounts the same PVC. It needs its own manifest, because kubectl run can’t attach a PVC:

test-pod-2.yaml

apiVersion: v1
kind: Pod
metadata:
  name: test-pod-2
spec:
  restartPolicy: Never
  containers:
    - name: debian
      image: debian:stable-slim
      command: ["/bin/sh", "-c", "sleep infinity"]
      volumeMounts:
        - name: shared-storage
          mountPath: /mnt/shared
  volumes:
    - name: shared-storage
      persistentVolumeClaim:
        claimName: shared-pvc               # Same PVC as the first pod

Create the second pod and read the file from it:

# Create the second pod
kubectl apply -f test-pod-2.yaml

# Exec into it
kubectl exec -it test-pod-2 -- bash

# The file you created from the first pod is visible here
cat /mnt/shared/test.txt

Run GPU workloads in containers

To run GPU workloads or work with your data volumes, deploy a pod with GPU and storage access, then exec into it. Use a CUDA-enabled base image, such as nvidia/cuda or pytorch/pytorch, instead of installing CUDA yourself. A pod gets the GPUs it requests with nvidia.com/gpu, so always request GPUs explicitly. On older clusters, a pod that requests none can still see every GPU on its node if its image sets NVIDIA_VISIBLE_DEVICES, as CUDA images do. The host provides the NVIDIA kernel driver, and your container image provides the CUDA libraries. In an image without CUDA, such as a debian or ubuntu base image, you can’t run CUDA workloads until you install CUDA. This pod mounts the shared-pvc claim from Deploy pods with storage, so create that PVC first:

gpu-pod.yaml

apiVersion: v1
kind: Pod
metadata:
  name: gpu-workload-pod
spec:
  restartPolicy: Never
  containers:
    - name: pytorch
      image: pytorch/pytorch:2.8.0-cuda12.8-cudnn9-runtime  # CUDA-enabled image
      command: ["/bin/bash", "-c", "sleep infinity"]
      resources:
        limits:
          nvidia.com/gpu: 1  # Request 1 GPU
      volumeMounts:
        - name: shared-storage
          mountPath: /mnt/shared
  volumes:
    - name: shared-storage
      persistentVolumeClaim:
        claimName: shared-pvc  # Must match your PVC name from earlier

Deploy the pod and open a shell in it:

# Deploy the pod
kubectl apply -f gpu-pod.yaml

# Wait for it to be running
kubectl wait --for=condition=Ready pod/gpu-workload-pod --timeout=10m

# Exec into the pod
kubectl exec -it gpu-workload-pod -- bash

# Inside the pod, you can now:
nvidia-smi                    # See GPU(s) allocated to this pod
ls /mnt/shared                # Access your mounted volumes
python train.py               # Run your GPU workloads

Monitor workloads with Grafana

The Observability add-on provides Grafana dashboards for cluster nodes and workloads. To open them:

Enable the Observability add-on. When you create a cluster, it’s under Cluster add-ons. On a running cluster, it’s in the Add-ons section of the cluster’s Overview tab.
On the cluster details page, select Open Grafana under Access.

Troubleshooting

Pods aren’t scheduled: Check node status with kubectl get nodes, confirm the pod’s resource requests don’t exceed available resources, and check for taints with kubectl describe node <node_name>.
Volumes don’t mount: Check that the PVC is bound with kubectl get pvc. For the shared PVC, check that volumeName matches your shared volume’s name. For the local PVC, check that its storage class exists.
GPU isn’t accessible in a container: Use a CUDA-enabled image such as pytorch/pytorch or nvidia/cuda.

Next steps

Cluster storage

Choose the right storage tier for datasets and checkpoints.

Manage clusters

Manage access, scale capacity, and monitor cluster health.

Volcano

Gang-schedule distributed GPU jobs with the Volcano scheduler.

Kueue

Queue and admit GPU jobs with Kueue quotas.

Preemptible compute

Schedule interruption-tolerant workloads onto discounted preemptible nodes.

GET STARTED

SERVERLESS

INFERENCE APIS

DEDICATED MODEL INFERENCE

DEDICATED CONTAINER INFERENCE

POST-TRAINING

FINE-TUNING

GPU CLUSTERS

CODE EXECUTION

ADMINISTRATION

Deploy pods with storage

Access the shared volume from multiple pods

Run GPU workloads in containers

Monitor workloads with Grafana

Troubleshooting

Next steps

Cluster storage

Manage clusters

Volcano

Kueue

Preemptible compute

​Deploy pods with storage

​Access the shared volume from multiple pods

​Run GPU workloads in containers

​Monitor workloads with Grafana

​Troubleshooting

​Next steps

Cluster storage

Manage clusters

Volcano

Kueue

Preemptible compute

Deploy pods with storage

Access the shared volume from multiple pods

Run GPU workloads in containers

Monitor workloads with Grafana

Troubleshooting

Next steps