> ## Documentation Index
> Fetch the complete documentation index at: https://docs.together.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Run workloads on Kubernetes

> Deploy pods with persistent storage, run GPU workloads, and monitor them with Grafana on a Kubernetes GPU cluster.

This page describes how to run workloads on a Together GPU cluster with Kubernetes: deploying pods with persistent storage, running GPU workloads in containers, and monitoring them with Grafana.

Before you start, [connect `kubectl` to your cluster](https://docs.together.ai/docs/gpu-clusters-quickstart#step-2-connect-to-the-cluster). For background on Kubernetes and how it works on Together, see [Concepts](https://docs.together.ai/docs/gpu-clusters-concepts.md#kubernetes).

## Deploy pods with storage

Together provides a static [PersistentVolume (PV)](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) with the same name as your shared volume. Data you write to the static PV persists across pod restarts, cluster operations, and, by default, cluster deletion.

Kubernetes connects storage to your containers through three objects:

* **PersistentVolume (PV):** The storage resource itself, managed by Together.
* **PersistentVolumeClaim (PVC):** A request for storage that your pods can use. You create this.
* **Pod with `volumeMounts`:** Mounts the PVC into your container at a specific path. You create this.

**1. Create PVCs**

For shared storage that multiple pods can read and write at the same time, create a PVC with [`ReadWriteMany` access](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes). Set `volumeName` to your shared volume's name so the PVC binds to the static PV:

```yaml shared-pvc.yaml theme={null}
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: shared-pvc                    # Name you'll reference in pods
spec:
  accessModes:
    - ReadWriteMany                   # Multiple pods can read/write simultaneously
  resources:
    requests:
      storage: 10Gi                   # Requested size (can be adjusted)
  volumeName: <shared volume name>    # Replace with your shared volume name from the console
```

> [!NOTE]
>
> The static PV uses your cluster's shared storage class, which is also the cluster's default storage class, so the PVC doesn't need a `storageClassName`. If the shared PVC stays `Pending`, check the PV's class with `kubectl get pv <shared volume name>` and add a matching `storageClassName` to the PVC.

For fast local NVMe storage that only one node can mount at a time, create a PVC with `ReadWriteOnce` access and the `local` storage class:

```yaml local-pvc.yaml theme={null}
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: local-pvc                     # Name you'll reference in pods
spec:
  accessModes:
    - ReadWriteOnce                   # Only one pod/node can mount at a time
  resources:
    requests:
      storage: 50Gi                   # Requested size
  storageClassName: local
```

Data on local NVMe is [ephemeral](https://docs.together.ai/docs/cluster-storage.md#ephemeral-storage-local-nvme-disks), so use the local PVC only for scratch files you can regenerate.

Apply both PVCs and check their status:

```bash theme={null}
kubectl apply -f shared-pvc.yaml -n default # change to your namespace
kubectl apply -f local-pvc.yaml -n default # change to your namespace

# Check PVC status
kubectl get pvc -A # across all namespaces
```

The shared PVC shows `Bound` right away. The local PVC stays `Pending` until a pod that mounts it is scheduled, which is expected for the `local` storage class.

**2. Define a pod that mounts the PVCs**

Define a pod that mounts both PVCs as [volumes](https://kubernetes.io/docs/concepts/storage/volumes/). Each `volumes` entry references a PVC by its `claimName`, and each `volumeMounts` entry uses a volume's `name` to mount it at a path inside the container:

```yaml pod-with-storage.yaml theme={null}
apiVersion: v1
kind: Pod
metadata:
  name: test-pod
spec:
  restartPolicy: Never
  containers:
    - name: debian
      image: debian:stable-slim
      command: ["/bin/sh", "-c", "sleep infinity"]  # Keeps pod running
      volumeMounts:                                   # Where to mount volumes inside container
        - name: shared-storage                        # References volume defined below
          mountPath: /mnt/shared                      # Path inside container
        - name: local-storage
          mountPath: /mnt/local
  volumes:                                            # Defines volumes from PVCs
    - name: shared-storage                            # Internal name for this volume
      persistentVolumeClaim:
        claimName: shared-pvc                         # Must match PVC name from Step 1
    - name: local-storage
      persistentVolumeClaim:
        claimName: local-pvc
```

**3. Deploy and access the pod**

Deploy the pod in the same namespace as its PVCs, then open a shell in it:

```bash theme={null}
# Deploy the pod
kubectl apply -f pod-with-storage.yaml -n default # should be same as the namespace in which PVC is deployed

# Wait for pod to be running
kubectl get pods -w

# Once STATUS shows "Running", access the pod
kubectl exec -it test-pod -- bash
```

**4. Verify mounted volumes**

Inside the pod, check that both volumes are mounted and that you can write to the shared volume:

```bash theme={null}
# Check mounted filesystems
df -h | grep /mnt

# List mounted directories
ls -la /mnt/shared
ls -la /mnt/local

# Test write access
echo "Hello from pod" > /mnt/shared/test.txt
cat /mnt/shared/test.txt
```

### Access the shared volume from multiple pods

Because the shared PVC uses `ReadWriteMany` access, multiple pods can mount it at the same time. To read the file you wrote from `test-pod`, create a second pod that mounts the same PVC. It needs its own manifest, because `kubectl run` can't attach a PVC:

```yaml test-pod-2.yaml theme={null}
apiVersion: v1
kind: Pod
metadata:
  name: test-pod-2
spec:
  restartPolicy: Never
  containers:
    - name: debian
      image: debian:stable-slim
      command: ["/bin/sh", "-c", "sleep infinity"]
      volumeMounts:
        - name: shared-storage
          mountPath: /mnt/shared
  volumes:
    - name: shared-storage
      persistentVolumeClaim:
        claimName: shared-pvc               # Same PVC as the first pod
```

Create the second pod and read the file from it:

```bash theme={null}
# Create the second pod
kubectl apply -f test-pod-2.yaml

# Exec into it
kubectl exec -it test-pod-2 -- bash

# The file you created from the first pod is visible here
cat /mnt/shared/test.txt
```

## Run GPU workloads in containers

To run GPU workloads or work with your data volumes, deploy a pod with GPU and storage access, then exec into it. Use a CUDA-enabled base image, such as `nvidia/cuda` or `pytorch/pytorch`, instead of installing CUDA yourself.

A pod gets the GPUs it requests with `nvidia.com/gpu`, so always request GPUs explicitly. On older clusters, a pod that requests none can still see every GPU on its node if its image sets `NVIDIA_VISIBLE_DEVICES`, as CUDA images do. The host provides the NVIDIA kernel driver, and your container image provides the CUDA libraries. In an image without CUDA, such as a `debian` or `ubuntu` base image, you can't run CUDA workloads until you install CUDA.

This pod mounts the `shared-pvc` claim from [Deploy pods with storage](#deploy-pods-with-storage), so create that PVC first:

```yaml gpu-pod.yaml theme={null}
apiVersion: v1
kind: Pod
metadata:
  name: gpu-workload-pod
spec:
  restartPolicy: Never
  containers:
    - name: pytorch
      image: pytorch/pytorch:2.8.0-cuda12.8-cudnn9-runtime  # CUDA-enabled image
      command: ["/bin/bash", "-c", "sleep infinity"]
      resources:
        limits:
          nvidia.com/gpu: 1  # Request 1 GPU
      volumeMounts:
        - name: shared-storage
          mountPath: /mnt/shared
  volumes:
    - name: shared-storage
      persistentVolumeClaim:
        claimName: shared-pvc  # Must match your PVC name from earlier
```

Deploy the pod and open a shell in it:

```bash theme={null}
# Deploy the pod
kubectl apply -f gpu-pod.yaml

# Wait for it to be running
kubectl wait --for=condition=Ready pod/gpu-workload-pod --timeout=10m

# Exec into the pod
kubectl exec -it gpu-workload-pod -- bash

# Inside the pod, you can now:
nvidia-smi                    # See GPU(s) allocated to this pod
ls /mnt/shared                # Access your mounted volumes
python train.py               # Run your GPU workloads
```

## Monitor workloads with Grafana

The **Observability** add-on provides Grafana dashboards for cluster nodes and workloads. To open them:

1. Enable the **Observability** add-on. When you create a cluster, it's under **Cluster add-ons**. On a running cluster, it's in the **Add-ons** section of the cluster's **Overview** tab.
2. On the cluster details page, select **Open Grafana** under **Access**.

## Troubleshooting

* **Pods aren't scheduled:** Check node status with `kubectl get nodes`, confirm the pod's resource requests don't exceed available resources, and check for taints with `kubectl describe node <node_name>`.
* **Volumes don't mount:** Check that the PVC is bound with `kubectl get pvc`. For the shared PVC, check that `volumeName` matches your shared volume's name. For the local PVC, check that its storage class exists.
* **GPU isn't accessible in a container:** Use a CUDA-enabled image such as `pytorch/pytorch` or `nvidia/cuda`.

## Next steps

> **[Cluster storage](https://docs.together.ai/docs/cluster-storage.md)**
>
> Choose the right storage tier for datasets and checkpoints.

> **[Manage clusters](https://docs.together.ai/docs/gpu-clusters-management.md)**
>
> Manage access, scale capacity, and monitor cluster health.

> **[Volcano](https://docs.together.ai/docs/volcano-on-gpu-clusters.md)**
>
> Gang-schedule distributed GPU jobs with the Volcano scheduler.

> **[Kueue](https://docs.together.ai/docs/kueue-on-gpu-clusters.md)**
>
> Queue and admit GPU jobs with Kueue quotas.

> **[Preemptible compute](https://docs.together.ai/docs/preemptible-compute.md)**
>
> Schedule interruption-tolerant workloads onto discounted preemptible nodes.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.