Skip to main content
Together GPU clusters provide persistent shared volumes, ephemeral NVMe disks local to each node, and a /home directory whose persistence depends on the cluster type.
Local NVMe disks and other node-local storage are ephemeral: Their data can be lost at any time during node migrations and recreations, maintenance, or other cluster operations. Always store data you need to keep on shared volumes (PVC-backed storage).

Storage types at a glance

  • Shared volumes (PVC): Persistent. Data survives pod restarts, node events (reboots, migrations, and recreations), cluster operations, and, by default, cluster deletion. Use shared volumes for training data, checkpoints, model weights, and anything else you can’t lose.
  • Local NVMe disks: Ephemeral. Fast local storage on each node. Its data can be lost during node migrations and recreations or cluster operations, so use it only for temporary scratch data, such as intermediate computation files.
  • /home directory: Depends on the cluster type. On Slurm clusters, /home is a persistent directory on the shared volume. On Kubernetes clusters, it’s local to each node and ephemeral.

Persistent storage: shared volumes

Shared volumes are remote-attached, high-speed filesystems that every node in the cluster can access. When you create a cluster, you either create a new shared volume with it or attach an existing one. Data on a shared volume persists across:
  • Pod restarts and rescheduling.
  • Node reboots, migrations, recreations, and maintenance.
  • Cluster scaling operations.
  • Cluster deletion, unless you choose to delete the volume along with the cluster.
When you delete a cluster but not its volume, the volume keeps billing as storage, and you can attach it to another cluster. A volume that draws its storage from a capacity pool moves to on-demand storage pricing once its reservation ends. How you access a shared volume depends on the cluster type:
  • Kubernetes clusters: The cluster provides a static PersistentVolume (PV) with the same name as your shared volume. Create a PersistentVolumeClaim (PVC) that references it, then mount it in your pods.
  • Slurm clusters: The shared volume is mounted at /data on all compute and login nodes.
Project ownership: By default, a new shared volume belongs to your default project. To assign it to a different project you have access to, pass project_id when you create the volume through the API or SDK. A volume you create with a cluster belongs to the cluster’s project instead, and the API ignores shared_volume.project_id in the create request.

Ephemeral storage: local NVMe disks

Each node has local NVMe disks for high-speed reads and writes. On Slurm clusters, this storage is mounted at /scratch on each compute node.
Data on local NVMe disks is not durable: It can be lost without warning during:
  • Node migrations and recreations (scheduled or unscheduled).
  • Cluster maintenance operations.
  • Hardware failures.
  • Pod rescheduling to a different node.
Do not rely on local NVMe for any data you need to keep. Use it only for temporary scratch files that can be regenerated.

/home directory

The behavior of /home differs between cluster types:

Slurm clusters

On Slurm clusters, /home is a persistent directory on your shared volume, available on every compute and login node. It links to /data/home, so home-directory data survives node events and counts toward your shared volume’s capacity. Use it for code, scripts, configuration files, logs, small datasets, model weights, and training data. Before you use /home, SSH into the Slurm login node to set up your user directory with the correct permissions.

Kubernetes clusters

On Kubernetes clusters, /home is local to each node and ephemeral. It is not shared across nodes and is subject to the same data loss risks as local NVMe storage.
On Kubernetes clusters, do not store important data in /home. Use a shared volume (PVC) instead.

Choose the right storage

Upload your data

To upload a small dataset, copy it into a pod that mounts your shared volume:
  1. Create a PVC that uses your shared volume’s name as its volumeName, and a pod that mounts the volume.
  2. Run kubectl cp LOCAL_FILENAME YOUR_POD_NAME:/mnt/shared/. If your pod mounts the volume at a different path, use that path instead of /mnt/shared/.
For a large dataset, schedule a pod on the cluster that downloads it directly from S3 or your data source, as in the example pod spec.

Next steps

Run workloads on Kubernetes

Create PVCs, mount volumes in pods, and run GPU workloads.

Manage GPU clusters

Upload data, resize storage, and monitor cluster health.