/home directory whose persistence depends on the cluster type.
Storage types at a glance
- Shared volumes (PVC): Persistent. Data survives pod restarts, node events (reboots, migrations, and recreations), cluster operations, and, by default, cluster deletion. Use shared volumes for training data, checkpoints, model weights, and anything else you can’t lose.
- Local NVMe disks: Ephemeral. Fast local storage on each node. Its data can be lost during node migrations and recreations or cluster operations, so use it only for temporary scratch data, such as intermediate computation files.
/homedirectory: Depends on the cluster type. On Slurm clusters,/homeis a persistent directory on the shared volume. On Kubernetes clusters, it’s local to each node and ephemeral.
Persistent storage: shared volumes
Shared volumes are remote-attached, high-speed filesystems that every node in the cluster can access. When you create a cluster, you either create a new shared volume with it or attach an existing one. Data on a shared volume persists across:- Pod restarts and rescheduling.
- Node reboots, migrations, recreations, and maintenance.
- Cluster scaling operations.
- Cluster deletion, unless you choose to delete the volume along with the cluster.
- Kubernetes clusters: The cluster provides a static PersistentVolume (PV) with the same name as your shared volume. Create a PersistentVolumeClaim (PVC) that references it, then mount it in your pods.
- Slurm clusters: The shared volume is mounted at
/dataon all compute and login nodes.
project_id when you create the volume through the API or SDK. A volume you create with a cluster belongs to the cluster’s project instead, and the API ignores shared_volume.project_id in the create request.
Ephemeral storage: local NVMe disks
Each node has local NVMe disks for high-speed reads and writes. On Slurm clusters, this storage is mounted at/scratch on each compute node.
/home directory
The behavior of /home differs between cluster types:
Slurm clusters
On Slurm clusters,/home is a persistent directory on your shared volume, available on every compute and login node. It links to /data/home, so home-directory data survives node events and counts toward your shared volume’s capacity. Use it for code, scripts, configuration files, logs, small datasets, model weights, and training data.
Before you use /home, SSH into the Slurm login node to set up your user directory with the correct permissions.
Kubernetes clusters
On Kubernetes clusters,/home is local to each node and ephemeral. It is not shared across nodes and is subject to the same data loss risks as local NVMe storage.
Choose the right storage
Upload your data
To upload a small dataset, copy it into a pod that mounts your shared volume:- Create a PVC that uses your shared volume’s name as its
volumeName, and a pod that mounts the volume. - Run
kubectl cp LOCAL_FILENAME YOUR_POD_NAME:/mnt/shared/. If your pod mounts the volume at a different path, use that path instead of/mnt/shared/.
Next steps
Run workloads on Kubernetes
Create PVCs, mount volumes in pods, and run GPU workloads.
Manage GPU clusters
Upload data, resize storage, and monitor cluster health.