> ## Documentation Index
> Fetch the complete documentation index at: https://docs.together.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Concepts

> Understand how nodes, Kubernetes, Slurm, storage, and preemptible compute fit together on a Together GPU cluster.

Every GPU cluster is built from the same stack: GPU nodes (virtual machines on bare-metal hosts), a Kubernetes layer that schedules work onto them, and, on Slurm clusters, a Slurm layer on top. This page explains each layer, where your data lives, and how preemptible compute works. The [glossary](#glossary) defines the terms used throughout the clusters docs.

## Architecture

![Diagram of a Together GPU cluster. The user reaches the Kubernetes control plane with kubectl, or Slurm on Kubernetes with ssh, sbatch, and srun. The control plane schedules pods on nodes, and Slurm schedules jobs on nodes. Both target the same GPU worker nodes, called compute nodes in Slurm, each with 8 GPUs and local NVMe, running pods and jobs side by side and connected by InfiniBand. A persistent shared volume that survives reboots is mounted on every node.](https://mintcdn.com/togetherai-52386018/3eXILkcCSB2Gej8O/images/gpu-clusters-architecture.png?fit=max\&auto=format\&n=3eXILkcCSB2Gej8O\&q=85\&s=54944a2600d8a0cb7e9a73265319508a)

* **Hardware:** GPU worker nodes connected by InfiniBand, each with a local NVMe disk, plus a persistent shared volume mounted on all of them. This is the layer you rent.
* **Kubernetes:** Every cluster runs Kubernetes, including Slurm clusters. A control plane managed by Together tracks cluster state and schedules pods onto the worker nodes. You reach it with `kubectl` after [downloading credentials](https://docs.together.ai/docs/gpu-clusters-quickstart.md#step-2-connect-to-the-cluster).
* **Slurm (optional):** On Slurm clusters, the Slurm stack runs on top of Kubernetes. You SSH into a login node and submit jobs, and the Slurm scheduler places them on the same GPU nodes.

Standard tooling works out of the box at every layer: `kubectl` and the Kubernetes ecosystem, Slurm's batch commands, and orchestration tools like [SkyPilot](https://docs.together.ai/docs/skypilot-on-gpu-clusters.md), [Volcano](https://docs.together.ai/docs/volcano-on-gpu-clusters.md), and [Kueue](https://docs.together.ai/docs/kueue-on-gpu-clusters.md).

Together monitors every node with [passive health checks](https://docs.together.ai/docs/health-checks.md#passive-health-checks) and repairs unhealthy nodes through [automated repair recommendations](https://docs.together.ai/docs/node-repair.md). You can also run [active health checks](https://docs.together.ai/docs/health-checks.md#active-health-checks) on demand.

## Kubernetes

Kubernetes is a container orchestration system that schedules containerized workloads across a cluster of machines. Together manages the control plane, and you get full `kubectl` access to schedule work onto the GPU worker nodes. For general Kubernetes usage, see the [Kubernetes documentation](https://kubernetes.io/docs/home/).

Kubernetes is declarative. You describe what you want to run, and Kubernetes finds nodes with the resources to run it, starts it, and restarts it if it fails.

These Kubernetes terms come up throughout the clusters docs:

* **Container:** Your workload packaged with its dependencies into an image, so it runs the same way on any node.
* **Pod:** The smallest unit of work Kubernetes schedules onto a node, made up of one or more containers.
* **Manifest:** A YAML file describing what you want to run. You apply it with `kubectl apply`, and Kubernetes makes the cluster match it.
* **Jobs and deployments:** The two most common workload types. A job runs pods to completion, for example to train a model. A deployment keeps pods running, for example to serve inference.

To deploy pods with persistent storage, see [Run workloads on Kubernetes](https://docs.together.ai/docs/kubernetes-workloads.md).

## Slurm

Slurm is a cluster management system that schedules jobs across a cluster of computers. On a Slurm cluster, Slurm comes preconfigured for distributed training. You submit jobs from the login node, and the scheduler assigns them to GPU nodes based on resource availability. For general Slurm usage, see the [Slurm quick start user guide](https://slurm.schedmd.com/quickstart.html).

Choose a Slurm cluster when your team works with high-performance computing (HPC) batch workflows, such as `sbatch` scripts, job arrays, and partitions. Choose a Kubernetes cluster when your workloads are containerized and you want to work with `kubectl` and the Kubernetes ecosystem directly. You select the cluster type when you [create the cluster](https://docs.together.ai/docs/gpu-clusters-quickstart.md). Both types run on the same hardware, storage, and networking.

Together runs Slurm on top of Kubernetes through Slinky. A Slurm cluster has the same layers underneath as a Kubernetes cluster, and Slurm adds its own vocabulary:

* **Login node:** The machine you SSH into to write, submit, and monitor jobs. You submit work from it rather than running work on it directly. The console labels it the head node, and the two terms are interchangeable.
* **Compute node:** Slurm's name for a GPU worker node. Each one is registered with both Kubernetes and Slurm.
* **Job:** Slurm's unit of work, submitted with `sbatch` (batch) or `srun` (interactive). A job plays the same role on a Slurm cluster that a pod plays on a Kubernetes cluster.
* **Partition:** A named group of compute nodes that jobs are submitted to. To change partitions, see [Slurm configuration](https://docs.together.ai/docs/slurm-configuration.md).

The Slurm controller itself runs as pods on the cluster, but you never manage those pods directly. You interact with Slurm entirely through its own commands. For common usage patterns and troubleshooting, see [Submit Slurm jobs](https://docs.together.ai/docs/slurm.md).

## Storage

Every cluster has these storage locations:

* **Shared volume:** A persistent, remote-attached filesystem mounted on every node. It survives reboots, node repairs, and, by default, cluster deletion. Store datasets, checkpoints, model weights, and anything else you can't lose on it.
* **Local NVMe:** A fast ephemeral disk on each node. Data on it can be lost during reboots, repairs, or rescheduling. Use it only for scratch files you can regenerate.
* **`/home`:** Persistence depends on the cluster type. On Slurm clusters, `/home` lives on the shared volume and persists. On Kubernetes clusters, it's local to each node and ephemeral.

For mount paths, upload instructions, and best practices, see [Cluster storage](https://docs.together.ai/docs/cluster-storage.md).

## Preemptible compute

You can request two kinds of compute on a cluster:

* **Standard:** The default. Standard nodes are provisioned up front when you create or scale a cluster, and they stay in the cluster until you scale down or delete it. Together never preempts them.
* **Preemptible:** Discounted nodes built from Together's spare GPU capacity. Because that capacity is spare, Together can reclaim, or *preempt*, a preemptible node whenever the capacity is needed elsewhere.

You don't provision preemptible nodes directly. Instead, you set a target GPU count, and Together adds nodes toward that target as spare capacity becomes available. The target is not guaranteed.

When a node is preempted, Together drains it for up to five minutes so running work can checkpoint and exit. After the node is removed, Together provisions replacements toward your target automatically.

In exchange for this uncertainty, preemptible usage is billed at a discounted rate, and only for the time a node is live.

For the full preemption contract, scheduling guidance, and checkpoint examples, see [Preemptible compute](https://docs.together.ai/docs/preemptible-compute.md).

## Glossary

| Term | What it means |
| - | - |
| Cluster | Everything you create: nodes, networking, storage, and the control plane. |
| Node | One GPU virtual machine (VM) in the cluster, running on a bare-metal host. |
| Control plane | The managed Kubernetes components that track cluster state and schedule pods. |
| Worker node | A GPU node that runs your workloads. |
| Compute node | Slurm's name for a GPU worker node. |
| Login node | The machine you SSH into on a Slurm cluster to submit jobs. The console labels it the head node. |
| Pod | The smallest unit of work Kubernetes schedules onto a node. |
| Job (Kubernetes) | A workload that runs pods to completion, such as a training run. |
| Job (Slurm) | The unit of work you submit to Slurm with `sbatch` or `srun`. |
| Partition | A named group of compute nodes that jobs run in. |
| Shared volume | Persistent storage mounted on every node. It survives reboots and, by default, cluster deletion. |
| Local NVMe | A fast ephemeral disk on each node. Data on it can be lost during reboots or repairs. |
| InfiniBand | High-bandwidth, low-latency networking that connects the nodes for multi-node training. |
| Reserved capacity | GPUs reserved for a fixed period and paid for upfront at discounted rates. |
| On-demand capacity | GPUs billed per hour of cluster runtime with no commitment. |
| Capacity pool | Pre-arranged capacity attached to a project. Scheduled capacity clusters draw GPUs from it for a fixed window. |
| Standard node | A node provisioned up front when you create or scale a cluster. Together never preempts it. |
| Preemptible node | A discounted node that Together can reclaim at any time with a five-minute drain window. [Preemptible compute](https://docs.together.ai/docs/preemptible-compute.md) is in preview on both Kubernetes and Slurm clusters. |

## Next steps

> **[Quickstart](https://docs.together.ai/docs/gpu-clusters-quickstart.md)**
>
> Create a cluster, connect to it, and run your first job.

> **[Run workloads on Kubernetes](https://docs.together.ai/docs/kubernetes-workloads.md)**
>
> Deploy pods with persistent storage and run GPU workloads.

> **[Submit Slurm jobs](https://docs.together.ai/docs/slurm.md)**
>
> Submit and manage batch jobs with the preconfigured Slurm scheduler.

> **[Cluster storage](https://docs.together.ai/docs/cluster-storage.md)**
>
> Choose the right storage tier for datasets and checkpoints.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.