Manage GPU clusters - Together AI docs

This page covers the administration tasks for GPU clusters, from checking region availability and creating a cluster to scaling, storage, access, and deletion. Most tasks have steps for three surfaces:

Console: The GPU clusters page in the Together AI console.
Together CLI: tg clusters commands. See the clusters CLI reference.
SDKs and REST API: client.clusters methods in the Python and TypeScript SDKs, or the HTTP API for custom integrations. See the GPU clusters API reference.

To run these operations from pipelines, cron jobs, and scripts, see Automate cluster operations. To run workloads on a cluster, see Run workloads on Kubernetes or Slurm. To connect to individual nodes, see Connect over SSH.

Check region availability

Region availability changes over time, so check the live inventory before you choose a region or GPU type. These lookups are read-only and don’t create or reserve anything.

Console
CLI
SDK

Open the GPU clusters page and select Create cluster. The create form shows availability as you make selections:

GPU type cards: A grayed-out card that shows Contact us has no capacity available. After you select a card, the line under Select GPU Count shows how many GPUs of that type are available.
Region: The selector disables regions that can’t host your selection.
Driver version: The selector lists the NVIDIA driver and CUDA combinations for the selected region, GPU type, and orchestration.

tg clusters list-regions

Pass --json for machine-readable output.

from together import Together

client = Together()

for region in client.clusters.list_regions().regions:
    print(region.name, region.supported_instance_types)

import Together from "together-ai";

const client = new Together();

const { regions } = await client.clusters.listRegions();
for (const region of regions) {
  console.log(region.name, region.supported_instance_types);
}

The CLI and SDK lookups list each region’s supported instance types and driver versions, but not prices. For current rates, see the GPU cluster pricing table.

Create a cluster

Every cluster has a capacity type, which sets how it bills:

On-demand (ON_DEMAND): Bills hourly from creation until you delete the cluster.
Reserved (RESERVED): Bills up front for a fixed duration.
Scheduled capacity (SCHEDULED_CAPACITY): Draws GPUs from a project capacity pool for a scheduled window. See Schedule a cluster.

To choose between them, see Capacity types. For how each type appears on invoices, see Pricing. Every cluster also needs at least 8 GPUs, in multiples of 8, and a shared volume.

Console
CLI
SDK

Open the create form

On the GPU clusters page, select Create cluster.

Choose the capacity type and GPUs

Select On-demand, Reserved, or Scheduled Capacity. For reserved, also set a Duration in days. Select a GPU type card (for example, H100-SXM), then set the count under Select GPU Count.

Configure the cluster

Select a Region, then fill in the remaining sections:

Cluster Details: Enter a Cluster name, choose Kubernetes or Slurm under Orchestration, and optionally select a Driver version.
Storage: Select an existing volume or + Create new storage.
Cluster add-ons: Optionally enable add-ons such as autoscaling.
Advanced configuration: Optionally enable acceptance tests.

Create the cluster

Select Create Cluster, review the details, then select Confirm & Create.

For a first-run walkthrough of every field, see the quickstart.

Run tg clusters create with no flags to walk through the required fields in an interactive prompt, or pass them as flags. If you don’t pass --volume, the prompt offers to create a shared volume with the cluster.To create a cluster from a script, pass --non-interactive. Non-interactive mode requires --volume, which attaches an existing volume (see Manage storage). Also pass a --nvidia-driver-version and --cuda-version pair that the region supports. The list-regions output shows the valid pairs for each region:

tg clusters create \
  --name my-cluster \
  --cluster-type KUBERNETES \
  --gpu-type H100_SXM \
  --num-gpus 8 \
  --region us-central-8 \
  --nvidia-driver-version 570 \
  --cuda-version 12.8 \
  --billing-type ON_DEMAND \
  --volume <VOLUME_ID> \
  --non-interactive

For reserved billing, pass --billing-type RESERVED --duration-days <DAYS>. To request preemptible capacity alongside standard nodes, pass --num-preemptible-gpus. For all flags, see the CLI reference.

The SDK create call requires these parameters:

Cluster settings: cluster_name, cluster_type, num_gpus, gpu_type, region, and billing_type.
NVIDIA driver: Either nvidia_version_id, or nvidia_driver_version together with cuda_version. Use a combination that the region lists in list-regions.
Shared volume: Pass volume_id to attach an existing volume, or shared_volume to create one with the cluster. To use a new volume, create it with shared_volume rather than on its own, so it lands in a datacenter that has your GPU type.

from together import Together

client = Together()

cluster = client.clusters.create(
    cluster_name="my-cluster",
    cluster_type="KUBERNETES",
    gpu_type="H100_SXM",
    num_gpus=8,
    region="us-central-8",
    nvidia_driver_version="570",
    cuda_version="12.8",
    billing_type="ON_DEMAND",
    volume_id="<VOLUME_ID>",
)
print(cluster.cluster_id)

import Together from "together-ai";

const client = new Together();

const cluster = await client.clusters.create({
  cluster_name: "my-cluster",
  cluster_type: "KUBERNETES",
  gpu_type: "H100_SXM",
  num_gpus: 8,
  region: "us-central-8",
  nvidia_driver_version: "570",
  cuda_version: "12.8",
  billing_type: "ON_DEMAND",
  volume_id: "<VOLUME_ID>",
});
console.log(cluster.cluster_id);

Provisioning usually takes 10 minutes to an hour, depending on cluster size and region. Wait for the cluster to reach Ready status before you submit work. To poll for status, see Inspect clusters.

Schedule a cluster

Scheduled capacity clusters draw GPUs from a project capacity pool for a fixed window that you set in advance. With the CLI, SDK, or API, set num_capacity_pool_gpus to the number of GPUs to draw from the pool. If you omit it, every GPU is booked as a prepaid reserved GPU outside the pool. To start the cluster now, omit the start time.

Console
CLI
SDK

Choose scheduled capacity

In the create cluster form, select Scheduled Capacity, which appears when your project has a capacity pool. Then select a pool under Capacity Pool. The pool sets the GPU type and region.

Pick the window

Under Scheduled Capacity Dates, select start and end dates within the capacity pool window. To provision immediately, check Start now and select an end date. Start now is available only when the selected capacity pool window includes today’s date.

Size and create the cluster

Set the GPU count under Select Capacity Pool GPU Count, up to the pool’s free capacity. Fill in the remaining fields, then select Create Cluster and Confirm & Create. The cost estimate covers the full selected window.

tg clusters create \
  --name scheduled-batch \
  --billing-type SCHEDULED_CAPACITY \
  --capacity-pool-id <CAPACITY_POOL_ID> \
  --reservation-start-time 2026-12-01T00:00:00Z \
  --reservation-end-time 2026-12-08T00:00:00Z \
  --num-gpus 8 \
  --num-capacity-pool-gpus 8 \
  --gpu-type H100_SXM \
  --region us-central-8 \
  --nvidia-driver-version 570 \
  --cuda-version 12.8 \
  --cluster-type KUBERNETES \
  --volume <VOLUME_ID> \
  --non-interactive

from together import Together

client = Together()

client.clusters.create(
    cluster_name="scheduled-batch",
    billing_type="SCHEDULED_CAPACITY",
    capacity_pool_id="<CAPACITY_POOL_ID>",
    reservation_start_time="2026-12-01T00:00:00Z",
    reservation_end_time="2026-12-08T00:00:00Z",
    num_gpus=8,
    num_capacity_pool_gpus=8,
    gpu_type="H100_SXM",
    region="us-central-8",
    nvidia_driver_version="570",
    cuda_version="12.8",
    cluster_type="KUBERNETES",
    volume_id="<VOLUME_ID>",
)

import Together from "together-ai";

const client = new Together();

await client.clusters.create({
  cluster_name: "scheduled-batch",
  billing_type: "SCHEDULED_CAPACITY",
  capacity_pool_id: "<CAPACITY_POOL_ID>",
  reservation_start_time: "2026-12-01T00:00:00Z",
  reservation_end_time: "2026-12-08T00:00:00Z",
  num_gpus: 8,
  num_capacity_pool_gpus: 8,
  gpu_type: "H100_SXM",
  region: "us-central-8",
  nvidia_driver_version: "570",
  cuda_version: "12.8",
  cluster_type: "KUBERNETES",
  volume_id: "<VOLUME_ID>",
});

Create a hybrid scheduled capacity cluster

A hybrid scheduled capacity cluster combines GPUs from your capacity pool with prepaid reserved GPUs outside the pool. With the CLI, SDK, or API, use SCHEDULED_CAPACITY billing and set num_gpus to the cluster’s total GPU count. Split it across num_capacity_pool_gpus and num_reserved_gpus. Both counts must be multiples of 8, and together they can’t exceed num_gpus. If you leave out num_reserved_gpus, the rest of num_gpus is reserved. A hybrid cluster starts now, so leave out the start time.

Console
CLI
SDK

In the create form with Scheduled Capacity selected, check Start now, then set the pool GPU count under Select Capacity Pool GPU Count. To add prepaid reserved GPUs from outside the pool, turn on Add compute outside your capacity pool, which appears only when the cluster starts now, then set Reserved GPUs.

tg clusters create \
  --name hybrid-scheduled \
  --billing-type SCHEDULED_CAPACITY \
  --capacity-pool-id <CAPACITY_POOL_ID> \
  --reservation-end-time 2026-12-08T00:00:00Z \
  --num-gpus 16 \
  --num-capacity-pool-gpus 8 \
  --num-reserved-gpus 8 \
  --gpu-type H100_SXM \
  --region us-central-8 \
  --nvidia-driver-version 570 \
  --cuda-version 12.8 \
  --cluster-type KUBERNETES \
  --volume <VOLUME_ID> \
  --non-interactive

See the clusters CLI reference for the full flag list.

from together import Together

client = Together()

client.clusters.create(
    cluster_name="hybrid-scheduled",
    billing_type="SCHEDULED_CAPACITY",
    capacity_pool_id="<CAPACITY_POOL_ID>",
    reservation_end_time="2026-12-08T00:00:00Z",
    num_gpus=16,
    num_capacity_pool_gpus=8,
    num_reserved_gpus=8,
    gpu_type="H100_SXM",
    region="us-central-8",
    nvidia_driver_version="570",
    cuda_version="12.8",
    cluster_type="KUBERNETES",
    volume_id="<VOLUME_ID>",
)

import Together from "together-ai";

const client = new Together();

await client.clusters.create({
  cluster_name: "hybrid-scheduled",
  billing_type: "SCHEDULED_CAPACITY",
  capacity_pool_id: "<CAPACITY_POOL_ID>",
  reservation_end_time: "2026-12-08T00:00:00Z",
  num_gpus: 16,
  num_capacity_pool_gpus: 8,
  num_reserved_gpus: 8,
  gpu_type: "H100_SXM",
  region: "us-central-8",
  nvidia_driver_version: "570",
  cuda_version: "12.8",
  cluster_type: "KUBERNETES",
  volume_id: "<VOLUME_ID>",
});

Extend a reservation

To keep a reserved cluster running past its decommission time, set a later reservation end time. The console also shows Extend Reservation on scheduled capacity clusters, but the request returns a 403 unless Together has enabled extensions for your account.

Console
CLI
SDK

Open the cluster on the GPU clusters page, select Extend Reservation, choose the new end date under Extend Duration, then select Review and Purchase. The review step shows the hourly rate and the estimated total cost of the extension.

tg clusters update <CLUSTER_ID> \
  --reservation-end-time 2027-01-15T15:00:00Z

from together import Together

client = Together()

client.clusters.update(
    "<CLUSTER_ID>",
    reservation_end_time="2027-01-15T15:00:00Z",
)

import Together from "together-ai";

const client = new Together();

await client.clusters.update("<CLUSTER_ID>", {
  reservation_end_time: "2027-01-15T15:00:00Z",
});

Inspect clusters

List your clusters and retrieve one cluster’s full state, including its status, node counts, and configuration.

Console
CLI
SDK

The GPU clusters page lists every cluster with its type, creator, provisioning and reservation end times, node counts, region, and status. Select a cluster to open its detail page, where the sidebar shows its resources, region, reservation dates, GPU breakdown, driver version, and add-on configuration.

tg clusters list
tg clusters retrieve <CLUSTER_ID>

Pass --json to either command for machine-readable output, for example to poll for "status": "Ready" in scripts.

from together import Together

client = Together()

for cluster in client.clusters.list().clusters:
    print(cluster.cluster_id, cluster.cluster_name)

cluster = client.clusters.retrieve("<CLUSTER_ID>")
print(cluster.status)

import Together from "together-ai";

const client = new Together();

const { clusters } = await client.clusters.list();
for (const cluster of clusters) {
  console.log(cluster.cluster_id, cluster.cluster_name);
}

const cluster = await client.clusters.retrieve("<CLUSTER_ID>");
console.log(cluster.status);

Inspect cluster GPU counts

When you retrieve a cluster, num_gpus is its standard (non-preemptible) GPU count. It splits into billed components:

num_reserved_gpus: Prepaid reserved GPUs on the cluster.
num_capacity_pool_gpus: GPUs drawn from the cluster’s capacity pool.

Any GPUs beyond those two components are billed on demand, for example after you scale up a reserved or scheduled capacity cluster. How the counts relate depends on the cluster:

Cluster without a capacity pool: num_capacity_pool_gpus is 0.
Fully reserved cluster: num_reserved_gpus equals num_gpus.
Fully on-demand cluster: num_reserved_gpus is 0.
Hybrid scheduled capacity cluster: num_gpus equals num_capacity_pool_gpus plus num_reserved_gpus.

Preemptible GPUs aren’t part of num_gpus. The cluster reports them separately, in desired_preemptible_gpus and allocated_preemptible_gpus.

Console
CLI
SDK

On a cluster with a capacity pool, the cluster detail sidebar lists Capacity Pool GPUs, plus Reserved GPUs and On-Demand GPUs when they’re above zero.

tg clusters retrieve <CLUSTER_ID> --json

from together import Together

client = Together()

cluster = client.clusters.retrieve("<CLUSTER_ID>")
print(
    cluster.num_gpus,
    cluster.num_reserved_gpus,
    cluster.num_capacity_pool_gpus,
)

import Together from "together-ai";

const client = new Together();

const cluster = await client.clusters.retrieve("<CLUSTER_ID>");
console.log(
  cluster.num_gpus,
  cluster.num_reserved_gpus,
  cluster.num_capacity_pool_gpus,
);

Scale a cluster

Scale a cluster up in real time with on-demand compute when workload demand spikes, then back down as demand decreases. You can also set a preemptible GPU target that Together fills as spare capacity becomes available. To scale, set the cluster’s new total GPU count, which can’t be lower than its reserved GPU count:

Console
CLI
SDK

Open the cluster on the GPU clusters page and select Scale. Choose the new GPU count, select Review, then Purchase, and enter the cluster name to confirm. The GPU size picker lists every full-node multiple of 8 GPUs, up to the capacity currently available for your GPU type and region, rather than a fixed upper limit.

tg clusters update <CLUSTER_ID> --num-gpus 16

from together import Together

client = Together()

client.clusters.update(
    "<CLUSTER_ID>",
    num_gpus=16,
)

import Together from "together-ai";

const client = new Together();

await client.clusters.update("<CLUSTER_ID>", {
  num_gpus: 16,
});

Update cluster GPU counts

To change how a cluster’s GPUs are split across billed components without changing num_gpus, update the individual component counts:

num_reserved_gpus: Increases the prepaid reserved GPU count on a reserved or scheduled capacity cluster. The value must be a multiple of 8 and can’t exceed num_gpus. You can’t decrease it. To reduce a reservation, contact your account team.
num_capacity_pool_gpus: Changes the number of GPUs drawn from the cluster’s capacity pool. It applies only to clusters created with a capacity pool. The value must be a multiple of 8, and its sum with num_reserved_gpus can’t exceed num_gpus.

Console
CLI
SDK

In the Scale dialog, reserved and scheduled capacity clusters show a Reserved GPUs selector next to On-Demand GPUs, and clusters created with a capacity pool also show Capacity Pool GPUs. The dialog sets num_gpus to their sum.

tg clusters update <CLUSTER_ID> --num-reserved-gpus 16

from together import Together

client = Together()

client.clusters.update(
    "<CLUSTER_ID>",
    num_reserved_gpus=16,
)

import Together from "together-ai";

const client = new Together();

await client.clusters.update("<CLUSTER_ID>", {
  num_reserved_gpus: 16,
});

For how reserved and on-demand usage appear on invoices, see Pricing.

Autoscaling

Cluster autoscaling adjusts the node count to match workload demand. It uses the Kubernetes Cluster Autoscaler, which:

Scales up when pods are pending due to insufficient resources.
Scales down by safely draining and removing nodes that stay underutilized for an extended period.
Keeps the GPU count between the cluster’s num_gpus and auto_scale_max_gpus.

You can’t turn autoscaling on or off after a cluster is created. On an autoscaled cluster, the autoscaler owns the GPU count, and any update through tg clusters update or clusters.update returns 403 “cluster update is not allowed when auto scale is enabled”. To enable autoscaling, turn it on when you create the cluster and set the maximum GPU count it can scale to, a multiple of 8 that’s at least num_gpus:

Console
CLI
SDK

When you create a Kubernetes cluster in the create cluster form, set Enable Autoscaling to Yes under Cluster add-ons, then set Max GPUs.

tg clusters create \
  --name autoscaled-cluster \
  --cluster-type KUBERNETES \
  --gpu-type H100_SXM \
  --num-gpus 8 \
  --region us-central-8 \
  --nvidia-driver-version 570 \
  --cuda-version 12.8 \
  --billing-type ON_DEMAND \
  --volume <VOLUME_ID> \
  --auto-scale \
  --auto-scale-max-gpus 32

from together import Together

client = Together()

client.clusters.create(
    cluster_name="autoscaled-cluster",
    cluster_type="KUBERNETES",
    gpu_type="H100_SXM",
    num_gpus=8,
    region="us-central-8",
    nvidia_driver_version="570",
    cuda_version="12.8",
    billing_type="ON_DEMAND",
    volume_id="<VOLUME_ID>",
    auto_scale=True,
    auto_scale_max_gpus=32,
)

import Together from "together-ai";

const client = new Together();

await client.clusters.create({
  cluster_name: "autoscaled-cluster",
  cluster_type: "KUBERNETES",
  gpu_type: "H100_SXM",
  num_gpus: 8,
  region: "us-central-8",
  nvidia_driver_version: "570",
  cuda_version: "12.8",
  billing_type: "ON_DEMAND",
  volume_id: "<VOLUME_ID>",
  auto_scale: true,
  auto_scale_max_gpus: 32,
});

Autoscaling works with both reserved and on-demand capacity. Scaling beyond reserved capacity provisions on-demand nodes, billed at standard hourly rates.

Targeted scale-down

To choose which nodes a scale-down removes, mark them for deletion before you scale down. Scale-down removes annotated and cordoned nodes before any others.

Mark the node for deletion

Mark the node with one of these methods:

Annotate (Kubernetes or Slurm, preferred): Annotating leaves the node schedulable for existing workloads until scale-down runs. On a Slurm cluster, run the command with the admin kubeconfig, using a node name from kubectl get nodes.
kubectl annotate node <NODE_NAME> node.together.ai/delete-node-on-scale-down=true
Cordon (Kubernetes): Cordon the node if you also want to stop new pods from being scheduled onto it immediately.
kubectl cordon <NODE_NAME>

Drain (Slurm): Drain the node with scontrol.

sudo scontrol update NodeName=<NODE_NAME> State=drain Reason="<REASON>"

Wait for the mark to register

In the Nodes tab of the console, wait until Marked for deletion appears next to the node. This confirms that Together has picked up the mark. Scaling down before it appears can cause a race condition that leaves the drain unfinished.

Trigger scale-down

Reduce the cluster’s total GPU count by one node’s worth of GPUs, with any of the methods in Scale a cluster. The marked node is removed first.

tg clusters update <CLUSTER_ID> --num-gpus <NEW_TOTAL_GPUS>

Scale down one node at a time, repeating these steps for each node you want to remove. This gives Together time to drain each node cleanly, and lets you stop if something goes wrong.

Delete a cluster

On-demand clusters bill hourly until you delete them. Shared volumes persist after you delete the cluster, and keep billing, unless you delete them too.

Console
CLI
SDK

Open the cluster on the GPU clusters page, select Delete, then enter the cluster name to confirm. To also remove the attached shared volume, check Delete attached shared volume in the same dialog.

tg clusters delete <CLUSTER_ID>

In an interactive session, the command asks you to confirm before deleting. Pass --force to delete without confirming. The global --non-interactive and --json flags also suppress the prompt, so scripted and CI deletes never wait for input.

from together import Together

client = Together()

client.clusters.delete("<CLUSTER_ID>")

import Together from "together-ai";

const client = new Together();

await client.clusters.delete("<CLUSTER_ID>");

Manage storage

All clusters include persistent shared volumes, fast ephemeral NVMe disks local to each node, and a /home directory whose persistence depends on the cluster type. For the full breakdown and best practices, see Cluster storage. Attach a volume when you create a cluster, through the Storage section of the create form in the console, the --volume flag in the CLI, or the volume_id parameter in the SDK. You can’t swap the volume on an existing cluster. You can attach a volume that outlives its cluster to a new cluster in the same region, as long as no other cluster uses it.

Create a volume

A volume you create on its own is always independent of a cluster’s lifecycle, so it persists, and keeps billing, after the cluster it’s attached to is deleted or decommissioned, unless you delete it along with the cluster. Delete it separately when you no longer need it. Volume names can contain only lowercase letters, digits, hyphens, and dots.

Console
CLI
SDK

On the GPU clusters page, select the Shared Filesystem tab, then Create Volume. Select a Region, enter a Volume Name (up to 44 characters) and Size (TiB), then select Create Volume. You can also create a volume with a cluster, from the Storage section of the create cluster form.

tg clusters storage create \
  --volume-name my-training-data \
  --size-tib 1 \
  --region us-central-8

from together import Together

client = Together()

volume = client.clusters.storage.create(
    volume_name="my-training-data",
    size_tib=1,
    region="us-central-8",
)
print(volume.volume_id)

import Together from "together-ai";

const client = new Together();

const volume = await client.clusters.storage.create({
  volume_name: "my-training-data",
  size_tib: 1,
  region: "us-central-8",
});
console.log(volume.volume_id);

To assign the volume to a specific project, pass project_id in the SDK or API. If you omit it, the volume goes to your default project.

List volumes

Console
CLI
SDK

The Shared Filesystem tab on the GPU clusters page lists every volume with its size, owner, region, and the clusters currently using it.

tg clusters storage list
tg clusters storage retrieve <VOLUME_ID>

from together import Together

client = Together()

for volume in client.clusters.storage.list().volumes:
    print(volume.volume_id, volume.volume_name, volume.size_tib)

import Together from "together-ai";

const client = new Together();

const { volumes } = await client.clusters.storage.list();
for (const volume of volumes) {
  console.log(volume.volume_id, volume.volume_name, volume.size_tib);
}

Upload data

Both upload methods go through a pod that mounts your shared volume. To create the PersistentVolumeClaim (PVC) and mount the volume in a pod, see Deploy pods with storage. For small datasets, copy files into the pod with kubectl cp:

# Copy a local file into a pod that mounts the shared volume
kubectl cp LOCAL_FILENAME POD_NAME:/mnt/shared/

For large datasets, schedule a pod on the cluster that downloads directly from S3 or your data source. This example mounts the volume through a PVC named shared-pvc. For a private bucket, give the container AWS credentials, for example from a Kubernetes Secret. For a public bucket, add --no-sign-request to the aws s3 cp command instead:

data-loader.yaml

apiVersion: v1
kind: Pod
metadata:
  name: data-loader
spec:
  restartPolicy: OnFailure
  containers:
    - name: downloader
      image: amazon/aws-cli
      command: ["aws", "s3", "cp", "s3://bucket/data", "/mnt/shared/", "--recursive"]
      volumeMounts:
        - name: shared-storage
          mountPath: /mnt/shared
  volumes:
    - name: shared-storage
      persistentVolumeClaim:
        claimName: shared-pvc

Resize storage

Increase a shared volume’s size as your data grows. The added capacity bills at the same rate as the rest of the volume. To decrease a volume’s size, contact your account team.

Console
CLI
SDK

Open the storage list

On the GPU clusters page, select the Shared Filesystem tab.

Edit the volume

Open the actions menu on the volume you want to resize and select Update Volume.

Set the new size

Enter the new size in TiB, then select Update Volume.

Find the volume ID, then set the new total size in TiB:

tg clusters storage list
tg clusters storage update <VOLUME_ID> --size-tib 4

from together import Together

client = Together()

client.clusters.storage.update(
    volume_id="<VOLUME_ID>",
    size_tib=4,
)

import Together from "together-ai";

const client = new Together();

await client.clusters.storage.update({
  volume_id: "<VOLUME_ID>",
  size_tib: 4,
});

Self-serve resizing isn’t enabled on every account. If the console dialog says the feature isn’t available, or the CLI or API returns a 403, contact your account team.

Delete a volume

Before you delete a volume, delete the cluster that uses it.

Deleting a volume permanently destroys its data.

Console
CLI
SDK

On the Shared Filesystem tab, open the actions menu on the volume and select Delete Volume, then enter the volume name to confirm. The action is disabled while a cluster still uses the volume.

tg clusters storage delete <VOLUME_ID>

In an interactive session, the command asks you to confirm before deleting. Pass --force to delete without confirming.

from together import Together

client = Together()

client.clusters.storage.delete("<VOLUME_ID>")

import Together from "together-ai";

const client = new Together();

await client.clusters.storage.delete("<VOLUME_ID>");

Manage cluster access

Cluster access follows Together’s project-based permission system. Anyone with access to a project can access all clusters and volumes in it. Projects have two roles:

admin: Can create and delete clusters, modify configurations, manage users, and use clusters.
editor: Can use clusters (SSH, kubectl, Slurm) but can’t create, delete, or modify infrastructure.

Download cluster kubeconfig

Both Kubernetes and Slurm clusters expose an admin kubeconfig with cluster-admin access to the cluster’s Kubernetes API through kubectl. Only project admins can get it, from the console, the Together CLI, or the API.

Console
CLI

Open your cluster on the GPU clusters page. The kubeconfig appears once the cluster is ready, and is hidden while the cluster is paused.
Find the Kubeconfig row in the cluster sidebar. When OpenID Connect (OIDC) is enabled, the row is labeled Admin Kubeconfig, and a separate OIDC Kubeconfig row appears.
Select View to copy the contents, or Download to save it as a file.

tg clusters get-credentials <CLUSTER_ID> --set-default-context

Find your cluster ID with tg clusters list. If you aren’t a project admin, the command prints “No kubeconfig found for cluster at this time.”

To give editors kubectl access, enable OIDC on the cluster and have a project admin share the OIDC kubeconfig with them. The file contains no credentials. Editors can’t download it themselves, because the console builds it from the admin kubeconfig. Keep the admin kubeconfig for administrative tasks such as role-based access control (RBAC) setup.

Add users to a cluster project

To give someone access to a cluster, add them as a collaborator on the cluster’s project. They must already belong to your organization. Open Settings > Project > Collaborators for the project that contains your cluster, then select Add collaborator. New collaborators get the editor role by default, except organization admins, who are admins on every project. To add external collaborators or remove collaborators, see Managing project collaborators. Removing a collaborator revokes their access to all clusters and volumes in the project, including SSH access, within minutes.

Monitor cluster health

Together monitors every node with passive health checks and raises repair recommendations for unhealthy ones. You can also run active health checks on demand.

Console
Kubernetes
Slurm

Open your cluster on the GPU clusters page to view its status (for example, Running tests, Ready, or Degraded). The detail page splits monitoring across these tabs:

Overview: Node health, GPU utilization, GPU memory, and network bandwidth metrics.
Nodes: Per-node state, hardware details, and a copyable SSH command.
Control plane or System nodes: The cluster’s control plane or system nodes, depending on the cluster’s architecture.
Health checks: Run active health checks and review results.
Repairs: Pending and past node repairs.

For detailed metrics, select Open Grafana under Access in the sidebar. This requires the Observability add-on. See Monitor workloads with Grafana.

kubectl get nodes           # Node status
kubectl top nodes           # Resource usage
kubectl get pods --all-namespaces  # All running workloads

sinfo                       # Node and partition status
squeue                      # Job queue
scontrol show node          # Detailed node info

Request capacity

If the console doesn’t show available capacity for the GPU type or configuration you need, request it from the create cluster view:

Select Need a different configuration? to open the Notify when available dialog.
Set the capacity you need:
- GPU type and Requested GPUs.
- Region, or leave Any region.
- Storage size (TiB) (optional).
- Requested start date for on-demand capacity, or Requested dates (start and end) for reserved or scheduled capacity.
Review the Estimated compute cost, then select Submit request.

Together uses these requests for demand planning, and reaches out if the capacity becomes available.

Due to high demand, submitting a request doesn’t guarantee fulfillment. If you need guaranteed GPU capacity for a fixed period, reach out to the sales team.

Best practices

Storage: Use shared volumes for training data, checkpoints, model weights, and application state. On Kubernetes, mount them in pods through a PVC. Local NVMe and node-local /home (on Kubernetes) are ephemeral and can be wiped during migrations, recreations, or maintenance, so never rely on them for data you can’t afford to lose. Use local NVMe only for temporary scratch files that you can regenerate. See Cluster storage.
Large datasets: Download them directly on the cluster rather than through your local machine. See Upload data.
Job scheduling: Use Kubernetes Jobs for batch processing and Slurm job arrays for embarrassingly parallel workloads. Set resource requests and limits in pod specs, and set appropriate timeouts and retry policies.
Scaling strategy: Start with reserved capacity for your baseline workload, add on-demand capacity for burst periods, and use targeted scale-down to control costs.

Troubleshooting

Create or scale fails with 409 “Out of stock”: The region doesn’t have room for the GPU type and count you requested. Create returns “Out of stock - insufficient capacity”, and scale returns “GPU out of stock”. Try another region that offers your GPU type, request fewer GPUs, or request capacity. list-regions shows which regions offer each GPU type, but not how much capacity they have.
Scale fails with 403 “less than the reservation state”: You requested fewer GPUs than the cluster has reserved. To reduce a reservation, contact your account team.
Scale-up or volume resize fails with 402: Your account balance is too low. Add credits, then retry. See Running out of credits.
Volume create or resize fails with 409 “Out of stock - no storage available”: The region has no room for the volume.
Slurm jobs aren’t running: Check node status with sinfo, verify the partition is available, and check job status with scontrol show job <JOB_ID>.
Workload issues on Kubernetes: For pod scheduling, storage mount, and GPU access problems, see Run workloads on Kubernetes.

Next steps

Run workloads on Kubernetes

Deploy pods with persistent storage and run GPU workloads.

Connect over SSH

Reach login, worker, and compute nodes with OIDC or key-based SSH.

Health checks

Monitor node health with active tests and continuous passive checks.

Automate cluster operations

Drive cluster operations from CI/CD pipelines, scripts, and cron jobs.