Clusters

Manage GPU clusters, node remediations, and storage volumes from the Together CLI.

Requires Together CLI 2.40+. Earlier versions used tg beta clusters. CLI 3.0 removed that path, so update any scripts that still call it. Check your version with tg --version. To install or upgrade, see Get started.

Create a cluster

tg clusters create

Parameters

Flag	Description
`--name [string]`	Name for the cluster.
`--num-gpus [integer]`	Number of GPUs to allocate in the cluster.
`--region [string]`	Region to create the cluster in. List valid regions with `tg clusters list-regions`.
`--billing-type [ON_DEMAND\|RESERVED\|SCHEDULED_CAPACITY]`	Billing type for the cluster. `ON_DEMAND` bills from the moment the cluster is created until you delete it. `RESERVED` starts billing immediately. The cluster is torn down automatically after `--duration-days` elapses. `SCHEDULED_CAPACITY` draws GPUs from a project capacity pool for a scheduled window. Set the window with `--reservation-start-time` and `--reservation-end-time`.
`--nvidia-driver-version [string]`	NVIDIA driver version. List valid versions with `tg clusters list-regions`.
`--cuda-version [string]`	CUDA version. List valid versions with `tg clusters list-regions`.
`--os [string]`	Operating system image for the driver and CUDA pair (for example, `ubuntu-24.04`). Requires `--nvidia-driver-version` and `--cuda-version`. When omitted, the API uses `ubuntu-22.04`.
`--nvidia-version-id [string]`	NVIDIA version catalog ID (the driver version’s `id` in `tg clusters list-regions` output). Selects an exact driver, CUDA, and OS combination in place of the individual version flags.
`--duration-days [integer]`	Number of days to reserve the cluster for. Applies only to `RESERVED` billing.
`--gpu-type [string]`	GPU type to use for the cluster. One of `H100_SXM`, `H200_SXM`, `RTX_6000_PCI`, `L40_PCIE`, `B200_SXM`, `B300_SXM`, `H100_SXM_INF`. Availability varies by region. List each region’s types with `tg clusters list-regions`.
`--cluster-type [KUBERNETES\|SLURM]`	Cluster workload manager or orchestrator.
`--volume [string]`	Storage volume ID to attach to the cluster. List existing volumes with `tg clusters storage list`.
`--num-reserved-gpus [integer]`	Number of prepaid reserved GPUs. With `RESERVED` or `SCHEDULED_CAPACITY` billing, defaults to `--num-gpus` minus `--num-capacity-pool-gpus`. For a hybrid `SCHEDULED_CAPACITY` cluster, set this to the reserved GPUs outside the pool. The sum of this value and `--num-capacity-pool-gpus` can’t exceed `--num-gpus`.
`--reservation-start-time [string]`	Reservation start time for `SCHEDULED_CAPACITY` billing.
`--reservation-end-time [string]`	Reservation end time for `SCHEDULED_CAPACITY` billing.
`--capacity-pool-id [string]`	Capacity pool to draw GPUs from with `SCHEDULED_CAPACITY` billing.
`--num-capacity-pool-gpus [integer]`	Number of GPUs to allocate from the capacity pool. For a hybrid `SCHEDULED_CAPACITY` cluster, set this to the pool portion of `--num-gpus` (a multiple of 8).
`--num-preemptible-gpus [integer]`	Number of preemptible GPUs to request alongside standard capacity. Must be a multiple of 8. Requires a project (pass the global `--project` flag or set `TOGETHER_PROJECT_ID`).
`--auto-scale/--no-auto-scale`	Enable or disable cluster autoscaling.
`--auto-scale-max-gpus [integer]`	Maximum number of GPUs the autoscaler can reach.
`--install-traefik/--no-install-traefik`	Install or skip the Traefik ingress controller.
`--slurm-image [string]`	Custom Slurm image for Slurm clusters.
`--slurm-shm-size-gib [integer]`	Shared memory size in GiB for Slurm clusters.
`--headlamp-addon`	Enable the Headlamp Kubernetes dashboard add-on.
`--slurm-web-addon`	Enable the Slurm Web add-on.

Run tg clusters create with no flags to launch an interactive prompt that walks through the required fields. Pass --non-interactive or --json to skip the prompts. The command also runs non-interactively when no terminal is attached, as in CI.

In non-interactive mode, select the NVIDIA driver in one of two ways:

Pass --nvidia-version-id, without --nvidia-driver-version, --cuda-version, or --os.
Pass --nvidia-driver-version and --cuda-version together, optionally with --os.

Update a cluster

tg clusters update [CLUSTER_ID]

Parameters

Flag	Description
`--num-gpus [integer]`	Number of GPUs to allocate in the cluster.
`--cluster-type [KUBERNETES\|SLURM]`	Accepted but ignored by the API, so the cluster keeps its type.
`--num-reserved-gpus [integer]`	Number of prepaid reserved GPUs on a reserved or scheduled capacity cluster. Must be a multiple of 8 and can’t exceed `num_gpus`. The count can only increase.
`--num-capacity-pool-gpus [integer]`	Desired number of capacity pool GPUs. Must be a multiple of 8 and can’t exceed `num_gpus`.
`--num-preemptible-gpus [integer]`	Desired number of preemptible GPUs for the cluster.
`--reservation-end-time [string]`	New decommission time for a reserved or scheduled capacity cluster, in RFC 3339 format (for example, `2026-12-01T00:00:00Z`). Must be later than the current decommission time. On a scheduled capacity cluster, the request returns a `403` unless Together has enabled reservation extensions for capacity pools.
`--headlamp/--no-headlamp`	Enable or disable the Headlamp Kubernetes dashboard add-on.
`--slurm-web/--no-slurm-web`	Enable or disable the Slurm Web add-on.

Retrieve a cluster

tg clusters retrieve [CLUSTER_ID]

Delete a cluster

tg clusters delete [CLUSTER_ID]

In an interactive session, the command asks you to confirm before deleting the cluster. Pass --force to delete without confirming. The global --non-interactive and --json flags also suppress the prompt, so scripted and CI deletions never wait for input.

Parameters

Flag	Description
`--force`	Delete without asking for confirmation.

List clusters

tg clusters list

The command shows 20 clusters at a time in text output. When a page is full, it also prints the tg clusters list --after <CLUSTER_ID> command that fetches the next page. With --json, it returns every cluster and ignores --after.

List regions

Get configuration information per region for creating a GPU cluster.

tg clusters list-regions

Example output

{
  "regions": [
    {
      "driver_versions": [
        {
          "id": "0195d44f-e3d0-7475-8197-2a7a7349ab65",
          "cuda_version": "12.8",
          "nvidia_driver_version": "570",
          "os": "ubuntu-22.04"
        },
        {
          "id": "019af67e-4ba5-7039-b8d8-de657e906261",
          "cuda_version": "12.9",
          "nvidia_driver_version": "575",
          "os": "ubuntu-22.04"
        }
      ],
      "name": "us-central-8",
      "supported_instance_types": [
        "H100_SXM",
        "H200_SXM"
      ]
    }
  ]
}

Each driver version’s id is its NVIDIA version catalog ID. Pass it to tg clusters create --nvidia-version-id to select that exact driver, CUDA, and OS combination.

SSH into a cluster

SSH into a Slurm cluster with a short-lived certificate. The command opens your browser to sign in with OpenID Connect (OIDC), requests a certificate from the cluster’s certificate authority, and connects through the bastion host. No API key or long-lived SSH key is required.

Requires Python 3.10+.

To get the command with your cluster’s values filled in, select OIDC under SSH access method in the Together AI console, then copy the command. If the copied command starts with tg beta clusters ssh, delete beta. Connect over SSH explains the <LOCATION> and <LOGIN> values.

tg clusters ssh https://dex.<LOCATION>.cloud.together.ai/t-<CLUSTER_ID> --login <LOGIN>

Parameters

Flag	Description
`DEX_URL` (positional)	Cluster Dex issuer URL, formatted as in the example command. required
`--login`, `-l`	POSIX login or SSH username on the cluster. required
`--host`	Target host reachable through the bastion. Default: `slurm-login`. For a compute node, copy the node’s command from the console, which sets `--host` for you.
`--client-id`	Dex public client ID. Default: `together-cli`.
`--scope`	OIDC scopes. Default: `openid email`.
`--key-type`	Ephemeral key type (`ecdsa` or `ed25519`). Default: `ecdsa`.
`--ca-root`	step-ca root certificate (PEM) for TLS verification.
`--cache/--no-cache`	Cache SSH keys and certificates, and reuse them while they’re valid. Default: `--cache`. `--print-ssh-command`, `--ssh-config-alias`, and `--write-ssh-config` require `--cache`.
`--refresh`	Force refresh the cached SSH certificate.
`--cache-dir`	Directory for cached SSH keys and certificates.
`--print-ssh-command`	Print the underlying `ssh` command instead of running it.
`--ssh-config-alias`	Print an `ssh_config` Host entry for this alias instead of running `ssh`.
`--write-ssh-config`	Write or update the alias in `~/.together/ssh/config` and include it from `~/.ssh/config`. Requires `--ssh-config-alias`.

Any other arguments after DEX_URL are passed through to ssh as a remote command.

On the bastion-to-target hop, the CLI disables SSH host key verification (StrictHostKeyChecking=no). Cluster hosts are reprovisioned frequently, so their host keys aren’t pinned. Authentication uses the short-lived step-ca user certificate from the OIDC flow, not the target host key.

Get cluster credentials

Download the cluster’s configuration and credentials to your local .kube/config file to manage Kubernetes resources.

tg clusters get-credentials [CLUSTER_ID]

Parameters

Flag	Description
`--file [Path\|-]`	Path to write the kubeconfig to. Pass `-` to print it to stdout instead. Default: `~/.kube/config`.
`--context-name [string]`	Name of the context to add to the kubeconfig. Default: the cluster name.
`--overwrite-existing`	On a conflict with the existing kubeconfig, overwrite it instead of raising an error.
`--set-default-context`	Change the current context for `kubectl` to the new context.

In --json mode, the command prints the target path as kubeconfig_path and doesn’t write or merge the kubeconfig. The CLI turns on JSON output automatically when it detects an AI agent, so pass --no-json to write the file from an agent session.

Create a node remediation

Create a node repair directly, without waiting for an automated recommendation.

tg clusters remediations create [CLUSTER_ID] [INSTANCE_ID] \
  --mode VM_ONLY \
  --reason "node unhealthy"

To find INSTANCE_ID, run tg clusters retrieve <CLUSTER_ID> --json and copy the node’s instance_id (not its node_id) from gpu_worker_nodes.

Parameters

Flag	Description
`--mode [VM_ONLY\|HOST_AWARE\|EVICT_WITHOUT_REPLACEMENT\|REBOOT_VM\|HOST_POWER_CYCLE]`	Remediation action to perform. required `VM_ONLY` (quick reprovision) deletes the VM and provisions a new one on any available host, without cordoning the current one. After two quick reprovisions for the same fault on the same host within seven days, the next one runs as a migration when at least two spare hosts are available in the same datacenter. `HOST_AWARE` (migrate to new host) cordons the host, deletes the VM, and provisions a new one on a different host. `EVICT_WITHOUT_REPLACEMENT` (remove) evicts the VM without provisioning a replacement and lowers the cluster’s GPU count. `REBOOT_VM` (reboot) reboots the VM in place. `HOST_POWER_CYCLE` (cycle) cordons and power-cycles the bare-metal host while preserving host and node identity. Available only on supported hardware and for accounts where Together has enabled it. Otherwise, the request returns a `403`. `HOST_AWARE` and `EVICT_WITHOUT_REPLACEMENT` return a `403` where host-level repair isn’t enabled for the node’s hardware. `HOST_AWARE` runs as `VM_ONLY` unless the node has a critical health check alert, firing now or on its host in the last 10 days.
`--reason [string]`	Reason for the remediation.

List node remediations

tg clusters remediations list [CLUSTER_ID] [INSTANCE_ID]

Omit INSTANCE_ID to list remediations for every node in the cluster. To match any of several values, repeat --mode, --state, or --trigger.

Parameters

Flag	Description
`--mode [mode]`	Filter by remediation mode. One of `VM_ONLY`, `HOST_AWARE`, `EVICT_WITHOUT_REPLACEMENT`, `REBOOT_VM`, `HOST_POWER_CYCLE`.
`--state [state]`	Filter by remediation state. One of `PENDING_APPROVAL`, `PENDING`, `RUNNING`, `SUCCEEDED`, `FAILED`, `CANCELLED`, `AUTO_RESOLVED`, `QUARANTINING`, `QUARANTINED`.
`--trigger [MANUAL\|AUTOMATED]`	Filter by how the remediation was triggered.
`--after [string]`	Pagination token from a previous response.

Retrieve a node remediation

tg clusters remediations retrieve [REMEDIATION_ID]

Approve a node remediation

Approve a node repair remediation that’s waiting for review. Find these remediations and their IDs with tg clusters remediations list, or in your cluster’s Repairs tab in the console.

tg clusters remediations approve [REMEDIATION_ID]

Parameters

Flag	Description
`--comment [string]`	Comment explaining the approval.
`--mode [VM_ONLY\|HOST_AWARE\|EVICT_WITHOUT_REPLACEMENT\|REBOOT_VM\|HOST_POWER_CYCLE]`	Overrides the remediation’s mode on approval. Takes the same modes as `remediations create`. When omitted, the remediation keeps its existing mode.

Approving a remediation that would run as HOST_POWER_CYCLE returns a 403 when host power cycle isn’t enabled for the node’s hardware. This applies whether you pass the mode with --mode or it’s the remediation’s existing mode.Overriding to --mode HOST_AWARE returns a 412 unless the node has a critical health check alert, firing now or on its host in the last 10 days.

Reject a node remediation

Reject a remediation that’s waiting for review.

tg clusters remediations reject [REMEDIATION_ID] \
  --comment "already handled"

Parameters

Flag	Description
`--comment [string]`	Comment explaining the rejection.

Cancel a node remediation

Cancel a remediation that hasn’t started running.

tg clusters remediations cancel [REMEDIATION_ID]

Create cluster storage

tg clusters storage create

Parameters

Flag	Description
`--region [string]`	Region to create the storage volume in. required
`--size-tib [integer]`	Size of the storage volume in TiB. required
`--volume-name [string]`	Name for the storage volume. required
`--is-lifecycle-independent / --no-is-lifecycle-independent`	Whether the volume persists independently of the clusters it’s attached to. Volumes created with this command are always lifecycle-independent, whichever value you pass, so they persist and keep billing after their cluster is deleted.

When you create a volume through the API or SDK, you can also pass project_id to assign it to a specific project. If you omit it, the volume belongs to your default project.

Update cluster storage

Resize a storage volume.

tg clusters storage update [VOLUME_ID] --size-tib 4

Parameters

Flag	Description
`--size-tib [integer]`	New total size of the storage volume in TiB. Volumes can only grow.

Retrieve cluster storage

tg clusters storage retrieve [VOLUME_ID]

List cluster storage

tg clusters storage list

The command shows 20 volumes at a time in text output. When a page is full, it also prints the tg clusters storage list --after <VOLUME_ID> command that fetches the next page. With --json, it returns every volume and ignores --after.

Delete cluster storage

tg clusters storage delete [VOLUME_ID]

In an interactive session, the command asks you to confirm before deleting the volume. Pass --force to delete without confirming. The global --non-interactive and --json flags also suppress the prompt, so scripted and CI deletions never wait for input.

Parameters

Flag	Description
`--force`	Delete without asking for confirmation.

TOGETHER CLI

COMMANDS

Create a cluster

Parameters

Update a cluster

Parameters

Retrieve a cluster

Delete a cluster

Parameters

List clusters

List regions

Example output

SSH into a cluster

Parameters

Get cluster credentials

Parameters

Create a node remediation

Parameters

List node remediations

Parameters

Retrieve a node remediation

Approve a node remediation

Parameters

Reject a node remediation

Parameters

Cancel a node remediation

Create cluster storage

Parameters

Update cluster storage

Parameters

Retrieve cluster storage

List cluster storage

Delete cluster storage

Parameters

​Create a cluster

​Parameters

​Update a cluster

​Parameters

​Retrieve a cluster

​Delete a cluster

​Parameters

​List clusters

​List regions

​Example output

​SSH into a cluster

​Parameters

​Get cluster credentials

​Parameters

​Create a node remediation

​Parameters

​List node remediations

​Parameters

​Retrieve a node remediation

​Approve a node remediation

​Parameters

​Reject a node remediation

​Parameters

​Cancel a node remediation

​Create cluster storage

​Parameters

​Update cluster storage

​Parameters

​Retrieve cluster storage

​List cluster storage

​Delete cluster storage

​Parameters

Create a cluster

Parameters

Update a cluster

Parameters

Retrieve a cluster

Delete a cluster

Parameters

List clusters

List regions

Example output

SSH into a cluster

Parameters

Get cluster credentials

Parameters

Create a node remediation

Parameters

List node remediations

Parameters

Retrieve a node remediation

Approve a node remediation

Parameters

Reject a node remediation

Parameters

Cancel a node remediation

Create cluster storage

Parameters

Update cluster storage

Parameters

Retrieve cluster storage

List cluster storage

Delete cluster storage

Parameters