Skip to main content
You can use SSH to connect to individual cluster nodes—for example, the Slurm login node for job submission, and worker or compute nodes for debugging and monitoring. You connect with key-based SSH or, on clusters that support OpenID Connect (OIDC) SSH, through the Together CLI, which needs no SSH key.

​
Choose an SSH access method on Slurm clusters

On Slurm clusters that support OIDC SSH, choose how you connect in the SSH access method selector. It’s in the sidebar of the cluster details page in the Together AI console, or at the top of the page on mobile. Your choice applies to the Slurm head node and all compute nodes. If the cluster doesn’t support OIDC SSH, the selector is hidden and only key-based SSH is available. OIDC SSH requires Together CLI 2.40+ and Python 3.10+. Install the CLI with uv:
The console shows the same command when you select OIDC and expand Set up CLI next to the head node command. If you installed the CLI with pip, follow the migration steps to move to the uv-managed install before you connect.

​
SSH to worker and compute nodes

To SSH into any GPU worker node on a Kubernetes cluster, or any compute node on a Slurm cluster, copy the node’s command from the console:
  1. Open your cluster on the GPU clusters page.
  2. On Slurm clusters that support OIDC SSH, select OIDC or Key-based in SSH access method.
  3. Select the Nodes tab.
  4. Open the ⋮ menu at the end of the node’s row and select Copy SSH command.
  5. Run the copied command in your terminal.
The copied command depends on your access method. On a Kubernetes cluster that supports OIDC SSH, it always uses OIDC. A key-based command connects through the cluster’s jump host:
An OIDC command connects through the Together CLI:
Copy SSH command fills in every value for you. <LOGIN> is the Login name in your SSH key settings, and you can set it only once. <LOCATION> is the datacenter part of the cluster’s node hostnames, such as s2.ca-west-1b, not the region name. Use direct node access to:
  • Check GPU utilization across all GPUs on the node with nvidia-smi.
  • Monitor node-level performance metrics (CPU, memory, disk, network).
  • Inspect system logs (journalctl, /var/log).
  • Debug node-level networking or storage issues.
  • Check kubelet status and logs on Kubernetes worker nodes.
  • View all processes running on the node.
What you can do on the node depends on the cluster type: To run GPU workloads or use PersistentVolumes on a Kubernetes cluster, exec into a pod with GPU and storage access.

​
SSH to Slurm login nodes

To submit jobs on a Slurm cluster, SSH into its login node, which the console calls the head node. Select Copy head node SSH command in the sidebar of the cluster details page, then run the copied command in your terminal.
The login node runs as a pod on the cluster’s GPU nodes. While the cluster has no GPU nodes, the console disables SSH access and shows “SSH is disabled when 0 GPUs are available.” SSH access returns when GPU nodes rejoin the cluster.
With key-based SSH, the command connects to slurm-login through the jump host:
With OIDC, it runs the Together CLI’s tg clusters ssh command:
Slurm nodes use these hostnames:
  • Login node: The host is always slurm-login.
  • Compute nodes: The host is <NODE_NAME>.slurm-compute.slurm, for example gpu-dp-hmqnh-nwlnj.slurm-compute.slurm. On Slinky v1.0 clusters, it’s <WORKER_HOSTNAME>.slurm-workers-slurm.slurm. The node’s Copy SSH command always uses the exact host.
Once you’re connected, submit and monitor jobs with standard Slurm commands. For the command set, memory-limit guidance, and job arrays, see Slurm.

​
Connect from VS Code

To use VS Code with your Slurm cluster, define a slurm-cluster SSH host for its login node. With key-based SSH, add the cluster’s jump host and login node to your ~/.ssh/config. Copy the jump host from the console’s key-based command into the together-jump entry:
With OIDC, have the CLI write the entry instead. This command signs you in, adds slurm-cluster to ~/.together/ssh/config, and includes that file from ~/.ssh/config:
Run the command again when its short-lived certificate expires. Then, in VS Code’s Remote SSH extension, connect to slurm-cluster.

​
Troubleshooting

  • Key-based SSH fails with Permission denied (publickey): Check that you added your SSH key at api.together.ai/settings/ssh-key before you created the cluster, then copy the command from the console again.
  • OIDC command not found: The tg clusters ssh command requires Together CLI 2.40+. Check your version with tg --version. To install or upgrade the CLI, run uv tool install "together[cli]" --upgrade.
  • A copied OIDC command fails with Unknown command "clusters": The console copies tg beta clusters ssh, but Together CLI 3.0 removed the beta path. Delete beta from the copied command and run it again.

​
Next steps

Slurm

Submit and manage batch jobs with the preconfigured Slurm scheduler.

Manage clusters

Manage access, scale capacity, and monitor cluster health.