> ## Documentation Index
> Fetch the complete documentation index at: https://docs.together.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Connect over SSH

> Reach your cluster's login, worker, and compute nodes with OIDC or key-based SSH.

You can use SSH to connect to individual cluster nodes—for example, the Slurm login node for job submission, and worker or compute nodes for debugging and monitoring. You connect with key-based SSH or, on clusters that support OpenID Connect (OIDC) SSH, through the Together CLI, which needs no SSH key.

## Choose an SSH access method on Slurm clusters

On Slurm clusters that support OIDC SSH, choose how you connect in the **SSH access method** selector. It's in the sidebar of the cluster details page in the [Together AI console](https://api.together.ai/clusters), or at the top of the page on mobile. Your choice applies to the Slurm head node and all compute nodes.

If the cluster doesn't support OIDC SSH, the selector is hidden and only key-based SSH is available.

| Method | What it does | Requirements |
| - | - | - |
| **OIDC** | Copies a Together CLI command that opens your browser to sign in, then connects with a short-lived certificate. | [Together CLI](https://docs.together.ai/reference/cli/getting-started.md) installed, login name chosen in the [console](https://api.together.ai/clusters). |
| **Key-based** | Copies a standard `ssh -J` command for your SSH client. | Uploaded SSH key at [api.together.ai/settings/ssh-key](https://api.together.ai/settings/ssh-key). |

OIDC SSH requires Together CLI 2.40+ and [Python 3.10+](https://www.python.org/). Install the CLI with uv:

```bash theme={null}
uv tool install "together[cli]"
```

The console shows the same command when you select **OIDC** and expand **Set up CLI** next to the head node command.

If you installed the CLI with `pip`, follow the [migration steps](https://docs.together.ai/reference/cli/getting-started.md#install-the-together-cli) to move to the `uv`-managed install before you connect.

## SSH to worker and compute nodes

To SSH into any GPU worker node on a Kubernetes cluster, or any compute node on a Slurm cluster, copy the node's command from the console:

1. Open your cluster on the [GPU clusters page](https://api.together.ai/clusters).
2. On Slurm clusters that support OIDC SSH, select **OIDC** or **Key-based** in **SSH access method**.
3. Select the **Nodes** tab.
4. Open the **⋮** menu at the end of the node's row and select **Copy SSH command**.
5. Run the copied command in your terminal.

The copied command depends on your access method. On a Kubernetes cluster that supports OIDC SSH, it always uses OIDC. A key-based command connects through the cluster's jump host:

```bash theme={null}
ssh -J <LOGIN>@ssh.t-<CLUSTER_ID>.<LOCATION>.cloud.together.ai <LOGIN>@<HOST>
```

An OIDC command connects through the Together CLI:

```bash theme={null}
tg clusters ssh https://dex.<LOCATION>.cloud.together.ai/t-<CLUSTER_ID> --login <LOGIN> --host <HOST>
```

**Copy SSH command** fills in every value for you. `<LOGIN>` is the **Login name** in your [SSH key settings](https://api.together.ai/settings/ssh-key), and you can set it only once. `<LOCATION>` is the datacenter part of the cluster's node hostnames, such as `s2.ca-west-1b`, not the region name.

Use direct node access to:

* Check GPU utilization across all GPUs on the node with `nvidia-smi`.
* Monitor node-level performance metrics (CPU, memory, disk, network).
* Inspect system logs (`journalctl`, `/var/log`).
* Debug node-level networking or storage issues.
* Check kubelet status and logs on Kubernetes worker nodes.
* View all processes running on the node.

What you can do on the node depends on the cluster type:

| Capability | Kubernetes worker node | Slurm compute node |
| - | - | - |
| Run GPU workloads | Not available (no direct GPU device access). | Available. |
| Access PersistentVolumes | Not available (mounted in pods only). | Available through the `/home` directory. |
| Best for | Node-level monitoring and debugging. | Node-level monitoring, debugging, and direct Slurm workflows. |

To run GPU workloads or use PersistentVolumes on a Kubernetes cluster, [exec into a pod with GPU and storage access](https://docs.together.ai/docs/kubernetes-workloads.md#run-gpu-workloads-in-containers).

## SSH to Slurm login nodes

To submit jobs on a [Slurm cluster](https://docs.together.ai/docs/slurm.md), SSH into its login node, which the console calls the head node. Select **Copy head node SSH command** in the sidebar of the cluster details page, then run the copied command in your terminal.

> [!NOTE]
>
> The login node runs as a pod on the cluster's GPU nodes. While the cluster has no GPU nodes, the console disables SSH access and shows "SSH is disabled when 0 GPUs are available." SSH access returns when GPU nodes rejoin the cluster.

With key-based SSH, the command connects to `slurm-login` through the jump host:

```bash theme={null}
ssh -J <LOGIN>@ssh.t-<CLUSTER_ID>.<LOCATION>.cloud.together.ai <LOGIN>@slurm-login
```

With OIDC, it runs the Together CLI's [`tg clusters ssh`](https://docs.together.ai/reference/cli/clusters#ssh-into-a-cluster) command:

```bash theme={null}
tg clusters ssh https://dex.<LOCATION>.cloud.together.ai/t-<CLUSTER_ID> --login <LOGIN>
```

Slurm nodes use these hostnames:

* **Login node:** The host is always `slurm-login`.
* **Compute nodes:** The host is `<NODE_NAME>.slurm-compute.slurm`, for example `gpu-dp-hmqnh-nwlnj.slurm-compute.slurm`. On Slinky v1.0 clusters, it's `<WORKER_HOSTNAME>.slurm-workers-slurm.slurm`. The node's **Copy SSH command** always uses the exact host.

Once you're connected, submit and monitor jobs with standard Slurm commands. For the command set, memory-limit guidance, and job arrays, see [Slurm](https://docs.together.ai/docs/slurm.md).

## Connect from VS Code

To use VS Code with your Slurm cluster, define a `slurm-cluster` SSH host for its login node.

With key-based SSH, add the cluster's jump host and login node to your `~/.ssh/config`. Copy the jump host from the console's key-based command into the `together-jump` entry:

```ssh-config theme={null}
# Keep connections alive
Host *
  ServerAliveInterval 60

# Together AI bastion host for your cluster
Host together-jump
  HostName ssh.t-<CLUSTER_ID>.<LOCATION>.cloud.together.ai
  User <LOGIN>

# Your Slurm login node
Host slurm-cluster
  HostName slurm-login
  ProxyJump together-jump
  User <LOGIN>
```

With OIDC, have the CLI write the entry instead. This command signs you in, adds `slurm-cluster` to `~/.together/ssh/config`, and includes that file from `~/.ssh/config`:

```bash theme={null}
tg clusters ssh https://dex.<LOCATION>.cloud.together.ai/t-<CLUSTER_ID> --login <LOGIN> \
  --ssh-config-alias slurm-cluster --write-ssh-config
```

Run the command again when its short-lived certificate expires.

Then, in VS Code's Remote SSH extension, connect to `slurm-cluster`.

## Troubleshooting

* **Key-based SSH fails with `Permission denied (publickey)`:** Check that you added your SSH key at [api.together.ai/settings/ssh-key](https://api.together.ai/settings/ssh-key) before you created the cluster, then copy the command from the console again.
* **OIDC command not found:** The `tg clusters ssh` command requires Together CLI 2.40+. Check your version with `tg --version`. To install or upgrade the CLI, run `uv tool install "together[cli]" --upgrade`.
* **A copied OIDC command fails with `Unknown command "clusters"`:** The console copies `tg beta clusters ssh`, but Together CLI 3.0 removed the `beta` path. Delete `beta` from the copied command and run it again.

## Next steps

> **[Slurm](https://docs.together.ai/docs/slurm.md)**
>
> Submit and manage batch jobs with the preconfigured Slurm scheduler.

> **[Manage clusters](https://docs.together.ai/docs/gpu-clusters-management.md)**
>
> Manage access, scale capacity, and monitor cluster health.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.