Run nanochat on a GPU cluster

nanochat is Andrej Karpathy’s end-to-end ChatGPT clone. It shows how to train and deploy a full conversational AI stack, from tokenizer to web UI. In this tutorial, you’ll train nanochat on an 8xH100 Together GPU cluster, then chat with it in your browser. The whole process covers:

Training a byte-pair encoding (BPE) tokenizer on FineWeb-Edu.
Pretraining a base transformer model.
Midtraining on curated tasks.
Supervised fine-tuning for conversational alignment.
Deploying a FastAPI web server with a chat interface.

Training takes about 4 hours and costs about $130 in GPU time at the on-demand H100 rate.

Requirements

Before you begin, make sure you have:

A Together AI account with access to GPU clusters.
The admin role on the cluster’s project. Only project admins can download the cluster kubeconfig in Step 7.
The Together CLI installed on your local machine.
kubectl installed on your local machine (installation guide).
Basic familiarity with SSH and the command line.

Step 1: Create a cluster

Create an 8xH100 cluster to train nanochat:

Sign in to the Together AI console.
Select GPU Clusters in the navigation sidebar.
Select Create cluster.
Select On-demand capacity.
Select the H100-SXM GPU type card, then set Select GPU Count to 8.
Choose your preferred Region.
Under Cluster Details, enter a Cluster name (for example, nanochat-training) and set Orchestration to Slurm.
Under Storage, open Storage Selection, select + Create new storage, and set Size (TiB) to at least 1.
Select Create Cluster, then select Confirm & Create.

The cluster takes 10 minutes to an hour to provision. Wait for its status to show Ready before you continue. For more on GPU cluster features and options, see the GPU clusters overview.

Step 2: SSH into the cluster

In the console, open your cluster’s details page and select Copy head node SSH command, then run the copied command in your terminal. With key-based SSH, the command connects through a jump host:

ssh -J <LOGIN>@ssh.t-<CLUSTER_ID>.<LOCATION>.cloud.together.ai <LOGIN>@slurm-login

The command opens a shell on your Slurm cluster’s login node, which the console calls the head node.

To keep the SSH connection from dropping during long stretches of training with no terminal input or output, add -o ServerAliveInterval=60 to the command. It pings the SSH server every 60 seconds.

Step 3: Clone nanochat and set up the environment

In the login node shell, clone the nanochat repository and set up your shell environment:

# Clone the repository
git clone https://github.com/karpathy/nanochat.git
cd nanochat

# Pin to the nanochat version this tutorial was written for
git checkout 48abd7d85f3b7a06fe8a457de2353047cff3d951

# Add ~/.local/bin to your PATH
export PATH="$HOME/.local/bin:$PATH"

# Source the Cargo environment. It exists only after speedrun.sh installs Rust.
[ -f "$HOME/.cargo/env" ] && source "$HOME/.cargo/env"

Then install Python 3.10 and its development headers, which nanochat requires:

# Update package manager and install Python dependencies
sudo apt-get update
sudo apt-get install -y python3.10-dev

# Verify Python installation
python3 -c "import sysconfig; print(sysconfig.get_path('include'))"

Step 4: Allocate GPUs

In the login node shell, start a screen session so training keeps running if your SSH connection drops. Inside it, use Slurm’s srun command to allocate 8 GPUs, which opens an interactive bash session on a compute node:

screen -L -Logfile speedrun.log -S speedrun
srun --gres=gpu:8 --pty bash

In the compute node shell, run nvidia-smi to verify GPU access:

nvidia-smi

You should see all 8 H100 GPUs listed with their memory and utilization stats. If your SSH connection drops, SSH back into the login node and run screen -r speedrun to reattach to the session.

Step 5: Configure the cache directory

Point the nanochat cache directory at /scratch, the fast NVMe storage local to each compute node, so that dataset streaming, checkpoints, and intermediate artifacts don’t bottleneck training. Without this change, model FLOPs utilization (MFU) during training is only about 13% instead of about 50%. Set the path in speedrun.sh, not in your shell. The script exports its own value of NANOCHAT_BASE_DIR near the top, which overrides any value you export in your shell. Edit that line to point at /scratch:

export NANOCHAT_BASE_DIR="/scratch/$USER/nanochat/.cache/nanochat"

Because /scratch is ephemeral, copy anything you want to keep long-term to /home, which lives on the persistent shared volume.

Step 6: Run the training pipeline

In the compute node shell inside your screen session, start the pipeline with nanochat’s speedrun.sh script:

bash speedrun.sh

The script runs these stages in order:

Tokenizer training: Trains a GPT-4-style BPE tokenizer on FineWeb-Edu data.
Base model pretraining: Trains the base transformer model with rotary embeddings and the Muon optimizer.
Midtraining: Fine-tunes on a mixture of SmolTalk, MMLU, GSM8K, synthetic identity conversations, and spelling tasks.
Supervised fine-tuning (SFT): Aligns the model for conversational interactions.
Evaluation: Runs CORE benchmarks and generates a report.

The full run takes about 4 hours on 8xH100 GPUs. As it runs, watch these metrics in the training output:

MFU: Expect about 50%. A lower value usually means a data loading bottleneck, so check the cache directory you set in Step 5.
tok/sec: Measures training throughput in tokens processed per second.
Step timing: Each training step should complete in a few seconds.

The scripts log progress and save checkpoints under $NANOCHAT_BASE_DIR, the /scratch path you set in Step 5. The remaining steps serve the trained model in nanochat’s chat UI. You run them from your local machine, reaching the cluster through kubectl. You can start Steps 7 and 8 while training runs, but wait for training to finish before you launch the web server in Step 9.

Step 7: Download the cluster kubeconfig

On your local machine, use the Together CLI to download the cluster’s kubeconfig, which kubectl needs to access the cluster. Find your cluster ID with tg clusters list, then write the kubeconfig to a local file, with the cluster as its default context:

tg clusters get-credentials <CLUSTER_ID> --file ~/.kube/nanochat-cluster-config --set-default-context --no-json

Step 8: Access the compute pod with kubectl

On your local machine, point kubectl at the kubeconfig from Step 7 and list the pods in the slurm namespace:

# Set the KUBECONFIG environment variable
export KUBECONFIG=~/.kube/nanochat-cluster-config

# List pods in the slurm namespace
kubectl -n slurm get pods

You should see your Slurm compute pods listed. Find the compute pod where training ran, then open a shell in it:

# Example output:
# NAME                              READY   STATUS    RESTARTS   AGE
# slurm-compute-production-abc123   1/1     Running   0          2h

# Exec into the pod
kubectl -n slurm exec -it <POD_NAME> -- /bin/bash

The pod shell runs as root, so it doesn’t have your login’s environment. Go to the nanochat directory you cloned in Step 3, point NANOCHAT_BASE_DIR at the /scratch path from Step 5, and activate the virtual environment that speedrun.sh created. Replace <LOGIN> with the login name from your SSH command in Step 2:

cd /home/<LOGIN>/nanochat
export NANOCHAT_BASE_DIR="/scratch/<LOGIN>/nanochat/.cache/nanochat"
source .venv/bin/activate

Step 9: Launch the web server

After training finishes, launch the FastAPI web server in the pod shell where you activated the virtual environment:

python -m scripts.chat_web

The server starts on port 8000 inside the pod. You should see output indicating the server is running.

Step 10: Forward the web server port

Leave the server running. In a new terminal window on your local machine, set up port forwarding to the web server:

# Set the KUBECONFIG (if not already set in this terminal)
export KUBECONFIG=~/.kube/nanochat-cluster-config

# Forward port 8000 from the pod to local port 6818
kubectl -n slurm port-forward <POD_NAME> 6818:8000

Port forwarding stays active as long as this terminal session is open.

Step 11: Chat with nanochat

Open this URL in your browser:

http://localhost:6818/

You should see the nanochat web interface, where you can chat with your trained model.

Congrats! You trained and deployed your own ChatGPT-style model on a Together GPU cluster.

Clean up to stop billing

On-demand clusters bill hourly until you delete them. When you’re done experimenting, delete the cluster:

tg clusters delete <CLUSTER_ID>

Your shared volume, including anything you saved to /home, persists after cluster deletion and continues to bill until you delete it too.

Costs and performance

Typical results for a full speedrun.sh run on an 8xH100 cluster:

Training time: About 4 hours.
MFU: About 50%.
Cost: About $130 at the on-demand H100 rate (8 GPUs for about 4 hours), plus storage.

For detailed metrics, read the report.md file that the pipeline writes to the nanochat directory when training completes.

Troubleshooting

nvidia-smi doesn’t show GPUs after srun: Rerun srun and request the GPUs on a single node explicitly:

srun --gres=gpu:8 --nodes=1 --pty bash

Out-of-memory (OOM) errors during training: The default batch sizes should work on H100 80GB GPUs. If you still see OOM errors:

Check that NANOCHAT_BASE_DIR is set to /scratch.
Make sure no other processes are using GPU memory.

Can’t connect to the web UI: Work through these checks:

Verify that the pod name in your port-forward command exactly matches one listed by kubectl -n slurm get pods.
Check the logs in the pod terminal to confirm the web server is running.
If port 6818 is already in use on your machine, forward a different local port and open that port in your browser.

Next steps

To keep going with nanochat:

Modify the SFT data or run additional reinforcement learning training for specific behaviors.
Extend chat_web.py with authentication and persistence layers.
Try the run1000.sh script for a larger model.

For more on nanochat’s architecture and training process, see the nanochat GitHub repository.

Slurm

Submit and manage batch jobs with the preconfigured Slurm scheduler.

Manage clusters

Manage access, scale capacity, and monitor cluster health.

GET STARTED

SERVERLESS

INFERENCE APIS

DEDICATED MODEL INFERENCE

DEDICATED CONTAINER INFERENCE

POST-TRAINING

FINE-TUNING

GPU CLUSTERS

CODE EXECUTION

ADMINISTRATION

Run nanochat on a GPU cluster

Requirements

Step 1: Create a cluster

Step 2: SSH into the cluster

Step 3: Clone nanochat and set up the environment

Step 4: Allocate GPUs

Step 5: Configure the cache directory

Step 6: Run the training pipeline

Step 7: Download the cluster kubeconfig

Step 8: Access the compute pod with kubectl

Step 9: Launch the web server

Step 10: Forward the web server port

Step 11: Chat with nanochat

Clean up to stop billing

Costs and performance

Troubleshooting

Next steps

Slurm

Manage clusters