Reinforcement learning with GRPO quickstart

Group relative policy optimization (GRPO) is a reinforcement learning (RL) method that trains a model from a reward signal instead of example responses. The model generates several answers to each prompt, a reward function scores them, and the update favors the answers that beat their group’s average. Use it when you can check whether an output is right but can’t write the ideal output yourself, as with math, code, or tool calls. In this tutorial, you train a model on the GSM8K grade-school math dataset with a ready-made recipe from the post-training cookbook. An exact-match reward checks each final answer against the reference. By the end, you have a trained model registered in your project and know what each step of the training loop does. Training runs for 200 steps and takes about 4 hours. The model resource costs about $56 per hour, about $200 for the run, and bills per GPU-minute from the moment it’s ready until the recipe stops it. Make sure your credit balance covers the run.

Requirements

Before you begin, make sure you have:

Python 3.11 or later and uv.
A Together AI account with a credit balance of at least $100, the minimum for creating a model resource. See Pricing.
Your API key exported as TOGETHER_API_KEY. Get one from API keys in the Together AI console.
A Weights & Biases API key exported as WANDB_API_KEY. Most cookbook recipes log to Weights & Biases, including the one in this quickstart. To run without an account, export WANDB_MODE=offline instead.
Optionally, a Hugging Face token exported as HF_TOKEN for gated models or datasets.

export TOGETHER_API_KEY=<your_key>
export WANDB_API_KEY=<your_wandb_key>     # or: export WANDB_MODE=offline
export HF_TOKEN=<your_huggingface_token>  # gated models and datasets only

Step 1: Clone the cookbook

Clone the cookbook and install its dependencies, including the Together Python SDK:

git clone https://github.com/togethercomputer/post-training-cookbook.git
cd post-training-cookbook
uv sync --locked

Step 2: Review the recipe

Open recipes/grpo/gsm8k.py:

recipes/grpo/gsm8k.py

from post_training_cookbook.grpo.config import GRPOConfig
from post_training_cookbook.grpo.train import train_single_turn

config = GRPOConfig(
    model="Qwen/Qwen3.6-35B-A3B",
    dataset="gsm8k",
    enable_thinking=False,
    num_examples=4000,
    num_val_examples=200,
    batch_size=16,
    group_size=8,
    max_sampling_attempts=3,
    num_steps=200,
    learning_rate=1e-5,
    max_model_length=1536,
    max_prompt_length=768,
    max_sample_tokens=768,
    training_checkpoint_interval=20,
    inference_checkpoint_interval=20,
    validation_interval=100,
    wandb_project="grpo-gsm8k",
)

if __name__ == "__main__":
    train_single_turn(config)

GSM8K reference answers end with #### <number>, and the prompt asks the model to end its solution the same way. The dataset’s reward function reads the number after #### in each completion, falling back to a \boxed{} answer, and returns 1.0 if it matches the reference and 0.0 otherwise. A completion with neither marker scores 0.0, even when it states the right number. The other key settings in the recipe:

Setting	What it does
`model="Qwen/Qwen3.6-35B-A3B"`	A mixture-of-experts model with 3B active parameters, so it samples quickly for its size.
`batch_size=16`, `group_size=8`	16 prompts and 8 completions per prompt, so 128 completions per step. Larger groups give steadier advantage estimates.
`max_sampling_attempts=3`	Turns on dynamic sampling, which the default (`1`) leaves off. If fewer than 16 groups have varied rewards, the step draws another batch, up to three batches in total. If no batch has any group with varied rewards, the recipe skips the step and doesn’t update the model.
`max_model_length=1536`, `max_prompt_length=768`, `max_sample_tokens=768`	GSM8K solutions are short. Prompts over 768 tokens are dropped, and completions are capped at 768 tokens. The overlong penalty lowers the reward of completions longer than 672 tokens.
`enable_thinking=False`	Turns off the model’s `<think>` output so completions stay inside the budget.
`training_checkpoint_interval=20`, `inference_checkpoint_interval=20`	Saves both checkpoint types every 20 steps.
`validation_interval=100`	Runs greedy validation every 100 steps and logs `val/accuracy`.

Under the hood, each step of train_single_turn samples group_size completions per prompt, scores them, and computes each completion’s advantage relative to its group. It then runs forward_backward with the GRPO loss, optim_step, and weights_sync, so the next step samples from the updated model. How post-training runs GRPO explains each piece.

Step 3: Run training

uv run python recipes/grpo/gsm8k.py

The recipe provisions a model resource with one generator replica, waits for it to become ready, and then prints a few lines per step. You should see output like this:

Dataset: gsm8k (openai/gsm8k)
Dataset size: 4000 train, 200 val
Training for 200 steps (1 epochs, 16 prompts x 8 samples per step)
[model-resources:9e4a1c7b-5d2f-4b8e-a3c6-7f0d2e9b1a54] created, waiting for READY status...
...
[model-resources:9e4a1c7b-5d2f-4b8e-a3c6-7f0d2e9b1a54] ready in 284.3s
[session:3f8b2c7d-91a4-4d2e-b6f0-5c1e9a7d4b23] created, waiting for RUNNING status...
...
[session:3f8b2c7d-91a4-4d2e-b6f0-5c1e9a7d4b23] ready in 63.8s
Session ready: 3f8b2c7d-91a4-4d2e-b6f0-5c1e9a7d4b23
Run artifacts: runs/Qwen3.6-35B-A3B_grpo-gsm8k_1.00e-05_16x8_20261001T153000Z
[3f8b2c7d] [step 0] VALIDATION accuracy=0.120 mean_len=312 elapsed=45.2s
[3f8b2c7d] [step 0] accuracy=0.188 reward=0.188 penalized=0.00% zero_var_skip=11.11% clipped=0.00% attempts=2
[3f8b2c7d] [step 1] loss=0.031200 lr=1.00e-06 elapsed=107.6s
[3f8b2c7d] [step 1] timing: collect=48.1s fb=9.6s optim=4.6s total=62.3s
[3f8b2c7d] [step 1] accuracy=0.203 reward=0.203 penalized=0.00% zero_var_skip=5.88% clipped=0.00% attempts=2
...

The bracketed prefix on each step line, [3f8b2c7d], is the first eight characters of the session ID.

Step 4: Monitor the run

The step log and the grpo-gsm8k project in Weights & Biases report the same metrics. On a first run, watch these:

Step log	Weights & Biases	What to look for
`VALIDATION accuracy`	`val/accuracy`	The fraction of validation problems answered correctly with greedy sampling. This is the main progress signal, and it should rise and then plateau.
`accuracy`	`accuracy`	Training accuracy, counted only over groups that trained, so it reads differently from validation accuracy. It should trend upward.
`zero_var_skip`	`zero_variance_skip_ratio`	The fraction of groups skipped because every completion scored the same, leaving no learning signal. High values mean the task is too easy or too hard for the model right now.

The loss line is a poor progress signal for GRPO, so track accuracy instead. Monitoring covers the other metrics and how to act on them.

Step 5: Retrieve the trained model

When training finishes, the recipe saves a final inference checkpoint and prints its ID:

[3f8b2c7d] Inference checkpoint created: 9c4e1a7b-2f3d-4e8a-b6c1-5d7f0e2a9b34

To download the weights, retrieve the session and pass the ID of its final inference checkpoint to download_checkpoint. The session ID is in the run’s manifest.json and on the Session ready line of the training output. The merged variant is the full model, about 72 GB for Qwen3.6 35B A3B, so check your free disk space first:

from together import Together
from together.post_training import download_checkpoint

client = Together()
session = client.post_training.sessions.retrieve("<session_id>")
checkpoint = session.inference_checkpoints[-1]

paths = download_checkpoint(
    client,
    checkpoint.id,
    variant="CHECKPOINT_VARIANT_MERGED",
    output_dir="./gsm8k_model",
)
print(f"Downloaded {len(paths)} files")

To serve the model on Together AI instead, see Deploy a checkpoint.

You trained a model with GRPO, watched it learn, and retrieved the result.

Confirm the resource stopped

The recipe stops the session and the model resource when it exits, so billing has already ended. If the run was interrupted, the resource may still be billing, or it may not be ready yet and start billing once it is. This script finds your resources that are waiting for capacity (PENDING), starting up (CREATING), or ready (READY), and stops each one:

from together import Together

client = Together()
for status in (
    "MODEL_RESOURCES_STATUS_PENDING",
    "MODEL_RESOURCES_STATUS_CREATING",
    "MODEL_RESOURCES_STATUS_READY",
):
    resources = client.post_training.model_resources.list(
        status=[status], created_by="me"
    ).data
    for resource in resources:
        print(resource.id, resource.status, resource.base_model)
        client.post_training.model_resources.stop(resource.id, force=True)

created_by="me" limits the list to resources you created, so the script leaves other people’s resources in a shared project alone. force=True also stops any session that’s still attached. Without it, the API refuses to stop a resource while a session is active. If you have other runs in progress, the script stops their resources too. To stop only this run’s resource, pass the ID from the [model-resources:<id>] created line in the run’s output to client.post_training.model_resources.stop("your_resource_id", force=True). You can also stop resources from the Model resources tab under Post-training in the console. Manage model resources covers stopping, idle timers, and what a resource costs while it runs.

Next steps