> ## Documentation Index > Fetch the complete documentation index at: https://docs.together.ai/llms.txt > Use this file to discover all available pages before exploring further. # Reinforcement learning with GRPO quickstart > Train a model to solve grade-school math with GRPO, monitor the run, and retrieve the trained model. [Group relative policy optimization (GRPO)](/docs/post-training/grpo) is a reinforcement learning (RL) method that trains a model from a reward signal instead of example responses. The model generates several answers to each prompt, a reward function scores them, and the update favors the answers that beat their group's average. Use it when you can check whether an output is right but can't write the ideal output yourself, as with math, code, or tool calls. In this tutorial, you train a model on the [GSM8K](https://huggingface.co/datasets/openai/gsm8k) grade-school math dataset with a ready-made recipe from the [post-training cookbook](https://github.com/togethercomputer/post-training-cookbook). An exact-match reward checks each final answer against the reference. By the end, you have a trained model registered in your project and know what each step of the training loop does. Training runs for 200 steps and takes about 4 hours. The model resource costs about \$56 per hour, about \$200 for the run, and bills [per GPU-minute](/docs/post-training/pricing) from the moment it's ready until the recipe stops it. Make sure your credit balance covers the run. ## Requirements Before you begin, make sure you have: * Python 3.11 or later and [uv](https://docs.astral.sh/uv/getting-started/installation/). * A [Together AI account](https://api.together.ai/) with a credit balance of at least \$100, the minimum for creating a model resource. See [Pricing](/docs/post-training/pricing#minimum-credit-balance). * Your API key exported as `TOGETHER_API_KEY`. Get one from [API keys](https://api.together.ai/settings/api-keys) in the Together AI console. * A Weights & Biases API key exported as `WANDB_API_KEY`. Most cookbook recipes log to Weights & Biases, including the one in this quickstart. To run without an account, export `WANDB_MODE=offline` instead. * Optionally, a Hugging Face token exported as `HF_TOKEN` for gated models or datasets. ```bash theme={null} export TOGETHER_API_KEY= export WANDB_API_KEY= # or: export WANDB_MODE=offline export HF_TOKEN= # gated models and datasets only ``` ## Step 1: Clone the cookbook Clone the cookbook and install its dependencies, including the Together Python SDK: ```bash theme={null} git clone https://github.com/togethercomputer/post-training-cookbook.git cd post-training-cookbook uv sync --locked ``` ## Step 2: Review the recipe Open `recipes/grpo/gsm8k.py`: ```python recipes/grpo/gsm8k.py theme={null} from post_training_cookbook.grpo.config import GRPOConfig from post_training_cookbook.grpo.train import train_single_turn config = GRPOConfig( model="Qwen/Qwen3.6-35B-A3B", dataset="gsm8k", enable_thinking=False, num_examples=4000, num_val_examples=200, batch_size=16, group_size=8, max_sampling_attempts=3, num_steps=200, learning_rate=1e-5, max_model_length=1536, max_prompt_length=768, max_sample_tokens=768, training_checkpoint_interval=20, inference_checkpoint_interval=20, validation_interval=100, wandb_project="grpo-gsm8k", ) if __name__ == "__main__": train_single_turn(config) ``` GSM8K reference answers end with `#### `, and the prompt asks the model to end its solution the same way. The dataset's reward function reads the number after `####` in each completion, falling back to a `\boxed{}` answer, and returns `1.0` if it matches the reference and `0.0` otherwise. A completion with neither marker scores `0.0`, even when it states the right number. The other key settings in the recipe: | Setting | What it does | | - | - | | `model="Qwen/Qwen3.6-35B-A3B"` | A mixture-of-experts model with 3B active parameters, so it samples quickly for its size. | | `batch_size=16`, `group_size=8` | 16 prompts and 8 completions per prompt, so 128 completions per step. Larger groups give steadier advantage estimates. | | `max_sampling_attempts=3` | Turns on dynamic sampling, which the default (`1`) leaves off. If fewer than 16 groups have varied rewards, the step draws another batch, up to three batches in total. If no batch has any group with varied rewards, the recipe skips the step and doesn't update the model. | | `max_model_length=1536`, `max_prompt_length=768`, `max_sample_tokens=768` | GSM8K solutions are short. Prompts over 768 tokens are dropped, and completions are capped at 768 tokens. The overlong penalty lowers the reward of completions longer than 672 tokens. | | `enable_thinking=False` | Turns off the model's `` output so completions stay inside the budget. | | `training_checkpoint_interval=20`, `inference_checkpoint_interval=20` | Saves both checkpoint types every 20 steps. | | `validation_interval=100` | Runs greedy validation every 100 steps and logs `val/accuracy`. | Under the hood, each step of `train_single_turn` samples `group_size` completions per prompt, scores them, and computes each completion's advantage relative to its group. It then runs `forward_backward` with the GRPO loss, `optim_step`, and `weights_sync`, so the next step samples from the updated model. [How post-training runs GRPO](/docs/post-training/grpo#how-post-training-runs-grpo) explains each piece. ## Step 3: Run training ```bash theme={null} uv run python recipes/grpo/gsm8k.py ``` The recipe provisions a model resource with one generator replica, waits for it to become ready, and then prints a few lines per step. You should see output like this: ```text theme={null} Dataset: gsm8k (openai/gsm8k) Dataset size: 4000 train, 200 val Training for 200 steps (1 epochs, 16 prompts x 8 samples per step) [model-resources:9e4a1c7b-5d2f-4b8e-a3c6-7f0d2e9b1a54] created, waiting for READY status... ... [model-resources:9e4a1c7b-5d2f-4b8e-a3c6-7f0d2e9b1a54] ready in 284.3s [session:3f8b2c7d-91a4-4d2e-b6f0-5c1e9a7d4b23] created, waiting for RUNNING status... ... [session:3f8b2c7d-91a4-4d2e-b6f0-5c1e9a7d4b23] ready in 63.8s Session ready: 3f8b2c7d-91a4-4d2e-b6f0-5c1e9a7d4b23 Run artifacts: runs/Qwen3.6-35B-A3B_grpo-gsm8k_1.00e-05_16x8_20261001T153000Z [3f8b2c7d] [step 0] VALIDATION accuracy=0.120 mean_len=312 elapsed=45.2s [3f8b2c7d] [step 0] accuracy=0.188 reward=0.188 penalized=0.00% zero_var_skip=11.11% clipped=0.00% attempts=2 [3f8b2c7d] [step 1] loss=0.031200 lr=1.00e-06 elapsed=107.6s [3f8b2c7d] [step 1] timing: collect=48.1s fb=9.6s optim=4.6s total=62.3s [3f8b2c7d] [step 1] accuracy=0.203 reward=0.203 penalized=0.00% zero_var_skip=5.88% clipped=0.00% attempts=2 ... ``` The bracketed prefix on each step line, `[3f8b2c7d]`, is the first eight characters of the session ID. ## Step 4: Monitor the run The step log and the `grpo-gsm8k` project in Weights & Biases report the same metrics. On a first run, watch these: | Step log | Weights & Biases | What to look for | | - | - | - | | `VALIDATION accuracy` | `val/accuracy` | The fraction of validation problems answered correctly with greedy sampling. This is the main progress signal, and it should rise and then plateau. | | `accuracy` | `accuracy` | Training accuracy, counted only over groups that trained, so it reads differently from validation accuracy. It should trend upward. | | `zero_var_skip` | `zero_variance_skip_ratio` | The fraction of groups skipped because every completion scored the same, leaving no learning signal. High values mean the task is too easy or too hard for the model right now. | The `loss` line is a poor progress signal for GRPO, so track accuracy instead. [Monitoring](/docs/post-training/monitoring) covers the other metrics and how to act on them. ## Step 5: Retrieve the trained model When training finishes, the recipe saves a final inference checkpoint and prints its ID: ```text theme={null} [3f8b2c7d] Inference checkpoint created: 9c4e1a7b-2f3d-4e8a-b6c1-5d7f0e2a9b34 ``` To download the weights, retrieve the session and pass the ID of its final inference checkpoint to `download_checkpoint`. The session ID is in the run's `manifest.json` and on the `Session ready` line of the training output. The merged variant is the full model, about 72 GB for Qwen3.6 35B A3B, so check your free disk space first: ```python theme={null} from together import Together from together.post_training import download_checkpoint client = Together() session = client.post_training.sessions.retrieve("") checkpoint = session.inference_checkpoints[-1] paths = download_checkpoint( client, checkpoint.id, variant="CHECKPOINT_VARIANT_MERGED", output_dir="./gsm8k_model", ) print(f"Downloaded {len(paths)} files") ``` To serve the model on Together AI instead, see [Deploy a checkpoint](/docs/post-training/deployment). You trained a model with GRPO, watched it learn, and retrieved the result. ## Confirm the resource stopped The recipe stops the session and the model resource when it exits, so billing has already ended. If the run was interrupted, the resource may still be billing, or it may not be ready yet and start billing once it is. This script finds your resources that are waiting for capacity (`PENDING`), starting up (`CREATING`), or ready (`READY`), and stops each one: ```python theme={null} from together import Together client = Together() for status in ( "MODEL_RESOURCES_STATUS_PENDING", "MODEL_RESOURCES_STATUS_CREATING", "MODEL_RESOURCES_STATUS_READY", ): resources = client.post_training.model_resources.list( status=[status], created_by="me" ).data for resource in resources: print(resource.id, resource.status, resource.base_model) client.post_training.model_resources.stop(resource.id, force=True) ``` `created_by="me"` limits the list to resources you created, so the script leaves other people's resources in a shared project alone. `force=True` also stops any session that's still attached. Without it, the API refuses to stop a resource while a session is active. If you have other runs in progress, the script stops their resources too. To stop only this run's resource, pass the ID from the `[model-resources:] created` line in the run's output to `client.post_training.model_resources.stop("your_resource_id", force=True)`. You can also stop resources from the **Model resources** tab under [Post-training](https://api.together.ai/custom-training/resources) in the console. [Manage model resources](/docs/post-training/model-resources) covers stopping, idle timers, and what a resource costs while it runs. ## Next steps Group size, clipping, dynamic sampling, overlong handling, and async training. Bring your own dataset and reward to the GSM8K loop. How work splits between your code and Together AI GPUs, and the model resources, sessions, and checkpoints behind each run. Deploy the trained model on dedicated model inference. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.