> ## Documentation Index
> Fetch the complete documentation index at: https://docs.together.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Run GRPO recipes

> Run the cookbook's single-turn GRPO recipes on a built-in dataset or your own dataset and reward, and tune how they sample, score, and train.

Run [group relative policy optimization (GRPO)](https://docs.together.ai/docs/post-training/grpo.md) with the single-turn recipes from the [post-training cookbook](https://docs.together.ai/docs/post-training/cookbook-overview.md), on a built-in dataset or on your own dataset and reward. To train a tool-use agent over multi-turn episodes, use the [multi-turn recipes](https://docs.together.ai/docs/post-training/multi-turn-recipes.md) instead.

## Requirements

* A [cookbook checkout](https://docs.together.ai/docs/post-training/cookbook-overview.md#clone-to-run-recipes).
* A [model with generator support](https://docs.together.ai/docs/post-training/supported-models.md#catalog). Models marked trainer-only in the catalog can't run GRPO.
* Your own dataset, unless you train on a built-in one. Each row needs a prompt and a target to score completions against.
* [Weights & Biases (W\&B) logging](https://docs.together.ai/docs/post-training/cookbook-overview.md#weights-&-biases) set up or turned off. `GRPOConfig` logs to W\&B by default, so without a W\&B setup, a run fails after it provisions the model resource. Export `WANDB_API_KEY`, set `WANDB_MODE=offline` to log locally, or pass `wandb_project=None` to turn logging off.

## Run a GRPO recipe

Pass a `GRPOConfig` to the cookbook's `train_single_turn` function. The config's `dataset` field selects a registered dataset, which supplies the prompt format and the reward. To train on your own data, [register your own dataset and reward](#register-a-dataset-and-reward).

> [!WARNING]
>
> Set `max_sample_tokens` or `max_prompt_length` explicitly, as the following example does. If you leave both unset, the [prompt limit](#length-budgets) is `0` and every prompt is dropped. With `num_steps` set, the run then fails after provisioning and billing have started. Without `num_steps`, it exits before provisioning and logs `no training needed`.

This example trains Qwen3.6 35B A3B on GSM8K, the dataset the [GRPO quickstart](https://docs.together.ai/docs/post-training/quickstart-rl.md) runs end to end:

```python theme={null}
from post_training_cookbook.grpo.config import GRPOConfig
from post_training_cookbook.grpo.train import train_single_turn

config = GRPOConfig(
    model="Qwen/Qwen3.6-35B-A3B",
    dataset="gsm8k",
    enable_thinking=False,
    max_sampling_attempts=3,
    num_steps=200,
    max_model_length=1536,
    max_sample_tokens=768,
)

train_single_turn(config)
```

`train_single_turn` provisions a model resource with `num_generator_replicas` generators (one by default) and one session, then stops both when it returns. To train on a [model resource that's already running](https://docs.together.ai/docs/post-training/model-resources.md), set `model_resources_id` instead of `model`.

Settings worth adjusting first, one at a time and in small increments:

* **`learning_rate`:** The default is `1e-5`. Try `5e-6` for steadier training or `2e-5` for faster convergence. To compare several rates at once, [train multiple adapters on one resource](https://docs.together.ai/docs/post-training/multi-lora.md).
* **`group_size` and `temperature`:** The defaults are `8` and `0.7`. [Rewards and advantages](https://docs.together.ai/docs/post-training/grpo.md#rewards-and-advantages) explains when to change them.

To change how the recipe samples, scores, and trains, see [Tune a GRPO recipe](#tune-a-grpo-recipe). The [cookbook config reference](https://docs.together.ai/docs/post-training/cookbook.md#grpoconfig) lists every `GRPOConfig` field.

## Built-in datasets

The cookbook registers four datasets, each paired with a reward. To train on one, set `dataset` on `GRPOConfig` to its name. Their entries in `post_training_cookbook/datasets/registry.py` are also working examples of the fields in [Register a dataset and reward](#register-a-dataset-and-reward).

| Name | Hugging Face dataset | Task | Reward |
| - | - | - | - |
| `gsm8k` | `openai/gsm8k` | Grade-school math word problems. The prompt asks for a final line of `#### <answer>`. | `1.0` when the number after the first `####`, or inside `\boxed{}` when there's no `####`, matches the reference. |
| `deepmath` | `zwhe99/DeepMath-103K` | Competition math. The prompt asks for the answer inside `\boxed{}`. | `1.0` when the boxed answer is mathematically equivalent to the reference. |
| `deepmath-lv5` | `zwhe99/DeepMath-103K` | The `deepmath` problems at difficulty 5 and below. | Same as `deepmath`. |
| `game24` | `nlile/24-game` | Combine four numbers with arithmetic to make 24, ending with a boxed expression. The registry entry filters out unsolvable rows. | Partial credit for a valid expression that uses each number once and for its closeness to 24, plus a bonus for hitting 24 exactly. Validation reports strict 0-or-1 accuracy instead. |

## Register a dataset and reward

To train on your own data, add an entry to the cookbook's dataset registry. Each entry maps a dataset name to how its rows become prompts, what each row's target is, and how to score completions against that target. These steps register a dataset of Python coding problems, where each row has a `prompt` and an `expected_output`.

**1. Write the reward function**

Add your reward function to `post_training_cookbook/datasets/rewards.py`. The recipe calls it with the model's decoded completion and the row's target, and expects a float back. This example runs the generated code and compares its standard output with the expected output:

````python theme={null}
import functools
import subprocess
import sys
import tempfile


@functools.lru_cache(maxsize=1024)
def code_output_reward(prediction: str, target: str) -> float:
    """Score 1.0 if the code's stdout matches the expected output."""
    code = _extract_code_block(prediction)
    if code is None:
        return 0.0
    try:
        with tempfile.NamedTemporaryFile(
            mode="w", suffix=".py", delete=True
        ) as f:
            f.write(code)
            f.flush()
            result = subprocess.run(
                [sys.executable, f.name],
                capture_output=True,
                text=True,
                timeout=5,
            )
        return 1.0 if result.stdout.strip() == target.strip() else 0.0
    except Exception:
        # A timeout, a crash, or unrunnable code all score 0.
        return 0.0


def _extract_code_block(text: str) -> str | None:
    """Extract the last ```python ... ``` block from text."""
    blocks = text.split("```python")
    if len(blocks) < 2:
        return None
    last_block = blocks[-1]
    end = last_block.find("```")
    return last_block.strip() if end == -1 else last_block[:end].strip()
````

> [!WARNING]
>
> This reward executes model output on your machine, with only a timeout to contain it. Run a reward like this in isolation, for example in a container or a [code sandbox](https://docs.together.ai/docs/together-code-sandbox.md).

The cookbook scores every completion twice: once with `compute_reward`, and once with `compute_metric`, which falls back to the reward when you don't set one. This example sets no metric, so the `lru_cache` decorator returns the cached score on the second call, and each completion's code runs only once.

Single-turn recipes score completions one at a time, so a reward that waits on an API call, a model, or a subprocess repeats that wait for every completion, including the extra ones that dynamic sampling and validation score. To score with something slow, such as a judge model, write a one-turn [environment](https://docs.together.ai/docs/post-training/multi-turn-recipes.md#write-an-environment) instead. The cookbook runs every episode's async `step` concurrently, so the waits overlap instead of adding up.

**2. Register the dataset**

Add a `DatasetConfig` entry to `post_training_cookbook/datasets/registry.py`:

````python theme={null}
from post_training_cookbook.datasets.rewards import code_output_reward

DATASETS["code_output"] = DatasetConfig(
    hf_name="my-org/python-coding-problems",
    hf_config=None,
    hf_split="train",
    format_question=lambda row: (
        f"{row['prompt']}\n\n"
        "Write Python code that solves this problem. "
        "Put your solution in a ```python code block."
    ),
    extract_target=lambda row: row["expected_output"],
    compute_reward=code_output_reward,
    few_shot_messages=[
        {
            "role": "user",
            "content": (
                "Write a function that returns the sum of two numbers, "
                "then print the result of calling it with 3 and 4.\n\n"
                "Write Python code that solves this problem. "
                "Put your solution in a ```python code block."
            ),
        },
        {
            "role": "assistant",
            "content": (
                "```python\n"
                "def add(a, b):\n"
                "    return a + b\n\n"
                "print(add(3, 4))\n"
                "```"
            ),
        },
    ],
)
````

| Field | Description |
| - | - |
| `hf_name`, `hf_config`, `hf_split` | The Hugging Face dataset repo or a local folder, its config (`None` if the dataset has none), and the split. |
| `format_question` | Takes a row and returns the prompt text. |
| `extract_target` | Takes a row and returns the ground truth the reward scores against. |
| `compute_reward` | Takes the decoded completion and the target, and returns the reward. |
| `compute_metric` | Optional. A separate accuracy function for validation. Falls back to `compute_reward` when `None`. |
| `filter_fn` | Optional. Takes a row and returns `True` to keep it. |
| `few_shot_messages` | Optional. Examples prepended to every prompt as earlier user and assistant turns. Write them in the output format the reward expects. |

To train on data that isn't on Hugging Face, load it from a local folder:

1. Save the rows as a JSON Lines file named `train.jsonl`, one row per line, in a folder of its own.
2. Set `hf_name` to the folder's path, `hf_config` to `None`, and `hf_split` to `"train"`. The cookbook loads data with the Hugging Face `datasets` library, which reads `train.jsonl` into the `train` split. A relative path, such as `"./my_problems"`, resolves from the directory you run the recipe in.

**3. Test the reward**

Run the reward on hand-written completions before you train on it:

````python theme={null}
from post_training_cookbook.datasets.rewards import code_output_reward

assert code_output_reward('```python\nprint("hello")\n```', "hello") == 1.0
assert code_output_reward('```python\nprint("world")\n```', "hello") == 0.0
assert code_output_reward("The answer is hello", "hello") == 0.0
````

Add the same cases to `tests/datasets/test_rewards.py`, then run `uv run pytest tests/datasets`. Include one edge case per failure mode: an empty completion, a very long completion, output in an unexpected format, and, for this reward, a timeout.

**4. Create and run the recipe**

Create `recipes/grpo/code_output.py`:

```python theme={null}
from post_training_cookbook.grpo.config import GRPOConfig
from post_training_cookbook.grpo.train import train_single_turn

config = GRPOConfig(
    model="Qwen/Qwen3.6-35B-A3B",
    dataset="code_output",
    num_examples=5000,
    num_val_examples=200,
    batch_size=8,
    group_size=8,
    num_epochs=3,
    learning_rate=1e-5,
    max_model_length=2048,
    max_prompt_length=512,
    max_sample_tokens=1024,
    enable_thinking=False,
    temperature=0.8,
    wandb_project="grpo-code-output",
)

if __name__ == "__main__":
    train_single_turn(config)
```

A code solution takes more tokens than a math answer, so this recipe sets `max_sample_tokens` higher than the GSM8K recipe does. It also raises `temperature` from the default `0.7` to `0.8`, so the completions in each group vary more.

Run the recipe from your cookbook checkout:

```bash theme={null}
uv run python recipes/grpo/code_output.py
```

## Report a separate metric

To train on one reward and report a different metric, set both `compute_reward` and `compute_metric` on the dataset's registry entry. For example, train on a smooth, [tiered reward](https://docs.together.ai/docs/post-training/grpo.md#reward-shapes) that gives GRPO a gradient to follow, and report a strict binary metric that tells you how often the model is actually right:

```python theme={null}
DATASETS["my_task"] = DatasetConfig(
    ...,
    compute_reward=tiered_reward,
    compute_metric=exact_match_accuracy,
)
```

`compute_reward` drives the advantages, and validation reports `compute_metric` as `val/accuracy`. The built-in `game24` dataset works this way, pairing a partial-credit reward with a strict 0-or-1 metric.

## Tune a GRPO recipe

These `GRPOConfig` settings change how the recipe samples, scores, and trains. [Multi-turn training](https://docs.together.ai/docs/post-training/multi-turn-recipes.md) uses the same settings, except where noted, and adds [async training](https://docs.together.ai/docs/post-training/multi-turn-recipes.md#async-training).

Length budgets, overlong handling, and dynamic sampling run in the cookbook's loop, not on the post-training service, so a loop you [write with the SDK](https://docs.together.ai/docs/post-training/training-loop.md) has them only if you implement them. Loss variants select one of the service's [built-in loss functions](https://docs.together.ai/docs/post-training/loss-functions.md#built-in-loss-functions), so your own loop can use the same objectives.

### Length budgets

`max_model_length` caps the prompt plus the completion, in tokens, and defaults to `2048`. The cookbook derives two budgets from it:

* **`max_sample_tokens`:** The maximum number of tokens generated per completion. It defaults to `max_model_length`, so set it lower to leave room for the prompt.
* **`max_prompt_length`:** The longest prompt that single-turn training keeps. Longer prompts are dropped before training. It defaults to `max_model_length - max_sample_tokens`, so if you set neither budget, [every prompt is dropped](#run-a-grpo-recipe).

Some models always think, and some reason at maximum effort by default. To avoid truncating their reasoning, set a generous completion budget, or lower `thinking_effort` on models that support it. [Supported models](https://docs.together.ai/docs/post-training/supported-models.md#thinking-and-chat-templates) lists each model's thinking default and effort levels.

A [`truncated_ratio`](https://docs.together.ai/docs/post-training/monitoring#cookbook-metrics) above about 30% indicates a tight budget or runaway-length output.

### Overlong responses

The cookbook has two independent controls for completions that approach the token budget:

* **Overlong penalty (on by default):** A reward penalty from the decoupled clip and dynamic sampling policy optimization (DAPO) paper. It ramps up linearly over the last `overlong_buffer_len` tokens of the budget. The buffer defaults to `max_sample_tokens // 8`. `overlong_penalty_factor` scales the penalty, and `0` turns it off.
* **Overlong masking (off by default):** Set `overlong_masking=True` to exclude truncated completions from training. Their rewards still count toward the group baseline. Single-turn training skips any step that masking leaves with fewer than `group_size` samples.

Masking doesn't turn the penalty off. To mask without penalizing, set `overlong_masking=True` and `overlong_penalty_factor=0`.

### Dynamic sampling

Dynamic sampling, also from the DAPO paper, keeps drawing fresh batches until it collects `batch_size` groups with varied rewards, or until it has drawn `max_sampling_attempts` batches. To turn it on, set `max_sampling_attempts` above its default of `1`. The cookbook's GSM8K recipe uses `3`.

Each attempt generates a full batch, so a step can cost up to `max_sampling_attempts` batches of generation. If no attempt produces a group with varied rewards, the cookbook skips the step. Single-turn training logs `skipped: 0/<groups> groups had non-zero reward variance`, and multi-turn training logs `samples=0`.

### Loss variants

The cookbook samples groups and computes advantages the same way for every objective, so switching objectives changes only the loss. These `GRPOConfig` fields select and configure the objective:

* **Policy loss:** `loss` defaults to `"grpo"`. The other [built-in loss functions](https://docs.together.ai/docs/post-training/loss-functions.md#built-in-loss-functions) are `"ppo"`, `"cispo"`, `"dppo"`, `"dro"`, and `"importance_sampling"`. They sum token losses instead of averaging them, so their loss grows with the number of trained tokens. When you switch to one, you may need to retune the learning rate.
* **Clip band:** `clip_low_threshold` and `clip_high_threshold` apply only to `"grpo"`, `"ppo"`, and `"cispo"`. Leave them `None` to use the service defaults, which are `[0.8, 1.2]` for GRPO and PPO, and `[0, 4]` for CISPO. To set your own bounds, see [Clipping](https://docs.together.ai/docs/post-training/loss-functions.md#clipping). CISPO's clipping scales a token's gradient instead of zeroing it.
* **DRO penalty:** `"dro"` requires `dro_beta`, the coefficient on its quadratic log-ratio penalty.
* **Loss aggregation:** `loss_agg_type` applies only to `"grpo"` and sets how its per-token losses reduce to a single value. Aggregation changes the effective learning rate, so keep it fixed when comparing runs.
  * `GRPO_LOSS_AGGREGATION_TYPE_FIXED_HORIZON`, the default, divides the masked sum by all batch tokens, including padding.
  * `TOKEN_MEAN` divides by trained tokens.
  * `SEQUENCE_MEAN` averages within each sequence, then across sequences, so each sequence carries equal weight.
* **GSPO:** To train with group sequence policy optimization (GSPO), keep the default `"grpo"` loss and set `ratio_type="GRPO_LOSS_RATIO_TYPE_SEQUENCE"`. GSPO computes the importance ratio and clips over the whole completion instead of per token.

Setting any of these fields for an objective that doesn't use it raises an error when you create the config.

Metric keys differ by objective. GRPO and PPO report `fb/loss/clip/*` and `fb/loss/ratio/mean`, described in [Service-emitted metrics](https://docs.together.ai/docs/post-training/monitoring.md#service-emitted-metrics). The other objectives use keys such as `fb/ratio_mean`, and CISPO reports `fb/clip_fraction`.

The cookbook config has no Kullback-Leibler (KL) reference-policy penalty. To add one, [add a KL penalty in your own loop](https://docs.together.ai/docs/post-training/loss-functions.md#kl-penalty). For an objective the built-in loss functions don't cover, [write a custom loss function](https://docs.together.ai/docs/post-training/custom-loss-functions.md).

## Continue a run

The recipe saves [training checkpoints](https://docs.together.ai/docs/post-training/checkpoints.md#training-checkpoints) only when you set `training_checkpoint_interval`, so set it on any run you might want to continue. To continue a run, rerun its recipe with the original settings, plus the ID of a training checkpoint the run saved and the original run's directory:

```python theme={null}
from post_training_cookbook.grpo.config import GRPOConfig
from post_training_cookbook.grpo.train import train_single_turn

config = GRPOConfig(
    model="Qwen/Qwen3.6-35B-A3B",
    dataset="gsm8k",
    # ...every other setting from the original run...
    num_steps=200,
    resume_from_checkpoint_id="<training_checkpoint_id>",
    resume_run_dir="runs/<original_run_name>",
)

train_single_turn(config)
```

The new session restores the adapter and optimizer state from the checkpoint. The cookbook then picks up the learning-rate schedule, validation cadence, and next batch of data from where the checkpoint was saved. It finds that point in the checkpoint's record in the run directory's `checkpoints.jsonl`, which holds the step the checkpoint was saved at and the run's position in the dataset. The server's step counter can't replace the record, because a step that collects no usable groups advances the schedule without an optimizer update.

* **Set `num_steps` to the total, not the remainder:** Continuing from a checkpoint saved at step 40 with `num_steps=200` trains 160 more steps. If the checkpoint is at or past `num_steps`, the run trains nothing and provisions nothing.
* **Keep the data and batch shape the same:** The dataset contents and order, `batch_size`, and single-turn or multi-turn mode must match the original run. The cookbook checks them against the record and fails before provisioning if they differ.
* **Keep the learning-rate and checkpoint settings the same:** If they differ, the continued run's schedule won't match the original's.
* **Keep the run directory:** Continuing requires the checkpoint's record. If the record is missing, or the checkpoint was saved before the cookbook started recording data positions, start a new run from the weights with `load_optimizer=False` instead.
* **Use JSON-serializable training inputs:** Any run that saves training checkpoints fingerprints its training data, so multi-turn task targets and metadata must be JSON-serializable.

A continued async multi-turn run regenerates any rollouts that were queued or in flight when the checkpoint was saved, using the restored model. It doesn't reproduce their original sampled tokens.

## Troubleshooting

* **`zero_variance_skip_ratio` stays above 50%:** Check `identical_nonzero_reward_skip_ratio` (single-turn only) to tell identical nonzero rewards apart from all-zero rewards. To recover more groups, [turn on dynamic sampling](#dynamic-sampling) by raising `max_sampling_attempts` above `1`, at the cost of extra sampling. To adjust task difficulty, temperature, or group size, see [GRPO troubleshooting](https://docs.together.ai/docs/post-training/grpo.md#troubleshooting).
* **Checkpoint creation fails with `You need to complete at least one training step before you can create an inference checkpoint`:** Every step was skipped. Check the log for `skipped:` (single-turn) or `samples=0` (multi-turn). If groups had identical rewards, see the previous item. If [overlong masking](#overlong-responses) left too few samples, adjust the [length budget](#length-budgets).
* **Out of memory or timeouts:** Reduce `batch_size`, `max_model_length`, or `group_size`. For timeouts, you can instead raise `timeout` on the config.

For problems that aren't specific to recipes, see [GRPO troubleshooting](https://docs.together.ai/docs/post-training/grpo.md#troubleshooting).

## Next steps

> [card] [Loss functions](https://docs.together.ai/docs/post-training/loss-functions.md)
>
> Compare the built-in loss functions behind each loss variant.

> [card] [Multi-turn recipes](https://docs.together.ai/docs/post-training/multi-turn-recipes.md)
>
> Train tool-use agents on an environment you write.

> [card] [Monitor training runs](https://docs.together.ai/docs/post-training/monitoring.md)
>
> Read the metrics a GRPO recipe logs and act on them.

> [card] [Cookbook config reference](https://docs.together.ai/docs/post-training/cookbook.md#grpoconfig)
>
> Look up every `GRPOConfig` field and its default.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.