Run GRPO recipes

Run group relative policy optimization (GRPO) with the single-turn recipes from the post-training cookbook, on a built-in dataset or on your own dataset and reward. To train a tool-use agent over multi-turn episodes, use the multi-turn recipes instead.

Requirements

A cookbook checkout.
A model with generator support. Models marked trainer-only in the catalog can’t run GRPO.
Your own dataset, unless you train on a built-in one. Each row needs a prompt and a target to score completions against.
Weights & Biases (W&B) logging set up or turned off. GRPOConfig logs to W&B by default, so without a W&B setup, a run fails after it provisions the model resource. Export WANDB_API_KEY, set WANDB_MODE=offline to log locally, or pass wandb_project=None to turn logging off.

Run a GRPO recipe

Pass a GRPOConfig to the cookbook’s train_single_turn function. The config’s dataset field selects a registered dataset, which supplies the prompt format and the reward. To train on your own data, register your own dataset and reward.

Set max_sample_tokens or max_prompt_length explicitly, as the following example does. If you leave both unset, the prompt limit is 0 and every prompt is dropped. With num_steps set, the run then fails after provisioning and billing have started. Without num_steps, it exits before provisioning and logs no training needed.

This example trains Qwen3.6 35B A3B on GSM8K, the dataset the GRPO quickstart runs end to end:

from post_training_cookbook.grpo.config import GRPOConfig
from post_training_cookbook.grpo.train import train_single_turn

config = GRPOConfig(
    model="Qwen/Qwen3.6-35B-A3B",
    dataset="gsm8k",
    enable_thinking=False,
    max_sampling_attempts=3,
    num_steps=200,
    max_model_length=1536,
    max_sample_tokens=768,
)

train_single_turn(config)

train_single_turn provisions a model resource with num_generator_replicas generators (one by default) and one session, then stops both when it returns. To train on a model resource that’s already running, set model_resources_id instead of model. Settings worth adjusting first, one at a time and in small increments:

learning_rate: The default is 1e-5. Try 5e-6 for steadier training or 2e-5 for faster convergence. To compare several rates at once, train multiple adapters on one resource.
group_size and temperature: The defaults are 8 and 0.7. Rewards and advantages explains when to change them.

To change how the recipe samples, scores, and trains, see Tune a GRPO recipe. The cookbook config reference lists every GRPOConfig field.

Built-in datasets

The cookbook registers four datasets, each paired with a reward. To train on one, set dataset on GRPOConfig to its name. Their entries in post_training_cookbook/datasets/registry.py are also working examples of the fields in Register a dataset and reward.

Name	Hugging Face dataset	Task	Reward
`gsm8k`	`openai/gsm8k`	Grade-school math word problems. The prompt asks for a final line of `#### <answer>`.	`1.0` when the number after the first `####`, or inside `\boxed{}` when there’s no `####`, matches the reference.
`deepmath`	`zwhe99/DeepMath-103K`	Competition math. The prompt asks for the answer inside `\boxed{}`.	`1.0` when the boxed answer is mathematically equivalent to the reference.
`deepmath-lv5`	`zwhe99/DeepMath-103K`	The `deepmath` problems at difficulty 5 and below.	Same as `deepmath`.
`game24`	`nlile/24-game`	Combine four numbers with arithmetic to make 24, ending with a boxed expression. The registry entry filters out unsolvable rows.	Partial credit for a valid expression that uses each number once and for its closeness to 24, plus a bonus for hitting 24 exactly. Validation reports strict 0-or-1 accuracy instead.

Register a dataset and reward

To train on your own data, add an entry to the cookbook’s dataset registry. Each entry maps a dataset name to how its rows become prompts, what each row’s target is, and how to score completions against that target. These steps register a dataset of Python coding problems, where each row has a prompt and an expected_output.

Write the reward function

Add your reward function to post_training_cookbook/datasets/rewards.py. The recipe calls it with the model’s decoded completion and the row’s target, and expects a float back. This example runs the generated code and compares its standard output with the expected output:

import functools
import subprocess
import sys
import tempfile


@functools.lru_cache(maxsize=1024)
def code_output_reward(prediction: str, target: str) -> float:
    """Score 1.0 if the code's stdout matches the expected output."""
    code = _extract_code_block(prediction)
    if code is None:
        return 0.0
    try:
        with tempfile.NamedTemporaryFile(
            mode="w", suffix=".py", delete=True
        ) as f:
            f.write(code)
            f.flush()
            result = subprocess.run(
                [sys.executable, f.name],
                capture_output=True,
                text=True,
                timeout=5,
            )
        return 1.0 if result.stdout.strip() == target.strip() else 0.0
    except Exception:
        # A timeout, a crash, or unrunnable code all score 0.
        return 0.0


def _extract_code_block(text: str) -> str | None:
    """Extract the last ```python ... ``` block from text."""
    blocks = text.split("```python")
    if len(blocks) < 2:
        return None
    last_block = blocks[-1]
    end = last_block.find("```")
    return last_block.strip() if end == -1 else last_block[:end].strip()

This reward executes model output on your machine, with only a timeout to contain it. Run a reward like this in isolation, for example in a container or a code sandbox.

The cookbook scores every completion twice: once with compute_reward, and once with compute_metric, which falls back to the reward when you don’t set one. This example sets no metric, so the lru_cache decorator returns the cached score on the second call, and each completion’s code runs only once.Single-turn recipes score completions one at a time, so a reward that waits on an API call, a model, or a subprocess repeats that wait for every completion, including the extra ones that dynamic sampling and validation score. To score with something slow, such as a judge model, write a one-turn environment instead. The cookbook runs every episode’s async step concurrently, so the waits overlap instead of adding up.

Add a DatasetConfig entry to post_training_cookbook/datasets/registry.py:

from post_training_cookbook.datasets.rewards import code_output_reward

DATASETS["code_output"] = DatasetConfig(
    hf_name="my-org/python-coding-problems",
    hf_config=None,
    hf_split="train",
    format_question=lambda row: (
        f"{row['prompt']}\n\n"
        "Write Python code that solves this problem. "
        "Put your solution in a ```python code block."
    ),
    extract_target=lambda row: row["expected_output"],
    compute_reward=code_output_reward,
    few_shot_messages=[
        {
            "role": "user",
            "content": (
                "Write a function that returns the sum of two numbers, "
                "then print the result of calling it with 3 and 4.\n\n"
                "Write Python code that solves this problem. "
                "Put your solution in a ```python code block."
            ),
        },
        {
            "role": "assistant",
            "content": (
                "```python\n"
                "def add(a, b):\n"
                "    return a + b\n\n"
                "print(add(3, 4))\n"
                "```"
            ),
        },
    ],
)

Field	Description
`hf_name`, `hf_config`, `hf_split`	The Hugging Face dataset repo or a local folder, its config (`None` if the dataset has none), and the split.
`format_question`	Takes a row and returns the prompt text.
`extract_target`	Takes a row and returns the ground truth the reward scores against.
`compute_reward`	Takes the decoded completion and the target, and returns the reward.
`compute_metric`	Optional. A separate accuracy function for validation. Falls back to `compute_reward` when `None`.
`filter_fn`	Optional. Takes a row and returns `True` to keep it.
`few_shot_messages`	Optional. Examples prepended to every prompt as earlier user and assistant turns. Write them in the output format the reward expects.

To train on data that isn’t on Hugging Face, load it from a local folder:

Save the rows as a JSON Lines file named train.jsonl, one row per line, in a folder of its own.
Set hf_name to the folder’s path, hf_config to None, and hf_split to "train". The cookbook loads data with the Hugging Face datasets library, which reads train.jsonl into the train split. A relative path, such as "./my_problems", resolves from the directory you run the recipe in.

Test the reward

Run the reward on hand-written completions before you train on it:

from post_training_cookbook.datasets.rewards import code_output_reward

assert code_output_reward('```python\nprint("hello")\n```', "hello") == 1.0
assert code_output_reward('```python\nprint("world")\n```', "hello") == 0.0
assert code_output_reward("The answer is hello", "hello") == 0.0

Add the same cases to tests/datasets/test_rewards.py, then run uv run pytest tests/datasets. Include one edge case per failure mode: an empty completion, a very long completion, output in an unexpected format, and, for this reward, a timeout.

Create and run the recipe

Create recipes/grpo/code_output.py:

from post_training_cookbook.grpo.config import GRPOConfig
from post_training_cookbook.grpo.train import train_single_turn

config = GRPOConfig(
    model="Qwen/Qwen3.6-35B-A3B",
    dataset="code_output",
    num_examples=5000,
    num_val_examples=200,
    batch_size=8,
    group_size=8,
    num_epochs=3,
    learning_rate=1e-5,
    max_model_length=2048,
    max_prompt_length=512,
    max_sample_tokens=1024,
    enable_thinking=False,
    temperature=0.8,
    wandb_project="grpo-code-output",
)

if __name__ == "__main__":
    train_single_turn(config)

A code solution takes more tokens than a math answer, so this recipe sets max_sample_tokens higher than the GSM8K recipe does. It also raises temperature from the default 0.7 to 0.8, so the completions in each group vary more.Run the recipe from your cookbook checkout:

uv run python recipes/grpo/code_output.py

Report a separate metric

To train on one reward and report a different metric, set both compute_reward and compute_metric on the dataset’s registry entry. For example, train on a smooth, tiered reward that gives GRPO a gradient to follow, and report a strict binary metric that tells you how often the model is actually right:

DATASETS["my_task"] = DatasetConfig(
    ...,
    compute_reward=tiered_reward,
    compute_metric=exact_match_accuracy,
)

compute_reward drives the advantages, and validation reports compute_metric as val/accuracy. The built-in game24 dataset works this way, pairing a partial-credit reward with a strict 0-or-1 metric.

Tune a GRPO recipe

These GRPOConfig settings change how the recipe samples, scores, and trains. Multi-turn training uses the same settings, except where noted, and adds async training. Length budgets, overlong handling, and dynamic sampling run in the cookbook’s loop, not on the post-training service, so a loop you write with the SDK has them only if you implement them. Loss variants select one of the service’s built-in loss functions, so your own loop can use the same objectives.

Length budgets

max_model_length caps the prompt plus the completion, in tokens, and defaults to 2048. The cookbook derives two budgets from it:

max_sample_tokens: The maximum number of tokens generated per completion. It defaults to max_model_length, so set it lower to leave room for the prompt.
max_prompt_length: The longest prompt that single-turn training keeps. Longer prompts are dropped before training. It defaults to max_model_length - max_sample_tokens, so if you set neither budget, every prompt is dropped.

Some models always think, and some reason at maximum effort by default. To avoid truncating their reasoning, set a generous completion budget, or lower thinking_effort on models that support it. Supported models lists each model’s thinking default and effort levels. A truncated_ratio above about 30% indicates a tight budget or runaway-length output.

Overlong responses

The cookbook has two independent controls for completions that approach the token budget:

Overlong penalty (on by default): A reward penalty from the decoupled clip and dynamic sampling policy optimization (DAPO) paper. It ramps up linearly over the last overlong_buffer_len tokens of the budget. The buffer defaults to max_sample_tokens // 8. overlong_penalty_factor scales the penalty, and 0 turns it off.
Overlong masking (off by default): Set overlong_masking=True to exclude truncated completions from training. Their rewards still count toward the group baseline. Single-turn training skips any step that masking leaves with fewer than group_size samples.

Masking doesn’t turn the penalty off. To mask without penalizing, set overlong_masking=True and overlong_penalty_factor=0.

Dynamic sampling

Dynamic sampling, also from the DAPO paper, keeps drawing fresh batches until it collects batch_size groups with varied rewards, or until it has drawn max_sampling_attempts batches. To turn it on, set max_sampling_attempts above its default of 1. The cookbook’s GSM8K recipe uses 3. Each attempt generates a full batch, so a step can cost up to max_sampling_attempts batches of generation. If no attempt produces a group with varied rewards, the cookbook skips the step. Single-turn training logs skipped: 0/<groups> groups had non-zero reward variance, and multi-turn training logs samples=0.

Loss variants

The cookbook samples groups and computes advantages the same way for every objective, so switching objectives changes only the loss. These GRPOConfig fields select and configure the objective:

Policy loss: loss defaults to "grpo". The other built-in loss functions are "ppo", "cispo", "dppo", "dro", and "importance_sampling". They sum token losses instead of averaging them, so their loss grows with the number of trained tokens. When you switch to one, you may need to retune the learning rate.
Clip band: clip_low_threshold and clip_high_threshold apply only to "grpo", "ppo", and "cispo". Leave them None to use the service defaults, which are [0.8, 1.2] for GRPO and PPO, and [0, 4] for CISPO. To set your own bounds, see Clipping. CISPO’s clipping scales a token’s gradient instead of zeroing it.
DRO penalty: "dro" requires dro_beta, the coefficient on its quadratic log-ratio penalty.
Loss aggregation: loss_agg_type applies only to "grpo" and sets how its per-token losses reduce to a single value. Aggregation changes the effective learning rate, so keep it fixed when comparing runs.
- GRPO_LOSS_AGGREGATION_TYPE_FIXED_HORIZON, the default, divides the masked sum by all batch tokens, including padding.
- TOKEN_MEAN divides by trained tokens.
- SEQUENCE_MEAN averages within each sequence, then across sequences, so each sequence carries equal weight.
GSPO: To train with group sequence policy optimization (GSPO), keep the default "grpo" loss and set ratio_type="GRPO_LOSS_RATIO_TYPE_SEQUENCE". GSPO computes the importance ratio and clips over the whole completion instead of per token.

Setting any of these fields for an objective that doesn’t use it raises an error when you create the config. Metric keys differ by objective. GRPO and PPO report fb/loss/clip/* and fb/loss/ratio/mean, described in Service-emitted metrics. The other objectives use keys such as fb/ratio_mean, and CISPO reports fb/clip_fraction. The cookbook config has no Kullback-Leibler (KL) reference-policy penalty. To add one, add a KL penalty in your own loop. For an objective the built-in loss functions don’t cover, write a custom loss function.

Continue a run

The recipe saves training checkpoints only when you set training_checkpoint_interval, so set it on any run you might want to continue. To continue a run, rerun its recipe with the original settings, plus the ID of a training checkpoint the run saved and the original run’s directory:

from post_training_cookbook.grpo.config import GRPOConfig
from post_training_cookbook.grpo.train import train_single_turn

config = GRPOConfig(
    model="Qwen/Qwen3.6-35B-A3B",
    dataset="gsm8k",
    # ...every other setting from the original run...
    num_steps=200,
    resume_from_checkpoint_id="<training_checkpoint_id>",
    resume_run_dir="runs/<original_run_name>",
)

train_single_turn(config)

The new session restores the adapter and optimizer state from the checkpoint. The cookbook then picks up the learning-rate schedule, validation cadence, and next batch of data from where the checkpoint was saved. It finds that point in the checkpoint’s record in the run directory’s checkpoints.jsonl, which holds the step the checkpoint was saved at and the run’s position in the dataset. The server’s step counter can’t replace the record, because a step that collects no usable groups advances the schedule without an optimizer update.

Set num_steps to the total, not the remainder: Continuing from a checkpoint saved at step 40 with num_steps=200 trains 160 more steps. If the checkpoint is at or past num_steps, the run trains nothing and provisions nothing.
Keep the data and batch shape the same: The dataset contents and order, batch_size, and single-turn or multi-turn mode must match the original run. The cookbook checks them against the record and fails before provisioning if they differ.
Keep the learning-rate and checkpoint settings the same: If they differ, the continued run’s schedule won’t match the original’s.
Keep the run directory: Continuing requires the checkpoint’s record. If the record is missing, or the checkpoint was saved before the cookbook started recording data positions, start a new run from the weights with load_optimizer=False instead.
Use JSON-serializable training inputs: Any run that saves training checkpoints fingerprints its training data, so multi-turn task targets and metadata must be JSON-serializable.

A continued async multi-turn run regenerates any rollouts that were queued or in flight when the checkpoint was saved, using the restored model. It doesn’t reproduce their original sampled tokens.

Troubleshooting

zero_variance_skip_ratio stays above 50%: Check identical_nonzero_reward_skip_ratio (single-turn only) to tell identical nonzero rewards apart from all-zero rewards. To recover more groups, turn on dynamic sampling by raising max_sampling_attempts above 1, at the cost of extra sampling. To adjust task difficulty, temperature, or group size, see GRPO troubleshooting.
Checkpoint creation fails with You need to complete at least one training step before you can create an inference checkpoint: Every step was skipped. Check the log for skipped: (single-turn) or samples=0 (multi-turn). If groups had identical rewards, see the previous item. If overlong masking left too few samples, adjust the length budget.
Out of memory or timeouts: Reduce batch_size, max_model_length, or group_size. For timeouts, you can instead raise timeout on the config.

For problems that aren’t specific to recipes, see GRPO troubleshooting.

Next steps

Loss functions

Compare the built-in loss functions behind each loss variant.

Multi-turn recipes

Train tool-use agents on an environment you write.

Monitor training runs

Read the metrics a GRPO recipe logs and act on them.

Cookbook config reference

Look up every GRPOConfig field and its default.

GET STARTED

SERVERLESS

INFERENCE APIS

DEDICATED MODEL INFERENCE

DEDICATED CONTAINER INFERENCE

POST-TRAINING

FINE-TUNING

GPU CLUSTERS

CODE EXECUTION

ADMINISTRATION

Requirements

Run a GRPO recipe

Built-in datasets

Register a dataset and reward

Report a separate metric

Tune a GRPO recipe

Length budgets

Overlong responses

Dynamic sampling

Loss variants

Continue a run

Troubleshooting

Next steps

Loss functions

Multi-turn recipes

Monitor training runs

Cookbook config reference

​Requirements

​Run a GRPO recipe

​Built-in datasets

​Register a dataset and reward

​Report a separate metric

​Tune a GRPO recipe

​Length budgets

​Overlong responses

​Dynamic sampling

​Loss variants

​Continue a run

​Troubleshooting

​Next steps

Loss functions

Multi-turn recipes

Monitor training runs

Cookbook config reference

Requirements

Run a GRPO recipe

Built-in datasets

Register a dataset and reward

Report a separate metric

Tune a GRPO recipe

Length budgets

Overlong responses

Dynamic sampling

Loss variants

Continue a run

Troubleshooting

Next steps