Reinforcement learning with GRPO

Group relative policy optimization (GRPO) trains a model from a reward signal instead of example responses. For each prompt, the model generates a group of completions and a reward function you write scores each one. The update then raises the probability of completions that scored above their group’s average and lowers the probability of the rest. Use it when you can measure how good an output is but can’t write the ideal output yourself, or when you want to push accuracy on a task past what your demonstrations reach. The RL quickstart runs a GRPO recipe end to end. This page covers the mechanics behind it, the settings that change its behavior, how to write your own loop, async training, and what to do when a run misbehaves.

Requirements

A model with generator support. GRPO samples completions as it trains, so the model resource needs num_generator_replicas of 1 or more. Models marked trainer-only on Supported models can’t run GRPO.
A dataset and a reward function. The cookbook ships four datasets with rewards. To bring your own, see Reward functions.
Weights & Biases set up, if you train with the cookbook. GRPOConfig logs to W&B by default, so export WANDB_API_KEY or turn logging off, as Monitoring describes. Otherwise the run fails after the model resource is provisioned.

How post-training runs GRPO

One GRPO step, which your code repeats. Step 1 calls sample, and the generator replicas on the model resource return completions and log-probabilities from the current policy. Steps 2 and 3 compute rewards and advantages in your own process. Steps 4, 5, and 6 call forward_backward, optim_step, and weights_sync on the trainer, which computes the loss and gradients, takes an optimizer step, and sends the new policy to the generator replicas. The base model stays frozen. SFT and DPO run only steps 4 and 5.

The generator replicas produce completions, and the trainer updates the weights. Each step runs this sequence:

sample: The generator replicas produce group_size completions for each prompt in the batch, along with the log-probability of each generated token.
Reward: Your reward function scores each completion.
Advantages: Your code subtracts each group’s mean reward from each completion’s reward. Rewards and advantages covers how the cookbook computes and filters them.
forward_backward: With LOSS_TYPE_GRPO, the trainer computes the clipped policy-gradient loss over samples that carry each completion’s tokens, advantage, and generator log-probabilities, and accumulates gradients.
optim_step: The optimizer applies the accumulated gradients. Its adam_params carry the learning rate, gradient clipping, and the other per-step optimizer settings.
weights_sync: The trainer publishes its new weights to the generator replicas, so the next sample comes from the updated policy. Async training covers the sync modes that return before the publish finishes.

Steps 2 and 3 run in your process, so the reward function can be any code you write. The other four steps run on the model resource.

Run a GRPO recipe

Every GRPO hyperparameter lives on GRPOConfig. To run the loop, pass the config to train_single_turn. This config sets only what the sections below discuss, and the RL quickstart walks through the full GSM8K recipe:

from post_training_cookbook.grpo.config import GRPOConfig
from post_training_cookbook.grpo.train import train_single_turn

config = GRPOConfig(
    model="Qwen/Qwen3.6-35B-A3B",
    dataset="gsm8k",
    enable_thinking=False,
    max_sampling_attempts=3,
    num_steps=200,
    max_model_length=1536,
    max_sample_tokens=768,
)

train_single_turn(config)

train_single_turn provisions a model resource for model with num_generator_replicas generator replicas (one by default), attaches one session to it, and stops both when it returns. To train on a model resource that’s already running, set model_resources_id instead of model. See Manage model resources. learning_rate defaults to 1e-5. Try 5e-6 for slower, steadier training or 2e-5 for faster convergence, and to compare several rates at once, train multiple adapters on one base model. Change one setting at a time, in small increments, so you can see how each one affects the run. Every GRPOConfig field is listed in the cookbook reference. The sections below cover the settings that most change a run’s behavior.

Rewards and advantages

How GRPO scores a group. One prompt produces group_size = 4 completions, c1 through c4. Your reward function scores them 1.0, 0.0, 1.0, and 0.0, for a group mean of 0.5. Each advantage is the reward minus the group mean, not divided by the group standard deviation, so c1 and c3 get +0.5 and are reinforced, and c2 and c4 get -0.5 and are suppressed. A zero-variance group, where every completion scores 1.0, carries no signal and is skipped, but its generation is still billed.

For each prompt, the generator produces group_size completions and the reward function scores each one. A completion’s advantage is its reward minus the group’s mean reward, so completions above the mean are reinforced and completions below it are suppressed. The cookbook departs from the GRPO paper in two ways. It doesn’t divide advantages by the group’s standard deviation, so they stay on the reward’s scale. It also skips any group where every completion scores the same, because that group carries no signal. No samples are built from a skipped group, but you still pay to generate its completions. The step log reports the fraction of skipped groups as zero_variance_skip_ratio. Group size: group_size defaults to 8. Larger groups, such as 16, give steadier advantage estimates and fewer zero-variance groups, at the cost of proportionally more sampling per step. Temperature: The group-mean baseline is only informative when the completions within a group differ. The default, temperature=0.7, balances exploration and exploitation. Raise it toward 1.0 when groups collapse to identical answers, or lower it toward 0.5 to exploit the current policy.

Write your own GRPO loop

When none of the cookbook recipes fit your use case, you can write your own training loop. This loop trains on an in-memory list of math problems whose prompts ask for the final answer after ####, and its reward function checks that answer:

from post_training_cookbook.codecs import get_codec
from post_training_cookbook.grpo.samples import (
    build_policy_sample,
    compute_group_advantages,
)
from post_training_cookbook.samples import make_prompt
from together.post_training import (
    AdamParams,
    ComputeConfig,
    GrpoLossParams,
    LoraConfig,
    LossConfig,
    ModelResourcesClient,
    SamplingParams,
)

MODEL = "Qwen/Qwen3.8-27B"
BATCH_SIZE = 16  # prompts per step
GROUP_SIZE = 8  # completions per prompt
problems = load_problems()  # a list of (messages, answer) pairs


def reward_fn(completion: str, answer: str) -> float:
    # 1.0 when the text after the last "####" is the answer, else 0.0
    return 1.0 if completion.split("####")[-1].strip() == answer else 0.0


# Load the model's chat template, with thinking off to keep completions short
codec = get_codec(MODEL, enable_thinking=False)

# Cap completion length, and use the cookbook's default temperature
sampling_params = SamplingParams(max_tokens=768, temperature=0.7)

# The GRPO loss, with the cookbook's default clip band from DAPO
loss = LossConfig(
    type="LOSS_TYPE_GRPO",
    grpo_params=GrpoLossParams(
        clip_low_threshold=0.8, clip_high_threshold=1.28
    ),
)

# Provision a model resource with one generator replica to sample from
with ModelResourcesClient.create(
    base_model=MODEL,
    compute_config=ComputeConfig(
        gpu_type="B200-SXM", num_generator_replicas=1
    ),
) as resources:

    # Start a training session that trains a LoRA adapter
    with resources.create_session(
        lora_config=LoraConfig(rank=16, alpha=16)
    ) as session:

        # Each step trains on the next BATCH_SIZE problems
        for step, start in enumerate(range(0, len(problems), BATCH_SIZE)):
            batch = problems[start : start + BATCH_SIZE]

            # Render each prompt with the chat template, ready for a reply
            prompts = [
                codec.encode_messages(messages, add_generation_prompt=True)
                for messages, _ in batch
            ]

            # 1. Sample GROUP_SIZE completions per prompt from the generator
            results = session.generator.sample_batch(
                [make_prompt(prompt) for prompt in prompts],
                num_samples=GROUP_SIZE,
                sampling_params=sampling_params,
            )

            samples, rewards = [], []
            for prompt, (_, answer), result in zip(prompts, batch, results):
                # Token IDs can come back as strings, so convert them
                group = [
                    ([int(t) for t in seq.tokens], list(seq.logprobs))
                    for seq in result.sequences
                ]

                # 2. Score each completion with the reward function
                group_rewards = [
                    reward_fn(codec.decode(tokens), answer)
                    for tokens, _ in group
                ]
                rewards.extend(group_rewards)

                # 3. Subtract the group's mean reward to get each advantage.
                # A group where every completion scored the same has no
                # signal, so skip it.
                advantages = compute_group_advantages(group_rewards).values
                if advantages is None:
                    continue

                # Build one sample per completion. Zeros over the prompt keep
                # it out of the loss, and the generator's logprobs let the
                # trainer compute the clipping ratio.
                prompt_zeros = [0.0] * len(prompt)
                for (tokens, logprobs), advantage in zip(group, advantages):
                    samples.append(
                        build_policy_sample(
                            tokens=prompt + tokens,
                            logprobs=prompt_zeros + logprobs,
                            loss_mask=prompt_zeros + [1.0] * len(tokens),
                            advantage=advantage,
                        )
                    )

            # The mean reward should rise as the policy improves
            print(f"[step {step}] reward={sum(rewards) / len(rewards):.3f}")

            # Skip the update when every group in the step was skipped
            if not samples:
                continue

            # 4. Compute the clipped policy-gradient loss and its gradients
            session.trainer.forward_backward(samples=samples, loss=loss)

            # 5. Update the adapter's weights
            session.trainer.optim_step(
                adam_params=AdamParams(learning_rate=1e-5, weight_decay=0.01),
            )

            # 6. Send the new weights to the generator, so the next step
            # samples from the updated policy
            session.trainer.weights_sync()

        # Save the adapter as an inference checkpoint you can deploy
        checkpoint = session.create_inference_checkpoint()
        print(checkpoint.id)

The reward runs in your process, so reward_fn can be any scoring code you write. Write a reward function covers what makes a good one. compute_group_advantages computes advantages as Rewards and advantages describes. For a group where every completion scored the same, its values is None, and the loop skips the group. build_policy_sample shifts the tokens into input and target positions and puts the advantage on every trained position. The loop keeps the generators busy by sending every prompt in a step in one sample_batch call and by getting each group from one request with num_samples. To overlap more, run independent calls concurrently with the _async variants. Sampling runs alongside other sampling and alongside the trainer’s forward_backward and optim_step, but a synchronous weights_sync waits for in-flight sampling to finish. sample_batch applies one SamplingParams to every prompt in the call. Its fields and defaults are in the API reference. Two of them need care:

stop: Not every model supports stop strings. A model that can’t match them rejects the request as invalid input, so omit stop and truncate completions in your code.
seed: Not supported yet, so leave it unset. When the generator runs more than one inference engine, the service rejects seed and the sample operation fails with an internal error.

To score fixed text without generating, call session.generator.compute_logprobs(prompt), or compute_logprobs_batch(prompts) for several prompts. They score with the weights the generator last received, so call weights_sync first if the trainer has stepped since. Entry 0 of the result is always 0, because the first token has no tokens before it. The same placeholder appears in prompt_logprobs when you pass prompt_logprobs=True to sample. The cookbook’s train_single_turn adds the rest of the recipe on top of this loop: a cosine learning-rate schedule, dynamic sampling, the overlong penalty, length budgets, validation, and periodic checkpoints.

Dynamic sampling

When too many groups collapse to identical rewards, a step can be left with little to train on. Dynamic sampling, from the DAPO paper, keeps drawing fresh batches and discarding zero-variance groups until the step has batch_size usable groups or reaches the attempt cap. To turn it on, set max_sampling_attempts above 1. The default is 1, so dynamic sampling is off unless you set it. The GSM8K recipe in the quickstart sets 3. Each attempt samples a full batch, so a step that uses all its attempts costs max_sampling_attempts batches of generation. If no attempt produces a group with varied rewards, the step has nothing to train on. The cookbook skips it. Single-turn runs log skipped: 0/<groups> groups had non-zero reward variance, and multi-turn runs log samples=0. Dynamic sampling doesn’t address why groups collapse. If zero_variance_skip_ratio stays high, also look at task difficulty and temperature.

Clipping

GRPO uses clipped importance sampling to keep any single step from moving the policy too far. For every token, the service computes the ratio between the trainer’s probability of that token and the generator’s probability when it sampled the token, then clamps the ratio to a band. A positive-advantage token whose ratio exceeds clip_high_threshold gets no gradient, because the model already favors it enough. A negative-advantage token whose ratio falls below clip_low_threshold gets no gradient, because the model already disfavors it enough. The thresholds are absolute bounds on the ratio, not offsets from 1.0, so a DAPO epsilon pair (ε_low, ε_high) maps to (1 - ε_low, 1 + ε_high). Move either bound toward 1.0 to clip harder. The cookbook defaults to 0.8 and 1.28, the asymmetric “Clip-Higher” band from DAPO. Its ceiling sits further from 1.0 than its floor, which leaves more room to raise the probability of tokens the policy currently underweights. If your own loop leaves the thresholds unset, the service uses 0.8 and 1.2 instead. forward_backward returns the clip rates in its metrics, and the cookbook logs them as fb/loss/clip/high_fraction and fb/loss/clip/low_fraction. Service-emitted metrics explains how to read them.

Overlong responses

Two independent settings handle completions that run into the token budget, and both can be on at once:

Overlong penalty (on by default): A soft penalty from DAPO, added to the reward, that ramps up linearly over the last overlong_buffer_len tokens of the completion’s token budget. overlong_buffer_len defaults to max_sample_tokens // 8. overlong_penalty_factor scales the penalty, and setting it to 0 turns the penalty off.
Overlong masking (off by default): With overlong_masking=True, a truncated completion produces no training sample at all, but its reward still counts toward the group’s baseline. In single-turn training, if masking leaves a step with fewer than group_size samples, the cookbook skips that step.

Turning masking on doesn’t turn the penalty off. To mask without penalizing, set both overlong_masking=True and overlong_penalty_factor=0.

Length budgets

Two settings bound sequence length, and both default to values derived from max_model_length (default 2048):

max_sample_tokens: The cap on tokens generated per completion. It defaults to max_model_length.
max_prompt_length: The hard filter on single-turn prompt length. It defaults to max_model_length - max_sample_tokens, and prompts over the limit are dropped before training.

For single-turn training, always set max_sample_tokens or max_prompt_length yourself. If you leave both at their defaults, max_prompt_length works out to 0 and every prompt is dropped. With num_steps set, the run then fails after the model resource has already started billing. Without it, the run exits before provisioning and reports that no training is needed. Setting max_sample_tokens well below max_model_length also keeps a single completion from filling the whole context. Watch truncated_ratio: a value above about 30% means the budget is too tight or the model is producing runaway-length output. GLM-5.3 needs a larger budget. It always thinks, and its chat template sets reasoning effort to maximum by default. Completions truncate mid-reasoning unless you set a generous max_sample_tokens or lower the effort with thinking_effort="high" or "low". The cookbook rejects enable_thinking=False for GLM-5.3.

Variants

The cookbook’s GRPO loop samples groups and computes advantages the same way whatever loss function it trains with. These GRPOConfig settings select related algorithms:

Policy loss function: loss picks the built-in loss function that trains on the group advantages. The default is "grpo", and the others are "ppo", "cispo", "dppo", "dro", and "importance_sampling", described in Built-in loss functions. Unlike GRPO, they sum their per-token losses instead of averaging them, so their loss grows with the number of trained tokens in the batch, and settings tuned for "grpo", such as the learning rate, may need retuning.
Clip band: clip_low_threshold and clip_high_threshold apply to "grpo", "ppo", and "cispo". The cookbook sends its own defaults of 0.8 and 1.28 with all three, not the service defaults, which are [0.8, 1.2] for PPO and [0, 4] for CISPO. To get a service default, set the thresholds to it.
Metrics: The fb/loss/clip/* and fb/loss/ratio/mean metrics in Clipping come from "grpo" and "ppo". The other loss functions log their own keys under fb/, such as fb/ratio_mean, and "cispo" logs fb/clip_fraction. "cispo" uses the band to scale each token’s gradient rather than to zero it.
DRO penalty: "dro" requires dro_beta, the coefficient on its quadratic log-ratio penalty.
Loss aggregation: loss_agg_type sets how the per-token losses reduce to one number. GRPO_LOSS_AGGREGATION_TYPE_FIXED_HORIZON (the default) divides the masked sum by every token in the batch, padding included. TOKEN_MEAN divides by the trained tokens only. SEQUENCE_MEAN averages within each sequence, then across sequences, so every sequence gets equal weight whatever its length. The denominator changes the effective learning rate, so keep it fixed when you compare runs.
GSPO: To compute the importance ratio per sequence instead of per token, set ratio_type="GRPO_LOSS_RATIO_TYPE_SEQUENCE". The clip band then applies to a whole completion at once.

loss_agg_type and ratio_type apply only to "grpo", and the config raises if you change them, the clip band, or dro_beta for a loss function that doesn’t use them. The cookbook has no setting for a KL penalty against a reference policy. In your own loop, set beta above 0 on GrpoLossParams, and give every sample a reference_logprobs row. For an objective none of the built-in loss functions cover, write a custom loss function.

Built-in loss functions

In your own loop, forward_backward takes a LossConfig. Its type selects the loss function, and the matching *_params field configures it. Setting a params field that belongs to a different loss function fails client-side. Each loss function reads its own rows from every sample’s loss_fn_inputs. The API reference lists each params field and its defaults.

`type`	Params	Sample rows required	Use
`LOSS_TYPE_CROSS_ENTROPY`	`cross_entropy_params`	`target_tokens`, `weights`	SFT. Weighted next-token cross-entropy. `weights` may be fractional.
`LOSS_TYPE_GRPO`	`grpo_params`	`target_tokens`, `advantages`, `logprobs`, plus `reference_logprobs` when `beta > 0`	GRPO, and GSPO with `ratio_type="GRPO_LOSS_RATIO_TYPE_SEQUENCE"`.
`LOSS_TYPE_PPO`	`ppo_params`	`target_tokens`, `advantages`, `logprobs`	Clipped surrogate with a token-level ratio and no KL term. The default clip range is `[0.8, 1.2]`.
`LOSS_TYPE_CISPO`	`cispo_params`	`target_tokens`, `advantages`, `logprobs`	Clipped ratio applied as a detached coefficient, so no token’s gradient is zeroed. The default clip range is `[0, 4]`.
`LOSS_TYPE_DRO`	`dro_params`	`target_tokens`, `advantages`, `logprobs`	Policy gradient plus a quadratic log-ratio penalty. `beta` is required.
`LOSS_TYPE_IMPORTANCE_SAMPLING`	None	`target_tokens`, `advantages`, `logprobs`	Unclipped `-(ratio * advantage)`.
`LOSS_TYPE_DPPO`	`dppo_params`	`target_tokens`, `advantages`, `logprobs`	Drops tokens whose probability moved past `delta_low` or `delta_high` in probability space, and trains the rest with the plain importance-sampling gradient. Both thresholds default to `0.15`.

Every built-in loss function except cross-entropy is advantage-based, and these loss functions also read an optional weights row as a mask of 0 or 1. If you omit it, every position trains, including the prompt. To train on the completion only, pass 1 on completion tokens and 0 everywhere else. Only cross-entropy accepts fractional weights.

Async training

Lockstep and async GRPO timelines. In lockstep, the default with max_steps_off_policy=None, the generators and the trainer take turns: the trainer is idle during rollouts, and the generators are idle during each optimizer step, so each side waits on the other. In async, with max_steps_off_policy=N, the generators roll out continuously with policy versions v1 through v4 while the trainer's steps produce v2 through v5 alongside them. A group trains until it is N steps old, then is dropped as stale.

Collection and training run in lockstep by default. Every rollout in a step finishes before the optimizer runs, so the generators sit idle during the forward and backward pass, and the trainer sits idle during sampling. Async training overlaps the two. The cookbook supports async training for multi-turn GRPO (train_multi_turn), where long, variable-length episodes make the idle gaps worst. train_single_turn rejects max_steps_off_policy. Three settings control async training:

max_steps_off_policy: Turns on async collection. None (the default) means lockstep. An integer starts a pool of rollout workers that keep collecting through the optimizer step, and sets how many optimizer steps old a group may be and still train. 0 is allowed and means async collection that trains only on groups started under the current policy.
groups_in_flight: The number of rollout groups the workers collect at once. It defaults to batch_size.
weight_sync_type: How new weights reach the generators after each step. WEIGHT_SYNC_TYPE_SYNCHRONOUS (the default) publishes inline. WEIGHT_SYNC_TYPE_BACKGROUND_PUBLISH hands the publish to a server-side worker and returns immediately. WEIGHT_SYNC_TYPE_PIPELINE also defers the publish and lets a generation already in flight finish under the new weights.

Async collection works with any weight-sync mode. The two deferred modes require max_steps_off_policy, so the drift they allow is bounded and measured.

WEIGHT_SYNC_TYPE_PIPELINE requires full-weight training (use_lora=False). The server rejects it on a low-rank adapter (LoRA) session, and the cookbook fails before provisioning when the two are combined.

To run async training, pass a configuration like this one to train_multi_turn, along with tasks and an environment factory as in Multi-turn environments:

from post_training_cookbook.grpo.config import GRPOConfig

config = GRPOConfig(
    model="Qwen/Qwen3.6-35B-A3B",
    max_turns=6,
    batch_size=4,
    episodes_per_task=8,
    max_steps_off_policy=1,
    groups_in_flight=8,
    num_generator_replicas=2,
)

Start with max_steps_off_policy=1. At 0, a worker’s next group is often one version behind by the time it arrives, so it gets dropped. To tell whether the overlap is paying off, watch the async collection metrics, starting with async/train_wait_s and async/dropped_stale_groups. Stale groups are counted separately from zero-variance groups, so they don’t inflate zero_variance_skip_ratio. For a runnable example, see recipes/grpo/tool_use_game_async.py.

Troubleshooting

zero_variance_skip_ratio stays above 50%: Most groups score identically. To tell “all right” from “all wrong”, check identical_nonzero_reward_skip_ratio, which only single-turn runs log. If the task is too easy, use a harder dataset or raise temperature. If it’s too hard, use an easier dataset or a stronger base model. You can also raise group_size to 16, or set max_sampling_attempts above 1 to recover the step at the cost of extra sampling.
The run fails at the end with You need to complete at least one training step before you can create an inference checkpoint: Every step was skipped, so nothing trained, but you’re still billed for the sampling. Before you rerun, check the step log for skipped: lines (single-turn) or samples=0 (multi-turn) to find the cause. If no group had varied rewards, adjust the task difficulty as the zero_variance_skip_ratio entry describes. If overlong masking left too few samples, adjust the length budget.
Accuracy stays at zero: The reward function probably can’t parse the model’s output. Run it by hand on a sampled completion, raise temperature, and confirm the few-shot examples match the format the reward expects.
fb/loss/ratio/mean drifts away from 1.0: Lower learning_rate and check that weights_sync runs after every step. Service-emitted metrics covers the system-side causes to rule out. With a loss function other than "grpo" or "ppo", watch fb/ratio_mean instead.
fb/loss/ratio/mean is far from 1.0 on the first step: Before the first update, the trainer and generator hold the same weights, so the ratio should be close to 1.0. In a loop that builds samples by hand, a ratio far from 1.0 this early usually means logprobs doesn’t line up with target_tokens. Check that you dropped the first value of every per-token array, as described in Build samples by hand. A misaligned advantages array doesn’t change this metric.
Out of memory or timeouts: Reduce batch_size, max_model_length, or group_size, or raise timeout on the config.
An operation fails because the session isn’t active: The session expired after an hour without an operation, was stopped, or hit an error. See Troubleshooting in Manage model resources.

Next steps

Reward functions

Bring your own dataset and reward to the loop.

Multi-turn environments

Train tool-use agents over whole episodes.

Monitoring

Every metric the loop logs and how to act on it.

Cookbook reference

Every GRPOConfig field and its default.

GET STARTED

SERVERLESS

INFERENCE APIS

DEDICATED MODEL INFERENCE

DEDICATED CONTAINER INFERENCE

POST-TRAINING

FINE-TUNING

GPU CLUSTERS

CODE EXECUTION

ADMINISTRATION

Reinforcement learning with GRPO

Requirements

How post-training runs GRPO

Run a GRPO recipe

Rewards and advantages

Write your own GRPO loop

Dynamic sampling

Clipping

Overlong responses

Length budgets

Variants

Built-in loss functions

Async training

Troubleshooting

Next steps

Reward functions

Multi-turn environments

Monitoring

Cookbook reference

​Requirements

​How post-training runs GRPO

​Run a GRPO recipe

​Rewards and advantages

​Write your own GRPO loop

​Dynamic sampling

​Clipping

​Overlong responses

​Length budgets

​Variants

​Built-in loss functions

​Async training

​Troubleshooting

​Next steps

Reward functions

Multi-turn environments

Monitoring

Cookbook reference

Requirements

How post-training runs GRPO

Run a GRPO recipe

Rewards and advantages

Write your own GRPO loop

Dynamic sampling

Clipping

Overlong responses

Length budgets

Variants

Built-in loss functions

Async training

Troubleshooting

Next steps