Run multi-turn GRPO recipes

Write an environment for a tool-use agent and train a model on it with multi-turn group relative policy optimization (GRPO), using the post-training cookbook. The cookbook’s train_multi_turn function runs episodes against your environment, turns them into training samples, and trains on them.

Requirements

A cookbook checkout, which provides the environment abstractions in post_training_cookbook.envs.
A model with generator support. Models marked trainer-only in the catalog can’t run GRPO.
Your own tasks and environment, unless you train on the cookbook’s bundled tool-use environment.
Weights & Biases (W&B) logging set up or turned off. GRPOConfig logs to W&B by default, and the example on this page sets wandb_project, so without a W&B setup, a run fails after it provisions the model resource. Export WANDB_API_KEY, set WANDB_MODE=offline to log locally, or pass wandb_project=None to turn logging off.

Write an environment

Subclass Env and implement its four members: reset, step, tool_specs, and messages. The environment owns the conversation history, tool dispatch, and the rule for when an episode ends. The training loop owns tokenization and sampling. The following environment gives the model a calculator tool. It ends the episode when the model replies without a tool call, and scores that reply against the task’s target:

import json

from post_training_cookbook.envs import (
    Env,
    StepResult,
    Task,
    Tool,
    ToolResult,
    tool_to_spec,
)
from post_training_cookbook.types import Message, ToolSpec

SYSTEM_PROMPT = (
    "Use the calculator tool, then reply with only the final number."
)


class CalculatorTool:
    name = "calculator"
    description = "Evaluate an arithmetic expression."
    parameters_schema = {
        "type": "object",
        "properties": {"expression": {"type": "string"}},
        "required": ["expression"],
    }

    async def execute(self, arguments: str) -> ToolResult:
        expression = json.loads(arguments)["expression"]
        # safe_eval is your own arithmetic evaluator. Define it before you
        # train. Until you do, every call returns a tool_error, because the
        # step's except catches the NameError. Never eval() model output.
        return ToolResult(content=str(safe_eval(expression)))


class MathEnv(Env):
    def __init__(self) -> None:
        self._messages: list[Message] = []
        self._task: Task | None = None
        self._tools: dict[str, Tool] = {"calculator": CalculatorTool()}

    async def reset(self, task: Task) -> list[Message]:
        self._task = task
        self._messages = [
            Message(role="system", content=SYSTEM_PROMPT),
            Message(role="user", content=task.prompt),
        ]
        return list(self._messages)

    async def step(self, message: Message) -> StepResult:
        self._messages.append(message)
        tool_calls = message.get("tool_calls", [])
        if not tool_calls:
            # No tool calls: treat the message as the final answer.
            correct = message.get("content", "").strip() == str(
                self._task.target
            )
            return StepResult(
                terminated=True,
                truncated=False,
                messages=list(self._messages),
                reward=1.0 if correct else 0.0,
            )
        for call in tool_calls:
            name = call["function"]["name"]
            try:
                tool = self._tools[name]
                result = await tool.execute(call["function"]["arguments"])
                content = result.content
            except Exception as exc:
                # Report a bad tool call to the model. An uncaught
                # exception here stops the whole training run.
                content = f"tool_error: {exc!r}"
            self._messages.append(
                Message(
                    role="tool",
                    content=content,
                    tool_call_id=call["id"],
                    name=name,
                )
            )
        return StepResult(
            terminated=False,
            truncated=False,
            messages=list(self._messages),
            reward=0.0,
        )

    @property
    def tool_specs(self) -> list[ToolSpec]:
        return [tool_to_spec(tool) for tool in self._tools.values()]

    @property
    def messages(self) -> list[Message]:
        return list(self._messages)

The example uses these types from post_training_cookbook.envs:

Task: The spec for one episode, with prompt, target, and metadata fields.
Tool: Any object with name, description, parameters_schema, and an async execute(arguments) -> ToolResult method. tool_to_spec converts a tool into the schema that the model’s chat template renders into the prompt.
ToolResult: The return value of execute. Set should_stop=True to ask the environment to end the episode. Your step decides how to honor the request, for example by returning terminated=True, as the cookbook’s reference environment does.
StepResult: The return value of step. terminated means task logic ended the episode, for example on a final answer. truncated means the episode was cut short for a reason other than task logic, such as a turn limit the environment enforces. The loop reads reward and metrics only from the step that ends the episode, so put diagnostics in the final step’s metrics. Intermediate steps usually return a reward of 0.0.

Override the optional close() method when reset starts background work, such as a subprocess or a connection. The loop calls close() whenever an episode ends, including an episode the loop truncates before step returns a terminal result. To build your own environment, start from a copy of the cookbook’s reference environment, ToolUseGameEnv in post_training_cookbook/envs/tool_use_game/. It provides three tools (list_fact_keys, lookup, and calculator) and a bundled task list. The training example trains on it.

Run multi-turn training

Call train_multi_turn with a config that sets max_turns, a list of tasks, and a factory that builds a fresh environment for each episode. The cookbook’s recipes/grpo/tool_use_game.py recipe trains on the reference environment’s bundled tasks:

from post_training_cookbook.envs.tool_use_game import (
    TOOL_USE_GAME_TASKS,
    ToolUseGameEnv,
)
from post_training_cookbook.grpo.config import GRPOConfig
from post_training_cookbook.grpo.train import train_multi_turn

tasks = list(TOOL_USE_GAME_TASKS)

config = GRPOConfig(
    model="Qwen/Qwen3.8-27B",
    batch_size=4,
    episodes_per_task=8,
    num_steps=200,
    validation_interval=10,
    max_turns=6,
    max_model_length=2048,
    learning_rate=2e-5,
    wandb_project="grpo-tool-use-game",
    sequential_validation=True,
    max_sampling_attempts=2,
)

train_multi_turn(
    config,
    tasks=tasks,
    env_factory=lambda task: ToolUseGameEnv(),
    val_tasks=tasks,
)

To train on your own environment, pass your Task list as tasks and a factory such as lambda task: MathEnv() as env_factory. The GRPOConfig reference lists the settings specific to multi-turn training. The shared GRPO settings for length budgets, overlong handling, dynamic sampling, and loss variants also apply to multi-turn training.

Handle truncated episodes

An episode is truncated when it ends without the environment returning terminated=True. The cookbook adds the soft overlong penalty to a truncated episode’s reward only when the episode ran out of context or turns:

Cause	What happened	Overlong penalty applies
`CONTEXT_OVERFLOW`	The conversation filled `max_model_length` before the episode ended.	Yes.
`MAX_TURNS`	The episode hit `max_turns` without terminating.	Yes.
`ENV`	The environment’s `step` returned `truncated=True`.	No.
`PARSE_FAILURE`	The sampled message couldn’t be parsed. The episode ends with `failed_parse_reward`.	No.

To leave truncated episodes out of training, set overlong_masking=True. The cookbook then builds no samples from truncated episodes of any cause, but their rewards still count toward the group baseline. To track how episodes end, watch the multi-turn metrics in the step log: terminated_ratio, truncated_ratio, context_overflow_ratio, and parse_failure_ratio.

Async training

By default, the cookbook runs in lockstep. Rollouts finish before training starts, so the generators sit idle during training and the trainer sits idle during sampling. Async training overlaps rollout collection with training.

Lockstep alternates sampling and training, leaving each side idle. Async overlaps rollout collection and optimizer steps, admitting groups within the configured policy-age limit and dropping older groups as stale.

The cookbook supports async training only for multi-turn runs, where long, variable-length episodes create larger idle gaps. Async rollout collection runs in the cookbook’s loop, not on the post-training service, so a loop you write with the SDK has it only if you implement it. Three settings control async training:

max_steps_off_policy: An integer turns on continuous rollout workers and sets the maximum age of a group, in optimizer steps since its rollout started. Older groups are dropped as stale. The default, None, keeps lockstep. 0 admits only groups started under the current policy, but groups often arrive one version behind and get dropped, so start with 1.
groups_in_flight: The number of rollout groups collected concurrently. It defaults to batch_size.
weight_sync_type: How new weights reach the generators after each step.
- WEIGHT_SYNC_TYPE_SYNCHRONOUS, the default, publishes the weights inline.
- WEIGHT_SYNC_TYPE_BACKGROUND_PUBLISH defers publication to a server worker and returns immediately.
- WEIGHT_SYNC_TYPE_PIPELINE also defers publication and lets generation already in flight finish under the updated weights.

Async training works with all three sync modes, but in the cookbook the two deferred modes require it, because max_steps_off_policy bounds and measures the drift they allow. WEIGHT_SYNC_TYPE_PIPELINE also requires full-weight training (use_lora=False), because the server rejects it on adapter sessions. The cookbook checks for this before provisioning. To turn on async training, pass a config like the following to train_multi_turn with your tasks and an environment factory, as in the training example:

from post_training_cookbook.grpo.config import GRPOConfig

config = GRPOConfig(
    model="Qwen/Qwen3.6-35B-A3B",
    max_turns=6,
    batch_size=4,
    episodes_per_task=8,
    max_steps_off_policy=1,
    groups_in_flight=8,
    num_generator_replicas=2,
)

For a complete script, run the cookbook’s recipes/grpo/tool_use_game_async.py recipe. To track how long each step waits for rollouts and how many groups go stale, watch async/train_wait_s and async/dropped_stale_groups in the async collection metrics. Stale groups are counted separately, so they don’t inflate zero_variance_skip_ratio.

Next steps

Tune a GRPO recipe

Adjust length budgets, overlong handling, dynamic sampling, and loss variants.

Monitor training runs

Read the multi-turn and async metrics a run logs.

Evaluate models

Score a hosted model on your environment before you train.

Cookbook config reference

Look up every multi-turn GRPOConfig field and its default.

GET STARTED

SERVERLESS

INFERENCE APIS

DEDICATED MODEL INFERENCE

DEDICATED CONTAINER INFERENCE

POST-TRAINING

FINE-TUNING

GPU CLUSTERS

CODE EXECUTION

ADMINISTRATION

Requirements

Write an environment

Run multi-turn training

Handle truncated episodes

Async training

Next steps

Tune a GRPO recipe

Monitor training runs

Evaluate models

Cookbook config reference

​Requirements

​Write an environment

​Run multi-turn training

​Handle truncated episodes

​Async training

​Next steps

Tune a GRPO recipe

Monitor training runs

Evaluate models

Cookbook config reference

Requirements

Write an environment

Run multi-turn training

Handle truncated episodes

Async training

Next steps