> ## Documentation Index
> Fetch the complete documentation index at: https://docs.together.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Run multi-turn GRPO recipes

> Write a cookbook environment for a tool-use agent and train on it with multi-turn GRPO, including async training.

Write an environment for a tool-use agent and train a model on it with [multi-turn group relative policy optimization (GRPO)](https://docs.together.ai/docs/post-training/multi-turn-environments.md), using the [post-training cookbook](https://docs.together.ai/docs/post-training/cookbook-overview.md). The cookbook's `train_multi_turn` function runs episodes against your environment, turns them into training samples, and trains on them.

## Requirements

* A [cookbook checkout](https://docs.together.ai/docs/post-training/cookbook-overview.md#clone-to-run-recipes), which provides the environment abstractions in `post_training_cookbook.envs`.
* A [model with generator support](https://docs.together.ai/docs/post-training/supported-models.md#catalog). Models marked trainer-only in the catalog can't run GRPO.
* Your own tasks and environment, unless you train on the cookbook's bundled tool-use environment.
* [Weights & Biases (W\&B) logging](https://docs.together.ai/docs/post-training/cookbook-overview.md#weights-&-biases) set up or turned off. `GRPOConfig` logs to W\&B by default, and the example on this page sets `wandb_project`, so without a W\&B setup, a run fails after it provisions the model resource. Export `WANDB_API_KEY`, set `WANDB_MODE=offline` to log locally, or pass `wandb_project=None` to turn logging off.

## Write an environment

Subclass `Env` and implement its four members: `reset`, `step`, `tool_specs`, and `messages`. The environment owns the conversation history, tool dispatch, and the rule for when an episode ends. The training loop owns tokenization and sampling.

The following environment gives the model a calculator tool. It ends the episode when the model replies without a tool call, and scores that reply against the task's target:

```python theme={null}
import json

from post_training_cookbook.envs import (
    Env,
    StepResult,
    Task,
    Tool,
    ToolResult,
    tool_to_spec,
)
from post_training_cookbook.types import Message, ToolSpec

SYSTEM_PROMPT = (
    "Use the calculator tool, then reply with only the final number."
)


class CalculatorTool:
    name = "calculator"
    description = "Evaluate an arithmetic expression."
    parameters_schema = {
        "type": "object",
        "properties": {"expression": {"type": "string"}},
        "required": ["expression"],
    }

    async def execute(self, arguments: str) -> ToolResult:
        expression = json.loads(arguments)["expression"]
        # safe_eval is your own arithmetic evaluator. Define it before you
        # train. Until you do, every call returns a tool_error, because the
        # step's except catches the NameError. Never eval() model output.
        return ToolResult(content=str(safe_eval(expression)))


class MathEnv(Env):
    def __init__(self) -> None:
        self._messages: list[Message] = []
        self._task: Task | None = None
        self._tools: dict[str, Tool] = {"calculator": CalculatorTool()}

    async def reset(self, task: Task) -> list[Message]:
        self._task = task
        self._messages = [
            Message(role="system", content=SYSTEM_PROMPT),
            Message(role="user", content=task.prompt),
        ]
        return list(self._messages)

    async def step(self, message: Message) -> StepResult:
        self._messages.append(message)
        tool_calls = message.get("tool_calls", [])
        if not tool_calls:
            # No tool calls: treat the message as the final answer.
            correct = message.get("content", "").strip() == str(
                self._task.target
            )
            return StepResult(
                terminated=True,
                truncated=False,
                messages=list(self._messages),
                reward=1.0 if correct else 0.0,
            )
        for call in tool_calls:
            name = call["function"]["name"]
            try:
                tool = self._tools[name]
                result = await tool.execute(call["function"]["arguments"])
                content = result.content
            except Exception as exc:
                # Report a bad tool call to the model. An uncaught
                # exception here stops the whole training run.
                content = f"tool_error: {exc!r}"
            self._messages.append(
                Message(
                    role="tool",
                    content=content,
                    tool_call_id=call["id"],
                    name=name,
                )
            )
        return StepResult(
            terminated=False,
            truncated=False,
            messages=list(self._messages),
            reward=0.0,
        )

    @property
    def tool_specs(self) -> list[ToolSpec]:
        return [tool_to_spec(tool) for tool in self._tools.values()]

    @property
    def messages(self) -> list[Message]:
        return list(self._messages)
```

The example uses these types from `post_training_cookbook.envs`:

* **`Task`:** The spec for one episode, with `prompt`, `target`, and `metadata` fields.
* **`Tool`:** Any object with `name`, `description`, `parameters_schema`, and an `async execute(arguments) -> ToolResult` method. `tool_to_spec` converts a tool into the schema that the model's chat template renders into the prompt.
* **`ToolResult`:** The return value of `execute`. Set `should_stop=True` to ask the environment to end the episode. Your `step` decides how to honor the request, for example by returning `terminated=True`, as the cookbook's reference environment does.
* **`StepResult`:** The return value of `step`. `terminated` means task logic ended the episode, for example on a final answer. `truncated` means the episode was cut short for a reason other than task logic, such as a turn limit the environment enforces. The loop reads `reward` and `metrics` only from the step that ends the episode, so put diagnostics in the final step's `metrics`. Intermediate steps usually return a `reward` of `0.0`.

Override the optional `close()` method when `reset` starts background work, such as a subprocess or a connection. The loop calls `close()` whenever an episode ends, including an episode the loop truncates before `step` returns a terminal result.

To build your own environment, start from a copy of the cookbook's reference environment, `ToolUseGameEnv` in `post_training_cookbook/envs/tool_use_game/`. It provides three tools (`list_fact_keys`, `lookup`, and `calculator`) and a bundled task list. The [training example](#run-multi-turn-training) trains on it.

## Run multi-turn training

Call `train_multi_turn` with a config that sets `max_turns`, a list of tasks, and a factory that builds a fresh environment for each episode. The cookbook's `recipes/grpo/tool_use_game.py` recipe trains on the reference environment's bundled tasks:

```python theme={null}
from post_training_cookbook.envs.tool_use_game import (
    TOOL_USE_GAME_TASKS,
    ToolUseGameEnv,
)
from post_training_cookbook.grpo.config import GRPOConfig
from post_training_cookbook.grpo.train import train_multi_turn

tasks = list(TOOL_USE_GAME_TASKS)

config = GRPOConfig(
    model="Qwen/Qwen3.8-27B",
    batch_size=4,
    episodes_per_task=8,
    num_steps=200,
    validation_interval=10,
    max_turns=6,
    max_model_length=2048,
    learning_rate=2e-5,
    wandb_project="grpo-tool-use-game",
    sequential_validation=True,
    max_sampling_attempts=2,
)

train_multi_turn(
    config,
    tasks=tasks,
    env_factory=lambda task: ToolUseGameEnv(),
    val_tasks=tasks,
)
```

To train on your own environment, pass your `Task` list as `tasks` and a factory such as `lambda task: MathEnv()` as `env_factory`.

The `GRPOConfig` reference lists the [settings specific to multi-turn training](https://docs.together.ai/docs/post-training/cookbook.md#multi-turn-only). The [shared GRPO settings](https://docs.together.ai/docs/post-training/grpo-recipes.md#tune-a-grpo-recipe) for length budgets, overlong handling, dynamic sampling, and loss variants also apply to multi-turn training.

## Handle truncated episodes

An episode is [truncated](https://docs.together.ai/docs/post-training/multi-turn-environments.md#truncation) when it ends without the environment returning `terminated=True`. The cookbook adds the [soft overlong penalty](https://docs.together.ai/docs/post-training/grpo-recipes.md#overlong-responses) to a truncated episode's reward only when the episode ran out of context or turns:

| Cause | What happened | Overlong penalty applies |
| - | - | - |
| `CONTEXT_OVERFLOW` | The conversation filled `max_model_length` before the episode ended. | Yes. |
| `MAX_TURNS` | The episode hit `max_turns` without terminating. | Yes. |
| `ENV` | The environment's `step` returned `truncated=True`. | No. |
| `PARSE_FAILURE` | The sampled message couldn't be parsed. The episode ends with `failed_parse_reward`. | No. |

To leave truncated episodes out of training, set `overlong_masking=True`. The cookbook then builds no samples from truncated episodes of any cause, but their rewards still count toward the group baseline.

To track how episodes end, watch the [multi-turn metrics](https://docs.together.ai/docs/post-training/monitoring.md#multi-turn-metrics) in the step log: `terminated_ratio`, `truncated_ratio`, `context_overflow_ratio`, and `parse_failure_ratio`.

## Async training

By default, the cookbook runs in lockstep. Rollouts finish before training starts, so the generators sit idle during training and the trainer sits idle during sampling. Async training overlaps rollout collection with training.

![Lockstep alternates sampling and training, leaving each side idle. Async overlaps rollout collection and optimizer steps, admitting groups within the configured policy-age limit and dropping older groups as stale.](https://mintcdn.com/togetherai-52386018/Ta7hn_4Cp_DKcb3y/images/post-training/lockstep-vs-async.png?fit=max\&auto=format\&n=Ta7hn_4Cp_DKcb3y\&q=85\&s=5fa3b6a82fa344275ed9163348b703c3)

The cookbook supports async training only for multi-turn runs, where long, variable-length episodes create larger idle gaps. Async rollout collection runs in the cookbook's loop, not on the post-training service, so a loop you [write with the SDK](https://docs.together.ai/docs/post-training/training-loop.md) has it only if you implement it.

Three settings control async training:

* **`max_steps_off_policy`:** An integer turns on continuous rollout workers and sets the maximum age of a group, in optimizer steps since its rollout started. Older groups are dropped as stale. The default, `None`, keeps lockstep. `0` admits only groups started under the current policy, but groups often arrive one version behind and get dropped, so start with `1`.
* **`groups_in_flight`:** The number of rollout groups collected concurrently. It defaults to `batch_size`.
* **`weight_sync_type`:** How new weights reach the generators after each step.
  * `WEIGHT_SYNC_TYPE_SYNCHRONOUS`, the default, publishes the weights inline.
  * `WEIGHT_SYNC_TYPE_BACKGROUND_PUBLISH` defers publication to a server worker and returns immediately.
  * `WEIGHT_SYNC_TYPE_PIPELINE` also defers publication and lets generation already in flight finish under the updated weights.

Async training works with all three sync modes, but in the cookbook the two deferred modes require it, because `max_steps_off_policy` bounds and measures the drift they allow. `WEIGHT_SYNC_TYPE_PIPELINE` also requires full-weight training (`use_lora=False`), because the server rejects it on adapter sessions. The cookbook checks for this before provisioning.

To turn on async training, pass a config like the following to `train_multi_turn` with your tasks and an environment factory, as in the [training example](#run-multi-turn-training):

```python theme={null}
from post_training_cookbook.grpo.config import GRPOConfig

config = GRPOConfig(
    model="Qwen/Qwen3.6-35B-A3B",
    max_turns=6,
    batch_size=4,
    episodes_per_task=8,
    max_steps_off_policy=1,
    groups_in_flight=8,
    num_generator_replicas=2,
)
```

For a complete script, run the cookbook's `recipes/grpo/tool_use_game_async.py` recipe.

To track how long each step waits for rollouts and how many groups go stale, watch `async/train_wait_s` and `async/dropped_stale_groups` in the [async collection metrics](https://docs.together.ai/docs/post-training/monitoring.md#async-collection-metrics). Stale groups are counted separately, so they don't inflate `zero_variance_skip_ratio`.

## Next steps

> [card] [Tune a GRPO recipe](https://docs.together.ai/docs/post-training/grpo-recipes.md#tune-a-grpo-recipe)
>
> Adjust length budgets, overlong handling, dynamic sampling, and loss variants.

> [card] [Monitor training runs](https://docs.together.ai/docs/post-training/monitoring.md#multi-turn-metrics)
>
> Read the multi-turn and async metrics a run logs.

> [card] [Evaluate models](https://docs.together.ai/docs/post-training/evaluate.md#score-a-multi-turn-environment)
>
> Score a hosted model on your environment before you train.

> [card] [Cookbook config reference](https://docs.together.ai/docs/post-training/cookbook.md#multi-turn-only)
>
> Look up every multi-turn `GRPOConfig` field and its default.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.