> ## Documentation Index
> Fetch the complete documentation index at: https://docs.together.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Run DPO recipes

> Run the cookbook's direct preference optimization recipe on HelpSteer3, or train on your own preference pairs.

Run [direct preference optimization (DPO)](https://docs.together.ai/docs/post-training/dpo.md) with the recipe from the [post-training cookbook](https://docs.together.ai/docs/post-training/cookbook-overview.md), on HelpSteer3 preference pairs or on your own.

## Requirements

* A [cookbook checkout](https://docs.together.ai/docs/post-training/cookbook-overview.md#clone-to-run-recipes).
* A [supported model](https://docs.together.ai/docs/post-training/supported-models.md). DPO doesn't need generator support.
* Your own preference dataset, unless you train on HelpSteer3. Each row needs a prompt and two responses to it, one preferred over the other.

## Run a DPO recipe

Pass a `DPOConfig` to the cookbook's `train` function, which handles dataset loading, the reference-policy snapshot, the training loop, validation, and checkpoints. To train on your own pairs, [use your own preference dataset](#use-your-own-preference-dataset).

The DPO recipe, `recipes/dpo/helpsteer3.py`, trains Qwen3.8 27B on [HelpSteer3](https://huggingface.co/datasets/nvidia/HelpSteer3) preference pairs:

```python theme={null}
from post_training_cookbook.dpo.config import DPOConfig
from post_training_cookbook.dpo.train import train

config = DPOConfig(
    model="Qwen/Qwen3.8-27B",
    enable_thinking=False,
    dataset_name="nvidia/HelpSteer3",
    dataset_split="train",
    dataset_subsample=1600,
    batch_size=8,
    learning_rate=5e-6,
    beta=0.1,
    max_length=4096,
    max_steps=200,
    precompute_ref_batch_size=512,
)

train(config)
```

Run the recipe from your cookbook checkout:

```bash theme={null}
uv run python recipes/dpo/helpsteer3.py
```

`train` provisions a trainer-only model resource and one session, then stops both when it returns. `DPOConfig` has no `model_resources_id` field, so to train on a resource that's already running, [attach to it through the SDK](https://docs.together.ai/docs/post-training/model-resources.md#attach-to-an-existing-resource).

The config sets `enable_thinking=False` because HelpSteer3 pairs contain no reasoning, and Qwen3.8 27B needs thinking off for such pairs. With thinking on, `train` raises `ValueError` before it provisions the model resource. To train a different model, check [how its chat template handles thinking](https://docs.together.ai/docs/post-training/supported-models.md#thinking-and-chat-templates) and set `enable_thinking` to match.

Settings worth adjusting first:

* **`beta`:** The temperature on the implicit reward. Higher values keep the policy closer to the reference.
* **`learning_rate`:** The default is `5e-6`, lower than for supervised fine-tuning (SFT). The rate follows a cosine decay over the run.
* **`label_smoothing`:** The weight the loss gives to the reversed preference, for conservative DPO. Off by default.
* **`precompute_ref_batch_size`:** The number of samples per forward pass when the recipe snapshots the reference policy and runs validation.
* **`val_interval`:** The number of steps between [validation runs on held-out pairs](https://docs.together.ai/docs/post-training/dpo.md#monitor-preference-accuracy).

The [cookbook config reference](https://docs.together.ai/docs/post-training/cookbook.md#dpoconfig) lists every `DPOConfig` field.

## Use your own preference dataset

Select your data with `dataset_name`, `dataset_config`, and `dataset_split`, which the recipe passes to the Hugging Face `datasets` loader.

By default, the recipe maps each row with `helpsteer3_pair` from `post_training_cookbook.dpo.train`, which picks the chosen response by HelpSteer3's `overall_preference` column and skips ties. For any other schema, pass `train` a `to_pair` function that maps each row to a `(chosen_messages, rejected_messages)` tuple, or to `None` to skip the row. This example maps a dataset with `prompt`, `chosen`, and `rejected` text columns:

```python theme={null}
def preference_pair(row):
    context = [{"role": "user", "content": row["prompt"]}]
    chosen = context + [{"role": "assistant", "content": row["chosen"]}]
    rejected = context + [{"role": "assistant", "content": row["rejected"]}]
    return chosen, rejected


train(config, to_pair=preference_pair)
```

Validation reads the dataset's `"validation"` split by default. If your dataset has no such split, set `val_split` to another held-out split, or turn validation off with `val_interval=0`. Don't validate on the training split, because the pairs the model trained on inflate `val/preference_accuracy`.

## Resume from a checkpoint

To start a DPO recipe from a training checkpoint, set `resume_from_checkpoint_id`. `DPOConfig` has no `resume_run_dir`. The recipe restores the adapter and optimizer from the checkpoint, but its learning-rate schedule and data order restart from step zero. It also snapshots the reference policy from the restored weights, so the checkpoint becomes the reference instead of the original model.

## Troubleshooting

* **The run fails before provisioning with `Nothing to train`:** The recipe skipped every row, either because `to_pair` returned `None` for it or because truncating to `max_length` left one side of its pair with no response tokens. The log reports how many it skipped for each reason. Check your `to_pair` mapping, or raise `max_length`.
* **Out of memory:** Lower `precompute_ref_batch_size`, especially if your pairs are long, or reduce `batch_size` or `max_length`.

For problems that aren't specific to recipes, see [Lifecycle, errors, and limits](https://docs.together.ai/docs/post-training/limits-and-errors.md#troubleshooting).

## Next steps

> [card] [SFT recipes](https://docs.together.ai/docs/post-training/sft-recipes.md)
>
> Warm up a model on demonstrations before preference training.

> [card] [Monitor training runs](https://docs.together.ai/docs/post-training/monitoring.md#dpo-metrics)
>
> Read the validation metrics a DPO recipe logs.

> [card] [Cookbook config reference](https://docs.together.ai/docs/post-training/cookbook.md#dpoconfig)
>
> Look up every `DPOConfig` field and its default.

> [card] [Deploy a checkpoint](https://docs.together.ai/docs/post-training/deployment.md)
>
> Deploy the trained model on dedicated model inference.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.