Skip to main content
Run direct preference optimization (DPO) with the recipe from the post-training cookbook, on HelpSteer3 preference pairs or on your own.

​
Requirements

  • A cookbook checkout.
  • A supported model. DPO doesn’t need generator support.
  • Your own preference dataset, unless you train on HelpSteer3. Each row needs a prompt and two responses to it, one preferred over the other.

​
Run a DPO recipe

Pass a DPOConfig to the cookbook’s train function, which handles dataset loading, the reference-policy snapshot, the training loop, validation, and checkpoints. To train on your own pairs, use your own preference dataset. The DPO recipe, recipes/dpo/helpsteer3.py, trains Qwen3.8 27B on HelpSteer3 preference pairs:
Run the recipe from your cookbook checkout:
train provisions a trainer-only model resource and one session, then stops both when it returns. DPOConfig has no model_resources_id field, so to train on a resource that’s already running, attach to it through the SDK. The config sets enable_thinking=False because HelpSteer3 pairs contain no reasoning, and Qwen3.8 27B needs thinking off for such pairs. With thinking on, train raises ValueError before it provisions the model resource. To train a different model, check how its chat template handles thinking and set enable_thinking to match. Settings worth adjusting first:
  • beta: The temperature on the implicit reward. Higher values keep the policy closer to the reference.
  • learning_rate: The default is 5e-6, lower than for supervised fine-tuning (SFT). The rate follows a cosine decay over the run.
  • label_smoothing: The weight the loss gives to the reversed preference, for conservative DPO. Off by default.
  • precompute_ref_batch_size: The number of samples per forward pass when the recipe snapshots the reference policy and runs validation.
  • val_interval: The number of steps between validation runs on held-out pairs.
The cookbook config reference lists every DPOConfig field.

​
Use your own preference dataset

Select your data with dataset_name, dataset_config, and dataset_split, which the recipe passes to the Hugging Face datasets loader. By default, the recipe maps each row with helpsteer3_pair from post_training_cookbook.dpo.train, which picks the chosen response by HelpSteer3’s overall_preference column and skips ties. For any other schema, pass train a to_pair function that maps each row to a (chosen_messages, rejected_messages) tuple, or to None to skip the row. This example maps a dataset with prompt, chosen, and rejected text columns:
Validation reads the dataset’s "validation" split by default. If your dataset has no such split, set val_split to another held-out split, or turn validation off with val_interval=0. Don’t validate on the training split, because the pairs the model trained on inflate val/preference_accuracy.

​
Resume from a checkpoint

To start a DPO recipe from a training checkpoint, set resume_from_checkpoint_id. DPOConfig has no resume_run_dir. The recipe restores the adapter and optimizer from the checkpoint, but its learning-rate schedule and data order restart from step zero. It also snapshots the reference policy from the restored weights, so the checkpoint becomes the reference instead of the original model.

​
Troubleshooting

  • The run fails before provisioning with Nothing to train: The recipe skipped every row, either because to_pair returned None for it or because truncating to max_length left one side of its pair with no response tokens. The log reports how many it skipped for each reason. Check your to_pair mapping, or raise max_length.
  • Out of memory: Lower precompute_ref_batch_size, especially if your pairs are long, or reduce batch_size or max_length.
For problems that aren’t specific to recipes, see Lifecycle, errors, and limits.

​
Next steps

SFT recipes

Warm up a model on demonstrations before preference training.

Monitor training runs

Read the validation metrics a DPO recipe logs.

Cookbook config reference

Look up every DPOConfig field and its default.

Deploy a checkpoint

Deploy the trained model on dedicated model inference.