> ## Documentation Index
> Fetch the complete documentation index at: https://docs.together.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Direct preference optimization
> Train on preference pairs with direct preference optimization, the worked example for any loss function you run client-side.
Direct preference optimization (DPO) trains a model on pairs of responses to the same prompt, where one response is preferred over the other. The model learns to raise the likelihood of the chosen response relative to the rejected one, measured against a frozen reference copy of itself. Unlike GRPO, DPO never generates its own responses during training. Use it when you have preference data, such as human ratings, reward-model scores, or comparisons between two systems.
DPO trains with a [custom loss function](/docs/post-training/custom-loss-functions) that runs on your own machine. [Fine-tuning jobs](/docs/fine-tuning/preference-tuning) also offer DPO, with a fixed data format and no code to write. For help choosing between the two, see [Post-training vs. standard fine-tuning](/docs/post-training/overview#post-training-vs-standard-fine-tuning).
## How post-training runs DPO
DPO runs as a custom loss function, which splits the work between Together AI GPUs and your local process:
1. **`forward`** scores a batch without accumulating gradients and returns the model's per-token log-probabilities of the target tokens. Each step opens with this pass. Before training starts, the loop also runs it over every training and validation pair to snapshot the reference policy (the model before any updates).
2. **Your local process** computes the loss in PyTorch from the live log-probabilities and the reference snapshot, then calls `.backward()` to get the gradient of the loss with respect to each token's log-probability.
3. **`custom_forward_backward`** sends those per-token gradients to the model resource's trainer, which runs the forward pass again and backpropagates them through the model to accumulate parameter gradients.
4. **`optim_step`** applies the accumulated gradients, as in every other loop.
[Write a custom loss function](/docs/post-training/custom-loss-functions) covers this pattern for any objective, including which call to make for the first pass and the other details your loss function and loop must get right.
## Run the DPO recipe
The cookbook's DPO loop trains on [HelpSteer3](https://huggingface.co/datasets/nvidia/HelpSteer3) preference pairs:
```python theme={null}
from post_training_cookbook.dpo.config import DPOConfig
from post_training_cookbook.dpo.train import train
config = DPOConfig(
model="Qwen/Qwen3.8-27B",
enable_thinking=False,
dataset_name="nvidia/HelpSteer3",
dataset_split="train",
dataset_subsample=1600,
batch_size=8,
learning_rate=5e-6,
beta=0.1,
max_length=4096,
max_steps=200,
precompute_ref_batch_size=512,
)
train(config)
```
Run it with `uv run python recipes/dpo/helpsteer3.py`. The `train` function provisions a trainer-only model resource for `model`, attaches one session to it, and stops both when it returns. `DPOConfig` has no `model_resources_id` field, so to train DPO on a model resource that's already running, attach to it from the SDK as described in [Manage model resources](/docs/post-training/model-resources#attach-to-an-existing-resource).
Settings worth knowing:
* **`beta`:** The temperature on the implicit reward. Higher values keep the policy closer to the reference.
* **`enable_thinking`:** Set it to `False` on Qwen models for preference data without reasoning, such as HelpSteer3. Every Qwen model in the catalog turns thinking on by default. With thinking on, the chat template can't render these pairs, so `train` raises a `ValueError` before it provisions anything. GLM-5.3 always thinks and renders these pairs without the setting, so leave `enable_thinking` unset for it, because the cookbook rejects `enable_thinking=False` on GLM-5.3.
* **`label_smoothing`:** Conservative-DPO smoothing. Off by default.
* **`learning_rate`:** The recipe default is `5e-6`, on a cosine schedule. DPO uses lower learning rates than SFT.
* **`precompute_ref_batch_size`:** The batch size for the reference log-probability forward passes that run before training, and for each validation pass. Lower it if `forward` runs out of memory on long pairs.
* **`val_interval`:** The number of steps between validation runs on held-out pairs.
The [cookbook reference](/docs/post-training/cookbook#dpoconfig) lists every `DPOConfig` field.
## Prepare preference pairs
Each example is a pair of full conversations that share everything up to the final assistant turn, including the prompt. One ends in the chosen response, and the other ends in the rejected response. To build a pair, call the cookbook's `build_preference_pair`. It renders both sides with the model's chat template and returns two SFT-style samples, with loss masks over the response tokens:
```python theme={null}
from post_training_cookbook.codecs import get_codec
from post_training_cookbook.dpo.samples import build_preference_pair
codec = get_codec("zai-org/GLM-5.3")
context = [{"role": "user", "content": "Explain recursion in one sentence."}]
chosen = context + [
{
"role": "assistant",
"content": "Recursion is when a function solves a problem by calling itself on a smaller version of it.",
}
]
rejected = context + [
{"role": "assistant", "content": "Recursion is a programming concept."}
]
pair = build_preference_pair(
chosen_messages=chosen,
rejected_messages=rejected,
codec=codec,
max_length=4096,
)
if pair is not None:
chosen_sample, rejected_sample = pair
```
The function returns `None` when `max_length` leaves either side with no trained token. Skip those pairs, because DPO needs both sides. It raises a `ValueError` when the chat template can't render a side, for the same reason as the `enable_thinking` setting above. For that data, pass `enable_thinking=False` to `get_codec`.
The training loop interleaves pairs into a flat batch, `[chosen_0, rejected_0, chosen_1, rejected_1, ...]`, so even indexes hold chosen responses and odd indexes hold their rejected counterparts.
### Use your own preference dataset
`train` turns each dataset row into a pair with its `to_pair` argument, a function that returns `(chosen_messages, rejected_messages)` for a row, or `None` to skip the row. The default, `helpsteer3_pair` in `post_training_cookbook.dpo.train`, picks the chosen side from HelpSteer3's `overall_preference` score and skips ties. For another preference dataset, set `dataset_name` to it and pass your own function. This one reads `prompt`, `chosen`, and `rejected` text columns:
```python theme={null}
def preference_pair(row):
context = [{"role": "user", "content": row["prompt"]}]
chosen = context + [{"role": "assistant", "content": row["chosen"]}]
rejected = context + [{"role": "assistant", "content": row["rejected"]}]
return chosen, rejected
train(config, to_pair=preference_pair)
```
Validation loads the dataset's `val_split`, which defaults to `"validation"`. If your dataset has no split by that name, set `val_split` to a held-out split it has, or set `val_interval=0` to turn validation off. Don't set `val_split` to the training split. Validation then scores pairs the run trains on, which inflates `val/preference_accuracy`.
## Write your own DPO loop
When none of the cookbook recipes fit your use case, you can write your own training loop. This loop trains on an in-memory list of preference pairs:
```python theme={null}
from post_training_cookbook.codecs import get_codec
from post_training_cookbook.custom_loss import (
compute_custom_forward_backward,
compute_logprobs,
)
from post_training_cookbook.dpo.loss import make_dpo_loss
from post_training_cookbook.dpo.samples import build_preference_pair
from post_training_cookbook.samples import extract_loss_mask
from together.post_training import (
AdamParams,
ComputeConfig,
LoraConfig,
ModelResourcesClient,
)
MODEL = "Qwen/Qwen3.8-27B"
PAIRS_PER_STEP = 8
REF_CHUNK_SIZE = 64
pairs = load_pairs() # a list of (chosen_messages, rejected_messages)
# Load the model's chat template, with thinking off for data without reasoning
codec = get_codec(MODEL, enable_thinking=False)
# Render each pair into a chosen sample and a rejected sample, interleaved
# so even indexes hold chosen responses and odd indexes hold rejected ones
samples = []
for chosen, rejected in pairs:
pair = build_preference_pair(
chosen_messages=chosen,
rejected_messages=rejected,
codec=codec,
max_length=4096,
)
# Skip pairs that truncation left with no response tokens
if pair is not None:
samples.extend(pair)
# The loss mask on each sample covers only its final response
masks = [extract_loss_mask(sample) for sample in samples]
# Provision a trainer-only model resource (DPO never samples)
with ModelResourcesClient.create(
base_model=MODEL,
compute_config=ComputeConfig(
gpu_type="B200-SXM", num_generator_replicas=0
),
) as resources:
# Start a training session that trains a LoRA adapter
with resources.create_session(
lora_config=LoraConfig(rank=16, alpha=16)
) as session:
# Before any update, score every sample to snapshot the reference
# policy. Keep each sample's summed logprob over its response.
ref_sums = []
for start in range(0, len(samples), REF_CHUNK_SIZE):
chunk = samples[start : start + REF_CHUNK_SIZE]
chunk_masks = masks[start : start + REF_CHUNK_SIZE]
for logprobs, mask in zip(
compute_logprobs(session, chunk), chunk_masks
):
ref_sums.append(sum(lp * m for lp, m in zip(logprobs, mask)))
# Each step trains on the next PAIRS_PER_STEP pairs. The step size is
# even, so a pair's two samples always land in the same batch.
batch_size = 2 * PAIRS_PER_STEP
for step, start in enumerate(range(0, len(samples), batch_size)):
batch = samples[start : start + batch_size]
# Bind the DPO loss to this batch's reference sums and masks
loss_fn = make_dpo_loss(
ref_sums[start : start + batch_size],
masks[start : start + batch_size],
beta=0.1,
)
# Score the batch, compute the loss and its gradients in PyTorch,
# and send the gradients to the trainer to accumulate
result = compute_custom_forward_backward(session, batch, loss_fn)
# Update the adapter's weights
session.trainer.optim_step(
adam_params=AdamParams(learning_rate=5e-6, weight_decay=0.1),
)
# The service doesn't report a loss for a custom loss function,
# so this is the only record of it
print(f"[step {step}] loss={result.loss:.4f}")
# Save the adapter as an inference checkpoint you can deploy
checkpoint = session.create_inference_checkpoint()
print(checkpoint.id)
```
The loop uses two cookbook helpers. `make_dpo_loss` builds the loss function for one batch, bound to that batch's reference sums and response masks. `compute_custom_forward_backward` runs `forward`, PyTorch autograd, and `custom_forward_backward` around it.
The loss function reduces each sample's per-token log-probabilities to a masked sum, forms the chosen-minus-rejected margin relative to the reference, and returns the mean of `-log σ(β · margin)` over the batch's pairs. To add conservative-DPO smoothing, pass `label_smoothing` to `make_dpo_loss`.
`compute_logprobs` scores everything you pass it in one forward pass, so the loop snapshots the reference in chunks of `REF_CHUNK_SIZE` samples. Lower it if `forward` runs out of memory on long pairs. The recipe does the same with `precompute_ref_batch_size`, and also adds a cosine learning-rate schedule, validation, and periodic checkpoints.
## Validate with preference accuracy
DPO has no token-level accuracy, so the recipe validates with implicit-reward preference accuracy instead. Every `val_interval` steps, it reports the fraction of held-out pairs where the policy's reward shift prefers the chosen response. Watch `val/preference_accuracy` and `val/mean_margin`, which should both rise. At step 0, preference accuracy reads `0`, because the policy still equals the reference and ties count as wrong. For details, see [DPO metrics](/docs/post-training/monitoring#dpo-metrics).
## Next steps
Warm up a model on demonstrations before preference training.
Read the DPO validation metrics.
Adapt the DPO pattern to your own objective.
Deploy the trained model.
This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.