Direct preference optimization

Direct preference optimization (DPO) trains a model on pairs of responses to the same prompt, where one response is preferred over the other. The model learns to raise the likelihood of the chosen response relative to the rejected one, measured against a frozen reference copy of itself. Unlike GRPO, DPO never generates its own responses during training. Use it when you have preference data, such as human ratings, reward-model scores, or comparisons between two systems.

DPO trains with a custom loss function that runs on your own machine. Fine-tuning jobs also offer DPO, with a fixed data format and no code to write. For help choosing between the two, see Post-training vs. standard fine-tuning.

How post-training runs DPO

DPO runs as a custom loss function, which splits the work between Together AI GPUs and your local process:

forward scores a batch without accumulating gradients and returns the model’s per-token log-probabilities of the target tokens. Each step opens with this pass. Before training starts, the loop also runs it over every training and validation pair to snapshot the reference policy (the model before any updates).
Your local process computes the loss in PyTorch from the live log-probabilities and the reference snapshot, then calls .backward() to get the gradient of the loss with respect to each token’s log-probability.
custom_forward_backward sends those per-token gradients to the model resource’s trainer, which runs the forward pass again and backpropagates them through the model to accumulate parameter gradients.
optim_step applies the accumulated gradients, as in every other loop.

Write a custom loss function covers this pattern for any objective, including which call to make for the first pass and the other details your loss function and loop must get right.

Run the DPO recipe

The cookbook’s DPO loop trains on HelpSteer3 preference pairs:

from post_training_cookbook.dpo.config import DPOConfig
from post_training_cookbook.dpo.train import train

config = DPOConfig(
    model="Qwen/Qwen3.8-27B",
    enable_thinking=False,
    dataset_name="nvidia/HelpSteer3",
    dataset_split="train",
    dataset_subsample=1600,
    batch_size=8,
    learning_rate=5e-6,
    beta=0.1,
    max_length=4096,
    max_steps=200,
    precompute_ref_batch_size=512,
)

train(config)

Run it with uv run python recipes/dpo/helpsteer3.py. The train function provisions a trainer-only model resource for model, attaches one session to it, and stops both when it returns. DPOConfig has no model_resources_id field, so to train DPO on a model resource that’s already running, attach to it from the SDK as described in Manage model resources. Settings worth knowing:

beta: The temperature on the implicit reward. Higher values keep the policy closer to the reference.
enable_thinking: Set it to False on Qwen models for preference data without reasoning, such as HelpSteer3. Every Qwen model in the catalog turns thinking on by default. With thinking on, the chat template can’t render these pairs, so train raises a ValueError before it provisions anything. GLM-5.3 always thinks and renders these pairs without the setting, so leave enable_thinking unset for it, because the cookbook rejects enable_thinking=False on GLM-5.3.
label_smoothing: Conservative-DPO smoothing. Off by default.
learning_rate: The recipe default is 5e-6, on a cosine schedule. DPO uses lower learning rates than SFT.
precompute_ref_batch_size: The batch size for the reference log-probability forward passes that run before training, and for each validation pass. Lower it if forward runs out of memory on long pairs.
val_interval: The number of steps between validation runs on held-out pairs.

The cookbook reference lists every DPOConfig field.

Prepare preference pairs

Each example is a pair of full conversations that share everything up to the final assistant turn, including the prompt. One ends in the chosen response, and the other ends in the rejected response. To build a pair, call the cookbook’s build_preference_pair. It renders both sides with the model’s chat template and returns two SFT-style samples, with loss masks over the response tokens:

from post_training_cookbook.codecs import get_codec
from post_training_cookbook.dpo.samples import build_preference_pair

codec = get_codec("zai-org/GLM-5.3")

context = [{"role": "user", "content": "Explain recursion in one sentence."}]
chosen = context + [
    {
        "role": "assistant",
        "content": "Recursion is when a function solves a problem by calling itself on a smaller version of it.",
    }
]
rejected = context + [
    {"role": "assistant", "content": "Recursion is a programming concept."}
]

pair = build_preference_pair(
    chosen_messages=chosen,
    rejected_messages=rejected,
    codec=codec,
    max_length=4096,
)
if pair is not None:
    chosen_sample, rejected_sample = pair

The function returns None when max_length leaves either side with no trained token. Skip those pairs, because DPO needs both sides. It raises a ValueError when the chat template can’t render a side, for the same reason as the enable_thinking setting above. For that data, pass enable_thinking=False to get_codec. The training loop interleaves pairs into a flat batch, [chosen_0, rejected_0, chosen_1, rejected_1, ...], so even indexes hold chosen responses and odd indexes hold their rejected counterparts.

Use your own preference dataset

train turns each dataset row into a pair with its to_pair argument, a function that returns (chosen_messages, rejected_messages) for a row, or None to skip the row. The default, helpsteer3_pair in post_training_cookbook.dpo.train, picks the chosen side from HelpSteer3’s overall_preference score and skips ties. For another preference dataset, set dataset_name to it and pass your own function. This one reads prompt, chosen, and rejected text columns:

def preference_pair(row):
    context = [{"role": "user", "content": row["prompt"]}]
    chosen = context + [{"role": "assistant", "content": row["chosen"]}]
    rejected = context + [{"role": "assistant", "content": row["rejected"]}]
    return chosen, rejected


train(config, to_pair=preference_pair)

Validation loads the dataset’s val_split, which defaults to "validation". If your dataset has no split by that name, set val_split to a held-out split it has, or set val_interval=0 to turn validation off. Don’t set val_split to the training split. Validation then scores pairs the run trains on, which inflates val/preference_accuracy.

Write your own DPO loop

When none of the cookbook recipes fit your use case, you can write your own training loop. This loop trains on an in-memory list of preference pairs:

from post_training_cookbook.codecs import get_codec
from post_training_cookbook.custom_loss import (
    compute_custom_forward_backward,
    compute_logprobs,
)
from post_training_cookbook.dpo.loss import make_dpo_loss
from post_training_cookbook.dpo.samples import build_preference_pair
from post_training_cookbook.samples import extract_loss_mask
from together.post_training import (
    AdamParams,
    ComputeConfig,
    LoraConfig,
    ModelResourcesClient,
)

MODEL = "Qwen/Qwen3.8-27B"
PAIRS_PER_STEP = 8
REF_CHUNK_SIZE = 64
pairs = load_pairs()  # a list of (chosen_messages, rejected_messages)

# Load the model's chat template, with thinking off for data without reasoning
codec = get_codec(MODEL, enable_thinking=False)

# Render each pair into a chosen sample and a rejected sample, interleaved
# so even indexes hold chosen responses and odd indexes hold rejected ones
samples = []
for chosen, rejected in pairs:
    pair = build_preference_pair(
        chosen_messages=chosen,
        rejected_messages=rejected,
        codec=codec,
        max_length=4096,
    )
    # Skip pairs that truncation left with no response tokens
    if pair is not None:
        samples.extend(pair)

# The loss mask on each sample covers only its final response
masks = [extract_loss_mask(sample) for sample in samples]

# Provision a trainer-only model resource (DPO never samples)
with ModelResourcesClient.create(
    base_model=MODEL,
    compute_config=ComputeConfig(
        gpu_type="B200-SXM", num_generator_replicas=0
    ),
) as resources:

    # Start a training session that trains a LoRA adapter
    with resources.create_session(
        lora_config=LoraConfig(rank=16, alpha=16)
    ) as session:

        # Before any update, score every sample to snapshot the reference
        # policy. Keep each sample's summed logprob over its response.
        ref_sums = []
        for start in range(0, len(samples), REF_CHUNK_SIZE):
            chunk = samples[start : start + REF_CHUNK_SIZE]
            chunk_masks = masks[start : start + REF_CHUNK_SIZE]
            for logprobs, mask in zip(
                compute_logprobs(session, chunk), chunk_masks
            ):
                ref_sums.append(sum(lp * m for lp, m in zip(logprobs, mask)))

        # Each step trains on the next PAIRS_PER_STEP pairs. The step size is
        # even, so a pair's two samples always land in the same batch.
        batch_size = 2 * PAIRS_PER_STEP
        for step, start in enumerate(range(0, len(samples), batch_size)):
            batch = samples[start : start + batch_size]

            # Bind the DPO loss to this batch's reference sums and masks
            loss_fn = make_dpo_loss(
                ref_sums[start : start + batch_size],
                masks[start : start + batch_size],
                beta=0.1,
            )

            # Score the batch, compute the loss and its gradients in PyTorch,
            # and send the gradients to the trainer to accumulate
            result = compute_custom_forward_backward(session, batch, loss_fn)

            # Update the adapter's weights
            session.trainer.optim_step(
                adam_params=AdamParams(learning_rate=5e-6, weight_decay=0.1),
            )

            # The service doesn't report a loss for a custom loss function,
            # so this is the only record of it
            print(f"[step {step}] loss={result.loss:.4f}")

        # Save the adapter as an inference checkpoint you can deploy
        checkpoint = session.create_inference_checkpoint()
        print(checkpoint.id)

The loop uses two cookbook helpers. make_dpo_loss builds the loss function for one batch, bound to that batch’s reference sums and response masks. compute_custom_forward_backward runs forward, PyTorch autograd, and custom_forward_backward around it. The loss function reduces each sample’s per-token log-probabilities to a masked sum, forms the chosen-minus-rejected margin relative to the reference, and returns the mean of -log σ(β · margin) over the batch’s pairs. To add conservative-DPO smoothing, pass label_smoothing to make_dpo_loss. compute_logprobs scores everything you pass it in one forward pass, so the loop snapshots the reference in chunks of REF_CHUNK_SIZE samples. Lower it if forward runs out of memory on long pairs. The recipe does the same with precompute_ref_batch_size, and also adds a cosine learning-rate schedule, validation, and periodic checkpoints.

Validate with preference accuracy

DPO has no token-level accuracy, so the recipe validates with implicit-reward preference accuracy instead. Every val_interval steps, it reports the fraction of held-out pairs where the policy’s reward shift prefers the chosen response. Watch val/preference_accuracy and val/mean_margin, which should both rise. At step 0, preference accuracy reads 0, because the policy still equals the reference and ties count as wrong. For details, see DPO metrics.

Next steps

Supervised fine-tuning

Warm up a model on demonstrations before preference training.

Monitoring

Read the DPO validation metrics.

Write a custom loss function

Adapt the DPO pattern to your own objective.

Deploy a checkpoint

Deploy the trained model.

GET STARTED

SERVERLESS

INFERENCE APIS

DEDICATED MODEL INFERENCE

DEDICATED CONTAINER INFERENCE

POST-TRAINING

FINE-TUNING

GPU CLUSTERS

CODE EXECUTION

ADMINISTRATION

Direct preference optimization

How post-training runs DPO

Run the DPO recipe

Prepare preference pairs

Use your own preference dataset

Write your own DPO loop

Validate with preference accuracy

Next steps

Supervised fine-tuning

Monitoring

Write a custom loss function

Deploy a checkpoint

​How post-training runs DPO

​Run the DPO recipe

​Prepare preference pairs

​Use your own preference dataset

​Write your own DPO loop

​Validate with preference accuracy

​Next steps

Supervised fine-tuning

Monitoring

Write a custom loss function

Deploy a checkpoint

How post-training runs DPO

Run the DPO recipe

Prepare preference pairs

Use your own preference dataset

Write your own DPO loop

Validate with preference accuracy

Next steps