Skip to main content
A loss function defines what a model learns during training. For each batch, it returns a single number, the loss, that measures how far the model’s outputs are from what you want the model to learn. Each training step updates the weights to lower the loss. Every loss function in post-training computes the loss from the model’s log-probabilities. A token’s log-probability is the logarithm of the probability the model assigns to that token, given the tokens before it. If the loss goes down as a token’s log-probability goes up, training makes the model more likely to produce that token. For supervised fine-tuning (SFT) and group relative policy optimization (GRPO), the service runs a built-in loss function. Direct preference optimization (DPO) trains with a custom loss function, one that your own process computes in PyTorch from the model’s log-probabilities. Your process then sends the gradients of the loss to the service. Cookbook recipes and your own loop train with the same loss functions. A recipe passes its loss settings to the same forward_backward call that your own loop makes, so the GRPOConfig fields in Loss variants map onto the loss functions and parameters on this page.

​
Built-in loss functions

To use a built-in loss function in your own loop, pass a LossConfig to forward_backward. Its type selects the objective, and the matching *_params field configures it. Setting params for any other type fails client-side. Each objective reads the loss_fn_inputs rows listed in the table, and the API reference lists each one’s params and defaults. The advantage-based objectives also accept an optional binary weights mask. Set it to 1 on completion tokens and 0 elsewhere. Without a mask, every position trains, including the prompt. For an objective the table doesn’t list, write a custom loss function.

​
Clipping

Clipping limits how far one update moves the policy. For each token, the service computes the ratio of the trainer’s probability to the generator’s probability at sampling time. It then suppresses gradients for positive-advantage tokens whose ratio exceeds clip_high_threshold, and for negative-advantage tokens whose ratio falls below clip_low_threshold. The GRPO, proximal policy optimization (PPO), and CISPO objectives take both thresholds. CISPO scales a clipped token’s gradient instead of suppressing it. Both thresholds are absolute ratio bounds, not offsets from 1.0, so move either one toward 1.0 for stronger clipping. Decoupled clip and dynamic sampling policy optimization (DAPO) writes its bounds as an epsilon pair (ε_low, ε_high), which maps to (1 - ε_low, 1 + ε_high). The cookbook leaves both thresholds unset (None), so the service’s [0.8, 1.2] default applies. To use DAPO’s asymmetric “Clip-Higher” band, set clip_low_threshold to 0.8 and clip_high_threshold to 1.28. The higher upper bound leaves more room to raise the probabilities of tokens the policy underweights. forward_backward.metrics reports clip rates, which the cookbook logs as fb/loss/clip/high_fraction and fb/loss/clip/low_fraction.

​
KL penalty

A KL penalty adds a term to the GRPO loss that grows as the policy moves away from a reference policy. The penalty’s weight is beta, so to add one in your own loop, set GrpoLossParams.beta > 0 and give each sample a reference_logprobs row. Cookbook recipes train without a KL penalty, because GRPOConfig has no KL setting.

​
When to use a custom loss function

Write a custom loss function when no built-in loss function matches your objective. Any method can use one, not only DPO. Common uses include:
  • Preference objectives: Train on chosen and rejected pairs with DPO or another preference objective. See Write a custom loss function for how to snapshot the reference policy.
  • Regularized SFT: Add a term to cross-entropy, such as a penalty for drifting from the starting model.
  • New reinforcement learning (RL) objectives: Train with a policy-gradient variant that the built-in loss functions don’t cover.
  • Research prototypes: Try a new objective, such as one from a recently published paper, without waiting for the service to add it.
A custom loss function sees only the log-probability that the model assigns to each target token, not the model’s full output distribution. You can’t use one for an objective that needs the probability of every token in the vocabulary, such as an exact entropy bonus. Before you write one, check whether a simpler option fits:
  • Built-in loss function: The built-in loss functions already cover clipped policy-gradient objectives such as PPO and CISPO, each in a single forward_backward call. A custom step takes longer than a built-in step on the same batch, because it runs the model’s forward pass twice, once in forward and again in custom_forward_backward.
  • Reward function: If your objective is a score per completion rather than a function of the model’s log-probabilities, use GRPO with a reward function instead. It’s simpler and runs on-policy.

​
How a custom loss function runs

Each training step with a custom loss function splits the work between your process and the model resource:
  1. forward: The model resource’s trainer scores the batch without accumulating gradients and returns the model’s log-probability of each target token.
  2. Your loss function: Your process runs the loss function on those log-probabilities in PyTorch, then calls .backward() on the loss to get its gradient with respect to each log-probability.
  3. custom_forward_backward: Your process sends those gradients to the trainer, which runs the forward pass again and backpropagates them through the model to accumulate parameter gradients.
  4. optim_step: The trainer applies the accumulated gradients, as in every other loop.
The model weights don’t change between forward and custom_forward_backward, so the parameter gradients match what the trainer would compute if it ran your loss function itself. Write a custom loss function shows the code for each step.