> ## Documentation Index > Fetch the complete documentation index at: https://docs.together.ai/llms.txt > Use this file to discover all available pages before exploring further. # Loss functions > Choose a built-in loss function, configure clipping and the KL penalty, or compute your own loss in PyTorch. A *loss function* defines what a model learns during training. For each batch, it returns a single number, the *loss*, that measures how far the model's outputs are from what you want the model to learn. Each training step [updates the weights to lower the loss](https://docs.together.ai/learn/how-llms-work.md#what-training-actually-does). Every loss function in post-training computes the loss from the model's *log-probabilities*. A token's log-probability is the logarithm of the probability the model assigns to that token, given the tokens before it. If the loss goes down as a token's log-probability goes up, training makes the model more likely to produce that token. For supervised fine-tuning (SFT) and group relative policy optimization (GRPO), the service runs a built-in loss function. Direct preference optimization (DPO) trains with a *custom loss function*, one that your own process computes in PyTorch from the model's log-probabilities. Your process then sends the gradients of the loss to the service. Cookbook recipes and your own loop train with the same loss functions. A recipe passes its loss settings to the same `forward_backward` call that your own loop makes, so the `GRPOConfig` fields in [Loss variants](https://docs.together.ai/docs/post-training/grpo-recipes.md#loss-variants) map onto the loss functions and parameters on this page. ## Built-in loss functions To use a built-in loss function in your own loop, pass a `LossConfig` to `forward_backward`. Its `type` selects the objective, and the matching `*_params` field configures it. Setting params for any other type fails client-side. Each objective reads the `loss_fn_inputs` rows listed in the table, and the [API reference](https://docs.together.ai/reference/post-training/operations-forward-backward-create.md) lists each one's params and defaults. | `type` | Params | Sample rows required | Use | | - | - | - | - | | `LOSS_TYPE_CROSS_ENTROPY` | `cross_entropy_params` | `target_tokens`, `weights` | [SFT](https://docs.together.ai/docs/post-training/sft.md). Weighted next-token cross-entropy. `weights` may be fractional. | | `LOSS_TYPE_GRPO` | `grpo_params` | `target_tokens`, `advantages`, `logprobs`, plus `reference_logprobs` when `beta > 0` | GRPO, and group sequence policy optimization (GSPO) with `ratio_type="GRPO_LOSS_RATIO_TYPE_SEQUENCE"`. | | `LOSS_TYPE_PPO` | `ppo_params` | `target_tokens`, `advantages`, `logprobs` | Clipped surrogate with a token-level ratio and no Kullback-Leibler (KL) term. The default clip range is `[0.8, 1.2]`. | | `LOSS_TYPE_CISPO` | `cispo_params` | `target_tokens`, `advantages`, `logprobs` | Clipped ratio applied as a detached coefficient, so no token's gradient is zeroed. The default clip range is `[0, 4]`. | | `LOSS_TYPE_DRO` | `dro_params` | `target_tokens`, `advantages`, `logprobs` | Policy gradient plus a quadratic log-ratio penalty. `beta` is required. | | `LOSS_TYPE_IMPORTANCE_SAMPLING` | None | `target_tokens`, `advantages`, `logprobs` | Unclipped `-(ratio * advantage)`. | | `LOSS_TYPE_DPPO` | `dppo_params` | `target_tokens`, `advantages`, `logprobs` | Drops tokens whose probability moved past `delta_low` or `delta_high` in probability space, and trains the rest with the plain importance-sampling gradient. Both thresholds default to `0.15`. | The advantage-based objectives also accept an optional binary `weights` mask. Set it to `1` on completion tokens and `0` elsewhere. Without a mask, every position trains, including the prompt. For an objective the table doesn't list, write a [custom loss function](#when-to-use-a-custom-loss-function). ### Clipping Clipping limits how far one update moves the policy. For each token, the service computes the ratio of the trainer's probability to the generator's probability at sampling time. It then suppresses gradients for positive-advantage tokens whose ratio exceeds `clip_high_threshold`, and for negative-advantage tokens whose ratio falls below `clip_low_threshold`. The GRPO, proximal policy optimization (PPO), and CISPO objectives take both thresholds. CISPO scales a clipped token's gradient instead of suppressing it. Both thresholds are absolute ratio bounds, not offsets from `1.0`, so move either one toward `1.0` for stronger clipping. Decoupled clip and dynamic sampling policy optimization (DAPO) writes its bounds as an epsilon pair (ε\_low, ε\_high), which maps to `(1 - ε_low, 1 + ε_high)`. The cookbook leaves both thresholds unset (`None`), so the service's `[0.8, 1.2]` default applies. To use DAPO's asymmetric "Clip-Higher" band, set `clip_low_threshold` to `0.8` and `clip_high_threshold` to `1.28`. The higher upper bound leaves more room to raise the probabilities of tokens the policy underweights. `forward_backward.metrics` reports [clip rates](https://docs.together.ai/docs/post-training/monitoring.md#service-emitted-metrics), which the cookbook logs as `fb/loss/clip/high_fraction` and `fb/loss/clip/low_fraction`. ### KL penalty A *KL penalty* adds a term to the GRPO loss that grows as the policy moves away from a reference policy. The penalty's weight is `beta`, so to add one in [your own loop](https://docs.together.ai/docs/post-training/training-loop.md#train-on-a-batch), set `GrpoLossParams.beta > 0` and give each sample a `reference_logprobs` row. Cookbook recipes train without a KL penalty, because `GRPOConfig` has no KL setting. ## When to use a custom loss function Write a custom loss function when no built-in loss function matches your objective. Any method can use one, not only DPO. Common uses include: * **Preference objectives:** Train on chosen and rejected pairs with DPO or another preference objective. See [Write a custom loss function](https://docs.together.ai/docs/post-training/custom-loss-functions.md) for how to snapshot the reference policy. * **Regularized SFT:** Add a term to cross-entropy, such as a penalty for drifting from the starting model. * **New reinforcement learning (RL) objectives:** Train with a policy-gradient variant that the built-in loss functions don't cover. * **Research prototypes:** Try a new objective, such as one from a recently published paper, without waiting for the service to add it. A custom loss function sees only the log-probability that the model assigns to each target token, not the model's full output distribution. You can't use one for an objective that needs the probability of every token in the vocabulary, such as an exact entropy bonus. Before you write one, check whether a simpler option fits: * **Built-in loss function:** The [built-in loss functions](#built-in-loss-functions) already cover clipped policy-gradient objectives such as PPO and CISPO, each in a single `forward_backward` call. A custom step takes longer than a built-in step on the same batch, because it runs the model's forward pass twice, once in `forward` and again in `custom_forward_backward`. * **Reward function:** If your objective is a score per completion rather than a function of the model's log-probabilities, use [GRPO with a reward function](https://docs.together.ai/docs/post-training/grpo.md#write-a-reward-function) instead. It's simpler and runs on-policy. ## How a custom loss function runs Each training step with a custom loss function splits the work between your process and the model resource: 1. **`forward`:** The model resource's trainer scores the batch without accumulating gradients and returns the model's log-probability of each target token. 2. **Your loss function:** Your process runs the loss function on those log-probabilities in PyTorch, then calls `.backward()` on the loss to get its gradient with respect to each log-probability. 3. **`custom_forward_backward`:** Your process sends those gradients to the trainer, which runs the forward pass again and backpropagates them through the model to accumulate parameter gradients. 4. **`optim_step`:** The trainer applies the accumulated gradients, as in every other loop. The model weights don't change between `forward` and `custom_forward_backward`, so the parameter gradients match what the trainer would compute if it ran your loss function itself. [Write a custom loss function](https://docs.together.ai/docs/post-training/custom-loss-functions.md) shows the code for each step. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.