> ## Documentation Index > Fetch the complete documentation index at: https://docs.together.ai/llms.txt > Use this file to discover all available pages before exploring further. # Monitor training runs > Track sessions and model resources in the console, and read the run artifacts and Weights & Biases metrics that show whether a training run is healthy. A post-training run reports in three places. The Together AI console shows the state of your model resources and sessions. The cookbook writes every step to a run directory and, when the config sets `wandb_project`, to Weights & Biases. Each `forward_backward` response also carries service-side metrics for the loss it computed. This page covers what to look at in each place and how to act on what you see. ## The console The [Post-training](https://api.together.ai/custom-training) page in the console has two tabs: * **Training sessions:** Every session in the project, filtered by default to the ones you created. Open a session to see its status, step counter, checkpoints, and the Weights & Biases run it reports to. * **Model resources:** Active resources with their status, hourly rate, and estimated spend, with past resources listed below them. Create a resource here, or stop one that's still running. Statuses on both tabs match the API's [lifecycle states](/docs/post-training/model-resources#lifecycle). ## Run artifacts Every cookbook recipe writes a record of the run to `runs//`: ```text theme={null} runs// config.yaml the config, round-trippable through .from_yaml manifest.json W&B URL, session ID, model, start time, git SHA metrics.jsonl one JSON line per step, mirroring what goes to W&B checkpoints.jsonl training and inference checkpoint IDs step_NNNN_*.json per-step completions or episodes, when export_samples=True val_NNNN_*.json validation completions, when export_samples=True ``` The cookbook always writes `metrics.jsonl`, so you can analyze a run offline without W\&B. The per-step completion dumps are the fastest way to see what the model is producing. ## Weights & Biases When a config sets `wandb_project`, the cookbook logs every step to that project. `GRPOConfig` defaults it to `post-training-cookbook`, and the SFT and DPO configs leave it unset. If `WANDB_API_KEY` isn't set, `wandb.init` raises an error, or waits at a login prompt in an interactive terminal, after the model resource is already provisioned. To run without a W\&B account, set `WANDB_MODE=offline` to keep the W\&B logs on disk, or set `wandb_project=None` to turn W\&B off. The cookbook also forwards `wandb_project`, `wandb_entity`, `wandb_group`, and `wandb_run_name` to the session as metadata, and the console shows them on the session's page. The cookbook doesn't send the W\&B run ID or URL, so the console shows these fields without a link to the run. The run's URL is in `manifest.json`. To compare related runs side by side, such as the arms of a sweep, give them the same `wandb_group`. ### What to watch first | Question | Metric | | - | - | | Is training making progress? | `val/accuracy` (single-turn) or `val/reward_mean` (multi-turn), then per-step `accuracy` or `reward_mean`. | | Is the system on-policy? | `fb/loss/ratio/mean`, which should sit near `1.0`. | | Are updates being clipped too hard? | `fb/loss/clip/high_fraction`, `fb/loss/clip/low_fraction`. | | Is the reward separating completions? | `zero_variance_skip_ratio`, `reward_std`, `completion_advantage_std`. | | Are completions hitting the length budget? | `truncated_ratio`, `mean_sampled_response_length`. | | Is the model reward-hacking? | `accuracy` rising while `val/accuracy` falls or `val/mean_response_length` climbs. | | Is SFT learning? | `loss`, which should fall. | | Is DPO separating chosen from rejected? | `val/preference_accuracy`, `val/mean_margin`. | ## GRPO metrics ### Service-emitted metrics Every `forward_backward` call with `LOSS_TYPE_GRPO` returns a `metrics` dict, which the cookbook logs under the `fb/` prefix. These are the health metrics for the update itself. | Metric | What it measures | How to read it | | - | - | - | | `fb/loss/ratio/mean` | Mean importance ratio over trained tokens: the trainer's probability of a token divided by the generator's. | Near `1.0` is healthy. Persistent drift to `0.85` or `1.2` usually points to a system-side issue, such as weight skew between the trainer and generator or stale rollouts. Rule these out before suspecting your reward. Continuous growth means the policy is moving faster than the generator's log-probabilities can stay fresh: lower `learning_rate`. | | `fb/loss/kl_policy/mean`, `fb/loss/kl_policy/k3/mean` | Approximate KL between the trainer and generator policies, from two different estimators. | Near `0` when on-policy. Spikes alongside ratio drift signal over-aggressive updates. | | `fb/loss/clip/high_fraction` | Fraction of trained tokens that have a positive advantage and a ratio above `clip_high_threshold`, so their gradient was suppressed. | Under 1% to 2% is healthy. Rising while accuracy plateaus means the model can't push already-likely tokens further. Above 10% means updates are being aggressively suppressed: lower `learning_rate` or widen the clip band. | | `fb/loss/clip/low_fraction` | The same measure for tokens that have a negative advantage and a ratio below `clip_low_threshold`. | Same thresholds as `high_fraction`. | | `fb/loss/clip/clipped_ratio/mean` | Mean of the clamped ratio. | Context for the two clip fractions. | | `fb/loss/kl_ref/mean` | KL to the reference policy. Emitted only when `beta > 0`. | A slow rise is expected and means the KL term is working. Spiking alongside ratio drift means the step was too large. | | `fb/loss/aggregate/active_fraction` | Fraction of the submitted tokens, prompts included, that contributed to the loss. Padding doesn't count. | A sharp drop means completions are short relative to their prompts, or many carry zero loss weight. | `loss` itself is a poor progress signal for GRPO. It can be flat, slightly negative, or non-monotonic while reward improves. Watch reward and accuracy for progress, and the metrics above for stability. ### Cookbook metrics The cookbook computes these after sampling and scoring. If you write your own loop, compute the equivalents yourself. | Metric | What it measures | How to read it | | - | - | - | | `accuracy` | Mean of the dataset's metric (or reward) over the completions in the groups that trained. Zero-variance groups are left out. | Should trend upward. Read `val/accuracy` for the unbiased number. | | `reward_mean`, `reward_std` | Mean and spread of `raw_reward + overlong_penalty`. | `reward_mean` below `accuracy` means the overlong penalty is lowering reward. Both cover only the groups that trained, so skipped groups don't pull `reward_std` toward zero. | | `zero_variance_skip_ratio` | Fraction of groups skipped because every completion scored the same. | Under 30% is healthy. Consistently above 50% means the task is currently too easy or too hard for the model. | | `identical_nonzero_reward_skip_ratio` | The subset of skipped groups where every completion succeeded. | A large value means the model has saturated this slice of the data. Near zero with a high skip ratio means the model fails every time. | | `truncated_ratio` | Fraction of completions cut off at the length budget. | Above 30% means the budget is too tight or the model is producing length-runaway output. | | `mean_sampled_response_length` | Average number of tokens the model produced. | Staying near `max_sample_tokens` means the model is running into the limit. | | `mean_response_budget` | Average token budget per completion: `max_sample_tokens`, or less when a long prompt leaves less room in `max_model_length`. Only single-turn runs log this. | A budget well below `max_sample_tokens` means long prompts are squeezing completions. | | `mean_trainable_response_length`, `post_budget_clip_ratio` | Average completion length kept for training, and the fraction of completions cut to the budget before training. Only single-turn runs log these. | `post_budget_clip_ratio` should stay near zero. | | `overlong_penalized_ratio`, `mean_overlong_penalty` | How many completions were penalized, and by how much. | A strongly negative mean penalty means the penalty is dominating reward: lower `overlong_penalty_factor` or raise `max_sample_tokens`. | | `completion_advantage_std` | Spread of the advantages that trained. | Near zero means tight reward distributions and small updates. | | `mean_logprob` | Mean per-token log-probability the generator assigned to its own samples. | Growing more negative over training can mean a temperature or exploration issue, or mode collapse onto rare tokens. | | `learning_rate`, `timing/step_*_s` | The scheduled learning rate and the wall-clock time of each phase. | Use `timing/step_total_s` to [estimate a run's cost](/docs/post-training/pricing#estimate-the-cost-of-a-training-run). | ### Dynamic sampling metrics The cookbook logs these metrics when `max_sampling_attempts` is above `1`, which turns on [dynamic sampling](/docs/post-training/grpo#dynamic-sampling). Async collection doesn't log them. | Metric | What it measures | How to read it | | - | - | - | | `dynamic_sampling/attempts` | Number of batches the step drew. | Regularly hitting `max_sampling_attempts` means most groups score identically. Look at task difficulty and `temperature` before raising the cap. | | `dynamic_sampling/total_groups_sampled` | Groups scored across all attempts, up to the one that filled the batch. The rest of that last batch isn't counted. | Every sampled group costs generation time, including discarded ones. | | `dynamic_sampling/kept_groups` | Groups with varied rewards that the step trained on. | Below `batch_size` means the step ran out of attempts and trained on a smaller batch. | ### Validation metrics Every `validation_interval` steps, the loop samples the validation set greedily and logs: * **Single-turn runs:** `val/accuracy`, `val/reward_mean`, and `val/mean_response_length`. * **Multi-turn runs:** Per-episode equivalents such as `val/reward_mean`, `val/num_turns_mean`, and `val/terminated_ratio`. `val/accuracy` is the headline metric. It should rise and then plateau. Plateaus are normal. Two patterns signal a problem: * **Reward hacking:** Validation accuracy falls while training reward keeps rising. * **Length runaway:** `val/mean_response_length` drifts upward, even when reward looks fine. ### Multi-turn metrics Multi-turn GRPO replaces per-completion stats with per-episode ones: `num_turns_mean`, `terminated_ratio` (episodes that ended on task logic), `truncated_ratio` (episodes cut short by the turn limit, context overflow, a parse failure, or the environment), `parse_failure_ratio`, and `context_overflow_ratio`. Two patterns point to specific fixes: * **Low `terminated_ratio` with high `truncated_ratio`:** Episodes run out of turns or context before finishing. Raise `max_turns` or `max_model_length`, or simplify the task. * **Rising `parse_failure_ratio`:** The model is emitting malformed tool calls. Check the system prompt and tool schemas. ### Async collection metrics The cookbook logs these metrics every step when `max_steps_off_policy` is set. Together, they show whether async collection is keeping the generators busy or throwing most of its rollouts away. | Metric | What it measures | How to read it | | - | - | - | | `async/accepted_groups` | Groups trained on this step. | Below `batch_size` means the step ran short. | | `async/staleness_mean`, `async/staleness_max` | Age of the trained groups in optimizer steps. | `staleness_max` never exceeds `max_steps_off_policy`. | | `async/dropped_stale_groups` | Groups discarded for exceeding the staleness budget. | Sustained non-zero values mean rollouts are slower than the optimizer: raise `max_steps_off_policy`, add generator replicas, or lower `groups_in_flight`. | | `async/dropped_zero_variance_groups` | Groups discarded for having no reward variance. | Hitting `batch_size * max_sampling_attempts` every step points to the reward, not to async. | | `async/queue_depth` | Completed groups waiting when the step began. | Persistently at `groups_in_flight` means collection outruns training. Persistently `0` with non-zero `train_wait_s` means training waits on collection. | | `async/train_wait_s` | Seconds the step spent blocked waiting for groups. | The direct measure of lost overlap. Near `0` is the goal. | | `async/mid_rollout_drift_ratio` | Fraction of generated tokens produced by a policy other than the one the group was scheduled against. | Non-zero whenever a publish lands during a rollout. | For the service's view of the policy versions, read `policy_state` from `session.retrieve()`: * `trainer_step`: The version the last optimizer step produced. * `target_weights_version`: The version the latest `weights_sync` promised the generators. * `applied_weights_version`: The version the generators are serving. ## SFT metrics | Metric | What it measures | | - | - | | `loss` | Mean cross-entropy per trained token. It's a real progress signal, unlike GRPO loss, and should fall steadily. Because it's normalized, it stays comparable across steps as the batch's token count changes. | | `loss_sum` | The unnormalized total the service returns. It scales with the batch's token count, so it isn't comparable across steps. | | `trained_tokens` | The weight mass of the batch's loss mask, which is the denominator of `loss`. | | `learning_rate` | The learning rate, which decays linearly. | | `samples` | Sequences trained this step. It can differ from `batch_size`, because a conversation splits into several sequences when its chat template [strips earlier reasoning](/docs/post-training/concepts#reasoning-in-conversation-history), and rows the template can't render are skipped. | ## DPO metrics Each step, DPO logs `loss`, which should trend down, and `pairs`. Every `val_interval` steps, validation reports: | Metric | What it measures | | - | - | | `val/preference_accuracy` | Fraction of held-out pairs where the policy's implicit reward prefers the chosen response. The headline DPO signal. | | `val/mean_chosen_delta`, `val/mean_rejected_delta` | Mean shift in each side's log-probability versus the reference. | | `val/mean_margin` | Mean `chosen_delta - rejected_delta`. It should turn positive and grow as the policy separates chosen from rejected responses. | ## Next steps The settings each metric points you toward. Turn `timing/step_total_s` into a cost estimate. Save the run once the metrics look right. The lifecycle states the console shows. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.