> ## Documentation Index
> Fetch the complete documentation index at: https://docs.together.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Evaluate models

> Score a baseline before training, watch validation during a run, and compare the trained candidates.

Evaluate a model at three points in a post-training project: score the base model before training to set a baseline, watch validation scores during a run, and compare the trained candidates to pick one for production.

## Check a baseline before training

To see how a model performs on your task before you spend GPU time training it, run one of the cookbook's `recipes/serverless_eval/` scripts from a [cookbook checkout](https://docs.together.ai/docs/post-training/cookbook-overview.md#clone-to-run-recipes). Each script scores a hosted chat model on a dataset's validation split, without a training session or model resource:

```bash theme={null}
uv run python recipes/serverless_eval/gsm8k.py
```

The scripts cover `gsm8k`, `deepmath`, `game24`, and the multi-turn tool-use environment. To score your own dataset, [register it](https://docs.together.ai/docs/post-training/grpo-recipes.md#register-a-dataset-and-reward), then copy a script and change its `dataset`.

Each run writes its episodes to `runs/serverless_eval_<label>_<model>_<timestamp>/episodes.json`. In the output, check `eval/length_capped`, the fraction of episodes cut off at `max_tokens`. If it's high, the score measures truncation rather than ability, so raise `max_tokens`, especially for a model with thinking on.

The baseline isn't directly comparable to training-time validation, because the scripts skip the codec (the cookbook's chat-template rendering), the token budget, and the prompt-length filter that training applies. To measure what training changed, [score the trained candidates](#compare-candidates) with the same script.

### Score a multi-turn environment

To get a baseline score for your own [environment](https://docs.together.ai/docs/post-training/multi-turn-recipes.md#write-an-environment), call `run_and_report` from the cookbook's `post_training_cookbook.serverless_eval` package. It runs any `Env` against a hosted chat model. For a worked example, see `recipes/serverless_eval/tool_use_game.py`.

### Score a model on a dedicated endpoint

The scripts call serverless inference by default. To score a model that serverless inference doesn't host, such as Qwen3.8 27B or a model you trained, deploy the model on [dedicated model inference](https://docs.together.ai/docs/dedicated-endpoints/overview.md) and point the script at it:

1. Deploy the model. For Qwen3.8 27B, deploy `Qwen/Qwen3.8-27B-FP8`.
2. Edit the script to set `model` to the endpoint string, since the scripts take no arguments. For a low-rank adapter (LoRA) attached to a [multi-LoRA deployment](https://docs.together.ai/docs/post-training/multi-lora-deployment.md), use the adapter's model string instead, such as `your-project-slug/my-sweep:ml_abc123`.
3. Run the script with `TOGETHER_BASE_URL` set to the dedicated inference host:

```bash theme={null}
export TOGETHER_BASE_URL=https://api-inference.together.ai/v1
uv run python recipes/serverless_eval/gsm8k.py
```

[Stop the deployment](https://docs.together.ai/docs/dedicated-endpoints/manage.md#stop-a-deployment) when you're done, because it bills while it runs.

## Watch validation during training

The cookbook's group relative policy optimization (GRPO) and direct preference optimization (DPO) recipes score held-out data at regular intervals during a run. The GRPO [validation metrics](https://docs.together.ai/docs/post-training/monitoring.md#validation-metrics) and the [DPO metrics](https://docs.together.ai/docs/post-training/monitoring.md#dpo-metrics) explain what each recipe logs and which patterns signal a problem.

## Sample a checkpoint

To read a saved checkpoint's outputs, see [Sample a saved checkpoint](https://docs.together.ai/docs/post-training/checkpoints.md#sample-a-saved-checkpoint).

## Compare candidates

Compare the trained candidates on the metric that matters for your task. If they're adapters, [attach them to a multi-LoRA deployment](https://docs.together.ai/docs/post-training/multi-lora-deployment.md) first. Then score them in one of two ways:

* **Rerun your baseline script:** If you scored the base model with a `serverless_eval` script before training, run the same script against each candidate [on a dedicated endpoint](#score-a-model-on-a-dedicated-endpoint). The scores then compare directly with the baseline.
* **Run an evaluation:** [Evaluations](https://docs.together.ai/docs/ai-evaluations.md) can score a dedicated endpoint, or judge two models head to head with a `compare` evaluation. They reference a dedicated model by its endpoint ID, so use them for candidates deployed to their own endpoints, not for adapters that share a multi-LoRA deployment.

Once you've picked a winner, [serve it on its own](https://docs.together.ai/docs/post-training/deployment.md#deploy-a-lora-adapter-in-merged-mode) for production and [stop the deployment](https://docs.together.ai/docs/post-training/multi-lora-deployment.md#stop-the-deployment).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.