Skip to main content
Evaluate a model at three points in a post-training project: score the base model before training to set a baseline, watch validation scores during a run, and compare the trained candidates to pick one for production.

​
Check a baseline before training

To see how a model performs on your task before you spend GPU time training it, run one of the cookbook’s recipes/serverless_eval/ scripts from a cookbook checkout. Each script scores a hosted chat model on a dataset’s validation split, without a training session or model resource:
The scripts cover gsm8k, deepmath, game24, and the multi-turn tool-use environment. To score your own dataset, register it, then copy a script and change its dataset. Each run writes its episodes to runs/serverless_eval_<label>_<model>_<timestamp>/episodes.json. In the output, check eval/length_capped, the fraction of episodes cut off at max_tokens. If it’s high, the score measures truncation rather than ability, so raise max_tokens, especially for a model with thinking on. The baseline isn’t directly comparable to training-time validation, because the scripts skip the codec (the cookbook’s chat-template rendering), the token budget, and the prompt-length filter that training applies. To measure what training changed, score the trained candidates with the same script.

​
Score a multi-turn environment

To get a baseline score for your own environment, call run_and_report from the cookbook’s post_training_cookbook.serverless_eval package. It runs any Env against a hosted chat model. For a worked example, see recipes/serverless_eval/tool_use_game.py.

​
Score a model on a dedicated endpoint

The scripts call serverless inference by default. To score a model that serverless inference doesn’t host, such as Qwen3.8 27B or a model you trained, deploy the model on dedicated model inference and point the script at it:
  1. Deploy the model. For Qwen3.8 27B, deploy Qwen/Qwen3.8-27B-FP8.
  2. Edit the script to set model to the endpoint string, since the scripts take no arguments. For a low-rank adapter (LoRA) attached to a multi-LoRA deployment, use the adapter’s model string instead, such as your-project-slug/my-sweep:ml_abc123.
  3. Run the script with TOGETHER_BASE_URL set to the dedicated inference host:
Stop the deployment when you’re done, because it bills while it runs.

​
Watch validation during training

The cookbook’s group relative policy optimization (GRPO) and direct preference optimization (DPO) recipes score held-out data at regular intervals during a run. The GRPO validation metrics and the DPO metrics explain what each recipe logs and which patterns signal a problem.

​
Sample a checkpoint

To read a saved checkpoint’s outputs, see Sample a saved checkpoint.

​
Compare candidates

Compare the trained candidates on the metric that matters for your task. If they’re adapters, attach them to a multi-LoRA deployment first. Then score them in one of two ways:
  • Rerun your baseline script: If you scored the base model with a serverless_eval script before training, run the same script against each candidate on a dedicated endpoint. The scores then compare directly with the baseline.
  • Run an evaluation: Evaluations can score a dedicated endpoint, or judge two models head to head with a compare evaluation. They reference a dedicated model by its endpoint ID, so use them for candidates deployed to their own endpoints, not for adapters that share a multi-LoRA deployment.
Once you’ve picked a winner, serve it on its own for production and stop the deployment.