Supervised fine-tuning

Supervised fine-tuning (SFT) trains a model to imitate example responses. Use it when you have high-quality demonstrations of the behavior you want: human-written answers, a target output format, or conversations from a system that already works. The SFT quickstart walks through a complete run. This page explains how SFT runs on post-training, how to bring your own data or write your own loop, and how to hand the trained adapter to a GRPO run.

Fine-tuning jobs also run SFT, with a fixed data format and no training loop to write. For help choosing between the two, see Post-training vs. standard fine-tuning.

How post-training runs SFT

SFT runs on a trainer-only model resource. Nothing samples from the model, so the resource runs with num_generator_replicas=0. Each step runs two operations:

forward_backward with LOSS_TYPE_CROSS_ENTROPY, which computes the weighted next-token cross-entropy over the batch and accumulates gradients.
optim_step, which applies the accumulated gradients with the step’s learning rate.

The next step trains directly on the updated weights.

Run an SFT recipe

The cookbook’s SFTConfig and train function handle dataset loading, tokenization, loss masks, the training loop, and checkpoints. This example trains Qwen/Qwen3.8-27B on the No Robots dataset, with every other setting at its default:

from post_training_cookbook.sft.config import SFTConfig
from post_training_cookbook.sft.train import train

config = SFTConfig(
    model="Qwen/Qwen3.8-27B",
    enable_thinking=False,
    dataset_name="HuggingFaceH4/no_robots",
    max_steps=200,
    wandb_project="sft-no-robots",
)

train(config)

train provisions a trainer-only model resource for model, attaches one session to it, and stops both when it returns. To train on a resource that’s already running, set model_resources_id instead of model. See Manage model resources. The config sets enable_thinking=False because No Robots conversations contain no reasoning, as Data for reasoning models explains. The config also sets wandb_project, so train logs to Weights & Biases. Export WANDB_API_KEY before you run it, or set WANDB_MODE=offline to log locally. Without either, the run fails after it provisions the model resource, or, in an interactive terminal, waits at a W&B login prompt while the resource bills. Settings worth adjusting first:

num_epochs and max_steps: train makes num_epochs passes over the dataset, one by default, and reshuffles the rows before each pass. max_steps caps the total number of steps, which keeps a test run short. Its default, None, sets no cap, so a config without it trains on every full batch: about 1,187 steps for No Robots at a batch_size of 8. The learning rate decays linearly toward zero over the run.
learning_rate: The right rate depends on the model and the task. For a low-rank adapter (LoRA) run, 1e-4 to 5e-4 is typical. Full-weight training needs a lower rate. The cookbook’s No Robots recipes use 2e-5 for full-weight training and 1e-4 for LoRA on the same model.
train_on_what: TRAIN_ON_ALL trains on every assistant turn. TRAIN_ON_LAST trains on only the final one, which suits multi-turn datasets where only the last response matters.
batch_size: Larger batches (16 to 32) produce smoother loss curves.
use_lora=False: Trains the full weights instead of an adapter. Not every model supports full-weight training, so check Supported models first. See LoRA vs. full-weight training.

The cookbook reference lists every SFTConfig field.

Prepare your data

train accepts any Hugging Face dataset with a messages column. In each row, messages is a list of {"role": ..., "content": ...} dicts:

{
  "messages": [
    {"role": "user", "content": "Write a haiku about programming."},
    {
      "role": "assistant",
      "content": "Code flows like water\nBugs hide in the morning light\nTests bring peace of mind"
    }
  ]
}

For a dataset with other columns, set dataset_name to it and pass a to_messages function to train. It maps one row to its list of messages. Put any instruction you want the model to learn, such as an answer format, in that mapping:

def qa_messages(row):
    return [
        {"role": "user", "content": row["question"]},
        {"role": "assistant", "content": row["answer"]},
    ]


train(config, to_messages=qa_messages)

Before it provisions a model resource or opens a session, train calls to_messages on the first row, so a schema mismatch, such as a missing messages column, fails before you’re billed. For data that isn’t on Hugging Face, save the rows as a JSON Lines file named train.jsonl, one row per line, in a folder of its own. Then set dataset_name to the folder’s path, such as "./my_data", and keep dataset_split="train". train loads the folder with the Hugging Face datasets library, which reads train.jsonl into the train split. A relative path resolves from the directory you run the recipe in.

Data for reasoning models

Two chat-template behaviors decide whether your rows train:

Turn thinking off for data without reasoning: Every Qwen model in the catalog turns thinking on by default. For conversations without <think> blocks, such as No Robots, set enable_thinking=False. With thinking on, the template renders each prompt differently from the full conversation, so train skips every row. The run then fails with nothing trained, after it has already provisioned the model resource. GLM-5.3 doesn’t need the setting: its template always thinks and renders these rows as they are, and the cookbook rejects enable_thinking=False for it.
Avoid consecutive assistant turns on GLM models: The GLM template ends an assistant turn with the role marker of the message after it, so an assistant turn followed by another assistant turn has no stop token. train skips those rows with a warning. Merge or split the turns before training.

For conversations that do contain reasoning, put each assistant turn’s reasoning in a reasoning_content field next to content. Qwen3.8-27B’s template reads reasoning only from that field, so train skips rows that carry it as <think> tags inside content. preserve_thinking controls whether earlier turns keep their reasoning when the history is rendered. See Thinking settings.

Loss masks

SFT doesn’t train on every token equally. The model’s chat template wraps each message in role headers and special tokens, and a loss mask gives each position a weight that sets how much it counts toward the loss:

Role headers, including the assistant’s (<|im_start|>assistant in Qwen’s chat template), and user content get weight 0.
Assistant content and the assistant’s end-of-turn token (<|im_end|> in Qwen’s template) get weight 1. Training on the end-of-turn token teaches the model when to stop.

The cookbook’s render_conversation builds the mask from the chat template and your train_on_what choice. Weights are float32, so you can down-weight some turns with fractional values instead of masking them out. Concepts explains how the mask lines up with the input and target tokens.

Write your own SFT loop

When none of the cookbook recipes fit your use case, you can write your own training loop. This loop trains on an in-memory list of conversations:

from post_training_cookbook.codecs import get_codec
from post_training_cookbook.samples import (
    build_cross_entropy_sample,
    is_trainable,
)
from post_training_cookbook.sft.train import TRAIN_ON_ALL, render_conversation
from together.post_training import (
    AdamParams,
    ComputeConfig,
    LoraConfig,
    LossConfig,
    ModelResourcesClient,
)

MODEL = "Qwen/Qwen3.8-27B"
BATCH_SIZE = 8
conversations = load_conversations()  # a list of message lists

# Load the model's chat template, with thinking off for data without reasoning
codec = get_codec(MODEL, enable_thinking=False)

# Provision a trainer-only model resource (SFT never samples)
with ModelResourcesClient.create(
    base_model=MODEL,
    compute_config=ComputeConfig(
        gpu_type="B200-SXM", num_generator_replicas=0
    ),
) as resources:

    # Start a training session that trains a LoRA adapter
    with resources.create_session(
        lora_config=LoraConfig(rank=16, alpha=16)
    ) as session:

        # Each step trains on the next BATCH_SIZE conversations
        for step, start in enumerate(range(0, len(conversations), BATCH_SIZE)):

            # Render each conversation into tokens and a loss mask
            # that trains only the assistant turns
            samples = []
            for messages in conversations[start : start + BATCH_SIZE]:
                for seq in render_conversation(
                    messages,
                    codec=codec,
                    train_on_what=TRAIN_ON_ALL,
                    max_length=4096,
                ):
                    # Skip sequences that truncation left with no trained token
                    if not is_trainable(seq.loss_mask):
                        continue
                    samples.append(
                        build_cross_entropy_sample(
                            tokens=seq.tokens, loss_mask=seq.loss_mask
                        )
                    )

            # Compute the cross-entropy loss and accumulate gradients
            result = session.trainer.forward_backward(
                samples=samples,
                loss=LossConfig(type="LOSS_TYPE_CROSS_ENTROPY"),
            )

            # Update the adapter's weights
            session.trainer.optim_step(
                adam_params=AdamParams(learning_rate=1e-4, weight_decay=0.1),
            )

            # The loss is summed over the trained tokens, not averaged
            print(f"[step {step}] loss_sum={result.loss:.1f}")

        # Save the adapter as an inference checkpoint you can deploy
        checkpoint = session.create_inference_checkpoint()
        print(checkpoint.id)

render_conversation returns a list of sequences, not a single one, because it splits a conversation wherever the chat template strips earlier reasoning. build_cross_entropy_sample shifts the tokens into input and target positions for you. For cross-entropy, result.loss is the sum over all trained positions, not a mean. Divide it by the batch’s total weight to get the per-token mean the recipes log. To train with a variant of cross-entropy, such as one with an added regularization term, write a custom loss function. Each step then uses forward and custom_forward_backward instead of forward_backward.

Chain SFT and GRPO

To continue an SFT adapter with GRPO, hand it over with a training checkpoint. Combine methods explains when this helps. The handoff takes three settings:

Set training_checkpoint_interval so the SFT run writes a training checkpoint at its last step. The cookbook writes one only when the interval fires, and train returns the latest checkpoint ID (or None).
Start the GRPO run with resume_from_checkpoint_id set to that ID and load_optimizer=False. The new session loads the adapter weights with a fresh optimizer and its step counter at zero.
Use the same LoRA rank and alpha in both phases. A checkpoint with a different rank fails to load, and a different alpha silently changes how strongly the adapter applies.

Because SFT is trainer-only and GRPO needs a generator, the two phases run in separate sessions, typically on a new model resource for the same base model:

from dataclasses import replace

from post_training_cookbook.datasets.registry import DATASETS
from post_training_cookbook.grpo.config import GRPOConfig
from post_training_cookbook.grpo.train import train_single_turn
from post_training_cookbook.sft.config import SFTConfig
from post_training_cookbook.sft.train import train as sft_train

MODEL = "Qwen/Qwen3.8-27B"


def gsm8k_messages(row):
    # Pose each question the way the GRPO phase does, so SFT teaches
    # the "#### <answer>" format that the GRPO reward reads.
    return [
        {"role": "user", "content": DATASETS["gsm8k"].format_question(row)},
        {"role": "assistant", "content": row["answer"]},
    ]


sft_config = SFTConfig(
    model=MODEL,
    dataset_name="openai/gsm8k",
    dataset_config="main",
    max_steps=20,
    enable_thinking=False,
    lora_rank=16,
    lora_alpha=16,
    training_checkpoint_interval=20,
)
grpo_config = GRPOConfig(
    model=MODEL,
    dataset="gsm8k",
    enable_thinking=False,
    num_steps=20,
    max_model_length=1536,
    max_prompt_length=768,
    max_sample_tokens=768,
    lora_rank=16,
    lora_alpha=16,
)

checkpoint_id = sft_train(sft_config, to_messages=gsm8k_messages)
train_single_turn(
    replace(
        grpo_config,
        resume_from_checkpoint_id=checkpoint_id,
        load_optimizer=False,
    )
)

GSM8K rows have question and answer columns, not messages, so the SFT phase maps each row with to_messages. The mapping poses each question in the GRPO prompt format, so SFT teaches the answer format that GRPO then scores. The cookbook’s recipes/sft_then_grpo.py is the complete worked example. Unlike SFTConfig, GRPOConfig logs to Weights & Biases by default, so export WANDB_API_KEY or WANDB_MODE=offline, or pass wandb_project=None to GRPOConfig, before you run this example.

Next steps

Direct preference optimization

Train on preference pairs with a custom loss function.

Reinforcement learning with GRPO

Continue the adapter with reinforcement learning.

Cookbook reference

Every SFTConfig field and its default.

Deploy a checkpoint

Deploy the trained model on dedicated model inference.

GET STARTED

SERVERLESS

INFERENCE APIS

DEDICATED MODEL INFERENCE

DEDICATED CONTAINER INFERENCE

POST-TRAINING

FINE-TUNING

GPU CLUSTERS

CODE EXECUTION

ADMINISTRATION

How post-training runs SFT

Run an SFT recipe

Prepare your data

Data for reasoning models

Loss masks

Write your own SFT loop

Chain SFT and GRPO

Next steps

Direct preference optimization

Reinforcement learning with GRPO

Cookbook reference

Deploy a checkpoint

​How post-training runs SFT

​Run an SFT recipe

​Prepare your data

​Data for reasoning models

​Loss masks

​Write your own SFT loop

​Chain SFT and GRPO

​Next steps

Direct preference optimization

Reinforcement learning with GRPO

Cookbook reference

Deploy a checkpoint

How post-training runs SFT

Run an SFT recipe

Prepare your data

Data for reasoning models

Loss masks

Write your own SFT loop

Chain SFT and GRPO

Next steps