Build samples and loss masks

When you write your own training loop with the Together Python SDK, you build the samples it trains on. Each sample pairs a token sequence with the per-token values that the loss function reads. This page shows how a sample’s arrays line up, how to build them by hand, and how to mask a chat conversation so only the reply trains.

How a sample’s arrays line up

A Sample holds its input tokens in model_input and its per-token arrays in a loss_fn_inputs mapping. The arrays line up so that each position is scored on predicting the next token:

Input tokens: model_input holds the prompt and response, minus the final token.
Target tokens: target_tokens holds the same sequence minus the first token, so position i is scored on predicting target_tokens[i], the token that follows model_input[i].
Weights: weights is the loss mask. A position with 1.0 trains on its target token, and a position with 0.0 doesn’t. To train on the response only, use 0.0 where the target token is part of the prompt and 1.0 where it’s part of the response.

For example, a six-token sequence, t0 through t5, with a three-token prompt gives five positions:

Position	`model_input`	`target_tokens`	`weights`
0	`t0`	`t1`	`0.0`
1	`t1`	`t2`	`0.0`
2	`t2`	`t3`	`1.0`
3	`t3`	`t4`	`1.0`
4	`t4`	`t5`	`1.0`

Position 2 reads the last prompt token, but its target is the first response token, so its weight is 1.0. The last position’s target is t5, so t5 trains like any other response token, even though it isn’t in model_input. Group relative policy optimization (GRPO) samples add two more arrays to loss_fn_inputs:

Advantages: advantages holds the completion’s advantage at every trained position and 0.0 elsewhere.
Log-probabilities: logprobs holds the log-probability the generator assigned to each completion token when it sampled the completion, and 0.0 on the prompt.

Build samples by hand

Every array in loss_fn_inputs needs one value per position. To build each array, start with one value per token in the sequence, then drop the first value, as target_tokens drops the first token. Use 0.0 for positions you don’t train on rather than leaving them out.

The service doesn’t shift arrays or check that they line up. An array trimmed from the wrong end still has the right length, so the run trains on the wrong tokens without an error. Trim every array in one function, as build_sample in the next example does.

The build_sample function builds a cross-entropy sample from a token sequence and a loss mask with one value per token:

from together.post_training import Sample


def build_sample(tokens: list[int], loss_mask: list[float]) -> Sample:
    # Drop the last token from the input, and the first value
    # from every per-token array
    return {
        "model_input": {"chunks": [{"encoded_text": {"tokens": tokens[:-1]}}]},
        "loss_fn_inputs": {
            "target_tokens": {"data": tokens[1:], "dtype": "int64"},
            "weights": {"data": loss_mask[1:], "dtype": "float32"},
        },
    }

Each array is a tensor that holds its values in data and declares its dtype, either int64 or float32. To build a GRPO sample from a sampled completion, also add the advantages and logprobs arrays. Build each one the same way as weights, from a per-token list with its first value dropped.

Build a loss mask from a chat template

To train on only the final reply in a chat conversation, build the loss mask from the model’s chat template. Render the conversation twice with the model’s tokenizer: once without the final reply, ready for the model to respond, and once in full. When the full rendering starts with the prompt rendering, every token after the prompt belongs to the reply. This example passes the resulting mask to build_sample:

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3.8-27B")


def render(messages: list[dict], **kwargs) -> list[int]:
    # Apply the chat template, with thinking off for data without reasoning
    return tokenizer.apply_chat_template(
        messages,
        tokenize=True,
        return_dict=True,
        enable_thinking=False,
        **kwargs,
    )["input_ids"]


messages = [
    {"role": "user", "content": "What is 2 + 2?"},
    {"role": "assistant", "content": "4"},
]

# Render the prompt as the model sees it before it replies,
# then the full conversation
prompt = render(messages[:-1], add_generation_prompt=True)
tokens = render(messages)
if tokens[: len(prompt)] != prompt:
    raise ValueError("The template renders the prompt differently")

# Train on the reply and its stop token, not on the prompt
loss_mask = [0.0] * len(prompt) + [1.0] * (len(tokens) - len(prompt))
sample = build_sample(tokens, loss_mask)

For a single-turn Qwen3.8 27B conversation, the example builds the same tokens and loss mask as the cookbook. To adapt it to other models or conversations, handle these chat template behaviors:

Thinking: Templates differ in whether they think by default and whether you can turn thinking off. For data without reasoning, set enable_thinking to fit the model’s thinking behavior. The enable_thinking=False in render turns thinking off for Qwen3.8 27B.
Stop tokens: Some templates don’t close the final reply with a stop token. For those, append the model’s stop token to tokens and 1.0 to loss_mask, or the model never learns to stop.
Multi-turn conversations: To train on every assistant turn, build one sample per turn from the conversation up to and including that turn. Some templates strip reasoning from earlier turns, so a later turn’s prompt doesn’t always start with the earlier turns as the model produced them. A sample per turn avoids the mismatch, because each turn trains against the prompt the template renders for it.

GET STARTED

SERVERLESS

INFERENCE APIS

DEDICATED MODEL INFERENCE

DEDICATED CONTAINER INFERENCE

POST-TRAINING

FINE-TUNING

GPU CLUSTERS

CODE EXECUTION

ADMINISTRATION

How a sample’s arrays line up

Build samples by hand

Build a loss mask from a chat template

​How a sample’s arrays line up

​Build samples by hand

​Build a loss mask from a chat template

How a sample’s arrays line up

Build samples by hand

Build a loss mask from a chat template