Write a training loop

Write your own loop when you need control over the data, sampling, rewards, or loss function that isn’t in a cookbook recipe. Your training loop calls the Together Python SDK to create a model resource and a session, then sends each step’s operations to the resource.

If you already have a Tinker training script, you can migrate it by changing its import line.

Install the SDK

Install the Together Python SDK, version 3.0.0 or later:

uv add "together>=3.0.0"

Run a minimal loop

This loop trains a low-rank adapter (LoRA) for Qwen3.8 27B on one sample for one step, then saves the adapter as an inference checkpoint. The model resource it creates bills per GPU-minute from when it’s ready until the loop exits:

from together.post_training import ComputeConfig, ModelResourcesClient
from together.post_training import AdamParams, LossConfig, Sample

# "The capital of France is" and " Paris." as Qwen3.8 token IDs
tokens = [760, 6511, 314, 9338, 369, 11751, 13]
mask = [0.0] * 5 + [1.0] * 2  # train only on the reply
sample: Sample = {
    "model_input": {"chunks": [{"encoded_text": {"tokens": tokens[:-1]}}]},
    "loss_fn_inputs": {
        "target_tokens": {"data": tokens[1:], "dtype": "int64"},
        "weights": {"data": mask[1:], "dtype": "float32"},
    },
}

# Create a model resource
with ModelResourcesClient.create(
    base_model="Qwen/Qwen3.8-27B",
    compute_config=ComputeConfig(num_generator_replicas=0),
) as resources:

    # Create a session
    with resources.create_session() as session:

        # Train on the sample
        session.trainer.forward_backward(
            samples=[sample], loss=LossConfig(type="LOSS_TYPE_CROSS_ENTROPY")
        )
        session.trainer.optim_step(adam_params=AdamParams(learning_rate=1e-4))

        # Save the checkpoint
        print(session.create_inference_checkpoint().id)

To build on this loop, learn how to:

Train on a batch

This loop trains on eight of your samples per step, for NUM_STEPS steps. You pass the learning rate on every optim_step, so it can follow a schedule, such as this loop’s linear decay:

for step in range(NUM_STEPS):
    batch = samples[step * 8 : (step + 1) * 8]
    result = session.trainer.forward_backward(
        samples=batch, loss=LossConfig(type="LOSS_TYPE_CROSS_ENTROPY")
    )
    learning_rate = 1e-4 * (1 - step / NUM_STEPS)
    session.trainer.optim_step(
        adam_params=AdamParams(learning_rate=learning_rate),
    )

    # Report the loss per trained token
    trained = sum(sum(s["loss_fn_inputs"]["weights"]["data"]) for s in batch)
    print(f"step {step + 1}: loss={result.loss / trained:.4f}")

For cross-entropy, result.loss is a sum over the batch’s trained tokens, not a mean. Divide it by the number of trained tokens for a per-token loss that stays comparable as the batch’s token count changes. To train with group relative policy optimization (GRPO), build GRPO samples and pass a policy-gradient loss. GrpoLossParams sets the clip band and the beta weight of a Kullback-Leibler (KL) penalty:

from together.post_training import GrpoLossParams, LossConfig

loss = LossConfig(
    type="LOSS_TYPE_GRPO",
    grpo_params=GrpoLossParams(
        clip_low_threshold=0.8,
        clip_high_threshold=1.28,
        beta=0.01,  # Needs reference_logprobs in every sample
    ),
)
session.trainer.forward_backward(samples=batch, loss=loss)

Sample from the generator

Pass all your prompts to sample_batch in one call, and set num_samples to request a group of completions for each prompt. In this example, prompts holds each prompt’s token IDs:

from together.post_training import SamplingParams

results = session.generator.sample_batch(
    [{"chunks": [{"encoded_text": {"tokens": p}}]} for p in prompts],
    num_samples=8,
    sampling_params=SamplingParams(max_tokens=768, temperature=0.7),
)

# One result per prompt, in order
for result in results:  
    for sequence in result.sequences:

        # Token IDs can come back as strings, so convert them
        tokens = [int(t) for t in sequence.tokens]
        logprobs = sequence.logprobs or []

# After each optim_step, publish the new weights to the generators
session.trainer.weights_sync()

Build each completion’s GRPO sample from its tokens and logprobs. Sampling can run at the same time as other sampling and trainer operations, but a synchronous weights_sync waits for in-flight sampling to finish. One SamplingParams applies to every prompt in a sample_batch call. The API reference lists its fields and defaults. Two fields need care:

stop: For models without stop-string support, omit stop and truncate completions in your code. A request with an unsupported stop string fails as invalid input.
seed: Leave it unset. It’s unsupported and causes an internal error when the generator uses multiple inference engines.

Compute log-probabilities for fixed text

Call session.generator.compute_logprobs(prompt) for one prompt, or compute_logprobs_batch(prompts) for several. If the trainer has stepped since the last sync, call weights_sync first, so the log-probabilities come from the updated weights. Entry 0 of the result is a 0 placeholder, because the first token has no preceding context. sample with prompt_logprobs=True returns the same placeholder.

Session operations

Every operation targets a session. Calls on session.trainer run on the model resource’s trainer. Calls on session.generator run on its generator replicas, which exist only on a resource created with num_generator_replicas above 0.

Call	What it does
`session.trainer.forward_backward`	Computes a built-in loss function over a batch of samples and accumulates gradients.
`session.trainer.forward`	Returns each target token’s log-probability without accumulating gradients, as the first pass of a custom loss function.
`session.trainer.custom_forward_backward`	Backpropagates the gradients your code computed for each log-probability, as the second pass of a custom loss function.
`session.trainer.optim_step`	Applies the accumulated gradients with the learning rate and optimizer parameters you pass.
`session.trainer.weights_sync`	Publishes the trainer’s current weights to the generators, so the next sample comes from the updated policy.
`session.generator.sample`, `sample_batch`	Samples completions from the generators’ current weights, with each token’s log-probability.
`session.generator.compute_logprobs`, `compute_logprobs_batch`	Returns each token’s log-probability for a fixed input under the generators’ current weights.
`session.create_inference_checkpoint`	Saves the weights so you can download or deploy them.
`session.create_training_checkpoint`	Saves the weights, optimizer state, and step counter, so a later session can resume the run.

The API reference lists every operation’s fields and defaults. weights_sync takes a weight_sync_type, which defaults to WEIGHT_SYNC_TYPE_SYNCHRONOUS and also accepts the deferred modes used for async training. Each call submits an operation, then waits until the operation finishes or the call’s timeout runs out. When an operation fails on the resource, the SDK raises an operation error.

Call operations asynchronously

Every call has an _async variant, such as forward_backward_async, sample_batch_async, or ModelResourcesClient.create_async. Use the _async variants in async code. The synchronous calls block the calling thread, so they raise RuntimeError when called from code that’s already running an event loop, such as an async application. Jupyter notebooks are the exception: a synchronous call works in a notebook cell but blocks the cell until it returns. The _async variants work in notebooks too. The _async variants also let independent work overlap. For example, one session can sample while another trains, or you can train several adapters at once.

Next steps

For complete loops, see the cookbook’s notebooks and direct preference optimization (DPO) loop, which build samples with the cookbook’s helpers.

SFT notebook

Supervised fine-tuning (SFT) on the No Robots dataset, step by step.

GRPO notebook

GRPO on GSM8K math problems, step by step.

DPO training loop

DPO on chosen and rejected pairs, built on a custom loss function.

Samples and loss masks

Line up a sample’s arrays and mask a chat prompt.

Custom loss functions

Train toward an objective you compute in PyTorch, such as DPO.

Model resources

Choose GPUs, attach to a running resource, and stop it.

GET STARTED

SERVERLESS

INFERENCE APIS

DEDICATED MODEL INFERENCE

DEDICATED CONTAINER INFERENCE

POST-TRAINING

FINE-TUNING

GPU CLUSTERS

CODE EXECUTION

ADMINISTRATION

Install the SDK

Run a minimal loop

Train on a batch

Sample from the generator

Compute log-probabilities for fixed text

Session operations

Call operations asynchronously

Next steps

SFT notebook

GRPO notebook

DPO training loop

Samples and loss masks

Custom loss functions

Model resources

​Install the SDK

​Run a minimal loop

​Train on a batch

​Sample from the generator

​Compute log-probabilities for fixed text

​Session operations

​Call operations asynchronously

​Next steps

SFT notebook

GRPO notebook

DPO training loop

Samples and loss masks

Custom loss functions

Model resources

Install the SDK

Run a minimal loop

Train on a batch

Sample from the generator

Compute log-probabilities for fixed text

Session operations

Call operations asynchronously

Next steps