Manage model resources

A model resource represents the set of GPUs allocated for training, plus the frozen base model loaded onto them. Every session trains on a model resource, and the resource is what Together AI bills you for. This page covers how to choose a configuration, create a resource or attach to an existing one, inspect its state, and stop it. It also documents the lifecycle states, timers, error codes, and limits that apply to resources, their sessions, and the operations you run on them. For more on how sessions relate to model resources, see Concepts. Cookbook recipes manage the resource for you. Set model in a recipe’s config to provision a resource that the recipe stops on exit, or, in GRPO and SFT recipes, set model_resources_id to attach to one that’s already running.

Choose a configuration

Six settings define a model resource: its base model, optional custom base weights, training mode (LoRA or full-weight), GPU type, number of generator replicas, and optimizer. All of these are fixed once the resource is created, and every session on the resource shares them, so to change one, provision a new resource.

Setting	SDK (`ModelResourcesClient.create`)	Cookbook config	Notes
Base model	`base_model`	`model`	A catalog model ID from Supported models.
Custom base weights	`base_weights_ref`, optional	Not exposed	A model in your project’s model registry whose weights the resource trains in place of the base model’s own, as `together://<object_id>` or `together://<object_id>@<revision_id>`. It must be a full model with the base model’s architecture, and `base_model` still selects the compute layout. Checkpoints resume only on resources created with the same weights.
Training mode	`lora_enabled`, default `True`	`use_lora`, default `True`	A low-rank adapter (LoRA) resource hosts up to five adapter sessions at once, or three for some models. A full-weight resource hosts one session. See LoRA vs. full-weight training.
GPU type	`ComputeConfig.gpu_type`	Not exposed.	Set `gpu_type="B200-SXM"`. Post-training currently runs only on B200 GPUs. See Choose a GPU type.
Generator replicas	`ComputeConfig.num_generator_replicas`	`GRPOConfig.num_generator_replicas`, default `1`. The SFT and DPO loops always provision `0`.	See Generator replicas.
Optimizer	`optimizer_config`	`GRPOConfig.optimizer`, `"adam"` or `"muon"`	Adam by default. For Muon, pass `OptimizerConfig(muon=MuonConfig())`. Muon requires full-weight training, and it falls back to Adam for embeddings, norm layers, and the output head. The learning rate isn’t a resource setting: you pass it on every `optim_step`.

For more details, see the API reference.

Generator replicas

A model resource runs a trainer and, optionally, generator replicas. The trainer computes losses and gradients and applies optimizer steps. Generators sample completions from the current policy, so only reinforcement learning needs them.

SFT and DPO: Set num_generator_replicas=0. The resource is trainer-only, which provisions less hardware and costs less per minute. The cookbook’s SFT and DPO loops do this for you.
GRPO: Set num_generator_replicas to 1 or more. More replicas raise sampling throughput when rollouts, rather than the optimizer step, are the bottleneck. Each replica adds GPUs and cost.

The API defaults to one generator replica whenever you omit num_generator_replicas, including when you pass a compute_config that sets only gpu_type, so pass num_generator_replicas=0 explicitly for trainer-only work. Models whose Supported models row says No generator can’t be provisioned with replicas at all, so creating one without num_generator_replicas=0 fails. Sessions inherit the resource’s layout. On a trainer-only resource, session.generator raises RuntimeError, so guard shared code paths with session.has_generator.

Create a resource

Your project needs a credit balance of at least $100, the minimum credit balance, to create a resource. ModelResourcesClient.create submits the request and waits until the resource is READY. The wait covers queueing for GPUs, starting them, and loading the base model, and it runs longer when capacity for the model’s GPU type is scarce. Billing starts the moment the resource becomes READY.

from together.post_training import ComputeConfig, ModelResourcesClient

with ModelResourcesClient.create(
    base_model="Qwen/Qwen3.8-27B",
    lora_enabled=True,
    compute_config=ComputeConfig(
        gpu_type="B200-SXM",
        num_generator_replicas=1,
    ),
) as resources:
    print(resources.model_resources_id)
    with resources.create_session() as session:
        ...  # your training loop

Used as a context manager, the client stops the resource on exit, even if your loop raises. Nest the session’s context manager inside it so the session stops first. create waits up to an hour by default. The service has no time limit for a resource waiting in the queue, so when capacity is scarce, the SDK’s timeout can run out while the resource is still queued. Pass timeout=None to keep waiting until the resource gets GPUs, or a smaller timeout to give up sooner. If creation fails (the wait times out, the resource enters ERROR, or you interrupt the process), the SDK stops the resource it created before raising, so a failed create doesn’t leave GPUs billing. If the process is killed instead, for example with kill -9 or by a machine crash, no cleanup runs. A resource still in the queue stays there until it gets GPUs, then becomes READY and bills until you stop it or it stops after an hour without an operation, so stop it yourself.

Every SDK operation has an _async variant, such as ModelResourcesClient.create_async. The synchronous calls block the calling thread, so they raise RuntimeError when called from code that’s already running an event loop, such as an async application. Jupyter notebooks are the exception: a synchronous call works in a notebook cell but blocks the cell until it returns. Use the _async variants in async code. They also work in notebooks, as the cookbook’s notebooks show.

You can also create a resource from the Model resources tab under Post-training in the console, then attach sessions to it from code.

Attach to an existing resource

To train on a resource that already exists, attach to it by ID. Use this for a resource you created in the console, one a teammate provisioned, or one you keep running across experiments. attach provisions nothing and doesn’t wait. It only confirms that the resource exists.

from together.post_training import ModelResourcesClient

resources = ModelResourcesClient.attach(
    model_resources_id="your_model_resources_id"
)
try:
    with resources.create_session() as session:
        ...  # your training loop
finally:
    resources.detach()  # releases the handle, leaves the resource running

Don’t use an attached handle as a context manager. Its exit calls stop(), the same as an owned handle’s, and stops the resource for everyone else using it. Call detach() instead. To attach a GRPO or SFT cookbook recipe instead, set model_resources_id on its config, as Attach to a running resource shows.

Inspect and list resources

To check a resource’s state, call retrieve on its client handle. It returns the resource’s current record, including its status and, when the status is ERROR, an error with the error code:

resource = resources.retrieve()
print(resource.status, resource.compute_config.gpu_type)

While a resource waits in the queue, resource.status_details.queue_position estimates its place: the number of older resources of the same GPU type still waiting for GPUs, plus one. It’s unset once the resource is placed on a cluster. To find resources without a handle, list them from the base client. Filter by status to see what’s billing right now:

from together import Together

client = Together()
for r in client.post_training.model_resources.list(
    status=["MODEL_RESOURCES_STATUS_READY"]
).data:
    print(
        r.id,
        r.base_model,
        r.compute_config.num_generator_replicas,
        r.created_by,
    )

Although status takes a list, the SDK sends several values in a form the API rejects with HTTP 400. To find resources in several states, call list once for each status. The Model resources tab in the console shows the same resources. The tg CLI can list and inspect them with tg training ls-resources, tg training ls-sessions, tg training ls-checkpoints, and tg training get <ID>, but it can’t create or stop them, so use the SDK or the console for that.

Stop a resource

A READY resource bills until it stops, even when nothing is training on it. Stop it as soon as you’re done:

resources.stop()

stop returns once the resource has left READY, which is when billing ends. The API refuses to stop a resource while any session on it is still active. Stop your sessions first, or pass force=True to stop the resource along with every session attached to it, including sessions that other processes own:

resources.stop(force=True)

To stop a resource you don’t have a handle for, such as one left behind by an interrupted run, stop it by ID from the base client:

from together import Together

client = Together()
client.post_training.model_resources.stop(
    "your_model_resources_id", force=True
)

You can also stop any running resource from the console’s Model resources tab. A resource also stops on its own after an hour without an operation, or when your credit balance runs out, as Timers describes. Don’t rely on either for cleanup, because the idle hour before an automatic stop is billed.

Lifecycle

A resource and its sessions each move through a small set of states. This page uses the short state names. The API returns them as full enum constants, such as MODEL_RESOURCES_STATUS_READY and TRAINING_SESSION_STATUS_RUNNING. Model resource: PENDING → CREATING → READY → STOPPING → STOPPED, or ERROR.

PENDING means the resource is waiting for GPUs. It first waits in a queue, with no time limit, for a cluster with free capacity for its GPU type. Then it waits for its pods to be scheduled on that cluster. While it’s queued, status_details.queue_position shows its place in line.
CREATING means the pods are scheduled and the base model is loading.
READY means sessions can attach. Billing runs only in this state.
ERROR is reachable from any active state, with the cause in error.code:
- MODEL_RESOURCES_ERROR_CODE_CAPACITY_WAIT_TIMEOUT: The resource left the queue for a cluster, but its pods weren’t scheduled within an hour. It was never billed.
- MODEL_RESOURCES_ERROR_CODE_PROVISIONING_FAILED: The resource’s pods were scheduled, but it didn’t finish starting, either because startup failed or because it wasn’t READY within an hour. It was never billed.
- MODEL_RESOURCES_ERROR_CODE_RESOURCE_UNAVAILABLE: A READY resource failed. A trailing window of its runtime is refunded. See Refunds on failure.

Session: CREATING → RUNNING → STOPPING → STOPPED, or EXPIRED or ERROR.

Stopping a session only detaches it. The resource keeps running for its other sessions, and billing continues until the resource itself stops.
When a resource goes one hour without an operation from any of its sessions, every session on it is marked EXPIRED and detached. An idle session doesn’t expire while another session on the same resource is still training. The check runs every minute, so sessions expire within about a minute of the one-hour mark.
ERROR means the session failed to start, to keep running, or to stop. Read the cause from session.retrieve().error, which holds a code, a message, and an occurred_at timestamp:
- TRAINING_SESSION_ERROR_CODE_RESOURCE_UNAVAILABLE: The session’s model resource is no longer available, so the session can’t continue.
- TRAINING_SESSION_ERROR_CODE_RESOURCE_AT_CAPACITY: The session couldn’t start, because the resource already hosts as many sessions as it accepts. Stop another session on the resource, or use a different resource.
- TRAINING_SESSION_ERROR_CODE_TIMED_OUT: The session didn’t become ready in time.
- TRAINING_SESSION_ERROR_CODE_SESSION_FAILED: The session failed for any other reason.

If a new session reaches STOPPING, STOPPED, EXPIRED, or ERROR instead of RUNNING, create_session raises RuntimeError with the session ID and that status. You don’t get a session handle in that case, so read the error by ID with client.post_training.sessions.retrieve("your_session_id").error. To continue training after a session error, create a new session, resuming from a training checkpoint if you saved one.

Timers

Timer	Value	What happens
Pod scheduling	1 hour after the resource leaves the queue for a cluster	The resource fails with `CAPACITY_WAIT_TIMEOUT`. Nothing was billed. Time spent in the queue doesn’t count, and the queue itself has no time limit.
Startup	1 hour in `CREATING`	The resource fails with `PROVISIONING_FAILED`. Nothing was billed.
Session idle expiry	1 hour without an operation on the resource	Every session on the resource is marked `EXPIRED` and detached.
Resource auto-stop	1 hour without an operation	The resource stops, in the same check that expires any sessions still attached.
Credit balance exhausted	Within about a minute of the balance reaching zero	The resource stops and its sessions end with it. See Minimum credit balance.

On long runs, save training checkpoints so that an expiry or unexpected stop loses only the work since your last checkpoint, not the whole run. See Training checkpoints.

Operation errors

When an operation such as forward_backward, optim_step, or sample fails on the model resource, the SDK raises OperationFailedError, a subclass of RuntimeError. Its message holds the operation ID and the operation’s error, which has a code and a message that says what went wrong:

Operation (<operation_id>) failed: OperationError(code='TRAINING_OPERATION_ERROR_CODE_INVALID_INPUT', message='Operation failed because the request contains invalid training inputs: ...')

together.post_training doesn’t export OperationFailedError, so catch RuntimeError. Over the REST API, the failed operation has the status TRAINING_OPERATION_STATUS_FAILED and the same error.

Code	What happened	What to do
`TRAINING_OPERATION_ERROR_CODE_INVALID_INPUT`	The request’s training inputs are invalid, or its batch size or sequence length is over the limits. The message usually names the sample and the field, for example a `loss_fn_inputs` row whose length doesn’t match the sample’s tokens, a row the loss function requires that’s missing, or a value that’s NaN or infinite.	Fix the inputs and submit the operation again.
`TRAINING_OPERATION_ERROR_CODE_NON_FINITE_LOSS`	`forward_backward` produced a loss or metric that’s NaN or infinite, which means the run diverged, or `optim_step` saw a NaN or infinite gradient norm.	Stop the session, lower the learning rate, and start a new session from your last training checkpoint. After a non-finite gradient norm, the update isn’t applied and the session refuses every later `optim_step`. On a full-weight resource, a new session continues from the resource’s current weights, and resuming from a checkpoint needs a new resource.
`TRAINING_OPERATION_ERROR_CODE_RESOURCE_EXHAUSTED`	The step ran out of GPU memory.	Shorten the sequences (shorter prompts, or a lower `max_tokens` when sampling), sample fewer completions per prompt with `num_samples`, or send fewer samples per `forward_backward` call.
`TRAINING_OPERATION_ERROR_CODE_TIMEOUT`	The operation timed out on the model resource. This differs from the SDK’s `TimeoutError`, which only means your code stopped waiting.	Submit the operation again. If it keeps failing, contact support with the operation ID.
`TRAINING_OPERATION_ERROR_CODE_SESSION_NOT_ACTIVE`	The session was stopped, expired, or hit an error after the operation was submitted.	Read `session.retrieve().status` and `session.retrieve().error`, then create a new session, resuming from a training checkpoint if you saved one.
`TRAINING_OPERATION_ERROR_CODE_INTERNAL_ERROR`	An unexpected failure on the model resource.	Submit the operation again. If it fails again, contact support with the operation ID.

If the session had already left RUNNING when you submitted the operation, the API doesn’t create the operation. It rejects the request with HTTP 400 and training session is not running (current status: ...), which the SDK raises as BadRequestError. Handle it the same way as SESSION_NOT_ACTIVE.

Limits

Limit	Value
Concurrent LoRA sessions per resource	Per model. See Supported models.
Full-weight sessions per resource	1.
Generator replicas per resource	8.
Allocated nodes per project	15 total across all model resources in the project.
Max batch per `forward_backward`	4,096 samples. This caps the request payload, not GPU memory. The trainer splits a step into micro-batches regardless.
Max sequence length	Per model and training mode. See Supported models.

For the credit needed to create a resource, see Minimum credit balance.

Rate limits

The service limits request rates per project for creating resources, starting sessions, and downloading checkpoints, and per session for training operations. A burst is how many calls can go through at once before the rate applies. When you exceed a limit, the API returns HTTP 429, and the SDK retries the call before it raises RateLimitError.

Operation	Scope	Rate	Burst
Create a model resource	Project	20 per hour	5
Start a training session	Project	20 per minute	5
`forward_backward`, `forward`, `custom_forward_backward`, `optim_step`, `weights_sync`	Session	20 per second each	40
`sample`, `sample_batch`	Session	2,000 per second	1,000
Save a training or inference checkpoint	Session	10 per minute each	5
Download a checkpoint	Project	10 per second	20

The SDK retries any request that fails with a connection error or an HTTP 408, 409, 429, or 5xx response, up to seven times for session calls, which include every operation. Each operation carries an idempotency key, so a retried submission doesn’t create a second operation. Don’t wrap operation calls in your own retry loop.

Troubleshooting

The resource stays PENDING for a long time: No B200 GPU capacity is free yet. The resource stays in the queue until GPUs free up or you stop it, and you aren’t billed while it waits. Check status_details.queue_position to see how many resources are ahead of it.
create raises TimeoutError while the resource is still PENDING: The SDK’s one-hour default ran out before the resource left the queue, so the SDK stopped it. To wait longer, create the resource again with a larger timeout, or timeout=None. The new resource joins the back of the queue, so when capacity is scarce, pass timeout=None from the start.
Create is rejected with rate limit exceeded, retry after ...: Your project has reached its create rate limit. The API returns HTTP 429, which the SDK raises as RateLimitError after retrying. Wait for the time in the message, then retry. To stay under the limit, reuse a running resource by attaching sessions to it instead of creating one for each run.
Create is rejected with organization GPU quota exceeded: Your organization has a quota on the total GPUs held by its PENDING, CREATING, and READY resources, and the new resource would exceed it. The API returns HTTP 429, which the SDK raises as RateLimitError, with a message such as organization GPU quota exceeded: 112 of 120 GPUs in use, requested 16. Stop resources you no longer need, or contact support to request a higher quota.
Create fails for the GPU type or training mode: The model doesn’t support that combination. Check the model’s row on Supported models. Generator replicas also aren’t available for every model.
Create is rejected for insufficient credit: Your project’s balance is below the $100 minimum for creating a resource. See Minimum credit balance.
stop is refused because a session is still active: Stop the sessions first, or pass force=True. The context-manager exit on an owned resource doesn’t force, so it fails the same way if a session you don’t own is still attached.
An operation call raises TimeoutError: The SDK submits each operation, such as forward_backward or sample, and polls until it finishes or its timeout runs out, 300 seconds by default or 7200 seconds for checkpoint saves. The operation keeps running on the service after the call raises, and there’s no call to cancel it. Don’t resubmit it. A second forward_backward is a new operation, so its gradients add to the first one’s. For slow steps on large models or large batches, pass a larger timeout, or timeout=None to wait indefinitely.
An operation raises OperationFailedError: Operation (...) failed: The operation failed on the model resource. Look up the code from the message in Operation errors.
An operation fails with SESSION_NOT_ACTIVE or training session is not running: The session was stopped, expired after an hour without an operation on its resource, or hit an error. Read session.retrieve().status and session.retrieve().error, then create a new session, resuming from a training checkpoint if you saved one.
A running resource stopped on its own: Either your balance ran out, or the resource went an hour without an operation from any session. Check your balance, then check the session statuses for EXPIRED. If neither explains the stop, contact support with the resource ID and the time it stopped, and resume from your last training checkpoint.

When you contact support about a resource or session, include the resource ID, the session ID, the operation ID from the error message if an operation failed, and the time of the problem in UTC.

Next steps

Train multiple adapters

Share one resource across several sessions to run a sweep for one base model’s worth of GPUs.

Pricing

How a resource bills, how to estimate a run, and what’s refunded on failure.

Post-training API reference

Every field for resources, sessions, and operations, with types and defaults.

Supported models

GPU types, training modes, and session caps per model.

GET STARTED

SERVERLESS

INFERENCE APIS

DEDICATED MODEL INFERENCE

DEDICATED CONTAINER INFERENCE

POST-TRAINING

FINE-TUNING

GPU CLUSTERS

CODE EXECUTION

ADMINISTRATION

Choose a configuration

Generator replicas

Create a resource

Attach to an existing resource

Inspect and list resources

Stop a resource

Lifecycle

Timers

Operation errors

Limits

Rate limits

Troubleshooting

Next steps

Train multiple adapters

Pricing

Post-training API reference

Supported models

​Choose a configuration

​Generator replicas

​Create a resource

​Attach to an existing resource

​Inspect and list resources

​Stop a resource

​Lifecycle

​Timers

​Operation errors

​Limits

​Rate limits

​Troubleshooting

​Next steps

Train multiple adapters

Pricing

Post-training API reference

Supported models

Choose a configuration

Generator replicas

Create a resource

Attach to an existing resource

Inspect and list resources

Stop a resource

Lifecycle

Timers

Operation errors

Limits

Rate limits

Troubleshooting

Next steps