> ## Documentation Index > Fetch the complete documentation index at: https://docs.together.ai/llms.txt > Use this file to discover all available pages before exploring further. # Manage model resources > Configure, create, attach to, inspect, and stop the GPU resource that post-training sessions run on, and learn its lifecycle states, timers, error codes, and limits. A model resource represents the set of GPUs allocated for training, plus the frozen base model loaded onto them. Every session trains on a model resource, and the resource is what Together AI bills you for. This page covers how to choose a configuration, create a resource or attach to an existing one, inspect its state, and stop it. It also documents the lifecycle states, timers, error codes, and limits that apply to resources, their sessions, and the operations you run on them. For more on how sessions relate to model resources, see [Concepts](/docs/post-training/concepts#model-resources-and-sessions). Cookbook recipes manage the resource for you. Set `model` in a recipe's config to provision a resource that the recipe stops on exit, or, in GRPO and SFT recipes, set `model_resources_id` to attach to one that's already running. ## Choose a configuration Six settings define a model resource: its base model, optional custom base weights, training mode (LoRA or full-weight), GPU type, number of generator replicas, and optimizer. All of these are fixed once the resource is created, and every session on the resource shares them, so to change one, provision a new resource. | Setting | SDK (`ModelResourcesClient.create`) | Cookbook config | Notes | | - | - | - | - | | Base model | `base_model` | `model` | A catalog model ID from [Supported models](/docs/post-training/supported-models). | | Custom base weights | `base_weights_ref`, optional | Not exposed | A model in your project's model registry whose weights the resource trains in place of the base model's own, as `together://` or `together://@`. It must be a full model with the base model's architecture, and `base_model` still selects the compute layout. Checkpoints resume only on resources created with the same weights. | | Training mode | `lora_enabled`, default `True` | `use_lora`, default `True` | A low-rank adapter (LoRA) resource hosts up to five adapter sessions at once, or three for some models. A full-weight resource hosts one session. See [LoRA vs. full-weight training](/docs/post-training/training-methods#lora-vs-full-weight-training). | | GPU type | `ComputeConfig.gpu_type` | Not exposed. | Set `gpu_type="B200-SXM"`. Post-training currently runs only on B200 GPUs. See [Choose a GPU type](/docs/post-training/supported-models#choose-a-gpu-type). | | Generator replicas | `ComputeConfig.num_generator_replicas` | `GRPOConfig.num_generator_replicas`, default `1`. The SFT and DPO loops always provision `0`. | See [Generator replicas](#generator-replicas). | | Optimizer | `optimizer_config` | `GRPOConfig.optimizer`, `"adam"` or `"muon"` | Adam by default. For Muon, pass `OptimizerConfig(muon=MuonConfig())`. Muon requires full-weight training, and it falls back to Adam for embeddings, norm layers, and the output head. The learning rate isn't a resource setting: you pass it on every `optim_step`. | For more details, see the [API reference](/reference/post-training/model-resources-create). ### Generator replicas A model resource runs a trainer and, optionally, generator replicas. The trainer computes losses and gradients and applies optimizer steps. Generators sample completions from the current policy, so only reinforcement learning needs them. * **SFT and DPO:** Set `num_generator_replicas=0`. The resource is trainer-only, which provisions less hardware and costs less per minute. The cookbook's SFT and DPO loops do this for you. * **GRPO:** Set `num_generator_replicas` to 1 or more. More replicas raise sampling throughput when rollouts, rather than the optimizer step, are the bottleneck. Each replica adds GPUs and cost. The API defaults to one generator replica whenever you omit `num_generator_replicas`, including when you pass a `compute_config` that sets only `gpu_type`, so pass `num_generator_replicas=0` explicitly for trainer-only work. Models whose [Supported models](/docs/post-training/supported-models#catalog) row says **No generator** can't be provisioned with replicas at all, so creating one without `num_generator_replicas=0` fails. Sessions inherit the resource's layout. On a trainer-only resource, `session.generator` raises `RuntimeError`, so guard shared code paths with `session.has_generator`. ## Create a resource Your project needs a credit balance of at least \$100, the [minimum credit balance](/docs/post-training/pricing#minimum-credit-balance), to create a resource. `ModelResourcesClient.create` submits the request and waits until the resource is `READY`. The wait covers queueing for GPUs, starting them, and loading the base model, and it runs longer when capacity for the model's GPU type is scarce. Billing starts the moment the resource becomes `READY`. ```python theme={null} from together.post_training import ComputeConfig, ModelResourcesClient with ModelResourcesClient.create( base_model="Qwen/Qwen3.8-27B", lora_enabled=True, compute_config=ComputeConfig( gpu_type="B200-SXM", num_generator_replicas=1, ), ) as resources: print(resources.model_resources_id) with resources.create_session() as session: ... # your training loop ``` Used as a context manager, the client stops the resource on exit, even if your loop raises. Nest the session's context manager inside it so the session stops first. `create` waits up to an hour by default. The service has no time limit for a resource waiting in the queue, so when capacity is scarce, the SDK's timeout can run out while the resource is still queued. Pass `timeout=None` to keep waiting until the resource gets GPUs, or a smaller `timeout` to give up sooner. If creation fails (the wait times out, the resource enters `ERROR`, or you interrupt the process), the SDK stops the resource it created before raising, so a failed create doesn't leave GPUs billing. If the process is killed instead, for example with `kill -9` or by a machine crash, no cleanup runs. A resource still in the queue stays there until it gets GPUs, then becomes `READY` and bills until you stop it or it stops after an hour without an operation, so [stop it yourself](#stop-a-resource). Every SDK operation has an `_async` variant, such as `ModelResourcesClient.create_async`. The synchronous calls block the calling thread, so they raise `RuntimeError` when called from code that's already running an event loop, such as an async application. Jupyter notebooks are the exception: a synchronous call works in a notebook cell but blocks the cell until it returns. Use the `_async` variants in async code. They also work in notebooks, as the cookbook's notebooks show. You can also create a resource from the **Model resources** tab under [Post-training](https://api.together.ai/custom-training/resources) in the console, then [attach sessions to it](#attach-to-an-existing-resource) from code. ## Attach to an existing resource To train on a resource that already exists, attach to it by ID. Use this for a resource you created in the console, one a teammate provisioned, or one you keep running across experiments. `attach` provisions nothing and doesn't wait. It only confirms that the resource exists. ```python theme={null} from together.post_training import ModelResourcesClient resources = ModelResourcesClient.attach( model_resources_id="your_model_resources_id" ) try: with resources.create_session() as session: ... # your training loop finally: resources.detach() # releases the handle, leaves the resource running ``` Don't use an attached handle as a context manager. Its exit calls `stop()`, the same as an owned handle's, and stops the resource for everyone else using it. Call `detach()` instead. To attach a GRPO or SFT cookbook recipe instead, set `model_resources_id` on its config, as [Attach to a running resource](/docs/post-training/multi-lora#attach-to-a-running-resource) shows. ## Inspect and list resources To check a resource's state, call `retrieve` on its client handle. It returns the resource's current record, including its `status` and, when the status is `ERROR`, an `error` with the error code: ```python theme={null} resource = resources.retrieve() print(resource.status, resource.compute_config.gpu_type) ``` While a resource waits in the queue, `resource.status_details.queue_position` estimates its place: the number of older resources of the same GPU type still waiting for GPUs, plus one. It's unset once the resource is placed on a cluster. To find resources without a handle, list them from the base client. Filter by status to see what's billing right now: ```python theme={null} from together import Together client = Together() for r in client.post_training.model_resources.list( status=["MODEL_RESOURCES_STATUS_READY"] ).data: print( r.id, r.base_model, r.compute_config.num_generator_replicas, r.created_by, ) ``` Although `status` takes a list, the SDK sends several values in a form the API rejects with HTTP `400`. To find resources in several states, call `list` once for each status. The **Model resources** tab in the [console](/docs/post-training/monitoring#the-console) shows the same resources. The `tg` CLI can list and inspect them with `tg training ls-resources`, `tg training ls-sessions`, `tg training ls-checkpoints`, and `tg training get `, but it can't create or stop them, so use the SDK or the console for that. ## Stop a resource A `READY` resource bills until it stops, even when nothing is training on it. Stop it as soon as you're done: ```python theme={null} resources.stop() ``` `stop` returns once the resource has left `READY`, which is when billing ends. The API refuses to stop a resource while any session on it is still active. Stop your sessions first, or pass `force=True` to stop the resource along with every session attached to it, including sessions that other processes own: ```python theme={null} resources.stop(force=True) ``` To stop a resource you don't have a handle for, such as one left behind by an interrupted run, stop it by ID from the base client: ```python theme={null} from together import Together client = Together() client.post_training.model_resources.stop( "your_model_resources_id", force=True ) ``` You can also stop any running resource from the console's **Model resources** tab. A resource also stops on its own after an hour without an operation, or when your credit balance runs out, as [Timers](#timers) describes. Don't rely on either for cleanup, because the idle hour before an automatic stop is billed. ## Lifecycle Model resource and session lifecycles. A model resource moves from PENDING to CREATING to READY to STOPPING to STOPPED. It's billed per GPU-minute only while READY, and startup and stopping are free. A session moves from CREATING to RUNNING to STOPPING to STOPPED. A running session can also end in EXPIRED, after its resource is idle for an hour, or in ERROR. Stopping a session detaches it from the resource, which keeps running and billing. A resource and its sessions each move through a small set of states. This page uses the short state names. The API returns them as full enum constants, such as `MODEL_RESOURCES_STATUS_READY` and `TRAINING_SESSION_STATUS_RUNNING`. **Model resource:** `PENDING` → `CREATING` → `READY` → `STOPPING` → `STOPPED`, or `ERROR`. * `PENDING` means the resource is waiting for GPUs. It first waits in a queue, with no time limit, for a cluster with free capacity for its GPU type. Then it waits for its pods to be scheduled on that cluster. While it's queued, `status_details.queue_position` shows its place in line. * `CREATING` means the pods are scheduled and the base model is loading. * `READY` means sessions can attach. Billing runs only in this state. * `ERROR` is reachable from any active state, with the cause in `error.code`: * `MODEL_RESOURCES_ERROR_CODE_CAPACITY_WAIT_TIMEOUT`: The resource left the queue for a cluster, but its pods weren't scheduled within an hour. It was never billed. * `MODEL_RESOURCES_ERROR_CODE_PROVISIONING_FAILED`: The resource's pods were scheduled, but it didn't finish starting, either because startup failed or because it wasn't `READY` within an hour. It was never billed. * `MODEL_RESOURCES_ERROR_CODE_RESOURCE_UNAVAILABLE`: A `READY` resource failed. A trailing window of its runtime is refunded. See [Refunds on failure](/docs/post-training/pricing#refunds-on-failure). **Session:** `CREATING` → `RUNNING` → `STOPPING` → `STOPPED`, or `EXPIRED` or `ERROR`. * Stopping a session only detaches it. The resource keeps running for its other sessions, and billing continues until the resource itself stops. * When a resource goes one hour without an operation from any of its sessions, every session on it is marked `EXPIRED` and detached. An idle session doesn't expire while another session on the same resource is still training. The check runs every minute, so sessions expire within about a minute of the one-hour mark. * `ERROR` means the session failed to start, to keep running, or to stop. Read the cause from `session.retrieve().error`, which holds a `code`, a `message`, and an `occurred_at` timestamp: * `TRAINING_SESSION_ERROR_CODE_RESOURCE_UNAVAILABLE`: The session's model resource is no longer available, so the session can't continue. * `TRAINING_SESSION_ERROR_CODE_RESOURCE_AT_CAPACITY`: The session couldn't start, because the resource already hosts as many sessions as it accepts. Stop another session on the resource, or use a different resource. * `TRAINING_SESSION_ERROR_CODE_TIMED_OUT`: The session didn't become ready in time. * `TRAINING_SESSION_ERROR_CODE_SESSION_FAILED`: The session failed for any other reason. If a new session reaches `STOPPING`, `STOPPED`, `EXPIRED`, or `ERROR` instead of `RUNNING`, `create_session` raises `RuntimeError` with the session ID and that status. You don't get a session handle in that case, so read the error by ID with `client.post_training.sessions.retrieve("your_session_id").error`. To continue training after a session error, create a new session, resuming from a [training checkpoint](/docs/post-training/checkpoints#training-checkpoints) if you saved one. ### Timers | Timer | Value | What happens | | - | - | - | | Pod scheduling | 1 hour after the resource leaves the queue for a cluster | The resource fails with `CAPACITY_WAIT_TIMEOUT`. Nothing was billed. Time spent in the queue doesn't count, and the queue itself has no time limit. | | Startup | 1 hour in `CREATING` | The resource fails with `PROVISIONING_FAILED`. Nothing was billed. | | Session idle expiry | 1 hour without an operation on the resource | Every session on the resource is marked `EXPIRED` and detached. | | Resource auto-stop | 1 hour without an operation | The resource stops, in the same check that expires any sessions still attached. | | Credit balance exhausted | Within about a minute of the balance reaching zero | The resource stops and its sessions end with it. See [Minimum credit balance](/docs/post-training/pricing#minimum-credit-balance). | On long runs, save training checkpoints so that an expiry or unexpected stop loses only the work since your last checkpoint, not the whole run. See [Training checkpoints](/docs/post-training/checkpoints#training-checkpoints). ## Operation errors When an operation such as `forward_backward`, `optim_step`, or `sample` fails on the model resource, the SDK raises `OperationFailedError`, a subclass of `RuntimeError`. Its message holds the operation ID and the operation's `error`, which has a `code` and a `message` that says what went wrong: ```text theme={null} Operation () failed: OperationError(code='TRAINING_OPERATION_ERROR_CODE_INVALID_INPUT', message='Operation failed because the request contains invalid training inputs: ...') ``` `together.post_training` doesn't export `OperationFailedError`, so catch `RuntimeError`. Over the REST API, the failed operation has the status `TRAINING_OPERATION_STATUS_FAILED` and the same `error`. | Code | What happened | What to do | | - | - | - | | `TRAINING_OPERATION_ERROR_CODE_INVALID_INPUT` | The request's training inputs are invalid, or its batch size or sequence length is over the [limits](#limits). The message usually names the sample and the field, for example a `loss_fn_inputs` row whose length doesn't match the sample's tokens, a row the loss function requires that's missing, or a value that's NaN or infinite. | Fix the inputs and submit the operation again. | | `TRAINING_OPERATION_ERROR_CODE_NON_FINITE_LOSS` | `forward_backward` produced a loss or metric that's NaN or infinite, which means the run diverged, or `optim_step` saw a NaN or infinite gradient norm. | Stop the session, lower the learning rate, and start a new session from your last [training checkpoint](/docs/post-training/checkpoints#resume-or-fork-a-run). After a non-finite gradient norm, the update isn't applied and the session refuses every later `optim_step`. On a full-weight resource, a new session continues from the resource's current weights, and resuming from a checkpoint needs a new resource. | | `TRAINING_OPERATION_ERROR_CODE_RESOURCE_EXHAUSTED` | The step ran out of GPU memory. | Shorten the sequences (shorter prompts, or a lower `max_tokens` when sampling), sample fewer completions per prompt with `num_samples`, or send fewer samples per `forward_backward` call. | | `TRAINING_OPERATION_ERROR_CODE_TIMEOUT` | The operation timed out on the model resource. This differs from the SDK's `TimeoutError`, which only means your code stopped waiting. | Submit the operation again. If it keeps failing, [contact support](/docs/support) with the operation ID. | | `TRAINING_OPERATION_ERROR_CODE_SESSION_NOT_ACTIVE` | The session was stopped, expired, or hit an error after the operation was submitted. | Read `session.retrieve().status` and `session.retrieve().error`, then create a new session, resuming from a training checkpoint if you saved one. | | `TRAINING_OPERATION_ERROR_CODE_INTERNAL_ERROR` | An unexpected failure on the model resource. | Submit the operation again. If it fails again, [contact support](/docs/support) with the operation ID. | If the session had already left `RUNNING` when you submitted the operation, the API doesn't create the operation. It rejects the request with HTTP `400` and `training session is not running (current status: ...)`, which the SDK raises as `BadRequestError`. Handle it the same way as `SESSION_NOT_ACTIVE`. ## Limits | Limit | Value | | - | - | | Concurrent LoRA sessions per resource | Per model. See [Supported models](/docs/post-training/supported-models). | | Full-weight sessions per resource | 1. | | Generator replicas per resource | 8. | | Allocated nodes per project | 15 total across all model resources in the project. | | Max batch per `forward_backward` | 4,096 samples. This caps the request payload, not GPU memory. The trainer splits a step into micro-batches regardless. | | Max sequence length | Per model and training mode. See [Supported models](/docs/post-training/supported-models). | For the credit needed to create a resource, see [Minimum credit balance](/docs/post-training/pricing#minimum-credit-balance). ### Rate limits The service limits request rates per project for creating resources, starting sessions, and downloading checkpoints, and per session for training operations. A burst is how many calls can go through at once before the rate applies. When you exceed a limit, the API returns HTTP `429`, and the SDK retries the call before it raises `RateLimitError`. | Operation | Scope | Rate | Burst | | - | - | - | - | | Create a model resource | Project | 20 per hour | 5 | | Start a training session | Project | 20 per minute | 5 | | `forward_backward`, `forward`, `custom_forward_backward`, `optim_step`, `weights_sync` | Session | 20 per second each | 40 | | `sample`, `sample_batch` | Session | 2,000 per second | 1,000 | | Save a training or inference checkpoint | Session | 10 per minute each | 5 | | Download a checkpoint | Project | 10 per second | 20 | The SDK retries any request that fails with a connection error or an HTTP `408`, `409`, `429`, or `5xx` response, up to seven times for session calls, which include every operation. Each operation carries an idempotency key, so a retried submission doesn't create a second operation. Don't wrap operation calls in your own retry loop. ## Troubleshooting * **The resource stays `PENDING` for a long time:** No B200 GPU capacity is free yet. The resource stays in the queue until GPUs free up or you stop it, and you aren't billed while it waits. Check `status_details.queue_position` to see how many resources are ahead of it. * **`create` raises `TimeoutError` while the resource is still `PENDING`:** The SDK's one-hour default ran out before the resource left the queue, so the SDK stopped it. To wait longer, create the resource again with a larger `timeout`, or `timeout=None`. The new resource joins the back of the queue, so when capacity is scarce, pass `timeout=None` from the start. * **Create is rejected with `rate limit exceeded, retry after ...`:** Your project has reached its [create rate limit](#rate-limits). The API returns HTTP `429`, which the SDK raises as `RateLimitError` after retrying. Wait for the time in the message, then retry. To stay under the limit, reuse a running resource by [attaching sessions to it](#attach-to-an-existing-resource) instead of creating one for each run. * **Create is rejected with `organization GPU quota exceeded`:** Your organization has a quota on the total GPUs held by its `PENDING`, `CREATING`, and `READY` resources, and the new resource would exceed it. The API returns HTTP `429`, which the SDK raises as `RateLimitError`, with a message such as `organization GPU quota exceeded: 112 of 120 GPUs in use, requested 16`. Stop resources you no longer need, or [contact support](/docs/support) to request a higher quota. * **Create fails for the GPU type or training mode:** The model doesn't support that combination. Check the model's row on [Supported models](/docs/post-training/supported-models#catalog). Generator replicas also aren't available for every model. * **Create is rejected for insufficient credit:** Your project's balance is below the \$100 minimum for creating a resource. See [Minimum credit balance](/docs/post-training/pricing#minimum-credit-balance). * **`stop` is refused because a session is still active:** Stop the sessions first, or pass `force=True`. The context-manager exit on an owned resource doesn't force, so it fails the same way if a session you don't own is still attached. * **An operation call raises `TimeoutError`:** The SDK submits each operation, such as `forward_backward` or `sample`, and polls until it finishes or its `timeout` runs out, 300 seconds by default or 7200 seconds for checkpoint saves. The operation keeps running on the service after the call raises, and there's no call to cancel it. Don't resubmit it. A second `forward_backward` is a new operation, so its gradients add to the first one's. For slow steps on large models or large batches, pass a larger `timeout`, or `timeout=None` to wait indefinitely. * **An operation raises `OperationFailedError: Operation (...) failed`:** The operation failed on the model resource. Look up the `code` from the message in [Operation errors](#operation-errors). * **An operation fails with `SESSION_NOT_ACTIVE` or `training session is not running`:** The session was stopped, expired after an hour without an operation on its resource, or hit an error. Read `session.retrieve().status` and `session.retrieve().error`, then create a new session, resuming from a training checkpoint if you saved one. * **A running resource stopped on its own:** Either your balance ran out, or the resource went an hour without an operation from any session. Check your balance, then check the session statuses for `EXPIRED`. If neither explains the stop, [contact support](/docs/support) with the resource ID and the time it stopped, and resume from your last training checkpoint. When you [contact support](/docs/support) about a resource or session, include the resource ID, the session ID, the operation ID from the error message if an operation failed, and the time of the problem in UTC. ## Next steps Share one resource across several sessions to run a sweep for one base model's worth of GPUs. How a resource bills, how to estimate a run, and what's refunded on failure. Every field for resources, sessions, and operations, with types and defaults. GPU types, training modes, and session caps per model. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.