> ## Documentation Index
> Fetch the complete documentation index at: https://docs.together.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Lifecycle, errors, and limits

> Look up the lifecycle states, timers, error codes, limits, and fixes for post-training model resources, sessions, and operations.

This page lists the lifecycle states, timers, error codes, and limits for post-training [model resources, sessions, and operations](https://docs.together.ai/docs/post-training/concepts.md#model-resources-and-sessions), plus fixes for common problems. The same states, errors, and limits apply whether you run a cookbook recipe or your own training loop.

## Lifecycle

![Model resource and session lifecycles. A model resource moves from PENDING to CREATING to READY to STOPPING to STOPPED. It's billed per GPU-minute only while READY, and startup and stopping are free. A session moves from CREATING to RUNNING to STOPPING to STOPPED. A running session can also end in EXPIRED, after its resource is idle for an hour, or in ERROR. Stopping a session detaches it from the resource, which keeps running and billing.](https://mintcdn.com/togetherai-52386018/Ta7hn_4Cp_DKcb3y/images/post-training/resource-and-session-lifecycle.png?fit=max\&auto=format\&n=Ta7hn_4Cp_DKcb3y\&q=85\&s=350fd1a123f02ad3a4100a746ed24d75)

This page uses short state names. The API returns each state as a full enum constant, such as `MODEL_RESOURCES_STATUS_READY` for `READY` or `TRAINING_SESSION_STATUS_RUNNING` for `RUNNING`.

**Model resource:** `PENDING` → `CREATING` → `READY` → `STOPPING` → `STOPPED`, or `ERROR`.

* `PENDING` means the resource is waiting for GPUs. It first waits in a queue, with no time limit, for a cluster with free capacity for its GPU type. Then it waits for its pods to be scheduled on that cluster. While it's queued, `status_details.queue_position` shows its place in line.
* `CREATING` means the pods are scheduled and the base model is loading.
* `READY` means sessions can attach. Billing runs only in this state.
* `ERROR` can follow any active state. `error.code` holds the cause:
  * `MODEL_RESOURCES_ERROR_CODE_CAPACITY_WAIT_TIMEOUT`: The resource left the queue for a cluster, but its pods weren't scheduled within an hour. It was never billed.
  * `MODEL_RESOURCES_ERROR_CODE_PROVISIONING_FAILED`: The resource's pods were scheduled, but it didn't finish starting, either because startup failed or because it wasn't `READY` within an hour. It was never billed.
  * `MODEL_RESOURCES_ERROR_CODE_RESOURCE_UNAVAILABLE`: A `READY` resource failed, and a trailing window of its runtime is [refunded](https://docs.together.ai/docs/post-training/pricing.md#refunds-on-failure).

**Session:** `CREATING` → `RUNNING` → `STOPPING` → `STOPPED`, or `EXPIRED` or `ERROR`.

* `STOPPED` means the session was stopped, which only detaches it from the resource. The resource and its other sessions keep running, and billing continues until the resource itself stops.
* `EXPIRED` means the session's resource reached its [idle timeout](https://docs.together.ai/docs/post-training/model-resources.md#idle-timeout) (one hour by default) without an operation from any of its sessions, so every session on it was detached. An idle session doesn't expire while another session on the same resource is still training. The check runs every minute, so sessions expire within about a minute of the timeout.
* `ERROR` means the session failed to start, to keep running, or to stop. Read the cause from `session.retrieve().error`, which holds a `code`, a `message`, and an `occurred_at` timestamp:
  * `TRAINING_SESSION_ERROR_CODE_RESOURCE_UNAVAILABLE`: The session's model resource is no longer available, so the session can't continue.
  * `TRAINING_SESSION_ERROR_CODE_RESOURCE_AT_CAPACITY`: The session couldn't start because the resource already hosts as many sessions as it accepts. Stop another session on the resource, or use a different resource.
  * `TRAINING_SESSION_ERROR_CODE_TIMED_OUT`: The session didn't become ready in time.
  * `TRAINING_SESSION_ERROR_CODE_SESSION_FAILED`: The session failed for any other reason.

If a new session reaches `STOPPING`, `STOPPED`, `EXPIRED`, or `ERROR` instead of `RUNNING`, `create_session` raises `RuntimeError` with the session ID and that status. You don't get a session handle in that case, so read the error by ID with `client.post_training.sessions.retrieve("your_session_id").error`. To continue training after a session error, create a new session, resuming from a [training checkpoint](https://docs.together.ai/docs/post-training/checkpoints.md#training-checkpoints) if you saved one.

### Timers

| Timer | Value | What happens |
| - | - | - |
| Pod scheduling | 1 hour after the resource leaves the queue for a cluster | The resource fails with `CAPACITY_WAIT_TIMEOUT`. The queue itself has no time limit. |
| Startup | 1 hour in `CREATING` | The resource fails with `PROVISIONING_FAILED`. |
| [Idle timeout](https://docs.together.ai/docs/post-training/model-resources.md#idle-timeout) | The resource's `inactive_timeout` without an operation on the resource, 1 hour by default | Every session on the resource is marked `EXPIRED` and detached, and the resource stops in the same check. |
| Credit limit exceeded or organization blocked | Checked approximately once per minute. Shutdown can take longer. | The resource stops when the billing system reports either condition, and its sessions end with it. Falling below the [creation minimum](https://docs.together.ai/docs/post-training/pricing.md#minimum-credit-balance) alone doesn't stop a running resource. |

On long runs, save [training checkpoints](https://docs.together.ai/docs/post-training/checkpoints.md#training-checkpoints) so that an expiry or unexpected stop loses only the work since your last checkpoint.

## Operation errors

When an operation such as `forward_backward`, `optim_step`, or `sample` fails on the model resource, the SDK raises `OperationFailedError`. The `together.post_training` module doesn't export that class, so catch its parent class, `RuntimeError`. The exception message holds the operation ID and the operation's `error`, which has a `code` and a `message` that says what went wrong:

```text theme={null}
Operation (<operation_id>) failed: OperationError(code='TRAINING_OPERATION_ERROR_CODE_INVALID_INPUT', message='Operation failed because the request contains invalid training inputs: ...')
```

In the REST API, the failed operation has the status `TRAINING_OPERATION_STATUS_FAILED` and the same `error`.

| Code | What happened | What to do |
| - | - | - |
| `TRAINING_OPERATION_ERROR_CODE_INVALID_INPUT` | The request's training inputs are invalid, or its batch size or sequence length is over the [limits](#limits). The message usually names the sample and the field, for example a `loss_fn_inputs` row whose length doesn't match the sample's tokens, a missing row that the loss function requires, or a value that's NaN or infinite. | Fix the inputs and submit the operation again. |
| `TRAINING_OPERATION_ERROR_CODE_NON_FINITE_LOSS` | `forward_backward` produced a loss or metric that's NaN or infinite, which means the run diverged. `optim_step` fails with the same code when it sees a NaN or infinite gradient norm. The update isn't applied, and the session refuses every later `optim_step`. | Stop the session, lower the learning rate, and start a new session from your last [training checkpoint](https://docs.together.ai/docs/post-training/checkpoints.md#resume-or-fork-a-run). On a full-weight resource, a new session started without a checkpoint begins from the base weights, so resume from a checkpoint to keep the run's progress. |
| `TRAINING_OPERATION_ERROR_CODE_RESOURCE_EXHAUSTED` | The operation ran out of GPU memory. | Shorten the sequences (shorter prompts, or a lower `max_tokens` when sampling), sample fewer completions per prompt with `num_samples`, or send fewer samples per `forward_backward` call. |
| `TRAINING_OPERATION_ERROR_CODE_TIMEOUT` | The operation timed out on the model resource. This isn't the SDK's `TimeoutError`, which only means your code stopped waiting. | Submit the operation again. If it keeps failing, [contact support](https://portal.usepylon.com/together-ai/forms/support-request) with the operation ID. |
| `TRAINING_OPERATION_ERROR_CODE_SESSION_NOT_ACTIVE` | The session was stopped, expired, or hit an error after the operation was submitted. | Read `session.retrieve().status` and `session.retrieve().error`, then create a new session, resuming from a training checkpoint if you saved one. |
| `TRAINING_OPERATION_ERROR_CODE_INTERNAL_ERROR` | The operation failed unexpectedly on the model resource. | Submit the operation again. If it keeps failing, [contact support](https://portal.usepylon.com/together-ai/forms/support-request) with the operation ID. |

If the session had already left `RUNNING` when you submitted the operation, the API rejects the request without creating an operation. It returns HTTP `400` with `training session is not running (current status: ...)`, which the SDK raises as `BadRequestError`. Handle it the same way as `SESSION_NOT_ACTIVE`.

## Limits

| Limit | Value |
| - | - |
| Concurrent low-rank adapter (LoRA) sessions per resource | Per model. See [Supported models](https://docs.together.ai/docs/post-training/supported-models.md). |
| Full-weight sessions per resource | 1. |
| Generator replicas per resource | 8. |
| GPUs per organization | Set per organization. GPUs held by `PENDING`, `CREATING`, and `READY` resources count toward it. See [Troubleshooting](#troubleshooting). |
| Max batch per `forward_backward` | 4,096 samples. This caps the request payload, not GPU memory. The trainer splits a step into micro-batches regardless. |
| Max sequence length | Per model and training mode. See [Supported models](https://docs.together.ai/docs/post-training/supported-models.md). |

Creating a resource also needs [enough credit to run it for four hours](https://docs.together.ai/docs/post-training/pricing.md#minimum-credit-balance).

### Rate limits

Rate limits apply per project or per session, depending on the call. A *burst* is how many calls can go through at once before the rate applies. When you exceed a limit, the API returns HTTP `429`.

| Call | Scope | Rate | Burst |
| - | - | - | - |
| Create a model resource | Project | 20 per hour | 5 |
| Start a training session | Project | 20 per minute | 5 |
| `forward_backward`, `forward`, `custom_forward_backward`, `optim_step`, `weights_sync` | Session | 20 per second each | 40 |
| `sample`, `sample_batch` | Session | 2,000 per second | 1,000 |
| Save a training or inference checkpoint | Session | 10 per minute each | 5 |
| Download a checkpoint | Project | 10 per second | 20 |

The SDK retries any request that fails with a connection error or an HTTP `408`, `409`, `429`, or `5xx` response. It retries session calls, which include every operation, up to seven times. If a call still gets a `429` after the last retry, the SDK raises `RateLimitError`. Each operation carries an idempotency key, so a retried submission doesn't create a second operation. Don't wrap operation calls in your own retry loop.

## Troubleshooting

* **The resource stays `PENDING` for a long time:** No B200 GPU capacity is free yet. The resource stays in the queue until GPUs free up or you stop it, and you aren't billed while it waits. Check `status_details.queue_position` to see its place in line.
* **`create` raises `TimeoutError` while the resource is still `PENDING`:** The `ready_timeout` you passed ran out before the resource left the queue, so the SDK stopped the resource. To wait longer, create the resource again with a larger `ready_timeout`, or leave it unset to wait until the resource gets GPUs. The new resource joins the back of the queue, so when capacity is scarce, leave `ready_timeout` unset from the start.
* **Create is rejected with `rate limit exceeded, retry after ...`:** Your project has reached its [create rate limit](#rate-limits). The API returns HTTP `429`, which the SDK raises as `RateLimitError` after retrying. Wait for the time in the message, then retry. To stay under the limit, reuse a running resource by [attaching sessions to it](https://docs.together.ai/docs/post-training/model-resources.md#attach-to-an-existing-resource) instead of creating one for each run.
* **Create is rejected with `organization GPU quota exceeded`:** Your organization has a quota on the total GPUs that its `PENDING`, `CREATING`, and `READY` resources hold, and the new resource would exceed it. The API returns HTTP `403`, which the SDK raises as `PermissionDeniedError`, with a message such as `organization GPU quota exceeded: 112 of 120 GPUs in use, requested 16`. Stop resources you no longer need, or [contact support](https://portal.usepylon.com/together-ai/forms/support-request) to request a higher quota.
* **Create fails for the GPU type, training mode, or generator replicas:** The model doesn't support that combination. Check the model's row on [Supported models](https://docs.together.ai/docs/post-training/supported-models.md#catalog).
* **Create is rejected with `project balance cannot cover 4 hours of this model resource`:** The API returns HTTP `402` (`insufficient_balance`) when your project's balance plus its credit limit can't [cover four hours of the resource](https://docs.together.ai/docs/post-training/pricing.md#minimum-credit-balance) at its hourly rate, or when the project is already past its credit limit. Add credits and retry, or lower the hourly rate with fewer generator replicas, LoRA instead of full-weight training, or a smaller base model.
* **`stop` is refused because a session is still active:** Stop the sessions first, or pass `force=True`. The context-manager exit on an owned resource doesn't force, so it fails the same way if a session you don't own is still attached.
* **An operation call raises `TimeoutError`:** The call's `timeout`, 300 seconds by default or 7200 seconds for checkpoint saves, ran out before the operation finished. The SDK stopped waiting, but the operation keeps running on the service, and there's no call to cancel it. Don't resubmit it, because a second `forward_backward` is a new operation whose gradients add to the first one's. For slow steps on large models or large batches, pass a larger `timeout`, or `timeout=None` to wait indefinitely.
* **An operation raises `OperationFailedError: Operation (...) failed`:** Look up the `code` from the message in [Operation errors](#operation-errors).
* **An operation fails with `SESSION_NOT_ACTIVE` or `training session is not running`:** The session was stopped, expired when its resource reached its [idle timeout](https://docs.together.ai/docs/post-training/model-resources.md#idle-timeout), or hit an error. Read `session.retrieve().status` and `session.retrieve().error`, then create a new session, resuming from a training checkpoint if you saved one.
* **A running resource stopped on its own:** The billing system reported that your project's credit limit was exceeded or your organization was blocked, or the resource reached its [idle timeout](https://docs.together.ai/docs/post-training/model-resources.md#idle-timeout) (one hour by default) without an operation from any session. Check your available credit and account status, then check the session statuses for `EXPIRED`. If neither explains the stop, [contact support](https://portal.usepylon.com/together-ai/forms/support-request) with the resource ID and the time it stopped. To continue training, resume from your last training checkpoint.

When you [contact support](https://portal.usepylon.com/together-ai/forms/support-request) about a resource or session, include the resource ID, the session ID, the operation ID from the error message if an operation failed, and the time of the problem in UTC.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.