> ## Documentation Index
> Fetch the complete documentation index at: https://docs.together.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Multi-LoRA deployment
> Attach the adapters from post-training inference checkpoints to one dedicated deployment of their base model and choose an adapter per request.
export const ConsoleButton = ({href, children}) => {
return
{children}
;
};
To compare several low-rank adapters (LoRAs) before you scale one up in production, serve them all from one [dedicated model inference](/docs/dedicated-endpoints/overview) deployment of their shared base model, called a multi-LoRA deployment. Each adapter comes from an inference checkpoint of a post-training session. Each request names an adapter in its `model` field, or leaves the adapter out to call the base model.
A multi-LoRA deployment serves only adapters from post-training inference checkpoints. You can't attach a training checkpoint, or adapters from the [fine-tuning service](/docs/fine-tuning/overview).
## Why serve multiple adapters from one deployment?
Training projects usually produce several candidate adapters, either from a [hyperparameter sweep](/docs/post-training/multi-lora) or from inference checkpoints at different steps of a single training run. To pick a winner, you run each candidate on the same prompts and compare the outputs. Serving every candidate from one deployment makes that comparison cheaper and faster:
* **Pay for one deployment, not one per adapter:** An adapter is a small set of weights on top of a frozen base model, so every candidate trained on the same base model can share one copy of that model. Five candidates on Qwen3.8 27B cost one deployment's GPUs, not five. Attaching an adapter adds nothing to the deployment's cost.
* **Add candidates without provisioning:** A new deployment can take 20 minutes or more to provision. Attaching an adapter to a running deployment usually takes under a minute, so you can test inference checkpoints as training produces them.
* **Hold the serving environment constant:** Every candidate runs on the same base model and hardware, so differences in output come from the adapters, not the serving setup. To switch candidates, change the `model` string.
A multi-LoRA deployment trades some per-request speed for this flexibility, as [How adapters load](#how-adapters-load) explains. Once you've picked a winner, [serve it on its own](/docs/post-training/deployment#deploy-a-lora-adapter) for production inference.
## How adapters load
Multi-LoRA deployments load adapters dynamically. When you attach an adapter, the deployment records it but doesn't load its weights yet. The first request that names the adapter waits while the replica downloads and loads the weights, and later requests run at normal speed. Each replica loads adapters independently, including replicas that autoscaling adds later.
Each replica keeps a limited number of adapters loaded. When a request names an adapter that isn't loaded and the replica is full, the replica evicts its least recently used adapter. The next request for the evicted adapter waits for it to load again.
## Supported models
You can attach an adapter only to a deployment of the base model it was trained on. These base models support multi-LoRA deployments:
| Training base model | Serving base model | Base model ID |
| - | - | - |
| `Qwen/Qwen3.5-9B` | `Qwen/Qwen3.5-9B-BF16` | `ml_CbuqU8KKAtoGrowco4nqK` |
| `Qwen/Qwen3.6-35B-A3B` | `Qwen/Qwen3.6-35B-A3B-FP8` | `ml_CbSJsoXB4sHXmSRLUm9es` |
| `Qwen/Qwen3.8-27B` | `Qwen/Qwen3.8-27B-FP8` | `ml_CeGmEZ22uUuWLHSKj43rJ` |
| `zai-org/GLM-5.3` | `zai-org/GLM-5.3-FP8` | `ml_CeVYmpAFoURrTarMeHu7h` |
| `moonshotai/Kimi-K3` | `moonshotai/Kimi-K3` (MXFP4) | `ml_CdxPRWoWYcT8mobKFkwm1` |
B200 serving requires a profile whose `accelerator_type` is `nvidia-b200-180gb` and `adapter_mode` is `dynamic`. List the base model's configs to check availability. For GLM-5.3, `cr_Cfoa4Ez74aGkvCetiTuyi` uses eight B200 GPUs. If a model has no B200 multi-LoRA profile, contact your account team for B200 serving support.
Adapters from a model's single-session variant, such as `Qwen/Qwen3.8-27B-single-session`, register against the same serving base model as the default model, so they attach to the same deployments.
GLM-5.3 adapters saved before October 9, 2026 record `zai-org/GLM-5.3-NVFP4` as their base, so they can't be attached. [Prepare them for FP4 inference](/docs/post-training/prepare-for-fp4-inference) instead.
The adapter must come from an inference checkpoint that registers an adapter (`registration.adapter`). A checkpoint that registers a merged model instead [deploys directly](/docs/post-training/deployment#deploy-a-merged-model).
## Find the adapter ID
You attach an adapter by its model registry ID (`ml_...`). Every LoRA inference checkpoint registers its adapter as a model, so there's no upload step. Read the ID from the inference checkpoint.
Read the ID from each of the session's inference checkpoints:
```python theme={null}
from together import Together
client = Together()
session = client.post_training.sessions.retrieve("")
for checkpoint in session.inference_checkpoints:
if checkpoint.registration and checkpoint.registration.adapter:
print(checkpoint.step, checkpoint.registration.adapter.id)
```
After the session ends, list its inference checkpoints with the checkpoints API. There, the IDs are in `inference_registration`:
```python theme={null}
page = client.post_training.checkpoints.list(
session_id="",
type="CHECKPOINT_TYPE_INFERENCE",
)
for checkpoint in page.data:
if (
checkpoint.inference_registration
and checkpoint.inference_registration.adapter
):
print(checkpoint.step, checkpoint.inference_registration.adapter.id)
```
Training sessions
Open the session, then copy the adapter ID from the inference checkpoint's row.
To learn how checkpoints are saved, see [Inference checkpoints](/docs/post-training/checkpoints#inference-checkpoints).
## Attach an adapter
Adapters attach to a running deployment of their base model whose deployment profile serves adapters, called a multi-LoRA deployment. The CLI creates that deployment for you when you deploy the first adapter. In the console, you create the deployment first, then attach adapters to it.
Run `tg beta endpoints deploy` with the adapter ID, an endpoint name, and `--attach-adapter`:
```bash CLI theme={null}
tg beta endpoints deploy ml_abc123 \
--endpoint my-sweep \
--attach-adapter \
--config cr_Cfoa4Ez74aGkvCetiTuyi
```
The CLI looks up the adapter's base model, then looks for a deployment of that model on the `my-sweep` endpoint whose profile serves adapters. If there isn't one, the CLI creates the endpoint and a one-replica deployment of the base model, then attaches the adapter. On success, it confirms the attachment and prints the endpoint, with the adapter listed under its deployment.
To size the new deployment, add any of the [`deploy`](/reference/cli/endpoints-beta#deploy) command's replica, autoscaling, placement, or `--inactive-timeout` flags. They apply only when the CLI creates the deployment. When it attaches the adapter to an existing deployment, the CLI ignores them and prints a warning.
`--config` chooses the deployment profile when the CLI creates the deployment. To list a base model's profiles, run `tg beta models configs` with the base model ID from [Supported models](#supported-models). Choose one whose `accelerator_type` is `nvidia-b200-180gb` and `adapter_mode` is `dynamic`, then pass its ID to `--config`:
```bash CLI theme={null}
tg beta models configs ml_CeVYmpAFoURrTarMeHu7h
```
The example uses the eight-B200 GLM-5.3 profile, which holds 12 adapters, as [Attach more adapters](#attach-more-adapters) explains. Use an adapter trained on GLM-5.3 with this config.
Endpoints
On the **Endpoints** page, select **New endpoint**. Enter an **Endpoint name** and a **Deployment name**, then under **Model**, select the base model from [Supported models](#supported-models), for example GLM-5.3 FP8.
Turn on **Enable LoRA adapters**, then select a B200 card under **Deployment profiles**. With adapters turned on, the console lists only the profiles that serve adapters. Select **Create endpoint**.
Open the endpoint, then open the deployment from the endpoint's **Overview** tab. Wait until the **Status** card shows **Ready**. You can't attach an adapter while the deployment is still provisioning.
On the [Training sessions](https://api.together.ai/custom-training/sessions) page, open the session, then select **Deploy LoRA** from the inference checkpoint's actions menu.
Under **Where to deploy**, select your deployment, then select **Next**. The list shows ready deployments of the adapter's base model that have LoRA adapters turned on.
Review the adapter and the deployment, then select **Deploy**. The dialog shows **Attaching adapter** while the deployment loads the adapter, then **Adapter attached** and the `model` string for calling it.
If you open **Deploy LoRA** before you have a ready deployment, the dialog links to **Create one**, which opens the create form with the base model and **Enable LoRA adapters** already set.
Like any deployment, a new multi-LoRA deployment can take 20 minutes or more to provision, and it bills per minute while its replica runs.
An attached adapter moves through `ADAPTER_LOAD_STATE_PENDING` and `ADAPTER_LOAD_STATE_LOADING` to `ADAPTER_LOAD_STATE_READY`, at which point it can serve requests. On a running deployment, it's usually ready within a minute. If the adapter reaches `ADAPTER_LOAD_STATE_FAILED` instead, its `reason` and `message` say why. To check an adapter's state, see [List attached adapters](#list-attached-adapters).
## Send requests to an adapter
To call an adapter, set `model` to the endpoint string followed by a colon and the adapter ID. Send the request to `https://api-inference.together.ai`:
```bash cURL theme={null}
curl -s -X POST https://api-inference.together.ai/v1/chat/completions \
-H "Authorization: Bearer $TOGETHER_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "your-project-slug/my-sweep:ml_abc123",
"messages": [{"role": "user", "content": "What is the capital of France?"}],
"max_tokens": 128
}'
```
```python Python theme={null}
from together import Together
client = Together(base_url="https://api-inference.together.ai/v1")
response = client.chat.completions.create(
model="your-project-slug/my-sweep:ml_abc123",
messages=[{"role": "user", "content": "What is the capital of France?"}],
max_tokens=128,
)
print(response.model)
print(response.choices[0].message.content)
```
After the colon, you can use the adapter's full model name instead of its ID. To call the base model with no adapter, pass the endpoint string alone (`your-project-slug/my-sweep`).
To call an adapter from any OpenAI-compatible client, set the client's base URL to `https://api-inference.together.ai/v1` and pass the adapter's model string as `model`. Serverless inference can't serve post-training adapters, so a deployment is the only way to call one.
If the adapter isn't attached to the deployment, or the ID after the colon doesn't exist, the request fails with HTTP `400` and the code `adapter_not_registered`. A request that leaves out the adapter suffix (`:ml_abc123`) succeeds, but the base model answers it. Before you score an adapter's outputs, check the `model` field in each response to confirm that the adapter answered. A response from the adapter echoes the full `your-project-slug/my-sweep:ml_abc123` string, and a response from the base model has no adapter suffix.
If you trained with thinking turned off, as the [SFT examples](/docs/post-training/sft) do for Qwen models, pass `"chat_template_kwargs": {"enable_thinking": false}` in the request (`chat_template_kwargs={"enable_thinking": False}` in Python) so the chat template matches training.
## Attach more adapters
To compare more candidates, attach each one to the same deployment. With the CLI, run the same `deploy --attach-adapter` command with each adapter ID. The CLI finds the deployment it created, so you can leave out `--config`. If the endpoint has more than one deployment that serves adapters, pass `--deployment-name` to choose one. In the console, select **Deploy LoRA** on each checkpoint. Each request then chooses its adapter with the `model` string.
Each deployment can hold a limited number of attached adapters, set by its deployment profile. The eight-B200 GLM-5.3 profile in the example holds 12 adapters. Check the selected profile's capacity for other models. Once a deployment is full, attaching another adapter fails until you [detach one](#detach-an-adapter). To make room and attach in one step, select **Deploy anyway** when the console reports that the deployment is at its adapter limit, or pass `force=True` to `client.beta.endpoints.adapters.create`. Either way, the deployment detaches its oldest attached adapter first.
## Compare the candidates
Once the candidates are attached, compare them on the metric that matters for your task:
* **Rerun your baseline script:** If you scored the base model with a cookbook `serverless_eval` script before training, as in [Check a baseline before training](/docs/post-training/reward-functions#check-a-baseline-before-training), run the same script against each candidate so the scores compare directly with the baseline. Set `TOGETHER_BASE_URL` to `https://api-inference.together.ai/v1`, and pass each adapter's model string, such as `your-project-slug/my-sweep:ml_abc123`, as `model`.
* **Run an evaluation:** [Evaluations](/docs/ai-evaluations) can score a dedicated endpoint, or judge two models head to head with a `compare` evaluation. They reference a dedicated model by its endpoint ID, so use them for candidates deployed to their own endpoints.
Once you've picked a winner, [serve it on its own](/docs/post-training/deployment#deploy-a-lora-adapter) for production and [stop the deployment](#stop-the-deployment).
## List attached adapters
`tg beta endpoints get my-sweep` lists each deployment's attached adapters. The CLI doesn't report their state yet, so check it with the SDK, using the endpoint and deployment IDs from `get`:
```python theme={null}
from together import Together
client = Together()
project_id = client.whoami().project_id
for adapter in client.beta.endpoints.adapters.list(
"ep_abc123",
"dep_abc123",
project_id=project_id,
):
for cluster in adapter.per_cluster:
print(adapter.adapter_model_id, adapter.id, cluster.state)
```
```text theme={null}
ml_abc123 dad_abc123 ADAPTER_LOAD_STATE_READY
```
Each entry has the adapter's model ID (`adapter_model_id`) and its attachment ID (`id`, which starts with `dad_`). You need the attachment ID to [detach the adapter](#detach-an-adapter). If a cluster's state is `ADAPTER_LOAD_STATE_FAILED`, its `reason` and `message` say why.
Endpoints
Open the endpoint, then open the deployment. The **LoRA adapters** card lists each attached adapter with its **Status**: **Loading**, **Ready**, or **Failed**.
## Detach an adapter
To detach an adapter, pass its attachment ID and current `etag` to `delete`. Both come from the deployment's list of attached adapters:
```python theme={null}
from together import Together
client = Together()
project_id = client.whoami().project_id
for adapter in client.beta.endpoints.adapters.list(
"ep_abc123",
"dep_abc123",
project_id=project_id,
):
if adapter.adapter_model_id == "ml_abc123":
client.beta.endpoints.adapters.delete(
adapter.id,
project_id=project_id,
endpoint_id="ep_abc123",
deployment_id="dep_abc123",
etag=adapter.etag,
)
break
```
`retrieve`, `update`, and `delete` take the attachment ID, not the adapter's model ID. Passing `ml_abc123` fails with HTTP `400` and "is not a canonical adapter id". If you detach an adapter and attach it again, it gets a new attachment ID.
## Stop the deployment
A running deployment bills per replica-minute whether or not requests name an adapter. When you've finished comparing candidates, stop it:
```bash CLI theme={null}
tg beta endpoints update dep_abc123 --min-replicas 0 --max-replicas 0
```
Endpoints
On the deployment's page, select **Stop deployment**.
Stopping keeps the endpoint, the deployment, and its attached adapters. After you restart the deployment, each replica loads each adapter again on the first request that names it. To restart or delete a deployment, see [Manage deployments](/docs/dedicated-endpoints/manage#stop-a-deployment).
## Troubleshooting
* **`deploy` fails with "... is a LoRA adapter. Choose how to deploy it":** Add `--attach-adapter` to attach the adapter to a multi-LoRA deployment.
* **`deploy` fails with "--attach-adapter only applies to LoRA adapter models":** The model ID isn't a LoRA adapter in the project you're deploying to. Pass the checkpoint's `registration.adapter.id`. A `registration.model.id` is a merged model, which [deploys without `--attach-adapter`](/docs/post-training/deployment#deploy-a-merged-model).
* **`deploy` fails with "No fixed or dynamic LoRA config found":** The adapter's base model doesn't support multi-LoRA deployments. Check [Supported models](#supported-models).
* **`deploy` fails with "Multiple configs found for model":** The base model has more than one profile that serves adapters, and the CLI can't prompt, for example with `--non-interactive`. Pass `--config` with a `dynamic` config.
* **`deploy` fails with "Config ... is not valid for model":** The config isn't one of the model's current deployment profiles. Run `tg beta models configs` with the base model ID and pass a `dynamic` config from its output.
* **`deploy` fails with "Config ... has adapter\_mode disabled":** The config you passed to `--config` doesn't serve adapters. Pass a config whose `adapter_mode` is `dynamic`.
* **`deploy` reports `Failed Precondition` or "Error attaching adapter" right after it creates a deployment:** The new deployment isn't ready to take adapters yet. It keeps running and billing, and so does the endpoint if `deploy` created it. To attach the adapter, wait until `tg beta endpoints get my-sweep` shows the deployment ready, then run the same `deploy` command again. To delete them instead, run `tg beta endpoints rm --force`, which deletes every deployment on the endpoint. If `rm` reports that the deployment must be stopped first, wait for the deployment to stop, then run `rm` again.
* **Attaching fails right after the checkpoint is saved:** Together validates each adapter before you can attach it. Wait a few minutes and retry.
* **Attaching fails because the deployment is full:** [Detach an adapter](#detach-an-adapter), select **Deploy anyway** in the console, or attach with `force=True` in the SDK.
* **Re-attaching fails with `409` right after a detach:** The detach is still finishing. Wait a few seconds and retry.
* **Adapter state is `ADAPTER_LOAD_STATE_FAILED`:** Read `reason` and `message`. `ENGINE_SHAPE_MISMATCH` and `ENGINE_INCOMPATIBLE_TARGET_MODULES` mean the adapter's rank or target modules don't fit the deployment profile.
* **Requests fail with `400` and `adapter_not_registered`:** The ID after the colon is wrong, or the adapter isn't attached to this deployment. [List the attached adapters](#list-attached-adapters) to check, and attach the adapter if it's missing.
* **Requests fail with `500` and `lora_load_failed`:** The deployment couldn't load the adapter, and the error message says why. For example, the adapter's LoRA rank can be incompatible with the deployment's LoRA memory pool, even when it's below the profile's maximum rank. [Detach the adapter](#detach-an-adapter).
* **Responses don't change with the adapter:** If the response's `model` field has no adapter suffix, the request didn't name an adapter, so the base model answered. Check that the `model` string in the request ends with a colon and the adapter ID.
* **Requests fail with "Adapter lookup failed":** Send adapter requests to `https://api-inference.together.ai`, not `https://api.together.ai`.
* **The adapter's answer appears in `reasoning_content`:** The base model is a thinking model. Pass `"chat_template_kwargs": {"enable_thinking": false}` if you trained with thinking off.
## Next steps
Deploy the winning adapter as a merged model for production.
Produce the candidates this page serves.
Serve a GLM or Kimi-K3 adapter as one merged FP4 model.
Scale, stop, and delete endpoints and deployments.
This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.
To compare several low-rank adapters (LoRAs) before you scale one up in production, serve them all from one [dedicated model inference](/docs/dedicated-endpoints/overview) deployment of their shared base model, called a multi-LoRA deployment. Each adapter comes from an inference checkpoint of a post-training session. Each request names an adapter in its `model` field, or leaves the adapter out to call the base model.
Training projects usually produce several candidate adapters, either from a [hyperparameter sweep](/docs/post-training/multi-lora) or from inference checkpoints at different steps of a single training run. To pick a winner, you run each candidate on the same prompts and compare the outputs. Serving every candidate from one deployment makes that comparison cheaper and faster:
* **Pay for one deployment, not one per adapter:** An adapter is a small set of weights on top of a frozen base model, so every candidate trained on the same base model can share one copy of that model. Five candidates on Qwen3.8 27B cost one deployment's GPUs, not five. Attaching an adapter adds nothing to the deployment's cost.
* **Add candidates without provisioning:** A new deployment can take 20 minutes or more to provision. Attaching an adapter to a running deployment usually takes under a minute, so you can test inference checkpoints as training produces them.
* **Hold the serving environment constant:** Every candidate runs on the same base model and hardware, so differences in output come from the adapters, not the serving setup. To switch candidates, change the `model` string.
A multi-LoRA deployment trades some per-request speed for this flexibility, as [How adapters load](#how-adapters-load) explains. Once you've picked a winner, [serve it on its own](/docs/post-training/deployment#deploy-a-lora-adapter) for production inference.
## How adapters load
Multi-LoRA deployments load adapters dynamically. When you attach an adapter, the deployment records it but doesn't load its weights yet. The first request that names the adapter waits while the replica downloads and loads the weights, and later requests run at normal speed. Each replica loads adapters independently, including replicas that autoscaling adds later.
Each replica keeps a limited number of adapters loaded. When a request names an adapter that isn't loaded and the replica is full, the replica evicts its least recently used adapter. The next request for the evicted adapter waits for it to load again.
## Supported models
You can attach an adapter only to a deployment of the base model it was trained on. These base models support multi-LoRA deployments:
| Training base model | Serving base model | Base model ID |
| - | - | - |
| `Qwen/Qwen3.5-9B` | `Qwen/Qwen3.5-9B-BF16` | `ml_CbuqU8KKAtoGrowco4nqK` |
| `Qwen/Qwen3.6-35B-A3B` | `Qwen/Qwen3.6-35B-A3B-FP8` | `ml_CbSJsoXB4sHXmSRLUm9es` |
| `Qwen/Qwen3.8-27B` | `Qwen/Qwen3.8-27B-FP8` | `ml_CeGmEZ22uUuWLHSKj43rJ` |
| `zai-org/GLM-5.3` | `zai-org/GLM-5.3-FP8` | `ml_CeVYmpAFoURrTarMeHu7h` |
| `moonshotai/Kimi-K3` | `moonshotai/Kimi-K3` (MXFP4) | `ml_CdxPRWoWYcT8mobKFkwm1` |
B200 serving requires a profile whose `accelerator_type` is `nvidia-b200-180gb` and `adapter_mode` is `dynamic`. List the base model's configs to check availability. For GLM-5.3, `cr_Cfoa4Ez74aGkvCetiTuyi` uses eight B200 GPUs. If a model has no B200 multi-LoRA profile, contact your account team for B200 serving support.
Adapters from a model's single-session variant, such as `Qwen/Qwen3.8-27B-single-session`, register against the same serving base model as the default model, so they attach to the same deployments.
GLM-5.3 adapters saved before October 9, 2026 record `zai-org/GLM-5.3-NVFP4` as their base, so they can't be attached. [Prepare them for FP4 inference](/docs/post-training/prepare-for-fp4-inference) instead.
The adapter must come from an inference checkpoint that registers an adapter (`registration.adapter`). A checkpoint that registers a merged model instead [deploys directly](/docs/post-training/deployment#deploy-a-merged-model).
## Find the adapter ID
You attach an adapter by its model registry ID (`ml_...`). Every LoRA inference checkpoint registers its adapter as a model, so there's no upload step. Read the ID from the inference checkpoint.