Multi-LoRA deployment

The three stages of a post-training project. Train: one model resource trains up to five adapters, A, B, and C, while you pay for only one base model. Evaluate: deploy the adapters to a dedicated endpoint and compare them on live traffic, where adapter B wins. Productionize: merge and quantize the winner, B, then scale it across several replicas for production inference.

To compare several low-rank adapters (LoRAs) before you scale one up in production, serve them all from one dedicated model inference deployment of their shared base model, called a multi-LoRA deployment. Each adapter comes from an inference checkpoint of a post-training session. Each request names an adapter in its model field, or leaves the adapter out to call the base model.

A multi-LoRA deployment serves only adapters from post-training inference checkpoints. You can’t attach a training checkpoint, or adapters from the fine-tuning service.

Why serve multiple adapters from one deployment?

Five candidate adapters, A through E, served two ways. Separate deployments give each candidate its own copy of the base model, which takes five times the GPUs, and each deployment takes several minutes to provision. One multi-LoRA deployment serves every candidate on one shared, frozen base model, which takes one deployment's GPUs, and attaching a new candidate takes under a minute.

Training projects usually produce several candidate adapters, either from a hyperparameter sweep or from inference checkpoints at different steps of a single training run. To pick a winner, you run each candidate on the same prompts and compare the outputs. Serving every candidate from one deployment makes that comparison cheaper and faster:

Pay for one deployment, not one per adapter: An adapter is a small set of weights on top of a frozen base model, so every candidate trained on the same base model can share one copy of that model. Five candidates on Qwen3.8 27B cost one deployment’s GPUs, not five. Attaching an adapter adds nothing to the deployment’s cost.
Add candidates without provisioning: A new deployment can take 20 minutes or more to provision. Attaching an adapter to a running deployment usually takes under a minute, so you can test inference checkpoints as training produces them.
Hold the serving environment constant: Every candidate runs on the same base model and hardware, so differences in output come from the adapters, not the serving setup. To switch candidates, change the model string.

A multi-LoRA deployment trades some per-request speed for this flexibility, as How adapters load explains. Once you’ve picked a winner, serve it on its own for production inference.

How adapters load

Multi-LoRA deployments load adapters dynamically. When you attach an adapter, the deployment records it but doesn’t load its weights yet. The first request that names the adapter waits while the replica downloads and loads the weights, and later requests run at normal speed. Each replica loads adapters independently, including replicas that autoscaling adds later. Each replica keeps a limited number of adapters loaded. When a request names an adapter that isn’t loaded and the replica is full, the replica evicts its least recently used adapter. The next request for the evicted adapter waits for it to load again.

Supported models

You can attach an adapter only to a deployment of the base model it was trained on. These base models support multi-LoRA deployments:

Training base model	Serving base model	Base model ID
`Qwen/Qwen3.5-9B`	`Qwen/Qwen3.5-9B-BF16`	`ml_CbuqU8KKAtoGrowco4nqK`
`Qwen/Qwen3.6-35B-A3B`	`Qwen/Qwen3.6-35B-A3B-FP8`	`ml_CbSJsoXB4sHXmSRLUm9es`
`Qwen/Qwen3.8-27B`	`Qwen/Qwen3.8-27B-FP8`	`ml_CeGmEZ22uUuWLHSKj43rJ`
`zai-org/GLM-5.3`	`zai-org/GLM-5.3-FP8`	`ml_CeVYmpAFoURrTarMeHu7h`
`moonshotai/Kimi-K3`	`moonshotai/Kimi-K3` (MXFP4)	`ml_CdxPRWoWYcT8mobKFkwm1`

B200 serving requires a profile whose accelerator_type is nvidia-b200-180gb and adapter_mode is dynamic. List the base model’s configs to check availability. For GLM-5.3, cr_Cfoa4Ez74aGkvCetiTuyi uses eight B200 GPUs. If a model has no B200 multi-LoRA profile, contact your account team for B200 serving support. Adapters from a model’s single-session variant, such as Qwen/Qwen3.8-27B-single-session, register against the same serving base model as the default model, so they attach to the same deployments. GLM-5.3 adapters saved before October 9, 2026 record zai-org/GLM-5.3-NVFP4 as their base, so they can’t be attached. Prepare them for FP4 inference instead. The adapter must come from an inference checkpoint that registers an adapter (registration.adapter). A checkpoint that registers a merged model instead deploys directly.

Find the adapter ID

You attach an adapter by its model registry ID (ml_...). Every LoRA inference checkpoint registers its adapter as a model, so there’s no upload step. Read the ID from the inference checkpoint.

SDK
Console

Read the ID from each of the session’s inference checkpoints:

from together import Together

client = Together()
session = client.post_training.sessions.retrieve("<session_id>")
for checkpoint in session.inference_checkpoints:
    if checkpoint.registration and checkpoint.registration.adapter:
        print(checkpoint.step, checkpoint.registration.adapter.id)

After the session ends, list its inference checkpoints with the checkpoints API. There, the IDs are in inference_registration:

page = client.post_training.checkpoints.list(
    session_id="<session_id>",
    type="CHECKPOINT_TYPE_INFERENCE",
)
for checkpoint in page.data:
    if (
        checkpoint.inference_registration
        and checkpoint.inference_registration.adapter
    ):
        print(checkpoint.step, checkpoint.inference_registration.adapter.id)

To learn how checkpoints are saved, see Inference checkpoints.

Attach an adapter

Adapters attach to a running deployment of their base model whose deployment profile serves adapters, called a multi-LoRA deployment. The CLI creates that deployment for you when you deploy the first adapter. In the console, you create the deployment first, then attach adapters to it.

CLI
Console

Run tg beta endpoints deploy with the adapter ID, an endpoint name, and --attach-adapter:

CLI

tg beta endpoints deploy ml_abc123 \
  --endpoint my-sweep \
  --attach-adapter \
  --config cr_Cfoa4Ez74aGkvCetiTuyi

The CLI looks up the adapter’s base model, then looks for a deployment of that model on the my-sweep endpoint whose profile serves adapters. If there isn’t one, the CLI creates the endpoint and a one-replica deployment of the base model, then attaches the adapter. On success, it confirms the attachment and prints the endpoint, with the adapter listed under its deployment.To size the new deployment, add any of the deploy command’s replica, autoscaling, placement, or --inactive-timeout flags. They apply only when the CLI creates the deployment. When it attaches the adapter to an existing deployment, the CLI ignores them and prints a warning.--config chooses the deployment profile when the CLI creates the deployment. To list a base model’s profiles, run tg beta models configs with the base model ID from Supported models. Choose one whose accelerator_type is nvidia-b200-180gb and adapter_mode is dynamic, then pass its ID to --config:

CLI

tg beta models configs ml_CeVYmpAFoURrTarMeHu7h

The example uses the eight-B200 GLM-5.3 profile, which holds 12 adapters, as Attach more adapters explains. Use an adapter trained on GLM-5.3 with this config.

Create a multi-LoRA deployment

On the Endpoints page, select New endpoint. Enter an Endpoint name and a Deployment name, then under Model, select the base model from Supported models, for example GLM-5.3 FP8.

Turn on LoRA adapters

Turn on Enable LoRA adapters, then select a B200 card under Deployment profiles. With adapters turned on, the console lists only the profiles that serve adapters. Select Create endpoint.

Wait for the deployment

Open the endpoint, then open the deployment from the endpoint’s Overview tab. Wait until the Status card shows Ready. You can’t attach an adapter while the deployment is still provisioning.

Open the checkpoint

On the Training sessions page, open the session, then select Deploy LoRA from the inference checkpoint’s actions menu.

Choose the deployment

Under Where to deploy, select your deployment, then select Next. The list shows ready deployments of the adapter’s base model that have LoRA adapters turned on.

Deploy

Review the adapter and the deployment, then select Deploy. The dialog shows Attaching adapter while the deployment loads the adapter, then Adapter attached and the model string for calling it.

If you open Deploy LoRA before you have a ready deployment, the dialog links to Create one, which opens the create form with the base model and Enable LoRA adapters already set.

Like any deployment, a new multi-LoRA deployment can take 20 minutes or more to provision, and it bills per minute while its replica runs. An attached adapter moves through ADAPTER_LOAD_STATE_PENDING and ADAPTER_LOAD_STATE_LOADING to ADAPTER_LOAD_STATE_READY, at which point it can serve requests. On a running deployment, it’s usually ready within a minute. If the adapter reaches ADAPTER_LOAD_STATE_FAILED instead, its reason and message say why. To check an adapter’s state, see List attached adapters.

Send requests to an adapter

To call an adapter, set model to the endpoint string followed by a colon and the adapter ID. Send the request to https://api-inference.together.ai:

curl -s -X POST https://api-inference.together.ai/v1/chat/completions \
  -H "Authorization: Bearer $TOGETHER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "your-project-slug/my-sweep:ml_abc123",
    "messages": [{"role": "user", "content": "What is the capital of France?"}],
    "max_tokens": 128
  }'

from together import Together

client = Together(base_url="https://api-inference.together.ai/v1")

response = client.chat.completions.create(
    model="your-project-slug/my-sweep:ml_abc123",
    messages=[{"role": "user", "content": "What is the capital of France?"}],
    max_tokens=128,
)
print(response.model)
print(response.choices[0].message.content)

After the colon, you can use the adapter’s full model name instead of its ID. To call the base model with no adapter, pass the endpoint string alone (your-project-slug/my-sweep). To call an adapter from any OpenAI-compatible client, set the client’s base URL to https://api-inference.together.ai/v1 and pass the adapter’s model string as model. Serverless inference can’t serve post-training adapters, so a deployment is the only way to call one. If the adapter isn’t attached to the deployment, or the ID after the colon doesn’t exist, the request fails with HTTP 400 and the code adapter_not_registered. A request that leaves out the adapter suffix (:ml_abc123) succeeds, but the base model answers it. Before you score an adapter’s outputs, check the model field in each response to confirm that the adapter answered. A response from the adapter echoes the full your-project-slug/my-sweep:ml_abc123 string, and a response from the base model has no adapter suffix. If you trained with thinking turned off, as the SFT examples do for Qwen models, pass "chat_template_kwargs": {"enable_thinking": false} in the request (chat_template_kwargs={"enable_thinking": False} in Python) so the chat template matches training.

Attach more adapters

To compare more candidates, attach each one to the same deployment. With the CLI, run the same deploy --attach-adapter command with each adapter ID. The CLI finds the deployment it created, so you can leave out --config. If the endpoint has more than one deployment that serves adapters, pass --deployment-name to choose one. In the console, select Deploy LoRA on each checkpoint. Each request then chooses its adapter with the model string. Each deployment can hold a limited number of attached adapters, set by its deployment profile. The eight-B200 GLM-5.3 profile in the example holds 12 adapters. Check the selected profile’s capacity for other models. Once a deployment is full, attaching another adapter fails until you detach one. To make room and attach in one step, select Deploy anyway when the console reports that the deployment is at its adapter limit, or pass force=True to client.beta.endpoints.adapters.create. Either way, the deployment detaches its oldest attached adapter first.

Compare the candidates

Once the candidates are attached, compare them on the metric that matters for your task:

Rerun your baseline script: If you scored the base model with a cookbook serverless_eval script before training, as in Check a baseline before training, run the same script against each candidate so the scores compare directly with the baseline. Set TOGETHER_BASE_URL to https://api-inference.together.ai/v1, and pass each adapter’s model string, such as your-project-slug/my-sweep:ml_abc123, as model.
Run an evaluation: Evaluations can score a dedicated endpoint, or judge two models head to head with a compare evaluation. They reference a dedicated model by its endpoint ID, so use them for candidates deployed to their own endpoints.

Once you’ve picked a winner, serve it on its own for production and stop the deployment.

List attached adapters

CLI
Console

tg beta endpoints get my-sweep lists each deployment’s attached adapters. The CLI doesn’t report their state yet, so check it with the SDK, using the endpoint and deployment IDs from get:

from together import Together

client = Together()
project_id = client.whoami().project_id

for adapter in client.beta.endpoints.adapters.list(
    "ep_abc123",
    "dep_abc123",
    project_id=project_id,
):
    for cluster in adapter.per_cluster:
        print(adapter.adapter_model_id, adapter.id, cluster.state)

ml_abc123 dad_abc123 ADAPTER_LOAD_STATE_READY

Each entry has the adapter’s model ID (adapter_model_id) and its attachment ID (id, which starts with dad_). You need the attachment ID to detach the adapter. If a cluster’s state is ADAPTER_LOAD_STATE_FAILED, its reason and message say why.

Detach an adapter

To detach an adapter, pass its attachment ID and current etag to delete. Both come from the deployment’s list of attached adapters:

from together import Together

client = Together()
project_id = client.whoami().project_id

for adapter in client.beta.endpoints.adapters.list(
    "ep_abc123",
    "dep_abc123",
    project_id=project_id,
):
    if adapter.adapter_model_id == "ml_abc123":
        client.beta.endpoints.adapters.delete(
            adapter.id,
            project_id=project_id,
            endpoint_id="ep_abc123",
            deployment_id="dep_abc123",
            etag=adapter.etag,
        )
        break

retrieve, update, and delete take the attachment ID, not the adapter’s model ID. Passing ml_abc123 fails with HTTP 400 and “is not a canonical adapter id”. If you detach an adapter and attach it again, it gets a new attachment ID.

Stop the deployment

A running deployment bills per replica-minute whether or not requests name an adapter. When you’ve finished comparing candidates, stop it:

CLI
Console

CLI

tg beta endpoints update dep_abc123 --min-replicas 0 --max-replicas 0

Stopping keeps the endpoint, the deployment, and its attached adapters. After you restart the deployment, each replica loads each adapter again on the first request that names it. To restart or delete a deployment, see Manage deployments.

Troubleshooting

deploy fails with ”… is a LoRA adapter. Choose how to deploy it”: Add --attach-adapter to attach the adapter to a multi-LoRA deployment.
deploy fails with “—attach-adapter only applies to LoRA adapter models”: The model ID isn’t a LoRA adapter in the project you’re deploying to. Pass the checkpoint’s registration.adapter.id. A registration.model.id is a merged model, which deploys without --attach-adapter.
deploy fails with “No fixed or dynamic LoRA config found”: The adapter’s base model doesn’t support multi-LoRA deployments. Check Supported models.
deploy fails with “Multiple configs found for model”: The base model has more than one profile that serves adapters, and the CLI can’t prompt, for example with --non-interactive. Pass --config with a dynamic config.
deploy fails with “Config … is not valid for model”: The config isn’t one of the model’s current deployment profiles. Run tg beta models configs with the base model ID and pass a dynamic config from its output.
deploy fails with “Config … has adapter_mode disabled”: The config you passed to --config doesn’t serve adapters. Pass a config whose adapter_mode is dynamic.
deploy reports Failed Precondition or “Error attaching adapter” right after it creates a deployment: The new deployment isn’t ready to take adapters yet. It keeps running and billing, and so does the endpoint if deploy created it. To attach the adapter, wait until tg beta endpoints get my-sweep shows the deployment ready, then run the same deploy command again. To delete them instead, run tg beta endpoints rm <endpoint_id> --force, which deletes every deployment on the endpoint. If rm reports that the deployment must be stopped first, wait for the deployment to stop, then run rm again.
Attaching fails right after the checkpoint is saved: Together validates each adapter before you can attach it. Wait a few minutes and retry.
Attaching fails because the deployment is full: Detach an adapter, select Deploy anyway in the console, or attach with force=True in the SDK.
Re-attaching fails with 409 right after a detach: The detach is still finishing. Wait a few seconds and retry.
Adapter state is ADAPTER_LOAD_STATE_FAILED: Read reason and message. ENGINE_SHAPE_MISMATCH and ENGINE_INCOMPATIBLE_TARGET_MODULES mean the adapter’s rank or target modules don’t fit the deployment profile.
Requests fail with 400 and adapter_not_registered: The ID after the colon is wrong, or the adapter isn’t attached to this deployment. List the attached adapters to check, and attach the adapter if it’s missing.
Requests fail with 500 and lora_load_failed: The deployment couldn’t load the adapter, and the error message says why. For example, the adapter’s LoRA rank can be incompatible with the deployment’s LoRA memory pool, even when it’s below the profile’s maximum rank. Detach the adapter.
Responses don’t change with the adapter: If the response’s model field has no adapter suffix, the request didn’t name an adapter, so the base model answered. Check that the model string in the request ends with a colon and the adapter ID.
Requests fail with “Adapter lookup failed”: Send adapter requests to https://api-inference.together.ai, not https://api.together.ai.
The adapter’s answer appears in reasoning_content: The base model is a thinking model. Pass "chat_template_kwargs": {"enable_thinking": false} if you trained with thinking off.

Next steps

Deploy a checkpoint

Deploy the winning adapter as a merged model for production.

Train multiple adapters on one base model

Produce the candidates this page serves.

Prepare for FP4 inference

Serve a GLM or Kimi-K3 adapter as one merged FP4 model.

Manage deployments

Scale, stop, and delete endpoints and deployments.

GET STARTED

SERVERLESS

INFERENCE APIS

DEDICATED MODEL INFERENCE

DEDICATED CONTAINER INFERENCE

POST-TRAINING

FINE-TUNING

GPU CLUSTERS

CODE EXECUTION

ADMINISTRATION

Why serve multiple adapters from one deployment?

How adapters load

Supported models

Find the adapter ID

Attach an adapter

Send requests to an adapter

Attach more adapters

Compare the candidates

List attached adapters

Detach an adapter

Stop the deployment

Troubleshooting

Next steps

Deploy a checkpoint

Train multiple adapters on one base model

Prepare for FP4 inference

Manage deployments

​Why serve multiple adapters from one deployment?

​How adapters load

​Supported models

​Find the adapter ID

​Attach an adapter

​Send requests to an adapter

​Attach more adapters

​Compare the candidates

​List attached adapters

​Detach an adapter

​Stop the deployment

​Troubleshooting

​Next steps

Deploy a checkpoint

Train multiple adapters on one base model

Prepare for FP4 inference

Manage deployments

Why serve multiple adapters from one deployment?

How adapters load

Supported models

Find the adapter ID

Attach an adapter

Send requests to an adapter

Attach more adapters

Compare the candidates

List attached adapters

Detach an adapter

Stop the deployment

Troubleshooting

Next steps