Prepare for FP4 inference

Dedicated model inference serves merged models on GLM-5.3 and Kimi-K3 in FP4, a 4-bit weight format that fits these large models on fewer GPUs and serves them faster than full precision. Training doesn’t run on those FP4 weights, so a low-rank adapter (LoRA) you train doesn’t match the FP4 model you deploy. An FP4 preparation job gives you a trained model in the format production serving expects:

Serve at FP4 speed and cost: An adapter served on top of its base model adds work to every request. A merged model has the adapter’s changes built into its weights, so it serves like the FP4 base model, on the same hardware.
Keep the adapter’s quality: Merging an adapter straight into FP4 weights would apply its changes to weights that already carry quantization error. The job merges into the full-precision (BF16) weights first, then calibrates and quantizes the merged model to FP4, so the adapter’s changes survive quantization.

Prepare adapters from post-training sessions. A job takes 6 to 18 hours, depending on the base model, and produces one model, so start one after you’ve chosen a winning adapter. To compare several candidates first, serve them on one deployment.

Supported models

The adapter’s base model must be one of the following:

Base model	Adapter base ID	Typical running time
GLM-5.3	`zai-org/GLM-5.3-FP8` or `zai-org/GLM-5.3-NVFP4`	About 6 hours
Kimi-K3	`moonshotai/Kimi-K3`	About 18 hours

The running time excludes any wait for GPU capacity before the job starts. Together sets the merge and quantization parameters for each model, and you can’t change them. A GLM-5.3 adapter records zai-org/GLM-5.3-FP8 as its base. Adapters saved before October 9, 2026 record zai-org/GLM-5.3-NVFP4. A Kimi-K3 job writes the model in the release’s own MXFP4 format. The adapter must be a LoRA adapter from a post-training session. Full-weight models aren’t accepted.

Find the adapter ID

A job takes the adapter’s model registry ID (ml_...) and, optionally, a revision ID (rv_...). If you omit the revision, the job uses the adapter’s latest revision. Read both from registration.adapter on the session’s inference checkpoint, as Find your model’s ID shows, or from inference_registration.adapter after the session ends, as Find a checkpoint shows. Only a checkpoint that registers an adapter can be prepared. A checkpoint that registers a merged model deploys directly.

Estimate the job

To check a job’s price and running time without creating anything, estimate it with the adapter ID:

tg training fp4 estimate ml_abc123 --adapter-revision-id rv_abc123

from together import Together

client = Together()

estimate = client.post_training.prepare_for_fp4_inference.estimate_cost(
    inputs={
        "adapter_object_id": "ml_abc123",
        "adapter_revision_id": "rv_abc123",
    },
)
print(estimate.price_usd, estimate.time_hours, estimate.allowed_to_proceed)

The CLI prints the price and the expected running time:

The estimated price of this job is $43.14 and it is expected to run for about 6.0h (excluding queue time).

The running time excludes the wait for capacity. It’s an estimate, not a time limit, so a job that runs longer than estimated keeps running. If your credit can’t cover the price, the CLI adds a warning, and in Python allowed_to_proceed is False. A successful estimate also confirms that the adapter is eligible, because estimating and creating a job accept the same adapters.

Calibrate on your own data

The job calibrates the quantized model on a general text corpus. To make calibration match your workload, add a conversation dataset with --calibration-file (-c). The job then uses your dataset for half of the calibration corpus and general text for the other half. The dataset must be a JSONL file of conversations. Each row has messages and, optionally, tools, in the same chat format as SFT data. The file must be at most 64 MiB, and it must belong to the same project as the adapter.

# A local file: the CLI uploads it with purpose "calibration"
tg training fp4 create ml_abc123 --calibration-file ./calibration.jsonl

# A file you uploaded earlier
tg training fp4 create ml_abc123 --calibration-file file-abc123

from together import Together

client = Together()

calibration = client.files.upload(
    file="calibration.jsonl", purpose="calibration"
)

job = client.post_training.prepare_for_fp4_inference.create(
    inputs={
        "adapter_object_id": "ml_abc123",
        "calibration_file_id": calibration.id,
    },
)

estimate accepts the same flag, and it checks that the file exists and passed validation before you start a job. It doesn’t count tokens. Once the model’s chat template is applied, your dataset must supply at least 598,016 tokens to fill its half of the calibration corpus, or the job fails with a user_error. Add more conversations and retry.

Start the job

Create the job with the same adapter ID:

tg training fp4 create ml_abc123 --adapter-revision-id rv_abc123

from together import Together

client = Together()

job = client.post_training.prepare_for_fp4_inference.create(
    inputs={
        "adapter_object_id": "ml_abc123",
        "adapter_revision_id": "rv_abc123",
    },
)
print(job.id, job.status)

Before it creates the job, the CLI shows the estimate and asks you to confirm. To skip the prompt in scripts, pass --confirm (-y). To wait for the job in the same command, add --watch (see Monitor the job). For every flag, see the tg training reference.

The CLI doesn’t ask for confirmation when it doesn’t run in an interactive terminal, for example in CI, with redirected input or output, or when it detects an AI agent. In these cases create starts the job at once. Run estimate first.

The new job has an ID in the form shp-quant-... and starts in pending. Save the ID to monitor or cancel the job. Each request starts a new job, even for an adapter you’ve already prepared. A repeat job uploads its result to the same model as the earlier job, so reuse that model instead of starting another.

Monitor the job

A job moves from pending to queued to running to completed. Watch it until it reaches a terminal status:

tg training fp4 get shp-quant-1a2b3c4d-5e6f --watch

import time

from together import Together

client = Together()

TERMINAL = {"completed", "cancelled", "error", "user_error"}

job = client.post_training.prepare_for_fp4_inference.retrieve(
    "shp-quant-1a2b3c4d-5e6f"
)
while job.status not in TERMINAL:
    time.sleep(300)
    job = client.post_training.prepare_for_fp4_inference.retrieve(job.id)

print(job.status)
if job.status == "completed":
    print(job.results.api_model_object_id, job.results.api_model_revision_id)

--watch prints each status change and new event until the job reaches a terminal status. The command exits with a non-zero code if the job failed. Stopping the command doesn’t stop the job, so run it again to resume watching.

Status	Meaning
`pending`, `queued`	Accepted and waiting for a GPU.
`running`	Merging, quantizing, or uploading the model.
`completed`	Done. `results` holds the new model’s IDs.
`cancel_requested`, `cancelled`	You cancelled the job.
`user_error`	The job failed because of its input. The job’s events say what to fix.
`error`	The job failed on Together’s side. Retry, or contact support if it fails again.

When the job reports completed, the prepared model is already validated for deployment. The CLI prints the model’s ID (ml_...) and revision ID (rv_...). In Python, read them from job.results.api_model_object_id and job.results.api_model_revision_id. --watch prints the job’s events as they arrive. To read a job’s events later, such as the reason it failed, list them:

tg training fp4 list-events shp-quant-1a2b3c4d-5e6f

from together import Together

client = Together()

events = client.post_training.prepare_for_fp4_inference.list_events(
    "shp-quant-1a2b3c4d-5e6f"
)
for event in events.data:
    print(event.created_at, event.type, event.message)

A failed job ends with either a SHAPING_USER_ERROR event or a SHAPING_ERROR event. A SHAPING_USER_ERROR event’s message says how to fix the input. To list every FP4 preparation job in your project, run tg training fp4 list or call client.post_training.prepare_for_fp4_inference.list().

Deploy the prepared model

The prepared model is a private model in your project, registered against the base model that serves it in FP4. For GLM-5.3, that’s zai-org/GLM-5.3-NVFP4, even when the adapter records zai-org/GLM-5.3-FP8. List its deployment profiles, then deploy it by its model_object_id:

CLI

tg beta models configs ml_def456

tg beta endpoints deploy ml_def456 \
  --endpoint my-prepared-model \
  --config cr_Cf3yApYdaWMrH6RsyVmh7 \
  --project your_project_id

Pass --project with your project ID from tg whoami, or set TOGETHER_PROJECT_ID, even for your default project. Without either, the CLI resolves a private model ID to its base model and deploys that instead. For --config, pass the config ID of an FP4 profile from the configs output, such as cr_Cf3yApYdaWMrH6RsyVmh7 for GLM-5.3 or cr_CfsRY8rFVfgH9cEWwywEa for Kimi-K3. If the model has only one compatible profile, you can omit --config. For help picking a profile, see Choose a deployment profile. The deploy command creates the endpoint, attaches a deployment, and routes all traffic to it. For the rest of the deployment lifecycle, see Manage deployments.

Cancel a job

To stop a job before it completes, cancel it:

tg training fp4 cancel shp-quant-1a2b3c4d-5e6f

from together import Together

client = Together()

job = client.post_training.prepare_for_fp4_inference.cancel(
    "shp-quant-1a2b3c4d-5e6f"
)
print(job.status)

The CLI asks you to confirm. Pass --force (-y) to skip the prompt. The job moves to cancel_requested right away and reaches cancelled once the run stops, so watch it to confirm. The CLI won’t cancel a job that’s already in cancel_requested. A job in a terminal status can’t be cancelled, and the CLI prints that the job isn’t cancellable.

Troubleshooting

“Preparing an adapter on base model … is not supported”: The adapter’s base model isn’t one of the supported models.
“model … is not an adapter” or “does not record a fine-tuning relation”: Pass the adapter’s ID (registration.adapter.id, or inference_registration.adapter.id from the checkpoints API), not a merged model’s ID. Full-weight models aren’t accepted.
“model … does not exist in this project”: The adapter belongs to a different project. Use an API key scoped to the project that trained the adapter.
“Calibration file … not found”: The file ID is wrong, or the file belongs to another project. Upload the file in the adapter’s project.
The estimate warns that the price exceeds your credit, or create fails with a 402 error: Your credit balance can’t cover the estimated price. Add credits and retry.
Create fails with a 403 error: Your project is blocked from starting jobs. Contact support.
“Shaping job … not found”: The job doesn’t exist, or it belongs to another project.
Job ends in user_error: List the job’s events. The SHAPING_USER_ERROR event’s message describes the problem with the input.

Next steps

Manage checkpoints

Save inference checkpoints from a post-training session.

Multi-LoRA deployment

Compare candidate adapters before you prepare the winner.

Choose a deployment profile

Pick the hardware and quantization a deployment runs on.

Manage deployments

Scale, stop, and delete endpoints and deployments.

GET STARTED

SERVERLESS

INFERENCE APIS

DEDICATED MODEL INFERENCE

DEDICATED CONTAINER INFERENCE

POST-TRAINING

FINE-TUNING

GPU CLUSTERS

CODE EXECUTION

ADMINISTRATION

Supported models

Find the adapter ID

Estimate the job

Calibrate on your own data

Start the job

Monitor the job

Deploy the prepared model

Cancel a job

Troubleshooting

Next steps

Manage checkpoints

Multi-LoRA deployment

Choose a deployment profile

Manage deployments

​Supported models

​Find the adapter ID

​Estimate the job

​Calibrate on your own data

​Start the job

​Monitor the job

​Deploy the prepared model

​Cancel a job

​Troubleshooting

​Next steps

Manage checkpoints

Multi-LoRA deployment

Choose a deployment profile

Manage deployments

Supported models

Find the adapter ID

Estimate the job

Calibrate on your own data

Start the job

Monitor the job

Deploy the prepared model

Cancel a job

Troubleshooting

Next steps