{caption}
}
Quantization is the same trick applied to a model. The model's weights are real numbers, and quantization stores each one using fewer bits. 16 bits per weight is the pristine, uncompressed version. 8 bits per weight is the analogue of dropping the photo to 4 bits per channel: a meaningful reduction in storage that, for most workloads, you can't tell apart from the original. 4 bits per weight is closer to the 2-bits-per-channel photo—aggressive enough that artifacts start showing up in subtle places, and you have to actually evaluate the quantized model to know whether they matter for your specific task.
The bigger the model is to start with, the more headroom you have to compress and quantize. A 70B model at 4 bits per weight will generally outperform a 13B model at 16 bits per weight, the same way a 12-megapixel photo at 4 bits per channel still beats a 4-megapixel photo at 8 bits per channel.
## Why precision matters
A model's weights are real numbers, and at inference time the GPU spends most of its work reading those weights from memory and multiplying them with inputs. The more bits per weight, the more memory bandwidth you burn per token of output you generate. The total memory budget for a model and its activations also scales directly with bits per weight, which determines what hardware you can fit it on in the first place.
Quantization is the family of techniques for storing the same weights using fewer bits each. You end up with the same number of weights, but each one is smaller. Less memory used overall, more weights read per second, faster inference, lower hosting cost. The cost you pay is precision: the weights become slightly approximate, and that approximation translates into a small drop in model quality if done naively.
## Quantization formats
These are the formats you'll likely encounter, in approximate order of decreasing precision:
* **fp32:** 32-bit float. The full-precision format from textbook training. Almost never used for inference because it uses too much memory for not enough quality gain.
* **fp16 / bf16:** 16-bit floats. The standard format that models are trained and shipped in. bf16 has more range while fp16 has more precision. Modern GPUs prefer bf16.
* **fp8 (E4M3 / E5M2):** 8-bit float. Supported natively on H100 and later GPUs. Usually near-indistinguishable from bf16 in quality, while requiring half the memory and bandwidth.
* **int8:** 8-bit integer. An older approach to 8-bit quantization, slightly more involved than fp8 because it requires per-channel scaling. Still common on GPUs that do not have native fp8 support.
* **fp4 (MXFP4 / NVFP4):** 4-bit float. The newer 4-bit format supported natively on Blackwell (B100/B200) GPUs. Closer in quality to fp8 than int4, because it preserves the exponent range.
* **int4:** 4-bit integer. Aggressive enough that quality starts to be impacted. Most modern int4 implementations (AWQ, GPTQ, GGUF) include calibration on a sample dataset to choose quantization parameters that minimize the performance hit.
On Together, many models are served in fp8 or fp4 variants (visible in the [model list](/docs/serverless/models)), and quantization is one of the decoding options you can choose from when configuring a [dedicated endpoint](/docs/dedicated-endpoints/settings).
## How quantization actually works
Take the weights in a single layer. They have some distribution: most are small, a few are large. Suppose the range of values is roughly -1.5 to +1.5. To store these weights in int4 (only 16 possible distinct values), you:
1. Pick a scale factor. For example, 0.2.
2. For each weight, round it to the nearest multiple of the scale factor.
3. Store the integer that corresponds to the rounded value. For scale 0.2 and range -1.5 to +1.5, this gives you the integers -7 through +7, plus 0.
4. At inference time, multiply each stored integer by the scale factor to recover the (approximate) original weight.
That's essentially all there is to it conceptually. The integer takes 4 bits instead of 16, which is a 4× memory reduction. The scale factor is one float per group of weights (usually one per channel or per block), which is negligible for the overall storage cost.