Quantization sounds like a storage trick, and at the surface level it is. But what it actually changes is subtle enough that “use a smaller model” and “quantize the model” get treated as the same idea when they are not. A quantized 7 billion parameter model is still a 7 billion parameter model. Every weight is still there. What changes is how many bits it takes to represent each one.
What gets reduced, exactly
A model’s weights, and often its activations, the intermediate values produced as data flows through the network, are normally stored as floating point numbers. Most training happens in 32-bit floating point (FP32, sometimes called “full precision”) or 16-bit floating point (FP16). FP simply stands for floating point, the IEEE 754 number format, “full precision” is a separate, informal name people use for the uncompressed baseline a quantized model gets compared against, usually FP32. Quantization converts those numbers into a lower-precision format, most commonly 8-bit integers (INT8) or 4-bit integers (INT4), using a mapping that squeezes the original range of values into a much smaller set of possible values.
That mapping needs a scale factor, which says how big a step is between adjacent integer values, and in some schemes a zero-point, which shifts the integer range to line up with where the original values actually cluster. Together they let you convert a float to its nearest integer approximation and back. The “approximation” part is the whole story: you are deliberately losing precision, and the entire discipline of quantization is about losing as little useful information as possible while doing it.
The actual math
The conceptual description above maps to a specific, small piece of arithmetic. To quantize a float x into an integer q, you compute:
q = round(x / scale) + zero_point
To get an approximation of the original float back, you reverse it:
x ≈ scale × (q - zero_point)
The scale itself comes directly from the range you are compressing:
scale = (max_value - min_value) / (q_max - q_min)
where max_value and min_value are the largest and smallest float values you are quantizing (from a calibration pass, or from the tensor itself), and q_max and q_min are the limits of the integer type you are targeting, -128 and 127 for signed INT8.
Here is what that looks like on one actual number. Say a weight matrix’s values range from -0.8 to 0.7, and you are quantizing to signed INT8, so q_min = -128 and q_max = 127.
scale = (0.7 - (-0.8)) / (127 - (-128)) = 1.5 / 255 ≈ 0.00588
zero_point = q_min - round(min_value / scale) = -128 - round(-136.0) = -128 - (-136) = 8
Now take one weight from that matrix, x = 0.42, and quantize it:
q = round(0.42 / 0.00588) + 8 = round(71.4) + 8 = 71 + 8 = 79
That 79 is what actually gets stored, one byte instead of four. Dequantizing it back to a float:
x ≈ 0.00588 × (79 - 8) = 0.00588 × 71 ≈ 0.4176
The original value was 0.42, the round trip gives back 0.4176, an error of about 0.0024, roughly half a percent. That gap is quantization error, and it exists on essentially every weight in the model. It is small enough per-weight to be invisible, but it accumulates across billions of weights and through every layer’s computation, which is exactly why the choices covered below, symmetric versus asymmetric, per-tensor versus per-channel, PTQ versus QAT, all exist: they are different strategies for keeping that accumulated error small enough not to matter.
Why the size difference actually matters
The memory math is straightforward, and it is worth doing once so the motivation is concrete rather than abstract. A 7 billion parameter model stored in FP32 needs roughly 28 gigabytes just to hold the weights, 4 bytes per parameter. Convert to FP16 and that drops to 14 gigabytes. Convert to INT8 and it is around 7 gigabytes. INT4 gets it down to roughly 3.5 gigabytes. That difference decides whether a model fits on a single consumer GPU, runs on a laptop, or has to stay behind an API call to a data center. Lower precision also means the hardware can move and multiply numbers faster, since integer arithmetic is cheaper than floating point arithmetic on most chips, so quantization usually speeds up inference too, not just shrinks the model on disk.
Two different ways to get there
Quantization can happen at two different points in a model’s life, and that choice matters more than the bit-width you eventually pick.
Post-training quantization (PTQ) takes a model that has already finished training at full precision and converts its weights afterward, usually by running a small calibration dataset through the model to see what range of values each layer actually produces, then choosing scale and zero-point values that fit that range well. It is fast, needs no retraining, and is the default choice for anyone who just wants a smaller model to run.
Quantization-aware training (QAT) instead simulates the rounding error that quantization will introduce, during training or fine-tuning itself, so the model’s weights adjust to tolerate that error before it is ever actually applied. It costs more, since it means training or fine-tuning again rather than converting a finished model, but it consistently retains more accuracy at the same bit-width than PTQ does. The choice is a straightforward tradeoff: PTQ if you want a smaller model quickly and can tolerate some quality loss, QAT if that loss is not acceptable and you can afford to retrain.
The details that decide how much you actually lose
Two design choices inside PTQ account for most of the quality difference between a good quantization job and a bad one.
The first is symmetric versus asymmetric mapping. Symmetric quantization assumes the original values are centered around zero and maps them onto a symmetric integer range, which is simpler and slightly faster to compute with. Asymmetric quantization does not assume that. It maps the actual minimum and maximum of the data, which fits better when the values genuinely are not centered around zero, activations coming out of certain non-linear functions are a common example of that.
The second is the granularity of the scale and zero-point. Per-tensor quantization uses one scale and zero-point for an entire weight matrix. Per-channel, or per-group, quantization computes a separate scale and zero-point for each row or column, or for small groups of weights within a matrix. Per-channel is more expensive to store and compute, but it captures the fact that different parts of a weight matrix can have very different value ranges, and forcing one shared scale onto all of them wastes precision on the parts that did not need it. Most modern LLM quantization methods default to per-channel or per-group schemes specifically because a single per-tensor scale loses too much accuracy at 4-bit precision.
The named methods you will actually run into
In practice you rarely implement quantization from scratch, you pick a method.
GPTQ quantizes a model layer by layer, and the math behind it is a genuine optimization problem, not just “round each weight.” For a layer with weight matrix W and calibration inputs X, GPTQ wants the quantized weights Ŵ that minimize the squared difference in the layer’s output, ‖ŴX − WX‖², rather than minimizing the error on the weights themselves. It computes the Hessian of that error, H = 2XX^T, from the calibration data, then quantizes one weight at a time and uses the inverse Hessian, H⁻¹, to work out exactly how much to adjust every remaining unquantized weight in that row to compensate for the rounding error it just introduced, weighting the adjustment by that weight’s entry in H⁻¹. Quantizing in a fixed order lets the same inverse Hessian, computed once via Cholesky decomposition, be reused for the whole layer instead of recalculated after every weight, which is what makes it fast enough to run on billion-parameter models.
AWQ leaves the arithmetic simpler and instead changes what gets rounded aggressively. Using a calibration pass, it measures the average activation magnitude flowing through each channel of a weight matrix, on the theory that the weights multiplied by consistently large activations matter more to the output and deserve more protection. For a channel scaling factor s > 1, it computes Q(W · s) · (X / s), scaling that channel’s weights up before rounding, which shrinks the relative rounding error on it, and dividing the matching activations by the same s so the overall multiplication is mathematically unchanged. It searches a small range of values, typically s as a power of the average activation magnitude, and keeps whichever minimizes the actual output error on the calibration data.
GGUF, by contrast, is less a quantization algorithm and more a file format, built for running quantized models efficiently on regular CPUs and consumer GPUs, and it is the format most local inference tools expect. None of these is universally the right choice. GPTQ and AWQ are usually picked for GPU inference where you want the smallest model that still performs well, GGUF is picked when the deployment target is a laptop or a machine without a dedicated GPU.
What you are actually trading away
The honest tradeoff is accuracy for size and speed, and how much accuracy you lose depends heavily on the task, not just the bit-width. Quantized models tend to hold up well on everyday conversational use and lose more ground on tasks that require precise multi-step reasoning or exact recall, since small rounding errors compound differently depending on how sensitive a task is to any single wrong intermediate value. INT8 is close enough to full precision that most people cannot tell the difference in normal use. INT4 is where the tradeoff becomes a real decision rather than a free win, and it is worth testing your specific use case against the full-precision model before assuming a 4-bit version is good enough for it.
Quick Recall
Quantization: lower the numeric precision of weights (and activations)
FP32/FP16 → INT8/INT4, using a scale (and often a zero-point) to map values
Why: smaller memory footprint, faster integer math, runs on smaller hardware
When to quantize
│
├── After training, no retraining budget → PTQ
│ (fast, calibrate on sample data, some accuracy loss)
│
└── PTQ's accuracy loss is unacceptable → QAT
(simulate quantization error during training, costs more, keeps more accuracy)
Accuracy depends on:
- symmetric vs asymmetric mapping
- per-tensor vs per-channel/per-group scale
- the bit-width itself (INT8 usually safe, INT4 needs real testing)
Common named methods: GPTQ, AWQ → GPU inference · GGUF → CPU / consumer hardwareEnjoyed this? Subscribe via RSS.