Quantization means reducing the numerical precision with which the weights – and sometimes the activations – of a neural network are stored. A model trained in a 16-bit format is converted to 8-bit or 4-bit integers, which cuts memory requirements by a factor of two to four and speeds up inference, because less data has to travel between memory and the compute unit. The conversion is an affine mapping of a range of real values onto a discrete grid using a scale and a zero point, usually computed separately for each channel or block of weights, so that a single outlier cannot degrade an entire tensor. A distinction is drawn between post-training quantization (PTQ), which works on a finished model with a small calibration sample, and quantization-aware training (QAT), which simulates the rounding error during training itself. For large language models, GPTQ and AWQ have become established – both weigh the sensitivity of individual weights when deciding how to round – together with the GGUF distribution format. The main pitfall is so-called outlier features: a handful of dimensions with extremely large amplitude, which cause naive quantization to damage quality badly.
It is the same principle as the difference between a studio master in FLAC and the same track as an MP3. The sound is still the sound; it is simply stored with fewer gradations of loudness. The file shrinks to a fraction of its size, playback is snappier, and most listeners never hear the difference. Push it too far and strip out too many gradations, though, and it starts to crackle. A model behaves the same way: instead of remembering every weight to many decimal places, it rounds it to one of sixteen possible values. That is what lets a model which would otherwise only fit on a server GPU run on an ordinary laptop.