{"id":9497,"date":"2026-07-27T12:30:29","date_gmt":"2026-07-27T10:30:29","guid":{"rendered":"https:\/\/www.kubicek.ai\/?post_type=lexicon&#038;p=9497"},"modified":"2026-07-27T12:56:44","modified_gmt":"2026-07-27T10:56:44","slug":"quantization","status":"publish","type":"lexicon","link":"https:\/\/www.kubicek.ai\/en\/lexicon\/quantization\/","title":{"rendered":"Quantization"},"content":{"rendered":"<p class=\"wp-block-paragraph\"><strong>Quantization<\/strong> means reducing the numerical precision with which the weights \u2013 and sometimes the activations \u2013 of a neural network are stored. A model trained in a 16-bit format is converted to 8-bit or 4-bit integers, which cuts memory requirements by a factor of two to four and speeds up inference, because less data has to travel between memory and the compute unit. The conversion is an affine mapping of a range of real values onto a discrete grid using a scale and a zero point, usually computed separately for each channel or block of weights, so that a single outlier cannot degrade an entire tensor. A distinction is drawn between post-training quantization (PTQ), which works on a finished model with a small calibration sample, and quantization-aware training (QAT), which simulates the rounding error during training itself. For large language models, GPTQ and AWQ have become established \u2013 both weigh the sensitivity of individual weights when deciding how to round \u2013 together with the GGUF distribution format. The main pitfall is so-called outlier features: a handful of dimensions with extremely large amplitude, which cause naive quantization to damage quality badly.<\/p>\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n<p class=\"wp-block-paragraph\">It is the same principle as the difference between a studio master in FLAC and the same track as an MP3. The sound is still the sound; it is simply stored with fewer gradations of loudness. The file shrinks to a fraction of its size, playback is snappier, and most listeners never hear the difference. Push it too far and strip out too many gradations, though, and it starts to crackle. A model behaves the same way: instead of remembering every weight to many decimal places, it rounds it to one of sixteen possible values. That is what lets a model which would otherwise only fit on a server GPU run on an ordinary laptop.<\/p>\n","protected":false},"featured_media":0,"template":"","class_list":["post-9497","lexicon","type-lexicon","status-publish","hentry"],"acf":[],"_links":{"self":[{"href":"https:\/\/www.kubicek.ai\/en\/wp-json\/wp\/v2\/lexicon\/9497","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.kubicek.ai\/en\/wp-json\/wp\/v2\/lexicon"}],"about":[{"href":"https:\/\/www.kubicek.ai\/en\/wp-json\/wp\/v2\/types\/lexicon"}],"wp:attachment":[{"href":"https:\/\/www.kubicek.ai\/en\/wp-json\/wp\/v2\/media?parent=9497"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}