An AI model is just billions of numbers. At full precision, a thirteen billion model weighs twenty six gigabytes — too fat for your graphics card. Quantization is how we put it on a diet.
It rounds every number down to fewer bits. Sixteen bits becomes four. Think JPEG, but for an AI's brain — you throw away detail so fine you can barely tell it's gone.
That twenty six gig model at four-bit — the one they label Q4 — drops to about eight gigs. Like vacuum-packing a thick winter duvet down to a cushion: same duvet, a third the size. Now it fits a gaming card.
The catch? You lose a sliver of accuracy — usually a point or two, like a photo saved one too many times. Push too far, down to two-bit, and it gets properly dopey and starts to ramble. Four-bit is the sweet spot most people run.
So when you see Q4, Q5, Q8 — that's the precision. Bigger number, sharper brain, more VRAM. That's quantization: the trick that fits a datacenter model on your desk. Off the Cloud.