Quantization: 8-Bit to 1-Bit — Where AI Models Break

The download was a lie of omission

Almost every model you have pulled from Hugging Face arrived pre-shrunk. The full sixteen-bit version of that brain is typically two to four times bigger than the file you actually downloaded, and it runs on your card at all because of one trick: quantization. Round the numbers, shrink the brain, hope nobody notices. This episode rides the elevator all the way down — eight-bit, four, three, two, one — with measured numbers at every floor, looking for exactly where the magic ends and the lobotomy begins. The short version: the elevator does not go all the way down.

What your card is actually doing

A model is billions of numbers, called weights, each stored at sixteen-bit precision. An eight-billion-parameter model is sixteen gigabytes before you say a word to it — too big for a twelve-gig card. Quantization rounds every weight down to eight bits, or four: the same brain, blurrier numbers. At Q8 you are at half the size and effectively lossless. At Q4 — the community default — you are at roughly a quarter of the size, just under five gigabytes, and suddenly the model fits on the card you already own. VRAM is the whole game in local AI, and quantization is how everyone cheats at it.

The quant alphabet, and why the defaults are fine

When Ollama pulls a model, you are handed Q4_K_M without being asked: the GGUF format's four-bit sweet spot, still the default in 2026. Around it swims an alphabet. K-quants tune for speed. I-quants squeeze more quality per bit below four. Unsloth's dynamic quants grade every layer individually and protect the important ones. AWQ won the four-bit serving war for the vLLM crowd. EXL3, the enthusiast option, claims a 70B stays coherent at 1.6 bits — allegedly. Google now trains Gemma with four-bit baked in, and this spring llama.cpp added NVFP4 for Blackwell cards. It is deep soup, but the defaults are genuinely fine.

The measured cliff

A January 2026 study ran thirteen quants of Llama 3.1 8B through five benchmarks. Full precision averaged 69.5. Q8 scored 69.4. Q4_K_M scored 69.2 — less than half a percent lost, at under a third of the size. Q5_0 actually edged out full precision, almost certainly noise, but a good measure of how little five bits costs. Then the floor gives way: Q3_K_S drops about four points overall and more than nine points on grade-school math. Math always dies first; casual chat barely notices. One more tell: three different five-bit quants posted identical perplexity scores, then landed more than three points apart on math. Perplexity charts are a vibe. Task benchmarks are the truth.

What four-bit buys you, card by card

A 5090 with thirty-two gigs runs the hot 27B class at five or six bits with headroom, but a 70B still only squeezes on at three-bit. Twenty-four gigs is the sweet spot of 2026: the whole 27-to-35B class at Q4. Sixteen gigs is the awkward middle child. Twelve gigs — the class of card rendering this very video — runs 8-to-14B at Q4 beautifully. But there is a hidden second bill: context. Filling 128K tokens of context on an older 8B costs sixteen extra gigabytes of KV cache. The file size is the entry fee, not the whole ticket.

Below three bits, the horror show

This is where it breaks. Crushed naively to 1.58 bits, DeepSeek R1 wrote "Pygame's Pygame's Pygame's" forever; a community 1.6-bit build of DeepSeek's newest Flash model loops the same way. The science is just as blunt: Llama 3 8B crushed to two bits scored as low as 34.9% on reasoning benchmarks where full precision scored 68.6% — half the brain, gone. Even the author of llama.cpp's one-bit format calls it "not very useful," and says he built it to spite the hype. And BitNet — the real 1.58-bit technology — has to be trained ternary from birth, and nothing bigger than a few billion parameters exists. The famous hundred-billion BitNet running on a CPU is a README fantasy about a model nobody has trained.

The loophole: don't crush evenly

Unsloth found the way through: do not crush uniformly. Keep the attention layers at four to six bits and the routing layers at full precision — the brain stem stays intact — and crush only the fat expert layers. On DeepSeek's giant mixture-of-experts model, their two-bit dynamic build kept 92% of its coding score while cutting size by more than sixty percent, and even the one-bit build still beat GPT-4.1. The fine print: this trick needs a huge MoE. Small models degrade brutally — an 8B crushed to two-bit will usually lose to a 4B kept at four-bit. And the twist the papers keep finding: the more a model was trained, the worse it quantizes. The denser the brain, the less fat there is to cut.

Where the dial stops

Four-bit is the floor for real work — Q4_K_M or better gets you ninety-plus percent of the brain at a third of the size, the closest thing to a free lunch in local AI. Three-bit is for emergencies, and you pick the M variant, not the S — that one letter is worth nearly five points on math. Two-bit is chat-only desperation: defensible on a giant MoE with dynamic quants, a lobotomy on anything small. One-bit is a party trick. And burn this in: nearly every headline benchmark number — the ones on the model card and the leaderboard — was measured on the full-precision model, the one you are not running. Stay local, stay skeptical.