Quantization vs VRAM: Exactly How Much Memory Each Level Saves

What quantization actually does

Quantization reduces the precision of model weights — the numbers that define what the model learned during training. A model trained at FP16 (16-bit floating point, 2 bytes per weight) can have its weights stored at lower precision: 8-bit, 4-bit, 3-bit, or even 2-bit. Each reduction cuts memory usage proportionally. The tradeoff is that lower precision means each weight carries less information, which degrades output quality.

The key insight is that model quality does not degrade linearly. Moving from FP16 to Q4_K_M (4-bit) loses only 1-3% on benchmarks — your outputs will look nearly identical. But moving from Q4 to Q2 loses another 5-10% and the degradation becomes visible in longer outputs, factual accuracy, and reasoning quality. For a rigorous technical analysis, see the GPTQ paper which established much of the quantization methodology used in GGUF formats today.

Qwen 3 8B: VRAM at every quantization level

Qwen 3 8B is one of the most popular small models for local inference in 2026. Here is exactly how much VRAM it needs at each quantization level with 4K context, including model weights, KV cache (~0.5 GB at 4K), and 10% overhead.

The practical takeaway: Q4_K_M is the sweet spot. It uses 68% less VRAM than FP16 while retaining 97% quality. Q5_K_M is worth considering if you have a few extra gigabytes — it gets you to 98% quality for only 1 GB more.

Qwen 3 32B: the model that makes 24 GB GPUs shine

Qwen 3 32B is arguably the best model for single 24 GB GPUs in 2026. At Q4_K_M it fits in ~19 GB with 4K context, leaving comfortable headroom. At Q5_K_M it still fits at ~24 GB, using the full VRAM budget for better quality. This is the model that makes an RTX 3090 or RTX 4090 feel like the right purchase.

Llama 3.3 70B: where quantization decides your GPU

For dense 70B models, quantization is not about optimization — it determines whether you can run the model at all. FP16 requires 140 GB, which is far beyond any consumer GPU. Even Q4_K_M needs ~38 GB, pushing past a single 24 GB GPU.

MoE models: quantization on the 2026 generation

MoE (Mixture of Experts) models like Qwen 3 235B-A22B and Llama 4 Scout 109B have changed the quantization calculus. These models store many experts but only activate a few per token. All experts must be quantized and loaded into VRAM, but inference speed depends only on the active experts. This means quantization is even more critical for MoE models — you are compressing parameters that are loaded but rarely used.

The key insight: MoE models at Q2_K often deliver better quality than dense models at Q4_K_M of the same VRAM footprint, because the 22B active parameters at Q2 are still higher quality than a 14B dense model at Q4. If you are choosing between a dense 14B at Q4 and an MoE 30B at Q2, the MoE model usually wins.

What quantization does NOT compress

Quantization only reduces model weight memory. It does not affect three other VRAM consumers that you must budget for separately. This is a common source of confusion — people see that Q4 cuts weights to 25% and assume their total VRAM drops by the same factor.

• -KV cache: The attention cache stores activations in FP16 regardless of model quantization. At long context lengths, this can exceed model weights. See our KV cache guide for the full breakdown.
• -Framework overhead: Inference engines allocate memory for the computation graph, tokenizer buffers, and CUDA workspace. This adds 10-20% on top of weights + cache.
• -Batch size: Serving multiple users simultaneously multiplies the KV cache. Local inference at batch size 1 avoids this, but API-style deployments need to budget for it.

Picking your quantization strategy

The right quantization depends on your GPU, your model target, and your quality tolerance. Here is a decision framework based on common hardware configurations in 2026.

Calculate your exact quantization-to-VRAM mapping

These numbers are accurate for the models listed, but every model has slightly different architecture parameters that affect exact memory usage. The get your exact numbers — pick any model and quantization level, set your context length, and get a VRAM breakdown showing model weights, KV cache, overhead, and which GPUs can run it.

Related Guides