How Much Precision Does Large Model Quantization Actually Lose? Q8 to Q2 Explained in One Table, with GGUF Selection Guide

Here's the rule of thumb; if you're in a hurry, just read this part:

• Q8_0: The difference with FP16 is so small it's at the level of measurement error, can be considered lossless, but the volume is only saved by half—usually not worth it unless your memory is abundant to the point of not being used up.
• Q6_K: Near lossless, perplexity (PPL) increases by about 0.3%. If you want a "nearly no loss" safe level, choose this.
• Q5_K_M / Q4_K_M: Cost-effectiveness sweet spot. Q4_K_M volume is less than a third of FP16, PPL increases by about 2-3%, barely noticeable in daily conversation and coding. The community's default recommendation for Q4_K_M is not without reason.
• Q3 level: Starts to feel stupid (PPL increase 8-10%), error rates rise in long reasoning, math, and code. A compromise option when memory is really insufficient, prioritize imatrix versions like IQ3/Q3_K_M.
• Q2 level and below: Obvious degradation (PPL increase 40%+), only suitable for "can run and take a look" verification purposes. 1-bit level (IQ1_S) PPL explodes to nearly 10 times the baseline, don't have illusions.

Here are the data basis and details for each level.

1. Where Does the Data Come From: llama.cpp Official Quantization Quality Measurement

Most online discussions about quantization loss are based on subjective feelings like "I feel it's about the same." The data in this article comes from systematic measurements published in the llama.cpp official perplexity tool documentation: Llama-3-8B full quantization levels, measured on RTX 4090 using Wikitext corpus for PPL (perplexity), KLD (KL divergence), and token probability shift, which is the most complete quantization quality comparison in publicly available materials.

The three metrics explained in plain terms:

• PPL (Perplexity): The "surprise level" of the model for text, lower is better. The more PPL increases compared to FP16 after quantization, the greater the loss.
• KLD (KL Divergence): The deviation between the probability distributions of the model's outputs before and after quantization, 0 = identical. More sensitive than PPL, can catch situations where "PPL is similar but behavior has changed."
• Mean Δp: The average percentage point drop in the probability of the correct token, closest to the intuition of "how much dumber it gets."

2. Core Data Table: How Much Loss Each Level from Q8 to IQ1

From the official full table, key levels are extracted (Llama-3-8B, FP16 baseline PPL ≈ 6.23, volume 14.97 GiB):

A few points worth pausing on:

1. The losses for Q8 → Q6 → Q5 are at the "only measurable by instruments" level. Mean Δp is less than 0.2%, indistinguishable in human conversation.
2. Q4_K_M is the last level before the inflection point: Correct token probability only drops by 0.4-0.6%, but volume is already compressed to 30%. This is the data basis for it becoming the community default.
3. Old formats like Q4_0 are clearly worse than K-quant at the same volume: Both 4-bit, Q4_0's loss is three times that of Q4_K_M. Skip old quantization formats without the K suffix.
4. There's a cliff between Q3 and Q2: Q3_K_M drops by 1.2%, Q2_K drops directly by 6.5%—not a linear decline, but a cliff.
5. 1-bit is performance art: IQ1_S's PPL is nearly 10 times the baseline, correct token probability drops by a third. Its existence proves "can be loaded," not "can be used."

3. The Newer the Model, the More Vulnerable to Quantization

The same official document also has a comparison that's easily overlooked: for the same quantization level, Llama 3's loss is significantly greater than Llama 2's. Under Q2_K, Llama 2 7B's PPL only increases to 1.11 times the baseline, while Llama 3 8B increases to 1.56 times; KLD is also universally worse.

The mainstream explanation is: models with more training data and more fully utilized parameters have less "redundancy" in weights, so brutally compressed information causes more damage. This has direct implications for today's practice—for models like Qwen3.8, Llama 4 that consume full training compute, the actual loss from low-bit quantization (below Q3) will be greater than the reputation of older models, don't apply the experience from two years ago that "Q2 can still make do" to new models.

4. How to Choose: An Actionable Decision Path

1. First, calculate the memory bill: Model file volume ≈ number of parameters × bytes per parameter (reverse-engineered from the table: Q4_K_M is about 0.6 bytes/parameter including overhead, Q8_0 about 1.06). On top of that, you need to leave room for KV cache and system memory.
2. Under the premise of fitting in memory, choose the highest level that fits. Order: Q6_K > Q5_K_M > Q4_K_M > IQ4_XS > Q3_K_M > give up and switch to a smaller model.
3. Within the same level, prioritize versions with imatrix (imatrix or IQ prefix on the release page). In the table, the difference for Q4_K_M with or without imatrix is about 35% (ΔPPL 0.151 vs 0.175), free optimization.
4. Below Q3, first consider switching to a smaller model, instead of continuing to compress the large model. Q2 27B usually can't beat Q5 14B—the former's probability distribution is severely distorted, while the latter remains basically intact. (This is a consensus experience judgment in the community; for specific model pairs, please test empirically.)
5. Choose familiar publishers: bartowski, unsloth, mlx-community, etc., high-frequency publishers have mature quantization processes, usually with imatrix, and model cards will specify calibration details.

Practical Reference: Real Memory Bill for 24GB Mac mini

Beyond theory, we fully ran through and triple-tested the 4-bit quantization of Qwen3.8-27B on a 24GB unified memory Mac mini M4: the model body is about 15GB, plus a 2B draft model and KV cache, the peak is 19.4GB, speed is 6.5 tok/s (pure autoregressive) to 11.7-12.2 tok/s (speculative decoding). The complete deployment commands and memory details see:

• 24G Mac mini runs Qwen3.8-27B + DFlash 2: Measured 1.8x—which also measured a counterintuitive conclusion: 8-bit draft model is worse than 4-bit (in speculative decoding, the draft only provides candidates; precision redundancy doesn't translate to acceptance rate, and instead slows down drafting).
• Qwen3.8 native MTP vs DFlash 2 side-by-side comparison on the same machine
• Same DFlash 2: MLX acceleration 1.9x, llama.cpp actually slows down

5. Frequently Asked Questions (FAQ)

How effective is 4-bit (int4/Q4) quantization?

Based on Q4_K_M: PPL increases by about 2-3%, correct token probability drops by about 0.4-0.6%, barely noticeable in daily use; for tasks sensitive to probability distribution like mathematical proofs and long-chain reasoning, the gap will be amplified a bit. Conclusion: Default choice for consumer hardware, use with confidence.

How much difference is there between Q4 and Q8?

Data-wise: ΔPPL 0.15 vs 0.003, Mean Δp -0.4% vs -0.02%—Q8 is indeed better, but both are in the "hard to notice in conversation" range. The gap mainly manifests in error rates on extreme tasks. If you have enough memory, go for Q8/Q6; if not, the psychological burden of dropping to Q4_K_M can be put aside.

Can the loss of Q3 be accepted?

Q3_K_M (with imatrix) PPL increases by about 8%, correct token probability drops by 1.2%—usable, but the decline in "cleverness" starts to be perceptible, with increased error rates in code and math tasks. Treat it as a "transitional solution when memory is insufficient," and prioritize switching to a smaller model's higher quantization level.

Which one to choose, IQ4_XS or Q4_K_M?

Q4_K_M has slightly better quality (ΔPPL 0.151 vs 0.228), IQ4_XS has slightly smaller volume (4.14 vs 4.58 GiB). If memory is at a critical point, choose IQ4_XS; otherwise, Q4_K_M. Note that IQ series has slightly higher decoding overhead on some old hardware/backends; on Apple Silicon, both work well.

What are FP8 and INT4/NVFP4? Are they the same as GGUF?

No, they are not the same. GGUF's Q/IQ levels are weight compression formats in the llama.cpp ecosystem, focused on CPU/Apple Silicon; FP8 and NVFP4 are floating-point low-precision formats natively supported by NVIDIA data center/Blackwell hardware, mainly used with vLLM/TensorRT-LLM for high-throughput inference on server GPUs. Floating-point formats are friendlier to activation value distributions but depend on specific hardware; the "4-bit" of both cannot be directly compared via PPL tables—choose based on your hardware.

Is the 1.58-bit (ternarized) model real?

BitNet-style 1.58-bit is a trained-from-scratch ternary weight architecture, not compressing existing models to 1.58 bits post-hoc—these two things are often confused. For post-hoc quantization of existing models to 1-2 bits, refer to the IQ1/IQ2 data above: IQ2_M is barely usable, IQ1_S has already collapsed. The "dynamic 1.58-bit" large model GGUFs in the community are mixed-precision schemes (preserving higher bits for key layers), better than pure IQ1, but still significantly weaker than Q4.

Does quantization affect speed?

Yes, and it usually makes it faster: local inference is a memory bandwidth bottleneck task; smaller weights mean less data to move per token. This is one reason why 4-bit 27B can run close to reading speed on 24GB Mac. The exception is that some IQ levels have slightly higher decoding computational overhead, which might offset bandwidth gains on weak old CPUs.

References

• llama.cpp perplexity tool documentation (including all quantization level measurement data)
• llama.cpp quantization PR and K-quants design discussion
• This site: 24G Mac mini runs Qwen3.8-27B + DFlash 2 measurement