The Quant Decoder

Quantization stores model weights with fewer bits. This reduces memory use and file size at some cost to quality. This guide explains the labels used by GGUF files.

The short answer

Start at Q4_K_M. It keeps ~96% of quality at ~55% of the size. Go Q6 if you have room to spare. Go Q3 only to squeeze onto smaller cards. Expect visible degradation below that.

Beaker's longer explanation follows. The box is the answer.

Quality kept vs size on disk

Quality stays close to the full precision model through Q4_K_M in the published measurements. Loss increases faster below Q4. Each point links to its source.

7B classmeasured on LLaMA-7B
100%95%90%85%the kneeQ2_K: 87.2% quality kept, 2.67 GB file. Source: llama.cpp k-quants PRQ2_K2.67 GBQ3_K_M: 96.0% quality kept, 3.06 GB file. Source: llama.cpp k-quants PRQ3_K_M3.06 GBQ4_K_M: 99.1% quality kept, 3.8 GB file. Source: llama.cpp k-quants PRQ4_K_M3.8 GBQ5_K_M: 99.8% quality kept, 4.45 GB file. Source: llama.cpp k-quants PRQ5_K_M4.45 GBQ6_K: 99.9% quality kept, 5.15 GB file. Source: llama.cpp k-quants PRQ6_K5.15 GB
14B classnearest published measurements: LLaMA-13B
100%95%90%85%the kneeQ2_K: 89.7% quality kept, 5.13 GB file. Source: llama.cpp k-quants PRQ2_K5.13 GBQ3_K_M: 96.4% quality kept, 5.88 GB file. Source: llama.cpp k-quants PRQ3_K_M5.88 GBQ4_K_M: 99.1% quality kept, 7.32 GB file. Source: llama.cpp k-quants PRQ4_K_M7.32 GBQ5_K_M: 99.8% quality kept, 8.6 GB file. Source: llama.cpp k-quants PRQ5_K_M8.6 GBQ6_K: 100.0% quality kept, 9.95 GB file. Source: llama.cpp k-quants PRQ6_K9.95 GB

Quality kept = FP16 perplexity ÷ quant perplexity (wikitext), from the measurements published in the llama.cpp k-quants PR. KL divergence studies show the same knee. Artefact2's measurements.

32B class: published per-quant coverage is thin. The same source measures the trend that quantization loss shrinks as models grow. Treat the 14B curve as the floor, not the ceiling, for a 32B.

Quant names, decoded

QuantIn plain EnglishGB for a 14B*When to pick it
Q8_08-bit, plain rounding. The near-archival jar. Practically the full model.14.9 GBMemory is no object, or you're comparing quants and want a reference.
Q6_K6-bit K-quant. Near-lossless in the published measurements.11.5 GBYou have room to spare and want the quality question gone.
Q5_K_M5-bit K-quant, medium mix. A small quality margin over Q4.10.0 GBQ4 fits with a few GB left over and you'd rather spend them on quality.
Q4_K_M4-bit K-quant, medium mix. The house default. The knee of the quality curve.8.5 GBStart here. Move only when a measurement gives you a reason.
Q3_K_M3-bit K-quant. Degradation is measurable and starts to be noticeable.≈6.3 GBSqueezing onto a smaller card when Q4 won't fit and offload is too slow.
Q2_K2-bit K-quant. The smallest jar. Expect obvious quality loss.≈5.5 GBLast resort. A smaller model at Q4 usually beats a big one at Q2.

* weights only, via the site's shared fit math (running needs KV cache + overhead on top. full formula). Q3/Q2 figures scale the published 13B file sizes.

The rest of the alphabet soup

K (K_S / K_M / K_L)
K-quants: bits are mixed per layer block. Important tensors get more, filler gets less. S/M/L are small/medium/large mixes of the same bit level.
IQ2 / IQ3 / …
i-quants: importance-weighted packing that holds up better at very low bits, at some CPU-speed cost. Same idea, finer scales.
imatrix
An importance matrix used while quantizing. A calibration pass, not a format. Any quant can be made 'with imatrix'. Quality usually improves a little.
Q4_0 / Q4_1 / Q5_0 …
The legacy round-to-nearest quants that predate K-quants. Superseded: at the same size, the K-quant measures better.
UD (dynamic)
Per-layer dynamic bit allocation (e.g. Unsloth Dynamic): sensitive layers keep more bits. Different labels, same trade as imatrix. Better quality per GB.
bpw
Bits per weight. Exl2's dial. Where GGUF names steps (Q4, Q5…), exl2 quantizes to any fraction, so "4.25 bpw" ≈ a Q4-class build.

The same quant across runtimes

Same contents, four labeling systems. Beaker finds this personally offensive, so here's the translation table.

Classllama.cpp / GGUFOllama tagMLXexl2
4-bitQ4_K_M:q4_K_M. This is the usual default when a tag names no quant4-bit (…-4bit builds)~4.0 to 4.5 bpw
5-bitQ5_K_M:q5_K_M. (no 5-bit. MLX jumps 4 to 6)~5.0 bpw
6-bitQ6_K:q6_K6-bit~6.0 bpw
8-bitQ8_0:q8_08-bit~8.0 bpw
← Back to the Field GuideWill it run on your card?