Every quantisation of a model, side by side: the measured bits per weight, the memory it needs at your context, whether it fits, and what it decodes at. Widths come from real GGUF file sizes rather than from the quant's name.
Worked setups you can load into the form above. Each one is a decision the generator makes differently, and the reason it makes it.
Several quantisations fit and the widest of them is the one to take. Memory left unused buys nothing, and accuracy given away does not come back.
The cache is what runs it out. At 32k tokens no quantisation of a 70B fits in 24 GiB, and squeezing below three bits is worse than choosing a smaller model.
Below about four bits per weight the trade stops being smooth, and it degrades faster on a small model than on a large one.
These are the ones that fail silently. The config is accepted, nothing raises an error, and the consequence arrives later.
K-quants are mixed precision, so Q4_K_M measures about 4.84 bits per weight. On a 70B model that is roughly 7 GB more than a flat four bits predicts, which is the difference between fitting on two 24 GB cards and not.
Instead:Use the measured width for the quantisation you are actually running. Even Q8_0 is 8.5 bits, because every block of 32 weights carries a scale.
The high-precision tensors are a fixed set, so their share depends on the shape of the model. The same quant lands a few percent wider on a model with a large vocabulary relative to its depth.
Instead:Treat the figure as a median with a range around it, and leave that margin when a decision is close.
The KV cache is not in the file. At long context it can exceed the weights, so two quants that look far apart on disk can be nearly identical in what they need to run.
Instead:Compare total memory at the context you intend to serve, which is what the table on this page reports.
From 8 to 5 bits the cost is small. Below about four it falls off sharply, and faster on small models than large ones, because a small model has less redundancy to lose.
Instead:Prefer a larger model at a narrower quant down to roughly three bits per weight. Below that, prefer the smaller model at a wider one.
Memory left unused buys nothing. Accuracy given away does not come back.
Instead:Take the widest quantisation that fits at your real context and batch size.
K-quants are mixed precision by design: some tensors are kept at higher accuracy than the name implies, so the file is always wider than the arithmetic suggests. Every width on this page is measured from real GGUF files rather than quoted, because the gap between the nominal number and the real one is routinely the difference between fitting on a card and not.
Each figure is the published size of an actual .gguf file divided by the parameter count of the model it was quantised from, taken across 8 models of different shapes and reported as a median with the spread. Q4_K_M lands near 4.84 bits per weight rather than 4, which on a 70B model is about 7 GB more than a flat four bits would predict. Q8_0 is 8.5 rather than 8, because each block of 32 weights carries a 16-bit scale.
The mixed-precision tensors are a fixed set, so their share of the total depends on the shape of the model. A model with a large vocabulary relative to its depth carries proportionally more high-precision weight, and the same quantisation lands a few percent wider on it. That is why this page reports a range beside each median rather than a single authoritative number, and why a figure from one model should not be applied to another without that margin.
Between 8 and 5 bits the quality cost is small and the memory saving is large, which makes it an easy trade. Below about four bits it inverts: quality falls off sharply, and it falls off faster on small models than on large ones, because a small model has less redundancy to lose. The practical rule that follows is worth stating plainly. A larger model at a narrower quant usually beats a smaller model at a wider one, until roughly three bits per weight, after which it usually does not.
The table below reports total memory at 8,192 tokens with an fp16 cache, not the file size, because a quantisation that fits the weights and not the cache has not fitted. On a small model at long context the cache is the larger of the two terms, which is the thing sizing from a model card cannot show you. Quantising the KV cache to q8_0 halves it and is usually the cheapest lever available.
Quality. Everything here is arithmetic about size and speed, and none of it says whether a given quantisation is good enough for your task, which depends on the model, the domain and how much of the answer is a single token. Perplexity comparisons exist and are worth reading; this page deliberately does not repeat them, because they are measurements this repository does not own and cannot check. Widths measured 2026-08-18, architectures fetched 2026-08-18.
Total GiB at 8,192 tokens of context with an fp16 KV cache, including the weights and the cache and excluding runtime overhead. A representative model per size band; use the calculator above for any of the 103 architectures and any context length.
| Model | Params | Q8_0 | Q6_K | Q5_K_M | Q4_K_M | IQ4_XS | Q3_K_M | Q2_K |
|---|---|---|---|---|---|---|---|---|
| bits per weight, measured | 8.50 | 6.57 | 5.68 | 4.84 | 4.33 | 3.91 | 3.04 | |
| Qwen3 0.6B | 0.6B | 1.5 | 1.3 | 1.3 | 1.2 | 1.2 | 1.1 | 1.1 |
| Llama 3.2 1B Instruct | 1.2B | 1.5 | 1.2 | 1.1 | 0.9 | 0.9 | 0.8 | 0.7 |
| Gemma 3 1B | 1.0B | 1.0 | 0.8 | 0.7 | 0.6 | 0.5 | 0.5 | 0.4 |
| Llama 3.2 3B Instruct | 3.2B | 4.1 | 3.3 | 3.0 | 2.7 | 2.5 | 2.3 | 2.0 |
| Qwen3 4B | 4.0B | 5.1 | 4.2 | 3.8 | 3.4 | 3.2 | 3.0 | 2.5 |
| Mistral 7B Instruct v0.3 | 7.2B | 8.2 | 6.5 | 5.8 | 5.1 | 4.7 | 4.3 | 3.6 |
| Llama 3.1 8B Instruct | 8.0B | 8.9 | 7.1 | 6.3 | 5.5 | 5.0 | 4.7 | 3.8 |
| Qwen3 8B | 8.2B | 9.2 | 7.4 | 6.5 | 5.7 | 5.2 | 4.9 | 4.0 |
| Gemma 2 9B | 9.2B | 11.1 | 9.0 | 8.1 | 7.2 | 6.6 | 6.2 | 5.2 |
| Mistral Nemo 12B Instruct | 12.2B | 13.4 | 10.6 | 9.3 | 8.1 | 7.4 | 6.8 | 5.6 |
| Gemma 3 12B | 12.2B | 12.9 | 10.1 | 8.9 | 7.7 | 7.0 | 6.4 | 5.1 |
| Phi-4 14B | 14.7B | 16.1 | 12.8 | 11.2 | 9.8 | 8.9 | 8.2 | 6.7 |
| Qwen3 14B | 14.8B | 15.9 | 12.5 | 11.0 | 9.6 | 8.7 | 8.0 | 6.5 |
| Mistral Small 3.2 24B | 24.0B | 25.0 | 19.6 | 17.1 | 14.8 | 13.3 | 12.2 | 9.7 |
| Gemma 3 27B | 27.4B | 28.2 | 22.0 | 19.2 | 16.5 | 14.8 | 13.5 | 10.7 |
| Qwen3 32B | 32.8B | 34.4 | 27.0 | 23.6 | 20.5 | 18.5 | 16.9 | 13.6 |
| Qwen2.5 Coder 32B | 32.8B | 34.4 | 27.0 | 23.6 | 20.5 | 18.5 | 16.9 | 13.6 |
| Yi 1.5 34B Chat | 34.4B | 35.9 | 28.2 | 24.6 | 21.2 | 19.2 | 17.5 | 14.0 |
| Llama 3.3 70B Instruct | 70.6B | 72.3 | 56.4 | 49.1 | 42.2 | 38.0 | 34.6 | 27.5 |
| Qwen3 30B A3B Instruct | 30.5B | 31.0 | 24.1 | 20.9 | 17.9 | 16.1 | 14.7 | 11.5 |
| gpt-oss 20B | 21.5B | 21.5 | 16.6 | 14.4 | 12.3 | 11.0 | 10.0 | 7.8 |
| Mixtral 8x7B Instruct | 46.7B | 47.2 | 36.7 | 31.9 | 27.3 | 24.5 | 22.3 | 17.5 |
A mixture of experts holds every expert in memory, so its row reflects the total parameter count rather than the active one. It will decode much faster than a dense model of the same size, which the calculator above reports.