Work out whether a model fits, and what to change when it does not. The weights are the easy half: this sizes the KV cache from the model's real architecture, at your context length and batch size, which is the half that moves.
Worked setups you can load into the form above. Each one is a decision the generator makes differently, and the reason it makes it.
The most-asked configuration, and it does not fit. The weights alone are 39.8 GiB against 24 GiB of card, so the answer is not close and no amount of KV quantisation reaches it.
A small model that is nonetheless memory-bound. At 128k tokens the KV cache is several times the size of the quantised weights, so sizing from the model card is sizing for the wrong number.
Splitting further to gain memory and losing it instead. A KV head cannot be split across two devices, so past eight GPUs the heads are replicated and the cache grows with the GPU count.
64 attention heads do not divide by 6. Tensor parallelism splits heads across devices, so this fails at startup with an arithmetic error rather than running slowly.
These are the ones that fail silently. The config is accepted, nothing raises an error, and the consequence arrives later.
GGUF K-quants are mixed precision, so `q4_K_M` averages about 4.85 bits per weight. On a 70B model that is 39.8 GiB rather than 32.9 GiB, and the 7 GiB difference decides whether it fits.
Instead:Use the real average for the quantisation you are running. Even `q8_0` is 8.5 bits, because every block of 32 weights carries a 16-bit scale.
Grouped-query attention shares one K and V across several queries, so the cache is sized by KV heads. Llama 3.1 70B has 64 attention heads and 8 KV heads, so the attention count overstates the cache eightfold.
Instead:Use `num_key_value_heads` from the model's config, not `num_attention_heads`.
The KV cache is linear in context length and in concurrent sequences, while the weights are fixed. At 128k context an 8B model needs several times more memory for its cache than for its weights.
Instead:Budget at the context and concurrency you intend to serve, not at the default. Quantising the cache to `q8_0` halves it and is usually the cheapest lever.
Tensor parallelism needs the attention head count to divide by the GPU count, and a KV head cannot be split at all. With more GPUs than KV heads the heads are replicated and the total cache grows rather than shrinking.
Instead:Keep the tensor-parallel size at or below the KV head count and use pipeline parallelism above it.
vLLM reserves a fraction of the device up front through `gpu_memory_utilization`, which defaults to 0.9, and a long prefill allocates activation buffers this arithmetic does not model.
Instead:Leave a real margin. A plan that needs the final 5% of the card will not start under default settings.
Weights are a fixed cost you can look up. The KV cache is the one that moves, and past a certain context length it is the larger of the two. Almost every VRAM rule of thumb in circulation only describes the first.
This is the most common error in published guidance, and it is not a small one. Llama 3.1 70B has 64 attention heads and 8 KV heads, because grouped-query attention shares one K and V across eight queries. An estimate built on 64 heads is eight times too large. Anything modern uses GQA, so if a calculation anywhere multiplies by the attention head count, it is wrong by the GQA ratio.
GGUF K-quants are mixed precision by design: attention and embedding tensors keep more accuracy than the name implies. q4_K_M averages about 4.85 bits per weight, so a 70B model is roughly 39.8 GiB rather than the 32.9 GiB that four bits would give. That 7 GiB gap is the difference between fitting on a 48 GiB pair of cards and not. Even q8_0 is 8.5 bits, because each block of 32 weights carries a 16-bit scale.
The KV cache is linear in both. Doubling the context doubles it exactly; serving two concurrent requests doubles it again. A Llama 3.1 8B at 128k context needs several times more memory for its cache than for its quantised weights, which is why a small model can still be the thing that runs out of VRAM. Quantising the cache to q8_0 halves it for a small quality cost, and it is usually the cheapest lever available.
Tensor parallelism splits attention heads across devices, so the head count has to divide by the number of GPUs, and vLLM refuses to start when it does not. Worse, a KV head cannot be split: when there are more GPUs than KV heads, the heads are replicated and the total cache grows with the GPU count. Eight KV heads across sixteen GPUs is twice the cache of eight across eight.
Vendors print GB and allocators work in GiB, and the two differ by 7.4%. That is larger than the margin most of these decisions come down to, and it runs in the direction that helps: 24 GiB is 25.77 GB of the unit a model file is sometimes quoted in. Every figure on this page is GiB, which is the unit that decides whether an allocation succeeds.
This is a budget computed from published architecture, not a measurement of your runtime. Weights and KV cache are exact arithmetic and the architectures were checked on 2026-08-18. The overhead figure is an allowance you set, because the real number depends on the CUDA context, the allocator, activation buffers during a long prefill, and whether the framework preallocates: vLLM takes a fraction of the card up front through gpu_memory_utilization, which defaults to 0.9. Treat the total as a floor, and note that fitting is not the same as being fast.