Build a vllm serve command and find out whether it will start. The KV cache arithmetic is computed from the model's real architecture, so you get the concurrency and the longest context that actually fit rather than the ones you asked for.
Worked setups you can load into the form above. Each one is a decision the generator makes differently, and the reason it makes it.
The comfortable case. 15 GiB of weights against a 72 GiB budget leaves 57 GiB of KV cache, which is 57 sequences at 8k or one at 467k.
131 GiB of weights against a 72 GiB budget. There is no KV cache to allocate because the model does not fit, and lowering max-model-len cannot help.
Three GPUs does not divide Llama 3.1's 32 attention heads. vLLM refuses, and the constraint is a property of the model rather than of the hardware.
Sixteen GPUs against eight KV heads means the heads are replicated, so the total KV cache doubles. Adding a card can leave less context per card than before.
These are the ones that fail silently. The config is accepted, nothing raises an error, and the consequence arrives later.
The refusal to allocate a KV cache has three terms: the memory fraction, the weights and the context. The message names only the last one, so the fix people reach for is the one that costs the most capability.
Instead:Work out which term has room. Raising gpu-memory-utilization, quantising the weights or adding a GPU all fix the same error without shortening the context.
Every current model uses grouped-query attention, so the cache is sized by KV heads. On Llama 3.1 that is 8 against 64 attention heads, so the estimate comes out eight times too large and a configuration that fits looks impossible.
Instead:Use kv_heads from the model's config.json. The eightfold difference is the whole reason the number surprises people in both directions.
It is a ceiling. A request only runs when there are free KV blocks, so the real limit is the cache size. Leaving it at 256 on a card whose cache holds four full-length sequences does not fail, it queues.
Instead:Compute how many sequences the cache holds at your max-model-len and treat that as the floor. Shorter requests use fewer blocks, so the real number sits above it.
The fraction is of TOTAL card memory, not free memory, and vLLM preallocates it. At 0.95 anything else on the card is competing for the remainder, and the failure is an out of memory error mid-request rather than at startup.
Instead:Leave it at 0.90 unless the card is doing nothing else. The CUDA context sits outside this fraction either way.
A KV head cannot be split across devices, so past the KV head count the heads are replicated and the total cache grows rather than the usable cache. Going from 8 GPUs to 16 on a model with 8 KV heads doubles the cache demand.
Instead:Keep tensor-parallel-size at or below the KV head count and use pipeline parallelism beyond it. Also check the size divides the attention head count, or vLLM refuses to start.
vLLM's most common startup failure is a refusal to allocate the KV cache. The message names max_model_len, so that is what people lower, and it is one of three terms that decide whether it fits.
vLLM takes gpu-memory-utilization of the card, subtracts the weights, and turns what is left into KV cache blocks. If that cannot hold one sequence of max-model-len tokens it refuses to start. So the fix is whichever of the three terms has room: lower the context, raise the fraction, quantise the weights, or add a GPU. The form on this page computes all four from the model's own architecture.
cache = utilisation x VRAM - weights
needed = max_model_len x max_num_seqs x bytes_per_token
bytes_per_token = 2 x layers x kv_heads x head_dim x elementEvery current model uses grouped-query attention, so the KV cache is sized by the KV head count rather than the attention head count. On Llama 3.1 that is 8 against 64, an eight-fold difference. Sizing from the attention heads is the single most common way to conclude a configuration cannot work when it comfortably can.
A request only runs when there are free KV blocks for it, so the effective concurrency is the cache size and not the setting. Leaving max-num-seqs at 256 on a card whose cache holds four sequences at full length does not fail, it queues. Requests shorter than max-model-len use fewer blocks, so the real number sits between the two and moves with your traffic.
vLLM preallocates it, and it is measured against the whole card rather than what is available. So anything else on the GPU, another process, a display server, a monitoring agent, is competing for the remainder, and the failure arrives as an out of memory error partway through a request rather than at startup. The CUDA context itself sits outside this fraction.
The size has to divide the attention head count, which is a property of the model. And a KV head cannot be split across two devices, so when there are more GPUs than KV heads the heads are replicated and the total KV cache grows. Adding a GPU can leave you with less usable context per card than before, which is the opposite of the reason for adding it.
Llama 3.1 8B: 32 heads, 8 kv heads
tp=4 divides 32, no replication
tp=3 does not divide 32, refuses to start
tp=16 KV heads replicated twice, cache doublesIt reuses the KV blocks of a shared prompt prefix across requests, which is most of the win on anything with a common system prompt or a few-shot block. Blocks are reference counted rather than duplicated, so it costs nothing when there is nothing to share. The one case where it hurts is a workload with no shared prefix at all and heavy memory pressure.
The memory figures cover weights and KV cache. They exclude activation buffers, the CUDA context, the allocator's fragmentation and anything CUDA graphs capture, all of which sit outside the utilisation fraction, so treat the result as the arithmetic rather than as a measurement. It also does not know your checkpoint: a model quantised on disk carries its own quantization_config and vLLM reads it, which is why no --quantization flag is emitted for AWQ or GPTQ. Run the command and read the KV cache size vLLM prints at startup, which is the real number.