Estimate tokens per second for a local model, with prefill and decode reported separately because they are bounded by different things. Includes time to first token, and when the model does not fit, the largest --n-gpu-layers that does plus what the split costs.
Worked setups you can load into the form above. Each one is a decision the generator makes differently, and the reason it makes it.
Decode is bandwidth divided by bytes read per token. The model fits, and the number that decides the speed is the memory bus rather than the shader count.
It does not fit, so llama.cpp keeps some layers on the GPU. Every token passes through every layer, so the rates combine harmonically and the curve is steep near the top.
The shader formula reproduces every NVIDIA figure and is wrong by exactly two on RDNA 3, so those devices carry no compute number and the page says so rather than printing half the truth.
These are the ones that fail silently. The config is accepted, nothing raises an error, and the consequence arrives later.
Decode reads every active weight once per token, so it is bounded by memory bandwidth rather than by arithmetic. A card with more compute and the same bandwidth generates at the same rate.
Instead:Compare memory bandwidth, not TFLOPS, for generation. Compute is what improves prefill and time to first token.
Prefill and decode are bounded by different resources and differ by orders of magnitude. A model that processes 3,000 prompt tokens a second may generate 40.
Instead:State both, and say which one your complaint is about. Lag before the first token is prefill; slow writing is decode.
Every token passes through every layer, so the times add and the rates combine harmonically. The last few layers on the CPU dominate the total, which is why 90% on the GPU can be twice the speed of 70%.
Instead:Solve for the largest `--n-gpu-layers` that fits and use it. Freeing memory for a few more layers is worth more than the count suggests.
All experts are resident and only a few are read per token, so memory follows the total and speed follows the active count. Treating them as one number is wrong in both directions.
Instead:Check the total against your memory and the active count against your bandwidth.
The bandwidth efficiency and compute utilisation used here are assumptions, and real runtimes vary with batch size, attention implementation and thermal headroom.
Instead:Set them from your own measurements if you have any. Estimates of this shape usually land within 10 to 15%, which is not the same as being right.
Almost every question about local inference speed is really one of two questions, and the answers point at different hardware. How long until it starts writing is compute. How fast it writes is memory bandwidth. A card that fixes one may do nothing at all for the other.
Every token requires reading every active weight once, plus the cache. That is a memory traffic problem, not an arithmetic one: the maths involved is trivial and the bytes are not. So decode speed is bandwidth divided by bytes read per token, and this is why a 70B is slow on a fast GPU, why quantising the weights speeds generation up rather than only saving memory, and why a newer card with the same memory bandwidth generates at the same rate as the one it replaced.
Processing the prompt happens in parallel across tokens, so it saturates the arithmetic units instead of the memory bus, at roughly two floating point operations per active parameter per token. That is the number that decides time to first token, which is what a user actually experiences as lag. A 4,000 token system prompt on a modest card is seconds of silence before anything appears, and it is paid on every turn that is not served from a cache.
All the experts must be resident, and only a few are read per token. So memory follows the total parameter count and speed follows the active count, and a 30B mixture with 3B active decodes at roughly the rate of a 3B model while occupying the memory of a 30B one. That is the single most useful fact for anyone whose model fits but is too slow, and it is why the active figure is computed from each model's own config here rather than taken from the name.
When a model does not fit, llama.cpp keeps some layers on the GPU and runs the rest on the CPU. Every token passes through every layer, so the times add and the rates combine harmonically. The practical consequence is dramatic: moving from 70% of layers on the GPU to 90% roughly doubles the speed, because the remaining CPU layers dominate the total. This page solves for the largest --n-gpu-layers that fits and prints the curve, because that is the number you type.
Shader throughput is derived here as cores times boost clock times two, which reproduces the published figure on every NVIDIA part checked: the 4090 at 82.6 TFLOPS, the A100 at 19.5, the H100 SXM at 66.9. It is wrong by exactly two on RDNA 3, whose shaders dual-issue, and no reachable source publishes it for Apple silicon. Rather than apply a per-architecture multiplier that would be a guess for the next architecture, 10 of the 21 devices carry a figure and the rest carry none, with prefill skipped rather than estimated. Enter the number yourself and it is used.
Every figure here is a bound computed from published architecture and hardware specifications, not a measurement of your machine. It cannot see Flash Attention or paged attention, speculative decoding or a draft model, batching effects, thermal and power limits on a laptop, or what your particular runtime does differently. The two efficiency figures are assumptions and are exposed as fields for that reason. Estimates of this shape usually land within 10 to 15% of measured, and the only authority is running it. Devices were fetched on 2026-08-18.