Generate an Ollama Modelfile with num_ctx set deliberately rather than left at a default far below what the model supports, plus the server environment variables that decide the rest, including the one that divides your context between parallel requests.
Worked setups you can load into the form above. Each one is a decision the generator makes differently, and the reason it makes it.
The model card says 131,072 and Ollama gives you 4,096. Everything past it is dropped from the front of the prompt, which is where the system prompt lives.
16k of context and four parallel requests is 4k each. The context is allocated once and divided between the slots, and both settings look correct on their own.
Ollama has no authentication and no key to set. Bound to 0.0.0.0 anyone on the network can run inference, pull models onto your disk, and delete the ones you have.
The KV cache is allocated up front. When the model plus the cache stop fitting, Ollama moves layers to the CPU and keeps serving an order of magnitude slower with nothing in the output to say so.
These are the ones that fail silently. The config is accepted, nothing raises an error, and the consequence arrives later.
It uses its own default, which is far below what any current model supports, and everything past it is dropped from the FRONT of the prompt. No error, no warning, nothing in the response. The system prompt is what disappears first, so the model stops following instructions it was never shown.
Instead:Set `PARAMETER num_ctx` explicitly to what you need. `ollama show <model>` prints the value actually in effect.
The context is allocated once and DIVIDED between the concurrent slots, so four parallel requests each get a quarter. Both settings look correct on their own and the truncation is the same silent one.
Instead:Multiply num_ctx by the parallelism, then check the memory that costs before committing to it.
When the model plus the KV cache stop fitting in VRAM, Ollama does not fail. It moves layers to the CPU and keeps serving an order of magnitude slower, with nothing in the output to say so.
Instead:Run `ollama ps` after loading and read the GPU and CPU split. Anything other than 100% GPU is the answer to why it got slow.
Ollama has no authentication at all, and there is no key to set. On every interface it lets anyone on the network run inference on your GPU, pull models onto your disk and delete the ones you have.
Instead:Keep it on 127.0.0.1 and put a reverse proxy in front if it has to be reachable. The network is the only control there is.
It makes sampling greedy, which removes one source of variation. GPU floating point non-determinism, batching effects and any change in surrounding context still move the output.
Instead:Use it for consistency rather than reproducibility, and pin the model tag: a bare tag follows the registry and moves under you.
Ollama has its own default, it is far below what any current model supports, and going past it drops the front of your prompt with no error and no warning. Almost every complaint that a local model has forgotten something is this.
The model card says 128k. Ollama gives you a few thousand unless told otherwise. Everything beyond that is cut from the beginning of the prompt, which is where the system prompt lives, so the model stops following instructions it was never shown. Nothing in the response says it happened, and the behaviour reads as the model being worse than it is.
ollama show llama3.1:8b prints the num_ctx in effect
context length 131072 the model
num_ctx 4096 what Ollama usesThis is the one that catches people who already fixed the first. The context is allocated once and split between the concurrent slots, so a server set to 16k with four parallel requests gives each request 4k. Both settings look right on their own, and the symptom is identical to the default problem: prompts quietly getting shorter under load, which is exactly when you are least likely to be reading them.
num_ctx 8192, OLLAMA_NUM_PARALLEL 4
-> each request gets 2048 tokensThe KV cache is allocated up front and scales linearly with it. When the model plus the cache stop fitting in VRAM, Ollama does not fail: it moves layers to the CPU and keeps serving, an order of magnitude slower, with nothing in the output to say so. A model that suddenly got slow after a config change is almost always this. ollama ps prints the GPU and CPU split, and anything other than 100% GPU is the answer.
A bare name follows :latest, which moves when the registry updates. So does a tag like llama3.1:8b, which tracks the current quantisation of that size. If you want the same weights next month, name the quantisation explicitly, and treat a model upgrade as a change worth evaluating rather than one that arrives on its own.
It makes sampling greedy, which removes one source of variation. Floating point non-determinism on a GPU, batching effects and any change in the surrounding context still move the output. Treating it as a guarantee is how a test suite becomes flaky in a way nobody can reproduce locally.
There is no key to set. Bound to 0.0.0.0 it lets anyone on the network run inference on your GPU, pull models onto your disk and delete the ones you have. The network is the only control, so keep it on loopback and put a reverse proxy in front if it has to be reachable.
A Modelfile plus the environment variables that belong on the server rather than in the file, because half of what decides Ollama's behaviour is not in the Modelfile at all. It does not set TEMPLATE: the base model's template is almost always right, and a wrong one produces output that looks reasonable and is subtly off, which is worse than an error. It also cannot see your hardware, so the memory advice here is a rule to check rather than a number: run ollama ps after loading and read the actual split.