Render a conversation into the exact string a model sees, for ChatML, Llama 3, Llama 2, Mistral or Gemma. Then check it against the three things that silently degrade output: a doubled BOS token, a system message the format has no room for, and a missing generation prompt.
Worked setups you can load into the form above. Each one is a decision the generator makes differently, and the reason it makes it.
Llama 3's template emits <|begin_of_text|> itself, and add_special_tokens defaults to True, so the model gets two of a token it never saw twice in training. Nothing errors; the output is merely worse.
Mistral's v1 and v3 templates have no system role. The instruction is folded into the first user turn or dropped, and which one happens is a property of the library rather than the model.
Gemma calls the model's turns model. A message with role assistant is either a template error or a silently skipped turn, and a skipped turn loses the model's own previous answer.
The trailing header is what tells the model it is its turn. Without it generation continues the last message, which reads as the model impersonating the user.
These are the ones that fail silently. The config is accepted, nothing raises an error, and the consequence arrives later.
Most templates emit their own BOS, so the tokenizer adds a second one the model never saw in training. Nothing errors and the output is merely worse, which makes it the hardest prompt bug to attribute: the same messages through a different library appear to work better for no visible reason.
Instead:Render with the template, then tokenize with `add_special_tokens=False`. Apply one or the other, never both.
Mistral v1 and v3 and Gemma have no system role at all. Depending on the library the message is folded into the first user turn or dropped silently, so an instruction that works in one stack disappears in another.
Instead:Put the instruction at the top of the first user message yourself when the format has no system role. Then the behaviour is the same everywhere.
Gemma calls the model's turns `model`. A mismatched role is either a template error or a silently skipped turn, and a skipped turn drops the model's own previous answer from the conversation while everything still appears to work.
Instead:Map roles per format rather than assuming the OpenAI names carry across.
Fine-tunes routinely ship a modified template and merged models sometimes ship the wrong one entirely, so the canonical Llama 3 template can be wrong for the Llama 3 fine-tune you are running.
Instead:Compare against `apply_chat_template` on the tokenizer you are actually about to use. `tokenizer_config.json` is the authority for your checkpoint.
The template in between is where three specific things go wrong, and all three produce output that is merely worse rather than an error, which makes them the hardest prompt bugs to attribute.
Most templates emit the beginning-of-sequence token themselves: <s> for Llama 2 and Mistral, <|begin_of_text|> for Llama 3. Tokenize the rendered string with add_special_tokens=True, which is the default nearly everywhere, and the model gets two. It never saw two in training. Nothing errors and the output is quietly worse, so the same messages through a different library appear to work better for no visible reason.
template renders <|begin_of_text|><|start_header_id|>user...
tokenizer adds <|begin_of_text|>
model receives <|begin_of_text|><|begin_of_text|>...
fix: tokenize with add_special_tokens=FalseMistral's v1 and v3 templates and Gemma's have none at all. A system message is either folded into the first user turn or dropped, and which one happens is a property of the library rather than the model. So a system prompt that works in one stack silently disappears in another. Put the instruction at the top of the first user message yourself and the behaviour is the same everywhere.
A message with role assistant does not match the template. Depending on the renderer that is a template error or a silently skipped turn, and a skipped turn means the model loses its own previous answer from the conversation while everything still appears to work.
<start_of_turn>user
Hi<end_of_turn>
<start_of_turn>model not "assistant"
The trailing header is what tells the model it is its turn. Leave it off and generation continues whatever the last message was, which usually reads as the model impersonating the user or restating the question before answering it.
Two user turns in a row is either a template error or a prompt shape the model did not see in training. Merge consecutive messages of the same role into one with a blank line between them, rather than relying on the renderer to do something sensible.
It renders the templates as the formats define them, and your model's tokenizer_config.json is the authority for your specific checkpoint: fine-tunes routinely ship a modified template, and a merged model sometimes ships the wrong one entirely. Compare this output against apply_chat_template on your actual tokenizer before trusting either. It also renders text only, so tool definitions, tool results and image parts are not shown, and it counts characters rather than tokens because the token count depends on the tokenizer you are about to use.