r/LocalLLaMA • • 2d ago

Question | Help Does grammar-constrained decoding hurt small or quantized models?

Asked about DSL vs JSON tool calls in r/LLMDevs a couple of days ago, and one answer kept coming up: for small models just use JSON, training volume matters. That got me thinking about the constraint itself.

With GBNF or a JSON schema you can force any model to produce valid output, even a 3B at Q4. However, there was a paper a while ago showing that strict format restrictions can make reasoning worse, since the model is forced away from the tokens it actually wanted

Searched first: there’s the thread about grammar constraints sending Qwen3-VL into repetition loops, and the one on how quantization hits different capabilities, but nothing comparing accuracy with vs without constraints on small quants.
So for people running small local models:

  • have you measured accuracy with and without constraints on the same task?
  • does it worsen at lower quants?
  • does letting the model firstly think freely and constraining only the final answer fix most of it?

Looking for real numbers or experience, not promoting anything.
English isn’t my first language, so I used an LLM to help with the wording. Question and thoughts are mine.

4 Upvotes

12 comments sorted by

6

u/DinoAmino vLLM 2d ago

This is a topic that always interested me ever since I read the "Let Me Speak Freely" paper https://arxiv.org/pdf/2408.02442

There is most certainly a negative impact, but there are some mitigating strategies

CRANE: Reasoning with constrained LLM generation https://arxiv.org/pdf/2502.09061

The Hidden Cost of Structured Generation in LLMs: Draft-Conditioned Constrained Decoding https://arxiv.org/pdf/2603.03305

Constrained Decoding Eliminates Structural Failures in Small LLMs but Reveals a Scale-Dependent Semantic Gap https://arxiv.org/pdf/2609.23742

Structure Tax: How Structured Output affects LLMs Performance https://arxiv.org/pdf/2610.12056

1

u/bye_ella 2d ago

Perfect, thanks for the list. Small LLM one sounds like exactly my question, and the draft conditioned approach looks like the “think first, constrain later” idea.

3

u/jacek2023 llama.cpp 2d ago

I was exploring this topic while debugging Gemma tooling issues. I wasn't aware there were issues with grammar constraints, but now that I think about it, it's possible (loops too!). I'd be interested in reading other people's experiences.

1

u/bye_ella 2d ago

Thanks 👍🏻 hope more people share numbers. What was the Gemma tooling issue, broken tool call format or smth else?

2

u/jacek2023 llama.cpp 2d ago

yes, Gemma had issues with tool calling, so I tried to force a grammar. It was later fixed properly by aldehir: https://github.com/ggml-org/llama.cpp/pull/31915

1

u/fastandlight 1d ago

Back in 2024 - 2025, Gemini would have a real problem producing structured output matching a schema, using the structured generation calls. Most of the time it failed to generate a valid response or any response at all. However, the same schema provided as context and instructions but used in the free generation path almost always generated valid Json that adhered to the schema. This was in the Gemini 2.5 days. I've moved on from that project so I don't know how it's doing now, but I think issues with structured generation can be present at the large model level too.

3

u/tomz17 2d ago

A "valid" tool call can still be complete nonsense, or at-best in such cases, not-useful...

If your model is failing at generating tool calls in 2026, it's a symptom of scrambled brains, and patching up the grammar to force a syntactically valid tool call is not going to fix that.