Every "JSON mode" and structured-output feature your agent stack depends on is the same trick underneath: at each decoding step, a grammar engine computes which tokens can legally come next and masks out everything else before sampling. The model never gets the chance to emit an invalid byte. That guarantee is what makes tool calls parseable on the first try — and it is also where a famous benchmark result went off the rails, with one model dropping from 86.51% to 23.44% on grade-school math the moment a JSON schema was enforced.

Both halves of that story — the mechanism and the benchmark fight — are worth understanding, because the difference between "structured outputs are free" and "structured outputs lobotomize my model" usually comes down to a single design decision you control: the order of keys in your schema.

The mechanism: a mask over the vocabulary, every token

Constrained (or "guided") decoding was put on a rigorous footing by Willard and Louf in Efficient Guided Generation for Large Language Models, the paper behind the Outlines library. Their reformulation: text generation is a walk through the states of a finite-state machine. Compile your constraint — a regex, or a JSON Schema lowered to a regex — into an FSM. At each step, the FSM's current state defines exactly which continuations are legal. Mask the rest, sample from what's left, advance the state.

The naive version of this is expensive. Checking "which of the model's tokens are valid here?" by testing each one is O(N) in vocabulary size, at every single generated token. Willard and Louf's contribution is an indexing step: precompute, once per grammar, a map from each FSM state to the set of token IDs that are valid transitions out of it. Generation then becomes an average-cost O(1) lookup per step — pull the allowed-token set for the current state, apply it as a logit mask, sample, follow the transition.

There's a subtlety that makes this harder than it sounds: tokens are not characters. A character-level FSM for "score": has a transition per character, but the tokenizer might cover that span with tokens like ", score, ":, or entirely different segmentations depending on context. A single token can traverse several FSM states at once, and many different token sequences spell the same string. The paper's answer is to build the automaton over the vocabulary itself — each precomputed transition already accounts for the multi-character path a token takes through the character-level machine. That's what the index actually stores.

Regular expressions run out at recursion

An FSM handles anything a regex can express, and a surprising amount of JSON fits. But JSON Schema is not regular: a schema with recursive $ref definitions — trees, nested comments, arbitrarily deep objects — describes a context-free language, and a finite automaton fundamentally cannot count nesting depth. Willard and Louf sketch the extension in the same paper: replace the FSM with a pushdown automaton, adding a stack that tracks nesting.

The engineering problem is that the stack ruins the precomputation trick. With an FSM you can enumerate every state ahead of time and cache a token mask per state; a pushdown automaton's state includes stack contents, which are unbounded. XGrammar (MLSys '25), now a common backend in open-source inference servers, attacks exactly this. Its observation: most tokens' validity doesn't depend on the stack at all. XGrammar splits the vocabulary into context-independent tokens, whose validity is determined by the local grammar position and can be prechecked and stored in an adaptive token-mask cache, and context-dependent tokens, the small remainder that must be checked at runtime against a persistent stack structure. Grammar transformations shrink the context-dependent set further. The paper reports up to a 100x speedup in mask generation over existing engines and near-zero overhead for structure in end-to-end serving.

So the "does constrained decoding slow inference down?" question has a good answer in 2026: with a precompiled grammar and a modern engine, the mask is effectively free per token. You pay a one-time grammar compilation cost, then lookups.

The benchmark fight: 63 points lost, then found

If the mask only removes tokens that would be invalid JSON, it shouldn't make a model dumber. Yet Let Me Speak Freely? (Tam et al., EMNLP 2024 Industry Track) measured exactly that. The authors compared three regimes across GPT-3.5-turbo, Claude-3-Haiku, Gemini-1.5-Flash, LLaMA-3-8B-Instruct, and Gemma-2-9B-Instruct: API-enforced JSON mode (constrained decoding), format-restricting instructions in the prompt (asking for JSON without enforcement), and a two-step natural-language-then-convert pipeline.

On reasoning benchmarks the damage was dramatic. On GSM8K, Claude-3-Haiku fell from 86.51% with free-text answers to 23.44% under a JSON schema; GPT-3.5-turbo fell from 75.99% to 49.25%. Stricter constraints correlated with worse reasoning. Meanwhile — and this is the underrated half of the paper — classification tasks moved the other way: on DDXPlus medical diagnosis, Gemini-1.5-Flash got a significant accuracy boost from JSON mode, because the constraint eliminates malformed answer selections. And parsing failures didn't explain the reasoning gap; the outputs were valid, the answers were just wrong.

The paper's own error analysis contains the smoking gun: in their setup, 100% of GPT-3.5-turbo's JSON-mode responses placed the answer key before the reason key. The model committed to a final answer before generating its reasoning. Constrained decoding didn't remove the model's ability to think — the schema ordering removed its opportunity to think first. Chain-of-thought only helps if the chain precedes the conclusion; a schema that forces {"answer": ..., "reasoning": ...} converts your carefully prompted CoT setup into zero-shot direct answering, token order being the only order a causal LM has.

The team behind Outlines pushed back harder in Say What You Mean, a direct response. Their critique: the paper's structured and unstructured runs used markedly different prompts, the JSON-mode prompts under-specified the expected schema, and unstructured outputs were graded via an "AI parser" (Claude-3-Haiku extracting answers) that hid parsing fragility. Re-running with Llama-3-8B-Instruct, identical prompts, and schemas that leave room to reason, structured generation matched or beat unstructured: 0.77 vs 0.77 on GSM8K, 0.77 vs 0.73 on Last Letter, 0.44 vs 0.41 on Shuffled Objects. Their framing is the right one: structured generation is not merely another name for JSON mode — the schema, the prompt, and the parser are one artifact and must be designed together.

Your schema is control flow

The practical synthesis, for anyone building agents on structured outputs:

  1. Put reasoning fields before answer fields. In a causal model, key order is execution order. {"reasoning": ..., "answer": ...} preserves chain-of-thought; the reverse silently discards it.
  2. State the schema in the prompt too. The mask guarantees syntax, but the model plans in token space — a model that hasn't been told the format fights the mask, and per Willard and Louf the mask can only renormalize among tokens the grammar allows, not restore probability mass the model wanted to spend elsewhere.
  3. Use enforcement freely for extraction and classification. That's the regime where Let Me Speak Freely found constraints helping — the failure mode being eliminated (malformed or off-menu answers) is exactly what the mask prevents.
  4. Check what your engine supports before writing recursive schemas. Pure FSM backends handle regular structure; recursion needs a pushdown-automaton engine like XGrammar's CFG path.

The mask itself was never the problem. A grammar engine that runs in O(1) per token and provably emits valid JSON is one of the cleaner wins in the LLM serving stack. The failure mode was always upstream, in schemas written as data models rather than as what they actually are: a script the model executes in order, one masked token at a time.

References

constrained-decodingstructured-outputsjson-schemallm-inferencechain-of-thought