Teaching Small Models to Reason Inside a Grammar Mask, Whatever the Prompt Says

Engineers who need a language model to emit valid JSON or another rigid format usually rely on two tools. The first is constrained decoding, where a parser blocks every token that would break the format. The second is careful hand-written instructions. Both are fragile in ways that compound. Forcing a small model into structural tokens from the first step distorts its reasoning: on GSM8K, a 1B model under standard grammar masking scores 15.2% strict accuracy, and 39.0% when it is first allowed to write a free-text draft (Draft-Conditioned Constrained Decoding). How much the mask hurts also depends on whether the prompt already steered the model toward the structure. That makes the hand-tuned format prompt a hidden, brittle component, and nobody has measured how sensitive constrained generation is to its wording.

We propose to distill the draft-then-format behavior into the model's own weights, so that the model reaches draft-level accuracy in a single constrained pass under a plain prompt and gives the same answer however the format request is phrased. The teacher is the same model given privileged context: an unconstrained draft plus several differently worded format instructions. The student samples its own outputs under the grammar mask. At every position it is trained toward the teacher's next-token distribution, restricted to the same set of grammar-legal tokens and averaged over instruction wordings. In short, the aim is to make grammar-constrained reasoning cheap and indifferent to how the format request is phrased, by teaching the student where the teacher would put its probability inside the legal token set.

Core Claim

The "projection tax" of grammar masking is largely a symptom of low model confidence at the start of structured output. ProSA ties prompt sensitivity to exactly this condition: small wording changes flip outputs when the next-token distribution is flat. SENSEL finds that sensitivity correlates negatively with correctness (Pearson r up to −0.40). If these are the same phenomenon, then training that sharpens the student's distribution over grammar-legal tokens should do two things at once. It should shrink the accuracy gap between single-pass constrained decoding and two-pass drafting, and it should lower sensitivity to paraphrases of the format instruction.

We predict both effects from one training signal. We also predict that the gains hold on instruction wordings and output formats never seen during training. If the accuracy gap closes but sensitivity does not fall, or the reverse, the claimed link is false. That result would be informative in its own right.

How This Differs From Existing Work

Three papers each supply one piece, and none combines them.

DCCD diagnoses the projection tax and fixes it at inference time with two passes and frozen weights. We reuse its two-pass pipeline as our teacher and remove the second pass at deployment.

SDPRO distills a prompt-optimized "self-teacher" into a plain-prompt student. The student learns from the teacher's token distributions on its own samples, using Jensen-Shannon divergence and a lightweight weight update (LoRA). Its tasks are unconstrained text, and it never measures sensitivity. We borrow its self-teacher recipe and change two things: the student's action space is pruned by a parser, and the teacher is an average over instruction wordings rather than a single optimized prompt.

PromptKD and ImitKD establish that the student should learn from its own imperfect outputs rather than only from teacher-written text, which avoids exposure bias. Neither considers what happens when a grammar deletes most of the vocabulary at each step. The teacher's unconstrained distribution then puts mass on tokens the student is forbidden to produce, and the training signal partly points at illegal moves. We correct this by renormalizing the teacher over the legal set.

On the sensitivity side, STED measures JSON consistency across sampling temperatures and synthetic schema changes, but not across prompt wording, and proposes no remedy. PSM optimizes a prompt suffix against a different kind of sensitivity, namely leakage of the system prompt. Both motivate the evaluation rather than supply the method.

Our transfer is deliberately one-directional: the teacher is never adapted to the student. The teacher is the student's own weights with extra context, so there is little capacity gap to bridge. We test that choice rather than assume it, as described under Baselines.

Research Program

The construction has three parts.

The first part is the teacher. For each training input $x$, we pick $K$ wordings $\pi_1, \dots, \pi_K$ of the format instruction. We draw them from ProSA's four template categories (plain input, emotional support, role play, output requirements). We also vary how the schema is presented (a raw JSON Schema, a typed interface, a prose description) and the order of any examples. The model writes one unconstrained draft $d$ under a default instruction. The teacher then conditions on $x$, $d$, and each wording $\pi_k$.

The second part is the student. It sees only $x$ and one plain instruction, and generates under the grammar mask. Let $s_t$ be the student's prefix at step $t$ and $V(s_t)$ the set of tokens the incremental parser of Grammar-Constrained Decoding allows next. The teacher target at that step is its wording-averaged distribution restricted to legal tokens:

$$q(v \mid s_t) = \frac{\mathbb{1}[v \in V(s_t)]\;\frac{1}{K}\sum_{k=1}^{K} p_\theta(v \mid x, d, \pi_k, s_t)}{Z(s_t)},$$

Here $p_\theta$ is the frozen base model's next-token probability and $Z(s_t)$ normalizes over $V(s_t)$.

The third part is the training objective. The student $r_\phi$, the same model with LoRA parameters $\phi$, is renormalized over the same legal set. It is trained on its own constrained samples $\tau \sim r_\phi$ to minimize

$$\mathcal{L}(\phi) = \mathbb{E}_{x,\tau \sim r_\phi}\sum_t \mathrm{JSD}\big(r_\phi(\cdot \mid s_t)\,\|\,q(\cdot \mid s_t)\big),$$

following SDPRO's choice of divergence. Averaging over wordings is what should make the student indifferent to phrasing. Restricting both distributions to the mask ensures the student is never pushed toward tokens the parser will veto.

Beyond the method itself, the program produces the first measurement of prompt sensitivity under a grammar mask. We report ProSA's PromptSensiScore (PSS, how often an instance's answer changes across prompt templates) and STED-based semantic consistency across held-out wordings. We report both for plain decoding, masked decoding, DCCD, and each trained student. This establishes whether masking by itself increases or decreases sensitivity, which the research file flags as under-characterized.

We evaluate two deployment regimes. In the fixed-schema regime, a downstream consumer dictates the schema and no free-text field is allowed. In the flexible-schema regime, the schema may include a leading "reasoning" string.

Baselines

The naive reference is the base model with a single hand-written format prompt and grammar-constrained decoding. On GSM8K with a 1B model, DCCD reports 15.2% strict structured accuracy for this setup. We re-run it on our models, with unconstrained decoding as a second reference point. Any trained student must clearly exceed this floor, or the method has failed outright.

The strongest existing method on accuracy is DCCD itself: two passes, frozen weights, a free draft, then masked extraction. It reports 39.0% on GSM8K at 1B. It is the upper target because it is the teacher's own behavior. We run it on the same models, grammars, and prompts. The student beats it in a meaningful sense if it matches DCCD's accuracy to within about two points with roughly half the generated tokens, and shows lower PSS across held-out wordings. Exceeding DCCD's accuracy is not required. Matching it at single-pass cost is the claim.

The strongest training competitor is offline supervised fine-tuning on DCCD outputs. We collect the teacher's final structured outputs under all $K$ wordings, pair each with a randomly chosen wording as the student's input, and fine-tune on them. It uses the same LoRA rank, the same number of training tokens, and the same hyperparameter search budget as our method. This baseline isolates whether on-policy training and full distribution matching matter at all. The file offers indirect evidence that they do: PromptKD reports ROUGE-L 26.9 against 24.9 for SFT on Dolly. There is no reported result on Lacuna for either method under grammar constraints. We therefore treat this baseline as a live threat.

We also run an SDPRO-style variant: a single GEPA-optimized prompt as teacher, no draft, and no mask renormalization. It receives the same optimization budget. There is no reported result on Lacuna for SDPRO on grammar-constrained tasks. Its HotpotQA figure (70.7% teacher, 66.0% student with Qwen3-8B) is from an unconstrained task and is not comparable.

Three ablations isolate the key ingredients. The first drops mask renormalization of the teacher, so the student matches the raw unconstrained teacher distribution. The second uses a single-wording teacher ($K=1$), which isolates whether wording averaging is what reduces sensitivity. The third drops the draft from the teacher's context, which isolates the projection-tax fix from the robustness effect.

Why not just…?

Why not just put a "reasoning" field first in the schema and decode once? In the flexible-schema regime this is the most serious threat. It is essentially DCCD compressed into one pass, and it may recover most of the accuracy for free. We include it as a baseline in that regime and train our student under the same schema, so any additional gain is attributable to training. The trick does not apply when the schema is fixed by the consumer. It also does nothing, as far as the file shows, for sensitivity to how the instruction is worded. If it matches our student on both accuracy and PSS in the flexible regime, we will report that the contribution is confined to fixed-schema deployment.

Why not just fine-tune on DCCD outputs with paraphrased prompts? This baseline is described above and run at matched compute. The mechanism predicts it will fall short in two ways. It trains only on teacher-written prefixes, so it cannot correct the student's own drift partway through a structure; this is the exposure-bias failure documented by ImitKD. It also learns from one sampled output per example rather than a full distribution over legal tokens, which is the part that should sharpen confidence. If it ties with our method, the simpler recipe wins, and we will recommend it.

Why not just search for the best format prompt with an optimizer? Tools such as DSPy or GEPA could search for a format prompt, even with a consistency term in the objective. They can find a good prompt for the developer. They cannot control how end users or upstream modules word their requests, and they leave the weights, and hence the flat confidence behind the tax, unchanged. SDPRO reports that GEPA alone plateaus 4.7 points below co-evolved weights on HotpotQA. We run a GEPA search with an added PSS penalty at matched evaluation budget, then test it on held-out wordings. If it matches the trained student there, the weight updates are unnecessary.

Why not just add a few-shot example and decode greedily? ProSA finds that moving from zero to one example is the single largest stabilizer. SENSEL shows that example order then becomes a new source of brittleness, and greedy decoding removes sampling noise but not wording sensitivity. We include one-shot greedy decoding as a baseline and also measure PSS across permutations of the examples.

Proposed Experiments

Tasks and models. Training and in-domain evaluation use GSM8K, MATH500, and FOLIO with DCCD's JSON answer schemas, so our numbers are directly comparable with its 15.2% and 39.0%. To test transfer, we evaluate on StructEval-T. There we train on JSON only and test on YAML, XML, and CSV, against the benchmark's reported average of 67.04% for Qwen3-4B. We also evaluate on SynthIE-text closed information extraction under the input-dependent grammar, where LLaMA-33B with grammar-constrained decoding reports F1 36.0. We use a 1B model, Qwen3-4B, and Qwen3-8B.

Metrics. For accuracy we report strict structured accuracy or F1 and syntax validity. For efficiency we report generated tokens and wall-clock latency per query. For robustness we report PSS and STED consistency, computed over two held-out sets: 12 instruction wordings never used in training, and one entire ProSA category held out.

Decisive checks. The first check is whether the single-pass student closes most of the gap between masked decoding and DCCD on GSM8K at 1B. We compare it with offline fine-tuning at equal compute. The second check is whether PSS on held-out wordings drops relative to both DCCD and the $K=1$ ablation. If it does not, wording averaging is not doing the work. The third check is whether removing mask renormalization measurably hurts. If it does not, the grammar-aware teacher signal is unnecessary. The fourth check tests the confidence hypothesis directly: across instances, we test whether the reduction in each instance's projection tax correlates with the reduction in its PSS and the increase in its top-token probability at the first structural tokens.

Falsification conditions. The proposal fails if the student cannot beat offline fine-tuning. It also fails in the flexible-schema regime if a reasoning-field schema alone matches it on held-out wordings. A narrower failure is if gains vanish on StructEval-T formats not seen in training, which would mean the student memorized JSON rather than learned to reason inside a mask.