Small language models fine-tuned on a narrow technical domain can beat much larger general models at domain tasks. Many of those tasks, such as API calls, tool invocations, and typed configuration, must end in a machine-readable object like valid JSON. Forcing a model to write that object while it is still reasoning costs accuracy, and there is strong evidence for this. A known fix is a two-stage pipeline: the model first writes a free-form draft of its reasoning, and then a second step, the projector, converts the draft into the required format under a grammar that masks invalid tokens. So far that fix has been studied almost entirely with frozen, untrained models. This proposal asks what happens when both stages are fine-tuned for a domain. The central idea is to train the projector on the drafts the fine-tuned drafter actually produces, including its flawed ones, rather than on clean teacher reasoning.
The claim is testable and could fail. We claim that domain fine-tuning does not remove the accuracy cost of structured output on multi-step tasks. We also claim that most of the benefit a trained second stage can add comes from exposing it to the drafter's own imperfect drafts. The strongest counter-hypothesis is that fine-tuning one model to "think, then emit JSON" in a single pass absorbs the whole cost. We treat that as the decisive comparison, and we would report a null result if it holds.
What prior work establishes and leaves open
Three papers motivate the proposal by documenting the cost of formatting. Let Me Speak Freely? shows that format-restricting instructions hurt multi-step reasoning. The damage is worst when the schema puts the answer before the reasoning: GPT-3.5-Turbo on Last Letter Concatenation drops from about 56% to 25%. The same paper finds that JSON mode helps single-step classification, raising Gemini-1.5-Flash on DDXPlus from 41.6% to 60.3%. That result is a paradox we use as a control condition. Draft-Conditioned Constrained Decoding supplies our inference method. It samples an unconstrained draft, then decodes the structured output under grammar constraints conditioned on that draft. This lifts a 1B model's strict structured GSM8K accuracy from 15.2% to 39.0%, and recovers a 14B model on MATH500 from 47% to 58.6% (unconstrained: 76%). The method is training-free, and the authors name projector distillation as unexplored work (companion version). CAT runs a similar think-then-choose split for event argument extraction, but only zero-shot.
Four papers train decoupled stages. CasCoD trains one step from question to rationale and a second from question plus rationale to answer. It improves out-of-domain accuracy from 41.8% to 49.6%. Its answer step, however, is trained only on teacher rationales, even though at test time it sees the student's own. That train/test mismatch is the specific gap we target. SGFT pairs a fine-tuned guidance model with an untrained answer model. LoRID runs several adapters through iterative consensus loops. DualTune comes closest in the tool domain. It trains separate adapters for tool selection and argument generation and raises Qwen-2.5-7B on MCP-Bench filesystem from 15.0% to 61.5%, where monolithic fine-tuning reaches 39%. It has no reasoning draft, though, and trains per-tool adapters on gold targets. None of these works trains a grammar-constrained projector on the drafter's own outputs.
DLoFT constrains our design. It shows that ordinary fine-tuning on reasoning traces overfits to problem-specific content once training passes a turning point. That suggests a projector trained only on gold pairs will learn to answer from the question and ignore the draft. We measure that failure directly rather than assume it away. RAFT shows that training on deliberately imperfect context (distractor documents) makes models robust to it: 74.00% on HuggingFace APIBench versus 42.59% for plain domain fine-tuning. Our on-policy projector applies the same logic to imperfect drafts instead of imperfect retrieved documents.
Core Claim
Let $X$ be the input, $D$ a draft sampled from the fine-tuned drafter, and $Y$ the structured output decoded under a grammar $G$. A projector trained on teacher drafts learns $P(Y \mid X, D_{\text{teacher}})$. At test time it receives $D \sim$ drafter, which is a different distribution containing hedges, partial errors, and malformed intermediate values. We claim three things. First, this mismatch, not the projection step itself, limits trained two-stage pipelines. Second, training on on-policy drafts, with execution-checked targets, closes most of the gap. Third, the benefit appears only on multi-step tasks and disappears on single-step classification like DDXPlus, which would resolve the paradox from Let Me Speak Freely? in operational terms.
Research Program
We build one backbone (Qwen2.5 at 1.5B and 7B) with two low-rank adapters. Low-rank adapters are small trainable weight add-ons that leave the base model frozen. The drafter adapter is fine-tuned on domain questions paired with free-text reasoning distilled from a teacher. The projector adapter is fine-tuned to emit the schema object under grammar-masked decoding. Sharing the backbone keeps memory similar to a single model, so the comparison is about training targets, not model count. A separate 0.5B projector is a secondary variant for the latency question.
The projector is trained on a mixture. For each training input we sample $K=4$ drafts from the trained drafter. Each draft is paired with a target structured output, and the target design is the main object of study. In the faithful variant, the target is the draft's own conclusion rendered in the schema, which teaches pure translation. In the repairing variant, the target is the gold output, verified by execution or AST match, even when the draft is wrong. A mixed variant uses the gold target only when the draft's conclusion is close to correct: it differs from gold by one field or one argument. This keeps the projector from learning to ignore the draft wholesale. At inference we sample several drafts and select among projections by feasible mass, the probability the projector assigns to grammar-valid tokens, following DCCD. We also check whether training makes that selection score better calibrated.
The analysis centers on one diagnostic, draft reliance. We re-run the projector with the draft replaced by an empty string, by a draft from a different question, and by a draft with one intermediate value corrupted. A projector that relies on its draft should fail on the first two and follow the corruption in the third. A projector that has taken a shortcut will be unaffected. Among projector outputs that disagree with their draft's conclusion, we record how often the disagreement is a correct repair and how often it introduces an error.
Baselines
The naive reference point is the base instruction-tuned backbone with grammar-constrained decoding from the first token and no fine-tuning. It shows the size of the formatting cost that everything else must recover. The second reference is the same model with DCCD and no training. The file reports 39.0% strict structured GSM8K accuracy for a 1B model with this method (DCCD). We re-run it on our backbones and domains, since no reported result was found on Lacuna for DCCD on APIBench or MCP-Bench.
The strongest competitor is monolithic reasoning-first fine-tuning. One adapter is trained on sequences of the form "reasoning, delimiter, JSON," with the grammar switched on only after the delimiter. This is a single-pass version of the two-stage idea. It uses the same data as our drafter and projector combined, the same total adapter parameters, and the same number of tuning trials. It is strongest because fine-tuning may teach the model the format so well that separating the stages adds nothing. Beating it means a statistically reliable gain in execution accuracy on the multi-step domains, across three seeds, at matched sampling budget: the same number of drafts, with best-of-K applied to both. For the tool setting, the nearest published anchor is DualTune's 61.5% on MCP-Bench filesystem, against 39% for monolithic fine-tuning (DualTune). We reproduce DualTune's selector/argument decoupling on our backbone as a second strong baseline. For APIBench, RAFT's 74.00% on HuggingFace APIBench (RAFT) is the reported reference, with retrieval held fixed across all conditions.
The key ablation isolates on-policy training. It is a CasCoD-style projector trained on teacher drafts only (CasCoD), with identical architecture, data volume, and steps. The only difference is where the conditioning drafts come from. A further ablation removes the drafter's fine-tuning, which tells us whether gains come from a better draft or from a better handoff.
Why not just…?
Why not just fine-tune one model to reason first and then emit the JSON? This is the real threat. The format cost reported in Let Me Speak Freely? and DCCD was measured on untrained models, and fine-tuning could erase it. The only evidence against it is indirect: monolithic fine-tuning reached 39% versus 61.5% for decoupled adapters on MCP-Bench (DualTune), though that comparison had no reasoning draft. The head-to-head experiment above settles the question. If the monolithic model matches us, we report that domain fine-tuning absorbs the formatting cost, which is itself a useful result for practitioners.
Why not just use DCCD with a fine-tuned drafter and leave the projector frozen? A frozen projector is untuned for domain schemas and for the drafter's style. It may still work well, because the draft already concentrates probability on valid tokens. The "trained drafter, frozen projector" condition measures exactly this. If it comes within noise of the on-policy projector, the projector-training claim fails.
Why not just train the projector on clean teacher rationales, as CasCoD does? It is cheaper, and CasCoD's out-of-domain gains show it can work. However, DLoFT's finding that content overfitting sets in during reasoning fine-tuning (DLoFT) predicts that a gold-trained projector will stop reading its input. The draft-reliance test checks whether it does.
Why not just use a bigger general model with constrained decoding? Specialized small models already beat GPT-4 on domain agent tasks: 82% versus 62% Grade A on ChatEDA-Bench (ChatEDA). DCCD also found that two small models outperform one larger constrained model. We include a 14B constrained-decoding reference to confirm this on our domains.
Proposed Experiments
We use four evaluation settings. Gorilla APIBench and the Berkeley Function Calling Leaderboard come via the gorilla repository, scored by AST and execution match. MCP-Bench filesystem is scored by tool-calling accuracy. GSM8K is scored by strict structured accuracy, with a JSON schema whose reasoning fields cannot be reordered, so results link to the DCCD numbers. DDXPlus serves as the single-step classification control. The main metrics are execution accuracy, schema validity, tokens and wall-clock time per query, and draft reliance.
Three results would decide the question. First, on APIBench, MCP-Bench, and GSM8K, the on-policy projector must beat both the monolithic reasoning-first model and the teacher-draft projector. If it beats neither, the central claim is false. Second, on DDXPlus, the two-stage gain should vanish or reverse, matching the prediction that projection helps only when there is multi-step reasoning to protect. Third, the draft-reliance test must show that the on-policy projector follows corrupted intermediate values more often than the gold-trained projector. Otherwise any accuracy gain comes from the projector re-solving the problem itself, not from a better handoff. We also report the repairing-versus-faithful trade-off as a curve of correct repairs against introduced errors, because a projector that repairs aggressively may also corrupt correct drafts.
References
- Let Me Speak Freely? A Study On The Impact Of Format Restrictions On Large Language Model Performance
- The Hidden Cost of Structured Generation in LLMs: Draft-Conditioned Constrained Decoding
- Draft-Conditioned Constrained Decoding for Structured Generation in LLMs
- How do LLMs' Preferences Affect Event Argument Extraction? CAT: Addressing Preference Traps in Unsupervised EAE
- Improve Student's Reasoning Generalizability through Cascading Decomposed CoTs Distillation
- Enhancing the Reasoning Capabilities of Small Language Models via Solution Guidance Fine-Tuning
- Can Large Models Teach Student Models to Solve Mathematical Problems Like Human Beings? A Reasoning Distillation Method via Multi-LoRA Interaction
- DualTune: Decoupled Fine-tuning for On-Device Agentic Systems
- DLoFT: Gradient-Decoupled Fine-Tuning for Generalizable Long Chain-of-Thought Reasoning
- RAFT: Adapting Language Model to Domain Specific RAG
- ChatEDA: A Large Language Model Powered Autonomous Agent for EDA