Breaking the Draft Cascade: Cross-Family Joint Decoding for Structured Outputs

When a language model has to answer in strict JSON or SQL, forcing the format from the first token hurts its reasoning. A recent fix is to let the model write a free-form draft first and then have a "formatter" pass copy the conclusion into the required structure. This recovers much of the lost accuracy. It also creates a two-stage pipeline, and the formatter faithfully transcribes whatever the draft got wrong: a misremembered entity, an arithmetic slip, a swapped role. We propose to stop treating the formatter as a single copier. Two formatters from different model families each read their own independent draft, and they jointly write one structured output, character by character, under a single shared grammar. Where they agree, nothing changes. Where they disagree, the output moves toward values both find plausible, or abstains explicitly in a null field instead of fabricating one.

The obstacle is mechanical, and it is the reason this has not been done. Models from different families split text into incompatible subword vocabularies, so their next-token probabilities cannot be multiplied directly. Grammar-constrained decoding, which blocks any token that would break the syntax, is built around one model's vocabulary. Both problems disappear at the level of characters. A grammar is naturally a character-level automaton, and any model's subword distribution can be summed into a next-character distribution. The research question is whether this joint decoding actually reduces the errors that formatting pipelines inherit from their drafts. Specifically, we ask whether it does so on compositional fields, such as SQL queries, free-text strings and justifications, where simply comparing two finished outputs is ill-defined. And we ask whether it beats spending the same compute on more samples from one model.

Core Claim

Two prior results are combined here. The first is the diagnosis and fix for the "reasoning tax." The Hidden Cost of Structured Generation in LLMs: Draft-Conditioned Constrained Decoding (DCCD) shows that standard constrained decoding drops a 14B model on MATH500 from 76% to 47%, and that conditioning the constrained pass on an unconstrained draft brings it back to 58.6%. DCCD motivates this work, but its second stage has no defense against a wrong draft. The second is tokenization-invariant ensembling. CharED: Character-wise Ensemble Decoding for Large Language Models sums subword probabilities into next-character probabilities and proves the resulting ensemble depends only on each model's distribution over strings. CharED supplies the mixing mechanism, but it was never combined with a grammar mask or aimed at pipeline errors.

Our claim is about the structured setting specifically. A character-level product of two cross-family, draft-conditioned formatters under one grammar lowers the rate at which draft errors reach the final output. It does so without giving up DCCD's recovery of reasoning accuracy, and at matched compute it does better than output-level voting on fields whose values are long or compositional. We concede up front that for short categorical fields, simple per-field voting is probably just as good. That regime is a sanity check, not the target.

The combined next-character distribution at each decoding step is

$$ q(c \mid y_{<t}) \;\propto\; m_G(c \mid y_{<t})\; p_A(c \mid x, d_A, y_{<t})^{w}\; p_B(c \mid x, d_B, y_{<t})^{1-w}, $$

where the symbols are as follows. $c$ is a candidate next character and $y_{<t}$ the structured output so far. $x$ is the source prompt. $d_A$ and $d_B$ are free-form drafts written independently by models from families A and B. $p_A$ and $p_B$ are each formatter's subword distribution summed to the character level. $m_G$ is 1 if the grammar automaton accepts $c$ in its current state and 0 otherwise. $w$ is a mixing weight. Using a product rather than an average is deliberate. A product puts little mass on a value that only one formatter supports, which is the signature of an error carried in from one draft.

We extend the schema so that every value field also admits an explicit abstain value. We flag a field when the summed per-character divergence between $p_A$ and $p_B$ inside that field exceeds a threshold tuned on held-out data. This makes formatter disagreement a structured, checkable abstention rather than a silent fabrication.

Research Program

The system has three parts.

The first is a character-level grammar engine that serves several models at once. For each model, we track which partial subword it is in the middle of, as CharED does. We handle prompt-boundary fragments the way byte-level sampling does in Anchored Decoding, where fusing models from different families at the byte level was shown to be practical with about 1.1x overhead when one model is small. The grammar mask is applied once, to characters, so the mismatch between subwords and grammar terminals never arises. That mismatch is the problem Flexible and Efficient Grammar-Constrained Decoding and Gram2Token solve with precompiled per-tokenizer tables. Their methods are faster but tied to a single vocabulary, which is exactly what we need to drop.

Pitfalls, Subtleties, and Techniques in Automata-Based Subword-Level Constrained Generation shows that constraints can push a model into token splits it would never produce on its own, which constrains our design. Each model is therefore re-scored on its own canonical tokenization of the emitted prefix, never on forced splits.

The second part is a speed path. Character-level decoding is slow because it may need several forward calls per word, and this is CharED's stated cost. When both formatters' top next-token strings share a common prefix that the grammar allows, we emit that prefix in one step. We drop to character-level mixing only inside spans where the two formatters diverge. We expect most of the output, including the syntax and the fields where the formatters agree, to take the fast path. We will measure the fraction that does.

The third part is the analysis that justifies using two families. We will measure how strongly draft errors are correlated within one family, comparing different samples and different sizes, against the correlation between families on the same items. The whole proposal depends on cross-family errors being less correlated. If they are not, a same-family product, which needs no tokenizer handling at all, will do just as well, and we will report that.

Baselines

The naive reference is standard single-pass grammar-constrained decoding with one model, with the masks produced by an existing engine. This shows the size of the reasoning tax on each benchmark. DCCD reports this tax as accuracy falling from 76% to 47% on MATH500 for a 14B model, and as 15.2% strict accuracy on GSM8K for a 1B model.

The strongest existing method is DCCD itself, run with each family's best model as both drafter and formatter. Its reported numbers are 58.6% MATH500 accuracy for a 14B model and 39.0% GSM8K strict structured accuracy for a 1B model. Before comparing, we will reproduce DCCD at those scales with the same schemas, so that its numbers are the reference our method must clear. It is the right competitor because it already solves the reasoning tax, and our contribution is only about what it leaves broken. Beating it means higher end-task accuracy and a lower measured error-inheritance rate, with syntactic validity unchanged and inference compute within the same budget.

The compute-matched voting baseline runs DCCD K times with the same total FLOPs our method uses. The K runs are either all from the stronger family or split across both families. Categorical and numeric fields are decided by per-field majority vote. Compositional fields are chosen by picking the whole output with the highest agreement with the other samples. Cross-family DCCD pairs with output-level agreement checking also falls under this baseline. It is the sentence-level fallback that Yangyifan Xu et al. describe as less expressive than combining token-level probabilities. Beating it requires a statistically significant gain on compositional fields, whether SQL execution accuracy or the factual precision of free-text fields. Parity on short fields is expected and acceptable.

Two ablations isolate the key ingredients. The first is a same-family product: two formatters sharing one tokenizer, each with its own draft, combined at the token level with no character machinery. If this matches the cross-family version, heterogeneity is not doing the work. The second replaces the product with CharED's weighted average. If the average performs as well, the claim that a product suppresses single-draft errors fails. Prompts, schemas, drafts and tuning budgets for $w$ and the abstention threshold, tuned on the same development split, are identical across all variants.

Why not just…?

Why not just sample more drafts from one model and vote? This is the most serious threat. For fields like a GSM8K answer or a FEVER verdict, voting probably captures most of the benefit, and we expect it to tie with our method there. Voting is undefined, or reduced to choosing a whole output, for SQL queries, evidence justifications and biography text, where two samples almost never match exactly. Its value also depends on samples from one model making different mistakes. The compute-matched comparison and the error-correlation analysis settle this directly. If same-family errors turn out to be nearly independent and voting wins on compositional fields, the proposal is wrong in its central claim.

Why not just mix the two models' logits without a grammar, then repair or reprompt on parse failure? Unconstrained cross-family mixing already exists in ALIGNBEAM and CharED, and parse-error reprompting is the approach of ASPIRO. Mixing without a grammar gives no validity guarantee. Reprompting is itself another serial stage that can inherit errors, and its cost is unbounded. StructEval reports that even GPT-4o averages only 76.02% on structured generation, so we do not expect unconstrained generation followed by repair to be reliable. We include this approach as a variant to measure, not to dismiss: unconstrained CharED mixing plus one ASPIRO-style retry, under the same budget.

Why not just constrain string fields to copy spans from the source? For purely extractive fields this is a strong, cheap defense, and we will add it to every system where the schema allows it. It cannot catch role swaps, wrong arithmetic, or invented SQL join logic. Those are the inherited errors this proposal targets, and they are reported separately.

Proposed Experiments

The models are pairs at the 1B and roughly 8B scales drawn from the Qwen, Llama and Gemma families. The tasks have JSON or SQL schemas. GSM8K and MATH500 use DCCD's answer schemas so the comparison is direct. BIRD and Spider use a SQL grammar and are scored by execution accuracy. UCS-SQL's 68.95% on BIRD with GPT-4o (UCS-SQL) serves as an external ceiling, not a matched baseline. FEVER is reformatted as JSON with a verdict, evidence sentence IDs and a free-text justification. For context only, Claim-Dissector's FEVER score of 76.5 (Claim-Dissector) comes from a trained, encoder-based system and is not comparable. A structured-biography task scores each JSON record with FActScore. StructEval-T is used to confirm validity and keyword scores do not regress. We report Qwen3-4B's 67.04% from the StructEval paper as a reference point.

The decisive check is a controlled draft-corruption test. In one of the two drafts, we replace the final number, a schema column name or an entity with a plausible wrong value. We then measure how often that value appears in the final output. Under DCCD the rate should be near 100%. If our method does not cut it sharply, while leaving outputs with uncorrupted drafts unchanged, the mechanism has failed regardless of aggregate accuracy.

Three further checks complete the picture. The first is selective accuracy: whether fields flagged by divergence-based abstention are in fact enriched for errors. The prior that this could work is Agrawal et al., whose evidence is that fabricated content shows inconsistency when probed indirectly. The second is per-field-type gains against compute-matched voting, which tests the claim that the advantage is concentrated in compositional fields. The third is wall-clock latency and the fraction of characters on the fast path, compared against single-model DCCD running on a precompiled grammar engine.