Reasoning Only Where the Hypotheses Disagree: Span-Anchored Deliberation for Speech Transcript Correction

Large language models can fix speech recognizer mistakes by reading the recognizer's top few guesses (its "N-best list") and writing what the speaker probably said. They can even produce words that appear in none of the guesses. The trouble is that they also rewrite text that was already right, drop words, and add chatty filler. Two remedies exist, and each has a cost. Letting the model "think out loud" before answering (chain-of-thought prompting) helps it reason about sound-alike words, but it is slow and makes hallucination worse. Forcing the output to follow a formal grammar at decoding time (grammar-constrained decoding, which blocks any token the grammar forbids) removes formatting failures, but it is known to hurt reasoning when applied from the first token.

We propose to build both remedies into a single per-utterance grammar shaped by the recognizer's own uncertainty. The output is a short program of edit commands. Wherever the N-best hypotheses agree, the grammar allows only a "keep these words" command. Wherever they disagree, it opens a small slot where the model may reason freely about pronunciation and context, and then it must emit a replacement that the grammar checks. Reasoning therefore costs tokens only at the few positions that need it, correct regions cannot be rewritten, and every output can be executed. The model can still insert a word that no hypothesis contains, which is how generative correction gets past the best single hypothesis in the list.

Core Claim

Prior work handles reasoning and constraints at the level of the whole utterance. CRANE generates free reasoning, then switches on grammar masking once a delimiter appears, and recovers accuracy on symbolic math (38% vs 29% on GSM-Symbolic). Draft-Conditioned Constrained Decoding writes a free draft and then a constrained second pass. Both were tested only on math, logic, and summarization. Both treat the "reasoning tax" (the accuracy lost when masking starts at token zero) as a single global switch.

We claim that in transcript correction the right unit is the uncertain span, not the utterance. The specific hypothesis has three parts. Interleaving short, unconstrained reasoning segments inside an input-dependent edit grammar, with each segment anchored to an N-best disagreement span, should match or beat the accuracy of global reason-then-constrain decoding. It should do so while generating several times fewer tokens. It should also nearly eliminate over-correction of regions where the hypotheses agree.

The new element is the grammar. It is built per utterance from an alignment of the N-best list, in the spirit of the input-dependent grammars of Grammar-Constrained Decoding for Structured NLP Tasks without Finetuning. Inside that grammar, reasoning becomes a legal, position-bound nonterminal rather than a separate phase.

Research Program

Building the grammar. For each utterance we align the N-best hypotheses into a sequence of agreement regions and disagreement spans. We then compile a grammar over a CEGER-style command language. That language comes from Context-Enhanced Granular Edit Representation, which trained PaLM 2 to emit commands such as move forward n, delete, insert, and replace, but decoded them without masking and from the 1-best transcript only.

The grammar enforces the following rules. Pointer arithmetic must stay within the source length. An agreement region can only be consumed by a move-forward command covering exactly that region. Each disagreement span must be consumed by one block of the form: an optional reasoning segment in free text, closed by a delimiter, followed by a replace command. The replacement text is either one of the span's aligned alternatives or an open-vocabulary insertion, marked as such so we can measure how often the model goes beyond the list.

Masking is implemented with Transformers-CFG. Its reported overhead of 1–4 ms per token on Penn Treebank parsing suggests the grammar check will not dominate latency.

Errors outside the disagreement spans. This is the obvious weakness. If every hypothesis contains the same mistake, as with an unfamiliar proper noun, that position is an agreement region and would be locked. HyPoradise reports that a correct token missing from the 1-best appears elsewhere in the N-best only about 34% of the time, so many errors may lie outside disagreement spans.

We therefore add one more grammar option: at any word boundary, the model may emit an "open slot" command that turns the next k words into a deliberation span. A penalty discourages this. Concretely, we add a constant $-\beta$ to the open-slot command's logit, where $\beta \ge 0$ is tuned on development data, so that slots are opened only when the model's preference is strong. Sweeping $\beta$ traces a curve between "never touch agreed text" and "free correction." Plotting that curve is itself a result: it shows how much of generative correction's gain comes from positions where the recognizer was uncertain.

Model and training. We use an open-weight model, because masking needs access to the token scores. To keep comparisons fair we use LLaMA-13B with low-rank adapters (LoRA, small trainable weight matrices added to a frozen model), as in HyPoradise's H2T-LoRA. Gold edit programs come from Levenshtein alignment to the reference transcript. Reasoning segments are distilled: a larger teacher is prompted with the four-step localize, pronounce, propose candidates, select sequence of RLLM-CF, and only rationales whose final edit matches the gold edit are kept. We also run an untrained few-shot variant, to test whether the grammar alone carries the method without fine-tuning.

Analysis. We will break errors down by type: entity, out-of-vocabulary, sound-alike, and grammatical, following the taxonomy in ASR Error Correction with Constrained Decoding on Operation Prediction, which corrected only 32.08% of similar-sound errors and largely failed on entities. We also run a faithfulness check in the spirit of Faithful and Unfaithful Error Recovery in Chain of Thought: does the replacement actually follow from the slot's stated candidate? We test this by corrupting the rationale and measuring whether the edit changes.

Baselines

The naive reference is the recognizer's own 1-best output, together with the oracle error rate of the N-best list. The oracle tells us which gains require synthesis rather than selection.

The cheap classical competitor is N-best rescoring, plus lattice- and N-best-constrained T5 decoding from N-best T5. That system reports a 7.2% WER reduction with 10-best input and up to 12% with lattice constraints on LibriSpeech. We will retrain it on our exact N-best lists.

The strongest accuracy competitor is H2T-LoRA from HyPoradise, a full-sentence rewrite with LLaMA-13B. It reports a 77.1% relative WER reduction on ATIS and 55.1% on WSJ. We will run it with the same base model, the same adapter rank, the same training data, and an equal hyperparameter search budget. For it to count as beaten, we must match its word error rate (WER) within noise and cut generated tokens and wall-clock latency substantially, or lower WER on the noisy and high-variance subsets (CHiME-4, CORAAL), where HyPoradise reports that full fine-tuning overfits. For CHiME-4 the file gives no isolated figure; HyPoradise reports only that generative correction beat rescoring "by a wide margin."

For the prompting regime the competitor is RLLM-CF. It reduced AISHELL-1 character error rate from 5.06% to 4.06% with GPT-4o, while naive prompting rose to 53.10%. We re-implement it on the same open model so that logit access and cost are equal.

Two ablations isolate the key ingredient. The first is grammar-constrained CEGER: the same edit grammar with masking but no reasoning slots. The second is span-agnostic reasoning before constraints: one CRANE-style free rationale for the whole utterance followed by a grammar-constrained edit program, plus a DCCD-style two-pass version. All ablations share the model, data, and tuning budget.

Why not just…?

Why not just put grammar masking on CEGER-style edits and skip reasoning entirely? This is the most serious threat. CEGER already cut USM's LibriSpeech test-clean WER from 6.6% to 2.6% without any reasoning, and masking alone would remove its remaining syntax failures. If the grammar-constrained CEGER ablation matches our method on all error types, then reasoning adds nothing here and we will report that. We expect reasoning to help mainly on entity and sound-alike errors in disagreement spans, so the per-type comparison is the test that decides this question.

Why not just reason once for the whole utterance, then constrain, as CRANE or DCCD do? That may be as accurate. It pays for a full rationale on every utterance, including the many that need no edit, and the free rationale has no anchor that stops it from targeting correct text. The span-agnostic ablation measures both costs directly: tokens per utterance, and edits applied inside agreement regions. If it wins on accuracy at similar cost, the locality claim is false.

Why not just fine-tune a full-sentence rewriter, as HyPoradise does? Full rewrites regenerate every correct word. CEGER found that edit commands cut output from 18–20 to 10–11 tokens and latency from 980 ms to 385 ms. Rewrites also give no structural guarantee against deletions, which Can Generative LLMs Perform ASR Error Correction? observed as truncation. If H2T-LoRA beats us on WER at all operating points of $\beta$, the method's case reduces to latency alone, and we will say so.

Why not just rescore the N-best list? Rescoring cannot exceed the oracle of the list. The open-insertion share of our edits, together with the gains on errors absent from all hypotheses, directly measures what rescoring cannot reach.

Proposed Experiments

We evaluate on the HyPoradise corpus (WSJ, ATIS, CHiME-4, CORAAL, CommonVoice, SwitchBoard), on LibriSpeech test-clean and test-other, and on AISHELL-1 for Mandarin.

The metrics are word or character error rate, generated tokens and latency per utterance, the rate of edits inside agreement regions (over-correction), grammar validity, the fraction of reductions below the N-best oracle, and per-type correction ratios.

Three checks decide the outcome. The first is whether span-anchored reasoning beats the grammar-constrained-CEGER ablation on entity and sound-alike errors. The second is whether it matches the global reason-then-constrain variant at a fraction of the tokens. The third is the $\beta$ sweep: how much WER is lost by locking agreement regions, which shows whether uncertainty-anchored deliberation leaves a meaningful share of errors uncorrected. A negative result on the first check would redirect the project toward masked edit grammars alone, which would still be a useful, cheaper finding.