Evidence-Cited Edits: Checkable Reasoning for LLM Correction of Speech Transcripts

Large language models can fix speech recognition errors after the fact. A recognizer produces a transcript, or a ranked list of its top guesses (an "n-best list"), and the language model rewrites it into something more plausible. The trouble is that these models also rewrite words that were already correct. Asking them to "think step by step" before answering, known as chain-of-thought prompting, helps somewhat. But the reasoning they write is free text that nobody checks, and it may not reflect what actually drove the final answer.

We propose changing the form of that reasoning. The model does not produce a free-form rationale followed by a rewritten sentence. It produces a short list of discrete edits, and each edit must cite its evidence: which alternative hypothesis contains the replacement word and at what aligned position, or, for words found in no hypothesis, what pronunciation the replacement shares with the original. A deterministic checker verifies each citation separately. Any edit that fails is reverted on its own, while the model's other edits stand. In two sentences: the reasoning becomes a set of claims about the recognizer's own evidence that a script can check. That lets us keep the useful corrections and reject hallucinated ones at the level of single edits rather than whole sentences.

Core Claim

Our central claim concerns the training-free and cross-recognizer settings. There, evidence-cited, per-edit-verified correction over n-best lists should achieve lower word error rate than two alternatives. The first is direct n-best regeneration. The second is the strongest existing structured chain-of-thought pipeline, which works only on the single best guess. We expect the gain to come mainly from fewer over-corrections of already-correct words, without giving up the ability to produce words that appear in no hypothesis.

A second, measurable claim is that the cited edits are faithful: removing the cited evidence from the input should change the model's decision. If it does not, the citations are post-hoc decoration, and the method reduces to a filter.

What Exists and Where This Sits

The closest work is Fewer Hallucinations, More Verification (RLLM-CF). It is a training-free pipeline with three stages. First it detects whether an error exists. Then it runs a four-step chain of thought over the single best transcript: locate the error, write its pronunciation, generate candidates, select one. Finally, a script checks that the output follows the template and reverts the whole utterance to the raw transcript otherwise. This paper motivates us in two ways. It shows naive prompting can blow AISHELL-1 character error rate up from 5.06% to 53.10%, and that structure brings it down to 4.06% with GPT-4o. We differ from it in three places:

  • Its verifier checks format, not content.
  • It reverts whole sentences.
  • It ignores the n-best list. The authors also report a residual rise in insertions and deletions.

Pinyin-Guided Chinese Speech Recognition likewise reasons over one best character sequence plus its predicted Pinyin romanization. It notes cases where the pronunciation is right but the model still picks the wrong homophone. LyricWhiz does reason with GPT-4 across multiple Whisper candidates, but in unstructured prose with no verification.

The n-best side supplies the evidence we cite. HyPoradise contributes two things:

  • An open n-best benchmark on which LoRA-tuned LLaMA-13B (LoRA is a light form of fine-tuning that trains small added weight matrices) reports 77.1% relative word error reduction on ATIS and 55.1% on WSJ.
  • The finding that a correct word missing from the top hypothesis appears elsewhere in the list 34% of the time.

Contextualized Speech Recognition shows zero-shot regeneration from 10-best lists takes SQuAD-SRC from 24.2% to 20.9% word error rate. Neither method exposes why a token changed.

ASR Error Correction Using Large Language Models constrains us. Its "n-best constrained" and "n-best closest" decoding already suppress hallucination cheaply, by restricting outputs to, or snapping them back onto, existing hypotheses. Any method we propose must beat that. The 34% statistic cuts both ways: most missing words are not in the list, so pure constraining leaves gains on the table. This is exactly where a pronunciation-checked edit from outside the list matters.

The reasoning literature supplies both method and warnings. Structure Enables Effective Self-Localization of Errors in LLMs shows that splitting reasoning into discrete steps lets a model localize and resample failed steps. It also shows that without external verifiers, models suffer "specificity collapse" and break correct steps. Our deterministic citation checker is that external verifier, and discrete edits are the step units. Faithful and Unfaithful Error Recovery in Chain of Thought shows written reasoning often fails to reflect what the model actually computed. This motivates our counterfactual faithfulness test rather than trusting the rationale.

Research Program

The model receives the n-best list together with a word-level alignment of the hypotheses to the top guess, computed with standard dynamic-programming alignment. It outputs an edit list. Each edit specifies:

  • a span of the top hypothesis;
  • a replacement;
  • an evidence type, which is either "hypothesis $k$ at aligned slot $j$" or "pronunciation match";
  • a one-line contextual justification.

No full rewritten sentence is generated. This also shortens output relative to regeneration.

The checker applies deterministic rules by evidence type:

  • Hypothesis-cited edits must match the cited slot exactly.
  • Pronunciation-cited edits are converted to phones: a grapheme-to-phoneme converter for English, Pinyin for Mandarin. They are accepted only if the phone edit distance to the original span falls under a threshold tuned on development data.
  • Insertions and deletions, the error types RLLM-CF left worse, require hypothesis support. Deletions must be supported by at least one hypothesis lacking the word.

Rejected edits are dropped individually. The model may then be re-queried once, conditioned on the accepted edits as a fixed prefix. This is the discrete-step resampling of Thought-ICS, applied to transcript edits.

We run the same edit format in two regimes. The first uses prompted models: GPT-4o, and an open 7B-class instruction model. The second is a LoRA-tuned variant trained on HyPoradise edit lists derived automatically from hypothesis-reference alignments. This separates the contribution of the output format from that of prompting versus fine-tuning, the unresolved tension between ASR-EC Benchmark and RLLM-CF.

Baselines

Raw recognizer output. The do-nothing reference is the top hypothesis of the recognizer. Every number is reported relative to it, with substitution, insertion and deletion rates separated.

Unanimity gate. A near-free baseline returns the top hypothesis unless the n-best list disagrees there. It then falls back to alignment-based majority voting across hypotheses. It uses no language model, so it tells us how much of any gain is just "trust agreement."

RLLM-CF, rerun under our conditions. This is the strongest structured chain-of-thought competitor. We reproduce its three-stage prompts with the same underlying LLM, on the same recognizer outputs, with matched API budgets. Its published AISHELL-1 figure (CER 4.06% with GPT-4o from a 5.06% baseline) and its LibriSpeech figures (9% relative word error reduction on test-clean and 11.4% on test-other) are reference points. We compare against our rerun, because our recognizer outputs will differ. Beating it means lower error at equal or fewer output tokens on both languages.

Direct n-best regeneration. This is the strongest method without reasoning. We run H2T-style prompting zero-shot, and H2T-LoRA in the fine-tuned regime, following HyPoradise. Each is tried with and without n-best closest decoding from ASR Error Correction Using Large Language Models. On CHiME-4 we additionally report the evolved-prompt result from Evolutionary Prompt Design, word error rate 4.88% from 7.49%. All systems share the same LLM, the same n-best lists (HyPoradise's WavLM and Whisper sources), and the same number of development-set tuning runs.

Ablations isolating the key ingredient. We run four:

  • evidence citations required but the checker disabled;
  • the checker applied with whole-utterance revert, as in RLLM-CF, instead of per-edit revert;
  • hypothesis-citations only, which is effectively constrained correction;
  • free-text chain of thought over the n-best list, LyricWhiz-style.

Why not just…?

Why not just use n-best closest or constrained decoding? This is the most serious threat. It kills hallucination at almost no cost. It cannot, however, produce a word absent from every hypothesis, and HyPoradise's statistic implies that is the majority of missing words. Our hypothesis-citations-only ablation is essentially this method in edit form. We compare it with the full system on the subset of reference words absent from all hypotheses. If the pronunciation-cited edits add nothing there, or add more errors than they fix, the honest conclusion is that constrained decoding suffices, and we would say so.

Why not just fine-tune an n-best-to-transcript model with LoRA? Where in-domain paired data exists, H2T-LoRA is very strong. ASR-EC reports LoRA reaching 7.88% character error rate where prompting inflated error. Its known weakness is transfer: Crossmodal ASR Error Correction shows correctors trained on Whisper outputs degrade on other recognizers. We therefore train on one HyPoradise source and test on the other, and on ASR-EC's two Kaldi systems. We also fine-tune our own edit format, so any advantage there comes from verifiable edits, not from comparing prompting against training.

Why not just add the n-best list to RLLM-CF's prompt? This may close much of the gap, and the free-text n-best ablation tests it directly. Our bet is that without content verification, extra hypotheses increase distraction, much like the "hint confusion" reported with more retrieved entities in Retrieval Augmented Correction of Named Entity Speech Recognition Errors. The comparison will show whether that bet holds.

Why not just gate on agreement and skip the LLM? The unanimity gate is included precisely to measure this. We also report what fraction of utterances our system leaves unedited, since that is a legitimate compute saving to combine with it.

Proposed Experiments

English evaluation uses HyPoradise's CHiME-4, WSJ, ATIS, Common Voice and SwitchBoard n-best lists, plus LibriSpeech test-other. Mandarin uses AISHELL-1 n-best lists that we generate with an open recognizer, and ASR-EC for cross-system transfer. We report five measures:

  • word or character error rate, split by error type;
  • an over-correction rate: the fraction of reference-correct words that the system changes;
  • per-edit precision and recall against reference-derived edits;
  • output tokens and wall-clock latency per utterance;
  • a faithfulness rate.

Faithfulness is measured by intervention. For each accepted hypothesis-cited edit, we delete the cited token from the cited hypothesis and re-query the model. The edit counts as faithful if the model then withdraws it or cites other evidence.

Three results would be decisive:

  1. Over-correction versus RLLM-CF. The full system should cut over-correction substantially relative to the RLLM-CF rerun while matching or improving recall. It should also reverse RLLM-CF's insertion/deletion increase.
  2. Recognizer transfer versus LoRA regeneration. Its drop under recognizer transfer should be smaller than that of H2T-LoRA.
  3. Faithfulness. Its faithfulness rate should be high. A low rate would mean citations are rationalizations, and the gains come from the filter alone. We would then report the method as a verification filter, not as reasoning.

A clean negative result is equally informative. The final revert-granularity ablation tests whether per-edit revert beats whole-utterance revert. If it shows no gain, the value lies entirely in the n-best evidence, not in step-level checking.