When a speech recognizer mishears a rare name, a language model can often guess the right word from the surrounding conversation. That same conversation also contains other plausible names that are wrong, and asking the language model to "think harder" or to "check its own work" tends to make it latch onto those wrong names. We propose to separate the two jobs. The language model only proposes candidate corrections, drawing on conversational context. An acoustic model, which scores how well each candidate matches the actual audio, makes every accept-or-reject decision. The claim is that this split changes how extra inference compute behaves. With a language-model judge, more compute should help on clean context and hurt when the context contains distractors. With an acoustic judge, more compute should give gains that keep rising with budget, even under heavy distraction.
This matters because two well-supported findings point in opposite directions. Spending more computation at answer time (more reasoning tokens, more samples, more revision rounds) is the main current route to better model outputs. Yet language models are poor judges of their own answers, and longer reasoning amplifies irrelevant context. Speech correction is a domain where a judge that does not depend on text plausibility comes for free: the recorded signal itself. That makes it a clean test of whether grounding the verifier in raw sensory evidence turns harmful test-time scaling into helpful scaling.
Core Claim
Test-time compute for speech error correction scales the right way when two conditions hold. Candidate generation must be allowed to draw on context, including words the recognizer never put in its candidate list, and candidate selection must be made by an acoustic scorer that ignores that context. If either piece is removed, one of two known failures returns. Removing context generation limits gains to rescoring the recognizer's own top hypotheses. Replacing the acoustic judge with a model-based judge brings back the inverse scaling and self-verification collapse documented in text.
What Prior Work Establishes and Leaves Open
Several papers motivate the proposal by showing that text-only judgment fails. Inverse Scaling in Test-Time Compute builds distractor-laden text tasks, such as misleading math and misleading Python, and shows that longer reasoning lowers accuracy, especially when models choose their own reasoning length. More Thinking, Less Seeing? shows the multimodal version. As vision-language reasoning chains grow, attention drifts from image tokens to the model's own text, and perception accuracy falls; the paper measures this with RH-Bench and an area-under-curve score.
Both papers diagnose the problem in forward generation. Neither tests whether moving the decision to a grounded scorer removes it. Neither studies audio.
Two further papers constrain the design. On the self-verification limitations of large language models shows that GPT-4 self-critique degrades formal-task accuracy, for example Graph Coloring falling from 16% to 2%. It also reports a 97.09% false-negative rate on Mystery Blocksworld, meaning the model rejected nearly all valid solutions. Independent sampling with a sound external verifier did far better (Graph Coloring 44% at $k=25$). GPT-4 Doesn't Know It's Wrong reaches the same conclusion for iterative prompting. We take their recipe directly: the model proposes and an external checker verifies.
Their verifiers, however, are exact symbolic checkers. Speech has no exact checker. Our open question is whether a noisy but context-independent acoustic scorer keeps the benefits of an exact one.
On the speech side, other papers supply the method and the strongest competitors.
- Text-only correction fails. ASR-EC Benchmark finds that zero-shot text-only correction makes Chinese transcripts worse, while adding audio helps.
- Fusing audio into the language model helps, in one pass. Whispering LLaMA feeds speech-encoder features into a frozen language model through small adapters, and shows that removing the audio features collapses correction. It decodes once, with no search and no revision.
- Two-pass decoding is the architectural ancestor. Deliberation Model Based Two-Pass End-to-End Speech Recognition introduced the second pass that reads both the first-pass hypotheses and the audio. Audio-attention discriminative language model for ASR rescoring introduced audio-aware rescoring.
- The proposer-checker split has precedent. Environment Augmented Generation argues that deterministic environment feedback prunes dead reasoning branches and speeds up test-time scaling. We treat the audio as that environment.
The tournament scaling proofs of Chen et al. predict that failure falls toward zero as compute rises, but they assume the judge is better than chance in every comparison. Distractors may break that assumption for a model-based judge while leaving an acoustic judge unaffected. That is the hypothesis we test.
Research Program
The pipeline. A first-pass recognizer produces a transcript and its list of top hypotheses (the n-best list). A proposer, which is an instruction-tuned language model, sees three things: the transcript, the hypotheses, and the conversational or document context. It emits $k$ candidate edits, each covering a word span. Most importantly, it may introduce words absent from every hypothesis.
Each edited transcript is then scored by an acoustic verifier $V(y \mid x)$, the log-likelihood of transcript $y$ given audio $x$, computed only over the edited span and its aligned audio window. To keep the verifier from importing its own text prior, we use a scorer with little or no built-in language model, such as a frame-level alignment model. We also report a decoder-based scorer as a variant. An edit is accepted only if
$$V(y_{\text{edit}} \mid x) - V(y_{\text{orig}} \mid x) > \tau,$$
where $\tau$ is a margin tuned once on a development set. Compute is scaled two ways: by more proposals $k$ (parallel) and by more accept-and-repropose rounds $r$ (sequential).
The diagnostic study. We measure how three judges behave when deciding whether an edit is correct. The judges are a text-only language model, an audio language model reading the audio directly, and the acoustic scorer. For each we report the false-negative rate (rejecting correct edits) and the false-positive rate (accepting wrong ones), as a function of distractor density in the context. As far as the file shows, this is the first such measurement of verification error rates on cross-modal input. It directly asks whether the failure profile from symbolic domains carries over to perception.
The distractor construction. We follow the inverse-scaling design. Into the context we inject near-sound-alike entities: names, product codes, and place names close in pronunciation to the true word. We also inject irrelevant conversational turns, at controlled densities. The true entity appears in context in half the items, so that context is sometimes necessary and sometimes misleading.
Baselines
The do-nothing reference is the raw first-pass transcript. On ASR-EC this reference has a reported CER (character error rate, the Chinese analogue of word error rate) of 12.42% on subset A* ASR-EC Benchmark. Every method must beat it, and the text-only cascades may not.
The text-only cascade sends the transcript plus context to a strong language model. We run it zero-shot, with extended reasoning at several budgets, and with $k$ samples plus self-selection. This is the configuration AHELM found competitive: the GPT-4o-mini Transcribe plus GPT-4o pipeline ranked 6th of 17 systems. ASR-EC reports that zero-shot prompting degrades CER to 20.73%–34.13%. All cascades use the same proposer model as our method, so they differ only in the judge.
Acoustic rescoring of the n-best list uses our verifier but allows only the recognizer's own hypotheses as candidates. This is the most important ablation. It isolates whether gains come from the proposer bringing in new words or merely from better reranking. We run it at matched verifier calls.
The strongest existing method is Whispering LLaMA trained on the same data. On ATIS it reports a WER drop from 13.76% to 1.77%, a 37.66% relative improvement over the n-best oracle Whispering LLaMA. We retrain it on our training split and charge its training compute to its budget. The paper reports that adding audio features increased training time by about 395%. The file gives no Whispering LLaMA number on GigaSpeech that we could quote, so on GigaSpeech we use our own reproduction.
For Chinese, the strongest reported result is multimodal augmentation with Qwen at 6.16% CER on ASR-EC B* (mixed subset). LoRA fine-tuning of Baichuan2 reports 7.88% on the same subset.
The self-correction controls take an audio language model and give it the same proposals and budget. In one version it judges them itself; in another it revises over $r$ rounds. These controls test whether simply having audio in the judge is enough, or whether the judge must also be shielded from context.
Fairness. All methods are compared at matched total inference compute, measured both in floating-point operations and in wall-clock time on the same hardware. All share one hyperparameter-tuning budget on the development split.
What counts as winning. Our method beats a baseline if, at matched compute, it achieves lower entity error rate and lower WER under distractors. It must also show a non-decreasing curve of error against compute where the language-model-judge baselines show a rising one. On clean in-domain ATIS, matching Whispering LLaMA without training adapters would already be a meaningful result. Beating it is not required there. It is required on distractor and out-of-domain entity splits.
Why not just…?
Why not just use a modular cascade with a frontier text model? AHELM shows such cascades are hard to beat on general robustness. But the cascade has no access to the sounds, so it cannot tell a context-supported correct name from a context-supported distractor. ASR-EC's 20.73%–34.13% CER under zero-shot text correction shows how badly that goes. Our distractor sweep is designed to expose this directly. If the cascade holds up under high distractor density, our core claim is wrong.
Why not just rescore the recognizer's n-best list with the acoustic model? This is the real threat. On utterances where the correct word is already in the candidate list, it should do about as well as our method at a fraction of the cost. We therefore report out-of-list recovery rate: the fraction of errors whose correct word was absent from every hypothesis and that our method fixed. The proposal is only worthwhile if that fraction is substantial on rare-entity speech. If it is not, the honest conclusion is that rescoring the existing candidates suffices, and the proposer adds nothing.
Why not just fine-tune an audio-fused corrector like Whispering LLaMA, or apply LoRA? In domain, these methods are very strong (1.77% WER on ATIS; 7.88% and 6.16% CER on ASR-EC B*). But they fit the error distribution and the entity vocabulary of their training data. Our test for this is a held-out split whose context-supplied entities never appear in training. A fine-tuned corrector has no way to introduce such an entity unless it reads the context. Our method can introduce it and then check it against the audio.
Why not just let a reasoning model think longer? The inverse-scaling paper and the attention-drift paper both show that longer reasoning amplifies distractors and weakens grounding in the sensory signal. We nonetheless include extended-reasoning runs at several budgets, so this explanation is tested in our setting rather than assumed.
Proposed Experiments
Data. English experiments use ATIS and GigaSpeech, following GigaSpeech and Whispering LLaMA. Chinese experiments use ASR-EC. Each gets our distractor-augmented context at four densities. A downstream check uses Spoken SQuAD, which reports an F1 drop of nearly 20% from clean to recognized transcripts. There we ask whether corrected transcripts recover question-answering F1.
Metrics. We report WER or CER, entity error rate, verifier false-negative and false-positive rates, and out-of-list recovery rate. To summarize scaling, we report the area under the accuracy-versus-compute curve, in the spirit of RH-AUC.
Decisive checks. Three results would decide the question.
- In the verifier study, the acoustic scorer's false-positive rate on distractor edits should stay roughly flat as distractor density rises, while the language-model judges' rates climb.
- Error-versus-compute curves should separate. For language-model judges, they should fall and then rise under distractors. For the acoustic judge, they should keep falling.
- Out-of-list recovery should be clearly non-zero and should account for most of the method's margin over n-best rescoring.
If the first check fails, sensory grounding does not transfer the benefit of a sound verifier. If the third fails, the simpler rescoring method wins and we would report that.
References
- Inverse Scaling in Test-Time Compute
- More Thinking, Less Seeing? Assessing Amplified Hallucination in Multimodal Reasoning Models
- On the self-verification limitations of large language models on reasoning and planning tasks
- GPT-4 Doesn’t Know It’s Wrong: An Analysis of Iterative Prompting for Reasoning Problems
- ASR-EC Benchmark: Evaluating Large Language Models on Chinese ASR Error Correction
- Whispering LLaMA: A Cross-Modal Generative Error Correction Framework for Speech Recognition
- Deliberation Model Based Two-Pass End-To-End Speech Recognition
- Audio-Attention Discriminative Language Model for ASR Rescoring
- a1: Steep Test-time Scaling Law via Environment Augmented Generation
- Provable Scaling Laws for the Test-Time Compute of Large Language Models
- AHELM: A Holistic Evaluation of Audio-Language Models
- GigaSpeech: An Evolving, Multi-Domain ASR Corpus with 10, 000 Hours of Transcribed Audio
- Spoken SQuAD: A Study of Mitigating the Impact of Speech Recognition Errors on Listening Comprehension