Listening Again When Reasoning Stalls: Entropy-Triggered Repair of Speech Transcripts

When a language model reasons step by step over a speech transcript, one misheard word can quietly derail everything that follows. A misheard digit in a word problem or a wrong account number in a banking request is enough. The usual remedies act before reasoning starts. Systems either correct the whole transcript or feed in several alternative transcriptions. These remedies spend effort on every recognition error, including the many that do not matter for the answer, and when a large language model rewrites a transcript without careful constraints it can make things much worse. We propose the opposite order. Let the model reason over its best transcript. Watch its predictive uncertainty (the entropy of its next-token distribution) as it goes. When uncertainty surges at a reasoning step, ask a cheap counterfactual question: would that uncertainty disappear if a low-confidence word in the transcript were replaced by the speech recognizer's runner-up guess? If so, the confusion is acoustic, and we branch only over that word's alternatives. If not, the confusion is logical, and the model keeps thinking.

The opportunity comes from two findings that have not yet been connected. Pulikodan et al. show that word error rate is a poor predictor of whether a downstream model answers correctly from a transcript. Many recognition errors are harmless and a few are fatal. Unveiling the Entropy Dynamics of Chain-of-Thought Reasoning shows that a reasoning model's entropy over a chain of thought has readable structure: an exploratory high-entropy phase followed by an abrupt drop into confidence. If entropy surges can be attributed to specific input spans, the reasoning process itself becomes a detector for the errors that matter.

Core Claim

The reasoning model's own entropy, combined with a span-level counterfactual sensitivity test, can separate "I am stuck because the problem is hard" from "I am stuck because the input is garbled." Using this to trigger targeted, lazy disambiguation of the transcript should match or beat brute-force multi-transcript voting on spoken reasoning accuracy at a fraction of the decoding cost. It should also avoid the hallucination failures of whole-transcript LLM correction.

Concretely, let $x$ be the 1-best transcript and $r_{<t}$ the reasoning generated so far. Let $u_j$ be a transcript span whose speech recognizer confidence falls below a threshold, with alternatives $a_{j,1},\dots,a_{j,K}$ taken from the recognizer's N-best list (its ranked list of top competing transcriptions). When a step-entropy surge is detected at step $t$, we compute for each candidate span

$$\Delta_j = \max_k \, \mathrm{KL}\big(p(\cdot \mid x, r_{<t}) \,\|\, p(\cdot \mid x[u_j \to a_{j,k}], r_{<t})\big),$$

where $p$ is the reasoning model's next-token distribution over the first $m$ tokens of the step and $x[u_j \to a_{j,k}]$ is the transcript with the span swapped. A surge is labeled acoustic when some $\Delta_j$ is large and the swapped context lowers step entropy. Only then do we fork the reasoning into at most $K$ branches, one per alternative for that span. The branches are scored with entropy-weighted voting in the style of Xu et al.. The new elements are the attribution test and the decision to defer transcript repair until the reasoning reveals that a span is load-bearing.

Research Program

The system has three parts, each reusing an existing technique.

Step boundaries come from entropy rather than formatting. Reasoning over conversational transcripts often produces long unbroken blocks. We therefore place step boundaries at low-confidence "decision tokens," following AdaptiveStep, which cut reasoning at tokens in the bottom 2% of model confidence. We then compute length-normalized step entropy as in Making Slow Thinking Faster.

Surges are detected with the relative-increase rule of Jin et al., which flags a boundary when entropy jumps relative to the preceding step rather than crossing an absolute threshold. We also test a CUSUM change-point detector, a classical method that accumulates small deviations until they signal a shift. Xu et al. found that simple per-step thresholds raised false alarms on transient dips, so we expect the same risk here and will measure it.

The attribution test uses recognizer confidence only to propose candidate spans. This is the role recognizer confidence played as a feature in Sudoh et al.. The reasoning model's sensitivity then decides which candidate, if any, gets repaired. Its cost is one forward pass of the prefix per alternative per candidate span, and it is paid only at surges.

We study a cascade setup: a Whisper recognizer that exposes N-best lists and word confidences, feeding DeepSeek-R1-Distill-Qwen-7B and Qwen3-14B, the reasoning models analyzed by Xu et al. We work in the cascade setting for two reasons. AHELM found modular transcribe-then-reason pipelines competitive with native audio models, ranking 6th of 17 systems. Can Speech LLMs Think while Listening? found text reasoning twice as token-efficient as reasoning in speech tokens. Beyond accuracy, we will analyze when the attribution test is right. To do this we align each triggered span against the reference transcript, which tells us whether it really contained a recognition error and whether fixing it changes the answer.

Baselines

The naive reference point is chain-of-thought reasoning over the 1-best transcript with no repair. Its upper bound is the same model on the reference transcript. The gap between the two is the error-propagation budget we are trying to recover.

The strongest competitor on accuracy is multi-transcript self-consistency. We run full chain-of-thought on each of the top-$K$ N-best transcripts and take a majority vote, or an entropy-weighted vote as in Xu et al., which reported beating standard majority self-consistency on clean benchmarks. This baseline is strongest because it is a superset of our branches: any correct alternative we would find, it also sees. Our method beats it only if we reach equal accuracy at clearly lower token cost, or higher accuracy at equal cost. The second outcome could happen because entropy weighting over many irrelevant transcript variants dilutes the vote. We will compare matched-compute curves, varying $K$ for the baseline and the trigger threshold for ours, under the same models, recognizer outputs, and tuning budget on a shared development split. No reported result was found on Lacuna for N-best self-consistency on spoken reasoning benchmarks, so we must run this ourselves.

The strongest existing correction method is RLLM-CF. It is a three-stage pipeline: the model first checks whether an error exists, then corrects it through phonetic candidate reasoning, and finally an external script verifies the output. It reduced AISHELL-1 character error rate from 5.06% to 4.06% with GPT-4o, while naive prompting raised it to 53.10%. We apply it to the transcript before reasoning, using the same reasoning model as corrector, to keep compute fair. Beating it means higher answer accuracy, since its target is transcript fidelity rather than answer correctness, and that is exactly the gap Pulikodan et al. identify.

Two ablations isolate the key ingredient. In the first, we branch at every entropy surge over all low-confidence spans, with no attribution test. In the second, we branch over low-confidence spans whenever the recognizer is unsure, with no reasoning-entropy trigger. If either matches the full method at similar cost, the sensitivity test is not earning its place.

Why not just…?

Why not just run self-consistency over the N-best transcripts? This is the real threat. If recognition errors are dense and nearly all of them matter for the answer, brute-force voting will match us, and lazy repair adds little beyond overhead. The matched-compute comparison settles this directly. Pulikodan et al.'s finding that lower word error rate often fails to lower answer error suggests that most errors are irrelevant, which is the regime where targeted branching should save compute. If the curves coincide, we will report that the attribution signal does not pay for itself.

Why not just mark low-confidence words in the prompt and let the model handle it? Text models do not know the acoustic confusion set unless they are told. The constrained pipeline of Yang et al. shows that loosely guided models over-edit badly. A marked-prompt variant, which lists low-confidence words together with their N-best alternatives inline, is cheap to run, and we include it as an extra baseline. It may do well on short utterances and degrade on longer reasoning, where alternatives clutter context early, before the model knows which ones matter.

Why not just correct the transcript first, with RLLM-CF? Correction optimizes the wrong objective. It spends iterations on every error, and it must decide on ambiguous spans without knowing what the downstream question needs. A misheard digit can be impossible to resolve from the sentence alone but obvious once the arithmetic fails to work out. Our comparison measures whether reasoning-time evidence resolves cases that pre-correction gets wrong.

Why not just use a native audio model and skip transcripts? MMSU reports that more than half of native audio model failures are perceptual, with the best models near 60% against 89.72% for humans. SpeechGym reports perceptual errors in 32% of audio-agent interactions versus 2% for text. The perception problem does not disappear end to end; it becomes harder to inspect.

Proposed Experiments and Decisive Checks

The main benchmark is the spoken GSM8K and ARC portions of SRQA from Can Speech LLMs Think while Listening?. We transcribe them with Whisper under clean conditions and under added noise at several signal-to-noise levels. Spoken math is well suited because single misheard numbers are both common and answer-critical. That paper reports a 2.4x average accuracy gain for its streaming reasoning system, but this is a different setup, so we treat it only as context. The second benchmark is τ-Voice with the banking domain from SpeechGym, where numeric slot errors cascade into tool failures. There, the GRPO-trained Qwen3-Omni-30B reached a pass@1 of 53% (up from 24%), with a 62% recovery rate after mishearing. We apply our method to a cascade agent and compare recovery rate directly. Metrics are answer accuracy, answer error rate in the sense of Pulikodan et al., reasoning tokens per problem, number of branches, and the precision and recall of attribution against reference-aligned error locations.

Three checks would decide whether the idea holds. First, on reference transcripts, the attribution test should almost never fire, even on hard problems with long uncertainty regions. If it fires often, it is confusing logical difficulty with acoustic confusion and the core claim fails. Second, in controlled corruption, we inject a recognizer substitution into either an answer-relevant span (a quantity) or an irrelevant span (a name in a word problem). Branching should occur far more often for the relevant span. Third, on the matched-compute frontier against N-best self-consistency, we must sit strictly above the baseline curve somewhere. As a secondary check on Audio-CoT's finding that chain of thought hurts on hard audio tasks, we will test whether the accuracy loss on the hardest problem tier shrinks when perceptual errors are intercepted mid-chain.