Listening Again Where Hypotheses Disagree: Raw-Waveform Verification for Speech Recognition Reranking

Speech recognizers usually produce a short list of candidate transcripts, and a second model then picks the best one. That second model is typically a text language model that never hears the audio, so it favors fluent sentences even when they don't match what was said. The candidates themselves, though, tell us where the recognizer was unsure. Two competing transcripts tend to agree on most words and differ in a few short stretches, such as "can" versus "can't", "a" versus "the", or a name versus a near-homophone. We propose a reranker that goes back to the raw audio waveform only in those stretches. A small network with trainable filters reads the raw samples there and asks which competing word the audio supports. Its filters are trained by the reranking objective itself, so they learn to separate the specific confusions the first-pass recognizer makes, rather than to recognize speech in general.

This matters because the two existing ways of letting a reranker "hear" are expensive and coarse. One fuses a full self-supervised speech encoder into a text rescorer. The other injects audio features from a log-mel spectrogram model, built on a fixed 25 ms window, into a 7-billion-parameter language model. Both process the whole utterance at the time resolution of the first-pass model. They are blind to the sub-frame detail that often decides a minimal-pair confusion: plosive bursts, stop closures, short unstressed function words. Our bet is that a cheap, targeted, high-resolution second listen captures most of the benefit of acoustic grounding at a fraction of the cost, and adds information the first-pass encoder threw away.

Core Claim

The claim is that the information needed to choose among speech recognition candidates is concentrated in short "disagreement spans," the time regions where the candidates differ. If so, a small verifier that reads raw waveform in these spans should beat both text-only rescoring and whole-utterance multimodal rescoring per unit of compute. The verifier's first layer is a time-convolution filterbank, meaning trainable 1D filters applied directly to audio samples. It is trained end-to-end with a list-level discriminative loss, and we expect it to learn filters that differ measurably from those learned for first-pass recognition. We also expect the gains to be largest exactly where spectrogram front-ends are weakest, namely noisy audio and transient-heavy word confusions.

Three lines of prior work set this up. RescoreBERT supplies the training method. It trains a distilled BERT to rerank the candidate list with losses that push the model's distribution over candidates toward their true word error distribution, but it reads only text. MATE motivates the proposal. It feeds WavLM acoustic features through adapters into a BERT rescorer and reports 4–16% relative word error rate reduction in-domain over text-only rescoring, which is direct evidence that acoustic blindness costs accuracy. However, MATE processes whole utterances with a large, essentially fixed encoder and never sends the ranking error back into the waveform filters. Learning the speech front-end with raw waveform CLDNNs supplies the front-end design: Gammatone-initialized time convolution, max-pooling for phase invariance, and log compression. It also constrains our expectations. Raw waveform alone only matched log-mel features (14.2% WER on noisy Voice Search), and stacking the two improved results only to 13.8%. The raw-waveform benefit is therefore complementary rather than dominant, and it is weakest with scarce data. That constraint is why we confine the waveform model to short spans and a narrow discrimination task, instead of asking it to "learn the ear" from scratch.

Research Program

The first-pass recognizer produces an $N$-best list (its top $N$ candidate transcripts) with word timestamps. CTC models give these through forced alignment. We align all candidates to the top one by edit distance and mark each region where they disagree. Each region is padded by about 50 ms on each side and mapped to a waveform segment. For every disagreement span $d$ and every distinct word sequence $w$ that some candidate proposes there, a verifier $v_\theta(x_d, w)$ scores how well the raw segment $x_d$ supports $w$.

The verifier has three parts. The first is a Gammatone-initialized time-convolution layer with roughly 2.5 ms stride, about four times finer than a 10 ms spectrogram hop, followed by short-window max-pooling and log compression. The second is a few lightweight convolution and attention layers. The third is a cross-attention head over the character or phone sequence of $w$. The candidate's total score is

$$s(Y_i) = a_i + \lambda\, \ell_i + \mu \sum_{d \in D(Y_i)} v_\theta(x_d, w_{i,d}),$$

where $a_i$ is the first-pass score, $\ell_i$ is a text rescorer score (RescoreBERT-style), $D(Y_i)$ is the set of disagreement spans, $w_{i,d}$ is candidate $i$'s words in span $d$, and $\lambda, \mu$ are learned weights. Training minimizes the matching-word-error-distribution loss of RescoreBERT over the whole list:

$$\mathcal{L} = \mathrm{KL}\big(\mathrm{softmax}_i(-\mathrm{WER}(Y_i)/T)\,\|\,\mathrm{softmax}_i(s(Y_i))\big),$$

where $T$ is a temperature. Gradients flow through $v_\theta$ into the time-convolution filters, which is the feedback MATE lacks.

To address data hunger, we pretrain the verifier on cheap synthetic confusions: forced-aligned words paired with phonetically close distractor words from a pronunciation lexicon. We then fine-tune on real $N$-best lists. Because the verifier only sees spans that typically total a small fraction of the utterance, its cost scales with how uncertain the recognizer is, not with how long the utterance is. We will measure that cost directly, as wall-clock time and as floating-point operations per utterance.

Baselines

The naive reference point is the first-pass top candidate with no reranking. Next to it we report the oracle, meaning the best candidate in the list, so every gain can be read as a fraction of what reranking could possibly recover. For a wav2vec 2.0 CTC first pass on LibriSpeech, wav2vec 2.0 reports 1.8% / 3.3% WER on test-clean/test-other with the full 960 hours. We reproduce this number and generate our candidate lists from it.

The text-only competitor is RescoreBERT with its best variant, distillation plus the matching-word-error-distribution loss. The paper reports 6.6% / 3.4% relative WER reduction on LibriSpeech clean/other over a non-discriminative BERT (RescoreBERT). We train it on our exact candidate lists, with the same tuning budget we give our system. Our system includes the same text score $\ell_i$, so any gain is purely acoustic.

The strongest existing competitor is MATE, because it already gives a bidirectional rescorer access to audio. The file contains no reported absolute WER for MATE on LibriSpeech, only 4–16% relative in-domain and 3–7% out-of-domain gains over text-only rescoring. We will therefore reimplement it on the same candidate lists with the same text backbone. We give it a matched number of training steps and, separately, a version whose WavLM encoder is fine-tuned rather than frozen. Beating MATE means equal or lower WER at substantially lower second-pass compute, or lower WER at matched compute, on LibriSpeech test-other and in the noisy conditions.

On AMI (the meeting corpus, individual headset microphone audio), the reference rescoring result is the attention-based rescorer of ISCA, which reduced WER from 29.2% to 23.8%. Our first pass differs from ISCA's, so we treat that figure as context and rerun an attention rescorer on our own lists.

Three ablations isolate the key ingredients. The first keeps the verifier but feeds it log-mel features computed on the same spans, at both 10 ms and 2.5 ms hops. This isolates whether the raw waveform itself matters. The second freezes the Gammatone filters, which isolates whether the rescoring loss improves the front end. The third replaces disagreement crops with whole-utterance input at equal compute, which isolates whether focusing on the spans matters.

Why not just…?

Why not just fine-tune WavLM inside MATE end-to-end? WavLM already reads raw waveform, so this is the most serious threat. It would test "rescoring feedback into a raw-audio encoder" at much larger scale. Its 20 ms latent stride and whole-utterance processing are exactly what we argue are wasteful and coarse, but that remains an empirical claim. We run this baseline. If it matches our verifier at comparable cost on test-other and on noisy speech, our contribution shrinks to an efficiency result, and we will report it as such.

Why not just use log-mel features at a finer hop for the second listen? Sainath et al. found raw waveform and log-mel only reached parity, with a small 3% relative gain from combining them (raw waveform CLDNNs). It is entirely plausible that span-focused verification is the real driver and the input type matters little. The fine-hop log-mel ablation settles this directly. Ma et al. report that raw-waveform front-ends degrade less than log-mel as noise increases, so we predict a raw-waveform advantage mainly at low signal-to-noise ratios. If no such advantage appears, we will say so.

Why not just reuse the first-pass encoder's own frames at the disagreement spans? This costs almost nothing, because the frames are already computed. It is the honest "second look with the same eyes" baseline. We include it as a verifier variant. Our hypothesis is that the first-pass representation already discarded the cue that caused the confusion, so re-attending to it will help less, especially for a spectrogram-based first pass.

Why not just use generative correction like Whispering LLaMA to escape the oracle bound? Whispering LLaMA does break the oracle bound, reaching 13.48% average WER on GigaSpeech. It does so with a 7B language model on log-mel Whisper features, and generators can invent ungrounded text, as Dale et al. document for translation. We target a different operating point, a bounded and cheap second pass, so we compare at the oracle-normalized level rather than claiming to beat it outright.

Proposed Experiments

We will run two first-pass generators: wav2vec 2.0 CTC, which reads raw waveform, and a log-mel Conformer. The log-mel first pass is where complementarity should be largest. Each produces $N \in \{4, 8, 16\}$ candidates on LibriSpeech (training on the 960 h set, testing on clean and other) and AMI headset audio. For robustness we use Speech Robust Bench (code), reporting WER and its normalized degradation across environmental-noise and digital-augmentation perturbations. We report no prior rescorer numbers there, because none are reported on Lacuna for rescorers on this benchmark. The metrics are WER, the fraction of the oracle gap closed, and second-pass compute and latency.

Four checks are decisive. First, the gain over RescoreBERT should be concentrated on errors that are acoustic minimal pairs, which we identify by phone-level edit distance between the competing words in a span, rather than spread uniformly. Second, the raw-waveform verifier should beat its fine-hop log-mel twin by more at low signal-to-noise ratios than on clean speech. Third, the learned filters should diverge from their Gammatone initialization toward higher-frequency, shorter-duration responses after rescoring training, and diverge more than the same layer trained for first-pass recognition. Fourth, the verifier's gain should shrink when the first pass already reads raw waveform. If the first two checks fail, the raw-waveform part of the claim is wrong, and the work reduces to a test of whether disagreement-focused acoustic verification is a cheap substitute for MATE.