Teaching Speech Recognizers Only What the Room Lets Them Hear

Speech recognizers are made robust to far-away microphones by simulating rooms. Clean recordings are convolved with simulated room impulse responses, which are the "echo fingerprint" of a room, and background noise is added. The model is then trained to output the original transcript. This works, but it carries a hidden assumption: that every word in the transcript can still be heard in the reverberant version. In a large, live room the tail of one syllable can bury the next. When that happens, training on the clean transcript asks the model to produce words that are not acoustically present. The only way to comply is to guess from linguistic context. Our hypothesis is that this pressure is a direct, measurable cause of the robustness-accuracy trade-off in speech. Models learn to fabricate plausible words, and that habit leaks into clean audio as insertions and hallucinations.

The opportunity is that, unlike real data, simulated data comes with its own ground truth about audibility. Because we build the reverberant signal ourselves, we can split it exactly into the useful early sound (direct path and first reflections) and the harmful part (late reverberation plus noise). We can then compute, word by word, how much of each word survived. We propose to use this free physical signal to scale the training loss on each transcript token by how audible that word actually is. The model is still trained on fully reverberant input, but it is no longer pushed to "hear" words the simulated room destroyed.

Core Claim

The recipe has three steps. First, compute a per-word recoverability score from the simulator's own decomposition of the signal. Second, use it to down-weight supervision on words that are physically unrecoverable. Third, measure what this does to clean accuracy and real far-field accuracy. We claim this improves real far-field accuracy over uniform multi-condition training, removes most of the clean-audio penalty, and reduces fabricated words. We also claim it beats three simpler options: discarding harsh rooms wholesale, down-weighting examples by model loss, and adding a clean-data refinement phase.

Concretely, write the simulated input as the sum of three parts. The first is the clean signal $x$ convolved with the first 50 ms of the impulse response $h_e$. The second is $x$ convolved with the remainder $h_l$. The third is noise $n$. For word $w$, with frames $T_w$ taken from a forced alignment of the clean utterance (alignment carries over because the direct-path delay is removed, as in pMCT), the recoverability is

$$\rho_w = 10\log_{10}\frac{\sum_{t\in T_w}\|(x*h_e)_t\|^2}{\sum_{t\in T_w}\|(x*h_l + n)_t\|^2}.$$

This is a time-local, word-level version of the clarity index used in room acoustics. Each token's cross-entropy is weighted by $\alpha_w=\sigma((\rho_w-\tau)/s)$, where $\sigma$ is the logistic function, $\tau$ is a threshold in dB and $s$ sets how soft the gate is. Below the threshold, the model is not asked to commit to the word.

Relation to Prior Work

Two papers motivate the claim that augmentation can corrupt the supervision target itself. Adaptive Augmentation-Aware Latent Learning for Robust LiDAR Semantic Segmentation shows that severe simulated weather can make a point-cloud object resemble a different class, so the original label becomes wrong and clear-weather accuracy suffers. Its fix detects such regions with a learned codebook of the model's typical confusions and switches to a latent distillation loss, at 1.7x training cost. We borrow the diagnosis but not the detector. In speech simulation the corruption is computable exactly from the simulator, with no learned prior and negligible overhead. Data Augmentation Revisited proves that heavy augmentation inflates clean training risk and distorts the singular value spectrum of late representations. It fixes this with a final clean "refinement" phase. We use its spectrum analysis as a diagnostic and its schedule as a baseline, but we target the cause, per-token label corruption, rather than undoing it afterward.

The closest speech work is pMCT: Patched Multi-Condition Training for Robust Speech Recognition. It splices clean and reverberant 1.0–1.5 s patches into hybrid utterances and lowers VOiCES word error rate (WER) from 12.86% to 9.89%, with clean WER at 2.41% versus 2.42% for standard multi-condition training. pMCT changes the input and keeps every label, chosen blind to acoustics. We keep fully reverberant inputs and change the label weights according to physics. The two are complementary, and we test their combination. That pMCT's clean WER barely moves under standard multi-condition training is a real constraint on our claim: on LibriSpeech-style training the clean penalty may be small in aggregate WER. Hence the first study below measures where the penalty lives (insertions, hallucinations, foundation-model fine-tuning) before trying to fix it.

Robust Automatic Speech Recognition via WavAugment Guided Phoneme Adversarial Training names the robustness-accuracy trade-off in speech recognition. It addresses it by putting adversarial perturbations on phoneme units, guided by simple augmentations, through costly minimax training. MIXPGD adds an optimal-transport consistency term to adversarial training against digital $l_\infty$ noise. Both treat robustness as a worst-case search problem. We ask a different question: which targets are legitimate under realistic, physically simulated rooms.

On the simulation side, FRA-RIR supplies our fast impulse-response generator. IR-GAN and Spatial LibriSpeech show that more realistic, more diverse rooms help far-field transfer. That is exactly the regime where harsher rooms, and therefore more corrupted targets, enter training. Muting Whisper shows that Whisper's decisions about when to emit or stop emitting tokens are fragile to small acoustic inputs. This motivates measuring fabrication and early stopping rather than WER alone.

Research Program

The work has three parts.

Validate the score. Using a clean-trained recognizer, we test whether $\rho_w$ predicts per-word errors on simulated reverberant speech better than utterance-level room descriptors such as reverberation time ($T_{60}$) or direct-to-reverberant ratio. If $\rho_w$ is not more predictive, the per-word gating idea loses its basis and we stop there.

Measure the penalty mechanistically. We train a Conformer attention encoder-decoder from scratch on LibriSpeech with FRA-RIR rooms and MUSAN noise. We sweep the upper limit of room harshness and record clean WER split into substitutions, deletions and insertions. We also record fabrication rate: words emitted for segments whose $\rho_w$ is far below threshold, and words emitted on non-speech audio. Finally, we compute the singular value spectrum of encoder outputs on clean speech, following Data Augmentation Revisited. We repeat the sweep for fine-tuning Whisper-small, where we expect the clean penalty to be larger because the starting point is already strong on clean speech.

Train with recoverability weighting. We select $\tau$ and $s$ on a held-out simulated set, then evaluate on real far-field data never used for tuning.

Baselines

The naive reference is a model trained on clean LibriSpeech only. It defines the clean accuracy ceiling against which every "clean penalty" is computed, and it shows how badly far-field audio breaks an unadapted model.

The main conventional baseline is uniform multi-condition training: the same FRA-RIR rooms, noise, data, architecture, epochs and tuning budget, with full cross-entropy on every token.

The strongest existing method is pMCT, which we reimplement with its 1.0–1.5 s patching and SpecAugment "Mid" policy on our architecture. Its reported numbers on its own setup are 9.89% WER on VOiCES and 2.41% on clean LibriSpeech, against 12.86% and 2.42% for standard multi-condition training (pMCT). Those figures are not directly comparable to our models, so we compare to our reproduction under identical compute. Our method beats it if it lowers VOiCES WER relative to reproduced pMCT, or if pMCT plus weighting beats pMCT alone, without a worse clean WER and with fewer fabricated words.

We also run the refinement schedule from Data Augmentation Revisited, switching to clean data at a low learning rate for the final epochs. This is the most direct existing fix for clean-risk inflation.

The key ablation keeps everything fixed but replaces the physical gate with a random gate that drops the same fraction of tokens. If random dropping matches physical gating, the gain is just label-dropout regularization, not recoverability.

For reference only, we include WAPAT's reported result of a 6.28% relative average WER reduction over SpeechLM on ESB (WAPAT). We do not re-run it, because its phoneme-unit SpeechLM setup and minimax cost fall outside our fixed compute budget. We say so rather than claim to beat it.

Why not just…?

Why not just cap room harshness, or discard utterances whose room is too reverberant? This is the most serious threat, and it may work nearly as well. The argument against it is that recoverability varies strongly within an utterance. A soft word right after a loud one can be lost in a room where the rest of the sentence is fine. Utterance-level filtering therefore either keeps corrupted words or throws away mostly-useful hard examples, which are exactly the ones that IR-GAN and Spatial LibriSpeech suggest drive far-field transfer. We run utterance filtering at the same total discarded-supervision budget as our gate, and at several thresholds. If it matches us on VOiCES and clean WER, we will report that the per-word mechanism is unnecessary.

Why not just down-weight tokens with high training loss, as in small-loss or confidence-based schemes? Loss cannot tell "hard but learnable" from "physically absent." Down-weighting by loss suppresses exactly the difficult reverberant words the model needs to learn. It also does nothing about confidently fabricated words, which have low loss once the model has learned to guess. We include a loss-quantile gate at matched drop rate. The decisive comparison is fabrication rate on low-$\rho_w$ segments and VOiCES WER.

Why not just add a clean refinement phase at the end? Data Augmentation Revisited shows refinement recovers clean accuracy in vision, but its schedule is a fixed heuristic and was never tested on reverberation. We expect it to restore clean WER at some cost in far-field WER, because it moves the model back toward the clean distribution. Our plot of clean WER against VOiCES WER will show whether refinement and weighting sit on the same trade-off curve or whether weighting shifts the curve. We also test the two combined.

Why not just put a dereverberation front end before a clean-trained model? Improving Character Error Rate is Not Equal to Having Clean Speech shows that enhancement tuned for perceptual quality can hurt downstream recognition through processing artifacts. Its recognition-tuned enhancer instead leaves residual noise and has worse perceptual scores. That makes a front end a weak competitor for recognition accuracy, so we cite it rather than spend compute on it.

Proposed Experiments

Training. LibriSpeech 960 h (100 h for the sweeps), FRA-RIR rooms sampled up to $T_{60}$ of 1.5 s, and MUSAN noise.

Real far-field tests. VOiCES, using the same LibriSpeech-to-VOiCES transfer setup as pMCT, and AMI single distant microphone, with AMI headset recordings as its clean counterpart.

Controlled real-room tests. LibriSpeech test-clean and test-other convolved with measured impulse responses from BUT ReverbDB and the AIR dataset, which are never used in training.

Metrics. WER with its substitution, deletion and insertion breakdown; clean penalty relative to the clean-only model; fabrication rate on low-recoverability segments and on non-speech audio; and, for Whisper, the rate of premature end-of-transcript.

Decisive checks. The proposal succeeds only if four things hold. First, $\rho_w$ predicts per-word errors beyond utterance-level room descriptors. Second, uniform multi-condition training shows a measurable clean penalty concentrated in insertions or fabrication, at least for Whisper fine-tuning. Third, physical gating beats the random gate, the loss gate and utterance filtering at matched supervision budget, on both VOiCES WER and clean penalty. Fourth, the gain survives on real rooms (VOiCES, AMI, BUT ReverbDB) and not only on simulated ones. If the second check fails because there is no clean penalty to remove, the contribution narrows to a far-field gain claim, which must then stand against pMCT alone.