TL;DR
Researchers at AWS AI Labs have developed a novel way to strip away the "internal bias" of CTC-based speech recognition models. By iteratively masking segments of audio during inference, they can estimate the model's internal language preferences and subtract them, leading to significantly better performance on new, unseen domains—even without any target-domain data.
What Problem Does This Paper Address?
End-to-end Automatic Speech Recognition (ASR) models are like sponges; they don't just learn how sounds map to letters (Acoustic Model), they also soak up the linguistic patterns of their training data (Internal Language Model). While this helps in familiar territory, it creates a "source domain bias." If a model trained on news data hears "The Republican main...", it might stubbornly predict "street" even if the speaker actually said "mainstream," simply because "main street" was common in its training set.
To fix this, researchers use Internal Language Model Estimation (ILME). The idea is to calculate the model's internal bias and subtract it during decoding, allowing an external, domain-specific language model to take the lead. However, standard ILME techniques only work for autoregressive models (like RNN-T or Attention-based Encoder-Decoders) where the acoustic and language components are easy to pull apart. Connectionist Temporal Classification (CTC) models are non-autoregressive and compute everything in parallel, making it historically "inconceivable" to decouple their internal components for ILME.
Methods
The authors propose a "black-box" approach called Iterative Input Masking to estimate the internal LM of a CTC model without changing its architecture.
- Partitioning: They divide the input audio into $K$ equal, non-overlapping segments (partitions).
- Iterative Masking: They perform multiple forward passes through the model. In each pass, one partition of the audio is "masked" (zeroed out).
- Probing for Bias: By comparing the model's output for masked audio vs. original audio, they identify which timesteps are being driven by "internal memory" rather than acoustic input. If the model still assigns high probability to a word even when the corresponding audio is gone, that's a clear sign of internal bias.
- Accumulation: They accumulate these "text-only" log-posteriors across all masked iterations to form a pseudo-likelihood of the Internal Language Model (ILM).
- Subtraction: During final decoding, this ILM score is subtracted from the model's total score, effectively "muting" the source-domain bias and letting the target-domain language model shine.
Datasets
The model was trained on a massive 50,000+ hour corpus of English speech. To test how well it adapts to new domains, the authors evaluated it on four distinct out-of-domain datasets:
- LibriSpeech (test-clean/other): Audiobook readings.
- VoxPopuli: European Parliament event recordings (political speech).
- Wikipedia: A custom dataset of spoken named entities generated via Text-to-Speech (TTS).
- Wall Street Journal (WSJ): News dictation.
Key Insights and Ideas
- Probing via Perturbation: The most clever insight is that you don't need to mathematically decouple a model to understand its bias. By systematically removing information (masking) and watching how the model reacts, you can "trick" a non-autoregressive model into revealing its internal linguistic priors.
- Simplicity Wins: While they experimented with complex masking (like masking at word boundaries), they found that simply dividing the audio into equal-length chunks worked best.
- The Zero-Shot Advantage: Traditionally, ILME is used when you have a language model for the target domain. This paper shows that even if you only have a language model for the source domain, removing the internal bias still helps. It prevents the model from being "doubly biased" by both its internal weights and the external source LM.
- Handling the "Blank" Token: CTC models rely heavily on a special "blank" token. The authors found that they should only subtract the ILM when the model is confident it's not a blank token, preventing the ILME process from interfering with the model's basic alignment mechanics.
Key Results
The proposed masking-based ILME consistently outperformed the standard "Shallow Fusion" baseline:
- Word Error Rate (WER): Achieved a relative reduction of up to 9.8% on unseen target domains.
- Zero-Shot Success: In scenarios with no target-domain data, ILME improved WER by up to 9.3% relative to shallow fusion.
- OOV Recovery: One of the biggest wins was in detecting Out-of-Vocabulary (OOV) terms. The method improved the F1 score for OOV terms by up to 24.6%. This suggests that by removing the bias toward common "source" words, the model is much more likely to correctly identify rare or new words in a target domain.
Limitations and Future Work
- Computational Cost: The primary drawback is inference speed. Because the method requires $K$ forward passes (the authors used $K=5$), it increases the computational load linearly. While this can be partially mitigated by batching the masked inputs together, it remains more expensive than standard decoding.
- Hyperparameter Sensitivity: The method relies on several thresholds (like $\gamma$ for masking impact and $\beta$ for blank filtering) that were tuned on an in-house development set.
- Future Directions: The authors suggest exploring "post-training" techniques where a smaller "student" model is trained to estimate the ILM of the CTC model in a single pass, which would eliminate the need for iterative masking during real-time inference.
Summary updated