Abstract
End-to-end Automatic Speech Recognition (ASR) models trained on large-scale datasets often develop an implicit bias toward the linguistic patterns of their training data. While Internal Language Model Estimation (ILME) has been successfully used to mitigate this bias in autoregressive models like RNN-T, applying it to non-autoregressive Connectionist Temporal Classification (CTC) models remains a challenge due to the difficulty of decoupling acoustic and language components. This paper introduces a novel ILME technique for CTC-based ASR that utilizes iterative input masking to estimate the internal language model (ILM). By masking audio segments and accumulating log-posteriors, the method effectively isolates the model's internal linguistic priors. Experimental results across multiple out-of-domain datasets demonstrate a Word Error Rate (WER) reduction of up to 9.8% and an Out-of-Vocabulary (OOV) F1-score improvement of up to 24.6% compared to standard Shallow Fusion.
Introduction
Modern ASR systems are essentially "sponges" that absorb both acoustic mappings and language semantics from their training corpora. This absorption creates a "source domain bias," where the model favors sequences common in its training data over the actual acoustic evidence in a new domain. For example, a model trained on news broadcasts might incorrectly transcribe a specialized technical term as a common news phrase simply because the latter has a higher internal probability.
To address this, researchers use Internal Language Model Estimation (ILME). The goal is to calculate the model's internal bias and subtract it during the decoding process, allowing an external, domain-specific language model (LM) to exert more influence. However, standard ILME techniques are designed for autoregressive models where the acoustic and language components are modular. CTC-based models, which are non-autoregressive and compute posteriors in parallel, do not allow for such easy decoupling. This work proposes a "black-box" approach to estimate the ILM of CTC models without requiring architectural modifications.
Methodology: Iterative Input Masking
The proposed method, Iterative Input Masking, estimates the internal LM by observing how the model behaves when acoustic information is systematically removed. The process follows five key steps:
- Partitioning: The input audio is divided into $K$ equal, non-overlapping segments.
- Iterative Masking: The model performs multiple forward passes. In each pass, one of the $K$ partitions is "masked" (zeroed out), effectively removing the acoustic signal for that segment.
- Probing for Bias: By analyzing the model's output for these masked segments, the system identifies which timesteps are being driven by internal memory rather than acoustic input. If the model maintains high probability for a specific token even when the corresponding audio is missing, it indicates a strong internal linguistic bias.
- Accumulation: The "text-only" log-posteriors from all masked iterations are accumulated to form a pseudo-likelihood of the Internal Language Model (ILM).
- Subtraction: During final decoding, this ILM score is subtracted from the model's total score. This "mutes" the source-domain bias, allowing the external target-domain LM to guide the transcription more effectively.
Handling the Blank Token
A critical nuance in CTC models is the use of the "blank" token for alignment. The authors found that subtracting the ILM indiscriminately could interfere with the model's basic alignment mechanics. Consequently, the ILM is only subtracted when the model is confident that the current timestep does not represent a blank token, preserving the structural integrity of the CTC output.
Experimental Setup
The ASR model was trained on a massive corpus of over 50,000 hours of English speech. To evaluate the generalization capabilities of the proposed ILME method, the researchers tested it on four distinct out-of-domain datasets:
- LibriSpeech (test-clean/other): Audiobook recordings.
- VoxPopuli: Political speeches from the European Parliament.
- Wikipedia: A custom dataset of spoken named entities generated via Text-to-Speech (TTS).
- Wall Street Journal (WSJ): News dictation.
Results and Discussion
The iterative masking approach consistently outperformed the standard Shallow Fusion baseline across all test sets.
Performance Gains
The method achieved a relative WER reduction of up to 9.8% in scenarios where target-domain text was available for the external LM. More impressively, in "zero-shot" domain adaptation—where no target-domain data was available and the model relied on a source-domain external LM—the method still improved WER by up to 9.3%. This suggests that removing internal bias is beneficial even if the external LM isn't perfectly matched to the target domain, as it prevents the model from being "doubly biased" by both its internal weights and the external source LM.
OOV Recovery
One of the most significant findings was the improvement in Out-of-Vocabulary (OOV) term detection. The method improved the F1-score for OOV terms by up to 24.6%. By stripping away the bias toward common words from the training set, the model becomes significantly more sensitive to rare or domain-specific terms that it would otherwise have ignored in favor of more "familiar" sequences.
Limitations and Future Work
While effective, the primary trade-off of this method is computational cost. Because the approach requires $K$ forward passes (with $K=5$ in these experiments), inference time increases linearly. Although batching masked inputs can mitigate some of this latency, it remains more resource-intensive than standard decoding. Future research may focus on "post-training" techniques, such as training a smaller student model to approximate the ILM in a single pass, thereby eliminating the need for iterative masking during real-time inference.
Conclusion
This work demonstrates that the internal bias of non-autoregressive CTC models can be effectively estimated and mitigated through iterative input masking. By treating the model as a black box and probing its reactions to missing acoustic data, the proposed method significantly enhances the domain-adaptive generalization of ASR systems, particularly for rare and out-of-vocabulary terms.
Summary updated