Mask The Bias: Improving Domain-Adaptive Generalization of CTC-based ASR with Internal Language Model Estimation

Nilaksh Das, Monica Sunkara, Sravan Bodapati, Jinglun Cai, Devang Kulshreshtha, Jeff Farris, Katrin Kirchhoff

Abstract

End-to-end Automatic Speech Recognition (ASR) models trained on large-scale datasets often develop an implicit bias toward the linguistic patterns of their training data. While Internal Language Model Estimation (ILME) has been successfully used to mitigate this bias in autoregressive models like RNN-T, applying it to non-autoregressive Connectionist Temporal Classification (CTC) models remains a challenge due to the difficulty of decoupling acoustic and language components. This paper introduces a novel ILME technique for CTC-based ASR that utilizes iterative input masking to estimate the internal language model (ILM). By masking audio segments and accumulating log-posteriors, the method effectively isolates the model's internal linguistic priors. Experimental results across multiple out-of-domain datasets demonstrate a Word Error Rate (WER) reduction of up to 9.8% and an Out-of-Vocabulary (OOV) F1-score improvement of up to 24.6% compared to standard Shallow Fusion.

Introduction

Modern ASR systems are essentially "sponges" that absorb both acoustic mappings and language semantics from their training corpora. This absorption creates a "source domain bias," where the model favors sequences common in its training data over the actual acoustic evidence in a new domain. For example, a model trained on news broadcasts might incorrectly transcribe a specialized technical term as a common news phrase simply because the latter has a higher internal probability.

To address this, researchers use Internal Language Model Estimation (ILME). The goal is to calculate the model's internal bias and subtract it during the decoding process, allowing an external, domain-specific language model (LM) to exert more influence. However, standard ILME techniques are designed for autoregressive models where the acoustic and language components are modular. CTC-based models, which are non-autoregressive and compute posteriors in parallel, do not allow for such easy decoupling. This work proposes a "black-box" approach to estimate the ILM of CTC models without requiring architectural modifications.

Methodology: Iterative Input Masking

The proposed method, Iterative Input Masking, estimates the internal LM by observing how the model behaves when acoustic information is systematically removed. The process follows five key steps:

  1. Partitioning: The input audio is divided into $K$ equal, non-overlapping segments.
  2. Iterative Masking: The model performs multiple forward passes. In each pass, one of the $K$ partitions is "masked" (zeroed out), effectively removing the acoustic signal for that segment.
  3. Probing for Bias: By analyzing the model's output for these masked segments, the system identifies which timesteps are being driven by internal memory rather than acoustic input. If the model maintains high probability for a specific token even when the corresponding audio is missing, it indicates a strong internal linguistic bias.
  4. Accumulation: The "text-only" log-posteriors from all masked iterations are accumulated to form a pseudo-likelihood of the Internal Language Model (ILM).
  5. Subtraction: During final decoding, this ILM score is subtracted from the model's total score. This "mutes" the source-domain bias, allowing the external target-domain LM to guide the transcription more effectively.

Handling the Blank Token

A critical nuance in CTC models is the use of the "blank" token for alignment. The authors found that subtracting the ILM indiscriminately could interfere with the model's basic alignment mechanics. Consequently, the ILM is only subtracted when the model is confident that the current timestep does not represent a blank token, preserving the structural integrity of the CTC output.

Experimental Setup

The ASR model was trained on a massive corpus of over 50,000 hours of English speech. To evaluate the generalization capabilities of the proposed ILME method, the researchers tested it on four distinct out-of-domain datasets:

  • LibriSpeech (test-clean/other): Audiobook recordings.
  • VoxPopuli: Political speeches from the European Parliament.
  • Wikipedia: A custom dataset of spoken named entities generated via Text-to-Speech (TTS).
  • Wall Street Journal (WSJ): News dictation.

Results and Discussion

The iterative masking approach consistently outperformed the standard Shallow Fusion baseline across all test sets.

Performance Gains

The method achieved a relative WER reduction of up to 9.8% in scenarios where target-domain text was available for the external LM. More impressively, in "zero-shot" domain adaptation—where no target-domain data was available and the model relied on a source-domain external LM—the method still improved WER by up to 9.3%. This suggests that removing internal bias is beneficial even if the external LM isn't perfectly matched to the target domain, as it prevents the model from being "doubly biased" by both its internal weights and the external source LM.

OOV Recovery

One of the most significant findings was the improvement in Out-of-Vocabulary (OOV) term detection. The method improved the F1-score for OOV terms by up to 24.6%. By stripping away the bias toward common words from the training set, the model becomes significantly more sensitive to rare or domain-specific terms that it would otherwise have ignored in favor of more "familiar" sequences.

Limitations and Future Work

While effective, the primary trade-off of this method is computational cost. Because the approach requires $K$ forward passes (with $K=5$ in these experiments), inference time increases linearly. Although batching masked inputs can mitigate some of this latency, it remains more resource-intensive than standard decoding. Future research may focus on "post-training" techniques, such as training a smaller student model to approximate the ILM in a single pass, thereby eliminating the need for iterative masking during real-time inference.

Conclusion

This work demonstrates that the internal bias of non-autoregressive CTC models can be effectively estimated and mitigated through iterative input masking. By treating the model as a black box and probing its reactions to missing acoustic data, the proposed method significantly enhances the domain-adaptive generalization of ASR systems, particularly for rare and out-of-vocabulary terms.

Summary updated

Abstract

End-to-end ASR models trained on large amount of data tend to be implicitly biased towards language semantics of the training data. Internal language model estimation (ILME) has been proposed to mitigate this bias for autoregressive models such as attention-based encoder-decoder and RNN-T. Typically, ILME is performed by modularizing the acoustic and language components of the model architecture, and eliminating the acoustic input to perform log-linear interpolation with the text-only posterior. However, for CTC-based ASR, it is not as straightforward to decouple the model into such acoustic and language components, as CTC log-posteriors are computed in a non-autoregressive manner. In this work, we propose a novel ILME technique for CTC-based ASR models. Our method iteratively masks the audio timesteps to estimate a pseudo log-likelihood of the internal LM by accumulating log-posteriors for only the masked timesteps. Extensive evaluation across multiple out-of-domain datasets reveals that the proposed approach improves WER by up to 9.8% and OOV F1-score by up to 24.6% relative to Shallow Fusion, when only text data from target domain is available. In the case of zero-shot domain adaptation, with no access to any target domain data, we demonstrate that removing the source domain bias with ILME can still outperform Shallow Fusion to improve WER by up to 9.3% relative.
BibTeX
@inproceedings{das2023mask,
  author = {Nilaksh Das and Monica Sunkara and Sravan Bodapati and Jinglun Cai and Devang Kulshreshtha and Jeff Farris and Katrin Kirchhoff},
  title = {{Mask The Bias: Improving Domain-Adaptive Generalization of CTC-based ASR with Internal Language Model Estimation}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=xUb0YmHWa5},
}
Figure