Mask the Bias: Improving Domain-Adaptive Generalization of CTC-Based ASR with Internal Language Model Estimation

Nilaksh Das, Monica Sunkara, Sravan Bodapati, Jinglun Cai, Devang Kulshreshtha, Jeff Farris, Katrin Kirchhoff

TL;DR

Researchers at AWS AI Labs have developed a novel way to strip away the "internal bias" of CTC-based speech recognition models. By iteratively masking segments of audio during inference, they can estimate the model's internal language preferences and subtract them, leading to significantly better performance on new, unseen domains—even without any target-domain data.

What Problem Does This Paper Address?

End-to-end Automatic Speech Recognition (ASR) models are like sponges; they don't just learn how sounds map to letters (Acoustic Model), they also soak up the linguistic patterns of their training data (Internal Language Model). While this helps in familiar territory, it creates a "source domain bias." If a model trained on news data hears "The Republican main...", it might stubbornly predict "street" even if the speaker actually said "mainstream," simply because "main street" was common in its training set.

To fix this, researchers use Internal Language Model Estimation (ILME). The idea is to calculate the model's internal bias and subtract it during decoding, allowing an external, domain-specific language model to take the lead. However, standard ILME techniques only work for autoregressive models (like RNN-T or Attention-based Encoder-Decoders) where the acoustic and language components are easy to pull apart. Connectionist Temporal Classification (CTC) models are non-autoregressive and compute everything in parallel, making it historically "inconceivable" to decouple their internal components for ILME.

Methods

The authors propose a "black-box" approach called Iterative Input Masking to estimate the internal LM of a CTC model without changing its architecture.

  1. Partitioning: They divide the input audio into $K$ equal, non-overlapping segments (partitions).
  2. Iterative Masking: They perform multiple forward passes through the model. In each pass, one partition of the audio is "masked" (zeroed out).
  3. Probing for Bias: By comparing the model's output for masked audio vs. original audio, they identify which timesteps are being driven by "internal memory" rather than acoustic input. If the model still assigns high probability to a word even when the corresponding audio is gone, that's a clear sign of internal bias.
  4. Accumulation: They accumulate these "text-only" log-posteriors across all masked iterations to form a pseudo-likelihood of the Internal Language Model (ILM).
  5. Subtraction: During final decoding, this ILM score is subtracted from the model's total score, effectively "muting" the source-domain bias and letting the target-domain language model shine.

Datasets

The model was trained on a massive 50,000+ hour corpus of English speech. To test how well it adapts to new domains, the authors evaluated it on four distinct out-of-domain datasets:

  • LibriSpeech (test-clean/other): Audiobook readings.
  • VoxPopuli: European Parliament event recordings (political speech).
  • Wikipedia: A custom dataset of spoken named entities generated via Text-to-Speech (TTS).
  • Wall Street Journal (WSJ): News dictation.

Key Insights and Ideas

  • Probing via Perturbation: The most clever insight is that you don't need to mathematically decouple a model to understand its bias. By systematically removing information (masking) and watching how the model reacts, you can "trick" a non-autoregressive model into revealing its internal linguistic priors.
  • Simplicity Wins: While they experimented with complex masking (like masking at word boundaries), they found that simply dividing the audio into equal-length chunks worked best.
  • The Zero-Shot Advantage: Traditionally, ILME is used when you have a language model for the target domain. This paper shows that even if you only have a language model for the source domain, removing the internal bias still helps. It prevents the model from being "doubly biased" by both its internal weights and the external source LM.
  • Handling the "Blank" Token: CTC models rely heavily on a special "blank" token. The authors found that they should only subtract the ILM when the model is confident it's not a blank token, preventing the ILME process from interfering with the model's basic alignment mechanics.

Key Results

The proposed masking-based ILME consistently outperformed the standard "Shallow Fusion" baseline:

  • Word Error Rate (WER): Achieved a relative reduction of up to 9.8% on unseen target domains.
  • Zero-Shot Success: In scenarios with no target-domain data, ILME improved WER by up to 9.3% relative to shallow fusion.
  • OOV Recovery: One of the biggest wins was in detecting Out-of-Vocabulary (OOV) terms. The method improved the F1 score for OOV terms by up to 24.6%. This suggests that by removing the bias toward common "source" words, the model is much more likely to correctly identify rare or new words in a target domain.

Limitations and Future Work

  • Computational Cost: The primary drawback is inference speed. Because the method requires $K$ forward passes (the authors used $K=5$), it increases the computational load linearly. While this can be partially mitigated by batching the masked inputs together, it remains more expensive than standard decoding.
  • Hyperparameter Sensitivity: The method relies on several thresholds (like $\gamma$ for masking impact and $\beta$ for blank filtering) that were tuned on an in-house development set.
  • Future Directions: The authors suggest exploring "post-training" techniques where a smaller "student" model is trained to estimate the ILM of the CTC model in a single pass, which would eliminate the need for iterative masking during real-time inference.

Summary updated

Abstract

End-to-end ASR models trained on large amount of data tend to be implicitly biased towards language semantics of the training data. Internal language model estimation (ILME) has been proposed to mitigate this bias for autoregressive models such as attention-based encoder-decoder and RNN-T. Typically, ILME is performed by modularizing the acoustic and language components of the model architecture, and eliminating the acoustic input to perform log-linear interpolation with the text-only posterior. However, for CTC-based ASR, it is not as straightforward to decouple the model into such acoustic and language components, as CTC log-posteriors are computed in a non-autoregressive manner. In this work, we propose a novel ILME technique for CTC-based ASR models. Our method iteratively masks the audio timesteps to estimate a pseudo log-likelihood of the internal LM by accumulating log-posteriors for only the masked timesteps. Extensive evaluation across multiple out-of-domain datasets reveals that the proposed approach improves WER by up to 9.8% and OOV F1-score by up to 24.6% relative to Shallow Fusion, when only text data from target domain is available. In the case of zero-shot domain adaptation, with no access to any target domain data, we demonstrate that removing the source domain bias with ILME can still outperform Shallow Fusion to improve WER by up to 9.3% relative.
BibTeX
@inproceedings{das2023mask,
  author = {Nilaksh Das and Monica Sunkara and Sravan Bodapati and Jinglun Cai and Devang Kulshreshtha and Jeff Farris and Katrin Kirchhoff},
  title = {{Mask the Bias: Improving Domain-Adaptive Generalization of CTC-Based ASR with Internal Language Model Estimation}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=xUb0YmHWa5},
}

Loading related content...

Figure