Internal Language Model Subtraction for Speech Recognition

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 242 papers64 conceptsfrozen 2026-10-04

Modern end-to-end (E2E) speech recognition models are powerful because they learn to map audio directly to text in a single, unified network. However, this elegance comes with a hidden complication: the model implicitly learns the linguistic patterns of its training data, effectively embedding an "internal language model" (ILM) within its weights.

When we try to adapt these models to a new domain—like moving from general conversation to medical terminology—we usually add an external language model (LM). But because the E2E model already has its own internal ideas about which words follow which, the two models often "double count" linguistic priors. This leads to a bias toward the original training data, causing the system to ignore the new domain-specific knowledge or fail on rare words.

To solve this, researchers have developed a suite of techniques to estimate and subtract this internal bias, allowing for a "clean" integration of external knowledge.

The Density Ratio Intuition

The core logic behind this research direction is based on the density ratio method. In a traditional modular ASR system, we combine an acoustic model $P(x|y)$ with a language model $P(y)$. In an E2E system, the model directly estimates $P(y|x)$. By Bayes' rule, $P(y|x)$ is proportional to $P(x|y)P(y)$.

If we simply add an external language model $P_{ext}(y)$ to the E2E model's output (a technique called "Shallow Fusion"), we are effectively calculating: $$\text{Score} \approx P(x|y) \cdot P_{internal}(y) \cdot P_{ext}(y)$$

The $P_{internal}(y)$ term is the culprit. It represents the model's inherent bias toward its training text. To properly integrate a new domain, we need to "neutralize" this term. As Zhong Meng, Sarangarajan Parthasarathy, and colleagues demonstrated in Internal Language Model Estimation for Domain-Adaptive End-to-End Speech Recognition, subtracting the log-probability of the internal LM during inference allows the external LM to guide the model without being drowned out by the original training distribution.

Estimating the Invisible

The challenge is that the internal language model isn't a separate component; it's baked into the decoder. Researchers use two primary strategies to isolate it:

  1. Zeroing out Acoustics: Zhong Meng et al. proposed estimating the ILM by passing a "zero" or "average" vector from the encoder to the decoder. This forces the decoder to rely solely on its own linguistic history to predict the next token, effectively revealing its internal bias.
  2. Architectural Disentanglement: The Hybrid Autoregressive Transducer (HAT), explored by Liang Lu and Jinyu Li in On Minimum Word Error Rate Training of the Hybrid Autoregressive Transducer, modifies the transducer architecture to mathematically separate the "blank" token probability from the label distribution. This makes the internal LM score explicitly accessible and easy to subtract.

Beyond Domain Adaptation

While the primary goal is often cross-domain accuracy, this mechanism has proven useful for several other persistent problems in AI:

  • Rare Word Recognition: E2E models are notoriously bad at recognizing rare names or technical jargon because their internal LM favors common sequences. By subtracting the ILM, Ruoming Pang and Arun Narayanan showed that models become more receptive to rare wordpieces, significantly improving on-device performance for personal entities like contact names Personalization of CTC-based End-to-End Speech Recognition.
  • Mitigating Social Bias: In speech translation, models often default to masculine inflections learned from biased training data. Dennis Fucci and Marco Gaido found that by subtracting the internal LM bias, they could use gender-specific external LMs to control output inflection, improving feminine form accuracy by up to 31 points.
  • Language Consistency: In multilingual Large Language Models, instruction tuning often creates a bias toward English. Liang Zhang and Qin Jin applied a similar "Prior Enhanced Decoding" to subtract the tuned model's language bias, forcing the model to respond in the user's prompted language.

Moving from Inference to Training

While early work focused on post-hoc subtraction during decoding, newer research integrates this "de-biasing" directly into the training loop. Yu Wu and Liang Lu introduced MWER-ILME, which incorporates the internal LM estimation into the Minimum Word Error Rate training objective Minimum Word Error Rate Training with Language Model Fusion. This ensures the model learns to be "aware" that it will be paired with an external LM later.

Similarly, the Residual Language Model approach by Emiru Tsunoo and Shinji Watanabe shifts the computational burden from inference to training. Instead of estimating the bias on the fly, they directly model the difference between the target domain and the internal bias, resulting in a system that is twice as fast during decoding while maintaining high accuracy.

Papers in this version

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 26 papers
  2. Version 1 · 2026-10-04 · 42 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_internal_language_model_subtraction_44226696,
  title = {internal language model subtraction},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/internal-language-model-subtraction/rdr_442266961fac44e98eaa51f8b5666db0},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_internal_language_model_subtraction_for__44226696_v1,
  title = {Internal Language Model Subtraction for Speech Recognition},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/internal-language-model-subtraction-for-speech-recognition/rdr_442266961fac44e98eaa51f8b5666db0/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}