Subtracting Internal Language Models for Domain Adaptation

39 concepts27 papers2 proposals

When you train an end-to-end (E2E) sequence-to-sequence model—such as an automatic speech recognition (ASR) system or a neural machine translation engine—the network implicitly learns two distinct skills at the same time. First, it learns to map input signals (like audio features) to output tokens. Second, it learns the statistical patterns and syntax of the language it is trained on.

This implicit linguistic knowledge acts as an internal language model (ILM). While this internal prior helps the model achieve high accuracy on data that matches its training distribution, it becomes a major liability when trying to adapt the model to a new domain.

The Entanglement Problem

Imagine an ASR model trained predominantly on conversational speech. If you want to deploy this model in a specialized medical or legal domain, it will struggle with rare terminology. The traditional way to fix this is shallow fusion: adding an external language model (LM) trained on target-domain text to help guide the decoding process.

However, shallow fusion frequently fails or yields sub-optimal results. As Tüske et al. point out, the E2E model's output is already heavily conditioned on its own internal linguistic distribution. Simply adding the external LM's scores results in a redundant "double counting" of language priors. The source-domain bias buried inside the E2E model ends up competing with and drowning out the target-domain knowledge of the external model.

To successfully integrate new knowledge, we must find a way to isolate and subtract the model's internal language prior.

The Intuition of Density Ratio Fusion

To understand why subtraction works, we can look at the problem through a noisy channel or density ratio framework, as explored by Riley and Rybach.

An E2E model computes a posterior probability $P(Y|X)$, where $X$ is the acoustic input and $Y$ is the text sequence. By Bayes' theorem, this is proportional to the acoustic likelihood multiplied by the source-domain text prior:

$$P_{\text{E2E}}(Y|X) \propto P(X|Y) \cdot P_{\text{internal}}(Y)$$

If we want to adapt this model to a new domain characterized by an external language model $P_{\text{external}}(Y)$, we want to replace the old prior with the new one. Mathematically, this means dividing out the internal prior and multiplying by the external prior:

$$P_{\text{adapted}}(Y|X) \propto \frac{P_{\text{E2E}}(Y|X)}{P_{\text{internal}}(Y)} \cdot P_{\text{external}}(Y)$$

In log-probability space, this density ratio formulation turns into a straightforward linear combination during decoding:

$$\log P_{\text{adapted}}(Y|X) = \log P_{\text{E2E}}(Y|X) - \beta \log P_{\text{internal}}(Y) + \gamma \log P_{\text{external}}(Y)$$

By explicitly subtracting $\log P_{\text{internal}}(Y)$, we neutralize the model's inherent source-domain bias. As Saon et al. demonstrate, this subtraction provides significantly greater robustness than standard shallow fusion, especially when there is a severe mismatch between the training and testing data distributions.

Estimating the Invisible Prior

The core challenge of this approach is that $P_{\text{internal}}(Y)$ is not a standalone component; it is tangled within the shared weights of the neural network. How do we compute a score for a model that doesn't explicitly exist?

The breakthrough solution is Internal Language Model Estimation (ILME), introduced by Meng et al.. The intuition is elegant: if we want to know what the decoder predicts based only on language context, we can simply eliminate the acoustic context during inference. By "zeroing out" the encoder outputs or setting them to a constant vector, the decoder is forced to emit next-token probabilities relying solely on the history of previously generated tokens.

Inference-Time ILME:
[Previous Tokens] ---> [ Decoder / Prediction Net ] ---> P_internal(Y)
                             ^
                             | (Acoustic Vector Zeroed Out)

This core idea has been adapted across various architectures:

  • CTC Models: Because Connectionist Temporal Classification models lack an autoregressive decoder, Das et al. estimate the internal prior by iteratively masking segments of the acoustic input during inference to reveal the model's pure token preferences.
  • Speech Translation: To eliminate systemic gender biases (such as defaulting to masculine pronouns), Fucci et al. substitute a global encoder average into the decoder. This isolates and subtracts the decoder's standalone linguistic bias, allowing external gender-control signals to work effectively.

Moving Subtraction into Training

While inference-time estimation is highly effective, it relies on the assumption that an E2E model behaves like a clean language model when its acoustic inputs are zeroed out. In practice, networks aren't explicitly trained this way, which can lead to imperfect estimations.

To resolve this, researchers developed Internal Language Model Training (ILMT). As proposed by Meng et al., ILMT alters the training phase by adding a secondary objective: the decoder is periodically fed zeroed-out acoustic features and optimized purely on a text language modeling loss. This forces the model to develop a truly modular, standalone internal language model.

Taking this further, Wu et al. integrated this subtraction directly into Minimum Word Error Rate (MWER) training loops. By subtracting the estimated linguistic prior during training, the network receives cleaner gradients that isolate acoustic signals from linguistic biases. To ensure the mathematical assumptions of density ratio fusion hold perfectly true, Jin et al. introduced an auxiliary Mean Squared Error (MSE) loss that explicitly minimizes the discrepancy between the model's joint and marginal logit distributions during training.

Making Subtraction Efficient

Computing an extra forward pass through a large decoder to estimate the internal LM at every single step of a beam search introduces significant computational latency. The community has developed two primary strategies to make subtraction viable for real-time, on-device applications.

1. Knowledge Distillation

Instead of running the heavy ASR decoder twice, Chen et al. proposed distilling the internal language bias of the large decoder into a tiny, lightweight standalone language model during training. At inference time, this ultra-lightweight model predicts $P_{\text{internal}}(Y)$ instantly, reducing the internal model parameter footprint by 95% while maintaining domain-adaptation accuracy.

2. Residual Language Modeling

Rather than performing subtraction dynamically during decoding, Tsunoo et al. pre-calculate the statistical behavior of the internal language model across a target text corpus. They construct a "Residual LM" that directly models the difference between the external target-domain LM and the internal bias. This allows for a single-pass inference process that completely bypasses the need for dynamic internal LM estimation.

Beyond Speech Recognition

The principle of internal language model subtraction is a general solution for any sequence-to-sequence task where conditional models become overly biased by their training text.

In Neural Machine Translation, Herold et al. demonstrated that compensating for the decoder's implicit language model prevents the double-counting of idioms and syntax when fusing external monolingual language models.

The technique has even found a home in the alignment of Large Language Models. When instruction-tuning an LLM, the model can develop a stubborn bias to respond in the dominant language of the instruction dataset (often English), even when prompted in another language. To mitigate this, Zhang et al. introduced Prior Enhanced Decoding. By treating the unaligned base model as the "internal prior" and subtracting its logit distribution from the instruction-tuned model, they filter out over-represented tokens and restore language consistency at inference time without requiring expensive multilingual retraining.

Key Foundational Work

Internal Language Model Estimation for Domain-Adaptive End-to-End Speech Recognition (2020) — contributed 3 concept elements to this direction.
A Hybrid Seq-2-Seq ASR Design for On-Device and Server Applications (2021) — contributed 2 concept elements to this direction.
Minimum Word Error Rate Training with Language Model Fusion for End-to-End Speech Recognition (2021) — contributed 2 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 26 papers
  2. Version 1 · 2026-10-04 · 42 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_internal_language_model_subtraction_44226696,
  title = {internal language model subtraction},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/internal-language-model-subtraction/rdr_442266961fac44e98eaa51f8b5666db0},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_internal_language_model_subtraction_44226696_v2,
  title = {internal language model subtraction},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/internal-language-model-subtraction/rdr_442266961fac44e98eaa51f8b5666db0/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake