When modern end-to-end (E2E) speech recognition models are trained on large audio-transcript datasets, they learn to do two things at once: map acoustic signals to characters or phonemes, and learn the statistical patterns of the language they are transcribing. While this dual capability makes architectures like Recurrent Neural Network Transducers (RNN-Ts) and attention-based encoder-decoders incredibly powerful, it introduces a hidden complication: an implicit, deeply embedded internal language model (ILM) bias.
This internal bias becomes a major liability when we try to adapt a speech model to a new domain—such as moving from conversational speech to medical dictation—or when we try to boost its performance using a large, external language model (ELM) trained on billions of text tokens. Because the primary model's internal linguistic preferences are entangled with its acoustic processing, simply adding an external language model during decoding leads to a mathematical conflict, causing the system to over-count linguistic priors and ignore crucial acoustic evidence.
To solve this, a prominent line of research focuses on Internal Language Model Estimation (ILME): techniques designed to isolate, estimate, and subtract this internal bias during inference, allowing external knowledge to be integrated cleanly.
The Problem: The Hidden Cost of Shallow Fusion
In classical, modular Automatic Speech Recognition (ASR) systems, the separation of powers was explicit. By Bayes' theorem, the most likely sequence of words $Y$ given audio features $X$ is determined by:
$$\arg\max_Y P(Y|X) = \arg\max_Y P(X|Y) P(Y)$$
Here, the acoustic model $P(X|Y)$ and the language model $P(Y)$ are entirely separate. If you want to deploy the system in a legal or medical domain, you simply swap out $P(Y)$ for a domain-specific language model.
Modern E2E models collapse this pipeline, directly modeling the posterior probability $P(Y|X)$ within a single neural network. As surveyed by Takaaki Hori et al. in End-to-End Speech Recognition: A Survey, this unification simplifies training but implicitly bakes a language prior—$P_{\text{ILM}}(Y)$—directly into the model's weights.
When practitioners attempt to incorporate a specialized external language model during decoding—a popular technique known as Shallow Fusion—they typically modify the decoding objective by adding a weighted log-probability of the external model:
$$\log P_{\text{final}}(Y|X) = \log P_{\text{E2E}}(Y|X) + \beta \log P_{\text{ELM}}(Y)$$
The flaw in this approach is rooted in the fact that $P_{\text{E2E}}(Y|X)$ already contains the internal prior $P_{\text{ILM}}(Y)$. Conceptually, the E2E model behaves like $P(X|Y)P_{\text{ILM}}(Y)$. Substituting this back into the Shallow Fusion equation reveals a "double-counting" phenomenon:
$$\log P_{\text{final}}(Y|X) \approx \log P(X|Y) + \log P_{\text{ILM}}(Y) + \beta \log P_{\text{ELM}}(Y)$$
If the training text distribution matches the target domain, this redundancy is merely inefficient. But if they differ, $P_{\text{ILM}}(Y)$ acts as an anchor to the source domain. As noted by Zhong Meng et al. in Internal Language Model Estimation for Domain-Adaptive End-to-End Speech Recognition, this internal bias overpowers the external model, causing the system to reject rare words or specialized terminology in favor of common phrases from the training set.
This issue is not unique to traditional ASR architectures; it even plagues modern, massive systems. For instance, Xun Gong 0005 et al. demonstrated in Contextual Biasing Speech Recognition in Speech-enhanced Large Language Model that when large language models are integrated into speech pipelines, their massive internal text-based knowledge can override actual acoustic evidence, causing the model to hallucinate common words in place of phonetically similar but rare terms.
The Solution: Density Ratio Fusion
To eliminate this redundancy, researchers use an approach called Density Ratio Fusion. The core intuition is simple: if the E2E model double-counts the language prior, we must explicitly estimate and subtract the internal language model score during decoding:
$$\log P_{\text{final}}(Y|X) = \log P_{\text{E2E}}(Y|X) + \beta \log P_{\text{ELM}}(Y) - \gamma \log P_{\text{ILM}}(Y)$$
By subtracting $\log P_{\text{ILM}}(Y)$, we effectively strip away the source-domain linguistic inertia, allowing the external language model to freely steer the predictions based purely on the residual acoustic evidence.
Standard Shallow Fusion:
[Audio X] ──> [ E2E Model (Acoustic + Internal LM) ] ──┬──> [ Combined Score ]
│
[ External Language Model ] ─────────────┘
Density Ratio Fusion:
[Audio X] ──> [ E2E Model (Acoustic + Internal LM) ] ──┬──> [ Combined Score ]
│
[ External Language Model ] ─────────────┼──> (Add)
│
[ Estimated Internal LM ] ───────────────┘──> (Subtract)
The primary challenge, however, is that $P_{\text{ILM}}(Y)$ is not a distinct component; it is an emergent property distributed across the network's parameters. How do you isolate a language model that doesn't technically exist as a separate entity?
How to Estimate the Invisible Prior
Researchers have developed two primary strategies to isolate the internal language model: inference-time estimation and architectural factorization.
1. Inference-Time Estimation (Post-Hoc ILME)
In their seminal work, Zhong Meng et al. proposed a clever, non-destructive way to estimate the ILM without retraining the model. During decoding, they "zero out" or apply a constant vector to the acoustic encoder's outputs before they enter the joint network. Without any acoustic variance to guide it, the decoder is forced to generate token probabilities based entirely on the text history it has already emitted. This isolated text-only probability distribution serves as $P_{\text{ILM}}(Y)$.
While highly effective for transducer and attention models, this zeroing technique does not translate perfectly to Connectionist Temporal Classification (CTC) models, which lack an explicit autoregressive decoder. To circumvent this, Nilaksh Das et al. developed an iterative audio masking technique in Mask The Bias: Improving Domain-Adaptive Generalization of CTC-based ASR with Internal Language Model Estimation. By systematically masking segments of the acoustic input during inference, they force the network to reveal its internal linguistic expectations, allowing CTC-based systems to benefit from Density Ratio fusion.
2. Architectural Factorization (The HAT Approach)
Instead of guessing the internal language model score after the fact, why not design the architecture to compute it explicitly? This is the core philosophy behind the Hybrid Autoregressive Transducer (HAT), introduced by Michael Riley et al. in Hybrid Autoregressive Transducer (hat).
HAT modifies the standard RNN-T architecture by mathematically separating the probability of emitting a "blank" token (which advances time) from the probability of emitting a "label" token (a word or character). By structuring the joint network such that the text-only distribution is factored out cleanly from the transition dynamics, HAT provides an exact, mathematically rigorous internal language model score that can be subtracted without heuristic approximations.
Refining and Scaling the Paradigm
While the basic formulation of Density Ratio Fusion is elegant, making it work reliably in production environments has required several key innovations.
Making Estimation Mathematically Consistent
Even in architectures like HAT that are designed for factorization, the model's training objective doesn't inherently force the internal components to behave like true, isolated probability distributions. To fix this, Sichen Jin et al. introduced an additional Mean Squared Error (MSE) loss term during training in A More Accurate Internal Language Model Score Estimation for the Hybrid Autoregressive Transducer. This loss forces the internal prediction network to strictly satisfy the mathematical assumptions required for ILM estimation, leading to massive reductions in word error rates during domain adaptation.
Internal Language Model Training (ILMT)
For models using post-hoc zeroing techniques, the decoder is never explicitly trained to act as a standalone language model. To address this limitation, Zhong Meng et al. developed Internal Language Model Training (ILMT) in Internal Language Model Training for Domain-Adaptive End-to-End Speech Recognition. During training, the model is optimized using a dual objective: the standard ASR loss on audio-transcript pairs, and a text-only language modeling loss on the transcripts alone with the acoustic features omitted. This forces the decoder to develop a highly modular internal structure, making post-hoc acoustic zeroing significantly more accurate during inference.
Computational Efficiency via Knowledge Distillation
One downside of post-hoc ILME is that evaluating the internal language model at every step of a beam search can double the computational overhead of the decoder. To alleviate this, Zhipeng Chen et al. proposed a Knowledge Distillation (KD) approach in Knowledge Distillation Approach for Efficient Internal Language Model Estimation. They used the complex, zeroed-out outputs of a large ASR decoder as a "teacher" to train a tiny, lightweight external language model. During inference, this compressed student model mimics the internal bias of the primary model at a fraction of the computational cost, reducing internal model parameters by up to 95% without sacrificing accuracy.
Generalization Beyond Speech Recognition
The intuition behind ILME—that end-to-end sequence models implicitly learn target-side language distributions that conflict with external knowledge—extends naturally to other domains.
In Improving Language Model Integration for Neural Machine Translation, Christian Herold et al. demonstrated that Neural Machine Translation (NMT) decoders suffer from the exact same double-counting issue when fused with external language models to improve target-language fluency. By adapting ILME techniques from speech to translation, the authors successfully neutralized the NMT decoder's implicit target-language prior, enabling cleaner integration of monolingual text data.
Similarly, in the realm of multilingual reasoning, He Zhu introduced LayAlign in LayAlign: Enhancing Multilingual Reasoning in Large Language Models via Layer-Wise Adaptive Fusion and Alignment Strategy. This work highlights how direct exposure to source-language text can cause models to rely on internal shortcut biases rather than specialized external representations, reinforcing the broader machine learning insight that managing a model's internal priors is essential for successful multi-source knowledge fusion.
Key Foundational Work
Discriminative Joint Modeling of Lexical Variation and Acoustic Confusion for Automated Narrative Retelling Assessment (2013) — contributed 1 concept elements to this direction.
Using Sub-word-level Information for Confidence Estimation with Conditional Random Field Models (2012) — contributed 1 concept elements to this direction.