External Language Model Integration for Neural Speech Recognition

191 concepts139 papers1 proposal

Modern end-to-end (E2E) speech recognition models, such as Neural Transducers and Attention-based Sequence-to-Sequence models, attempt to learn a direct mapping from audio to text. While elegant, these models are "data hungry" and limited by the amount of paired audio-transcription data available. In contrast, text-only data is nearly infinite. The research direction of shallow fusion and its variants focuses on integrating external language models (LMs) trained on massive text corpora into the decoding process of speech recognizers to provide a stronger linguistic prior.

The Linguistic Gap in End-to-End Models

In a traditional hybrid speech system, the acoustic model and language model are separate components. In E2E systems, the model implicitly learns an "internal" language model from the transcriptions in its training set. However, as Chiu et al. note, this internal knowledge often falls short when the model encounters rare words or domain-specific terminology that appeared infrequently in the paired audio data.

The core intuition behind external integration is to treat the speech recognizer as a noisy sensor and the language model as a source of "common sense" about the target language. By combining their scores, the system can resolve phonetic ambiguities—choosing "write" over "right" based on the surrounding sentence structure, even if the audio for both is identical.

The Mechanism of Shallow Fusion

The most prevalent method for this integration is shallow fusion. During the decoding process (typically a beam search), the system maintains a set of candidate transcriptions. At each step, the score for a candidate is calculated as a weighted sum of the log-probabilities from the E2E model and the external LM:

$$Score = \log P_{E2E}(y|x) + \beta \log P_{LM}(y)$$

where $\beta$ is a hyperparameter controlling the LM's influence. Breiner et al. demonstrate that this simple interpolation allows for effective personalization; by fusing a language model trained on a specific user's text data, the ASR system can significantly improve accuracy for that user without needing their voice recordings for training.

Resolving Ambiguity in Low-Resource Settings

External LMs are particularly transformative for low-resource and endangered languages where paired audio data is scarce. For languages like Yorùbá, Ògúnrèmí et al. found that integrating an n-gram language model could reduce word error rates by nearly 40% compared to using an acoustic model in isolation.

Similarly, in spontaneous or dialectal speech, such as Austrian German, Linke et al. show that linguistic constraints from lexicons and n-gram models help compensate for the lack of explicit word-level training data. This suggests that even "outdated" symbolic models provide a necessary anchor for modern neural encoders when data is thin.

Handling Rare Words and Domain Adaptation

One of the "final bosses" of speech recognition is the long-tail problem: correctly transcribing rare proper nouns, technical jargon, or code-switched speech.

  • Large Language Models (LLMs): Hu et al. scaled shallow fusion to 84 languages using massive Mixture-of-Experts (MoE) models, showing that LLMs are exceptionally good at providing context for rare entities that appear infrequently in audio training sets.
  • Contextual Biasing: For dynamic environments (like a user's contact list), Gandhe et al. proposed using likelihood ratios to surgically bias the decoder toward specific entities during the first pass of decoding.
  • Code-Switching: In Mandarin-English mixed speech, Lin argues that external LMs are essential for resolving uncertainty at the exact moment a speaker transitions between languages.

Advanced Integration and Internal Prior Subtraction

A known issue with shallow fusion is that the E2E model already contains an internal linguistic prior. Adding an external LM can lead to "over-counting" certain linguistic patterns. To combat this, Liang et al. and others utilize internal language model subtraction (ILMS), where the estimated probability of the E2E model's internal LM is subtracted before adding the external LM's score.

Furthermore, the field is moving toward more flexible fusion. Hsu et al. introduced Generative Fusion Decoding (GFD), which operates at the byte level. This allows a speech model and a language model to be fused even if they use entirely different tokenizers, enabling real-time error correction and "instruction-aware" recognition where the LM guides the output to follow specific formatting rules.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 139 papers
  2. Version 1 · 2026-10-04 · 257 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_shallow_fusion_66617f0a,
  title = {shallow fusion},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/shallow-fusion/rdr_66617f0a89bf4d4c80293ea5dbeaa90e},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 2, frozen text):

@misc{lacuna_shallow_fusion_66617f0a_v2,
  title = {shallow fusion},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/shallow-fusion/rdr_66617f0a89bf4d4c80293ea5dbeaa90e/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake