Use of contexts in language model interpolation and adaptation

Xunying Liu, Mark J. F. Gales, Philip C. Woodland

Abstract

Traditional language model interpolation in Automatic Speech Recognition (ASR) relies on global weights that fail to account for the local variability of data source utility. This paper introduces a context-dependent interpolation framework that dynamically adjusts source weights based on preceding word history. By integrating hierarchical smoothing, MAP estimation, and discriminative Minimum Bayes Risk (MBR) training, the proposed method achieves a 6% relative improvement in speech recognition accuracy on large-scale Mandarin broadcast tasks.

What Problem Does This Paper Address?

In ASR, Language Models (LMs) are typically constructed by blending multiple component models trained on diverse data sources, such as news transcripts, web data, or audio captions. Standard practice uses global interpolation weights—fixed values representing the overall utility of a source.

However, a source's usefulness is rarely constant; a model trained on political news may excel at formal jargon but fail on conversational fillers. Global weights ignore this local variability. Additionally, standard LMs are often optimized for perplexity rather than the actual metric of interest: the Character Error Rate (CER) in speech recognition.

Methods

The authors propose a framework for Context-Dependent Interpolation. Instead of a static weight $\lambda_m$ for each source, they utilize a weighting function $\phi_m(h)$ that adapts based on the word history ($h$).

Technical Components

  • Hierarchical Smoothing: To mitigate data sparsity, the model uses a tree-structured hierarchy. If a specific tri-gram context lacks sufficient data, the model "backs off" to bi-gram or uni-gram contexts to estimate the weight.
  • MAP Estimation: Maximum A-Posteriori (MAP) estimation updates weights during test-time adaptation, using training-data weights as a stable "prior."
  • Weight Combination Schemes: The paper evaluates several methods to merge training-data weights with test-data weights. The most effective was Weighted Log-linear Composition and Union, which combines both sets of weights and incorporates new contexts discovered during test supervision.
  • Discriminative Training: Beyond perplexity, the authors apply Minimum Bayes Risk (MBR) training to optimize weights specifically for error reduction.
The tree-structured hierarchy used for context-dependent weight estimation and smoothing.

Datasets and Experimental Setup

Experiments were conducted on a state-of-the-art Mandarin broadcast speech transcription task. The training data comprised 1.0 billion words from 10 diverse sources, including audio transcriptions (bcm, bnm), newswire (giga-xin, giga-cna), and web data from TV channels like CCTV and Phoenix. Evaluation was performed on three sets:

  • bn06: 3.4 hours of broadcast news.
  • bc05: 2.5 hours of broadcast conversation.
  • eval06: 1.8-hour GALE 2006 evaluation set.

Key Results

The proposed methods yielded significant improvements across all metrics. Context-dependent weights reduced perplexity by 12% to 23% relative to the global-weight baseline. In terms of accuracy, the best-performing system (incorporating MBR and Log-linear Union adaptation) achieved an absolute CER reduction of 0.5% to 0.6%. This corresponds to a 6% relative improvement on the bn06 task, a substantial gain for a mature ASR system.

Comparison of baseline global weights vs. context-dependent weights (npp) across different evaluation sets.

Key Insights and Ideas

  • Locality of Utility: The "usefulness" of a data source is highly local. Allowing the model to favor different sources depending on specific word histories provides necessary flexibility.
  • The Union Advantage: Including "uniquely observed" contexts from test data is critical. Even if a word history was absent in training, its appearance in the test set's initial hypothesis warrants a specific weight.
  • Log-Linear Robustness: Log-linear combination proved more robust than standard linear interpolation, as it can effectively "reject" erroneous test-time supervision if it deviates too far from reliable training priors.

Limitations and Future Work

The current approach focuses on discrete word histories, which leads to exponential growth in the state space. While hierarchical back-off structures manage this, they remain rigid. The authors suggest that future research should explore continuous forms of history weighting functions, foreshadowing the neural attention mechanisms that have since become standard in the field. Additionally, the system's performance remains sensitive to the quality of the initial "first-pass" transcription used for adaptation supervision.

Summary updated

BibTeX
@inproceedings{liu2009use,
  author = {Xunying Liu and Mark J. F. Gales and Philip C. Woodland},
  title = {{Use of contexts in language model interpolation and adaptation}},
  booktitle = {Interspeech},
  year = {2009},
  doi = {10.21437/interspeech.2009-115},
}
Figure