Abstract
Traditional language model interpolation in Automatic Speech Recognition (ASR) relies on global weights that fail to account for the local variability of data source utility. This paper introduces a context-dependent interpolation framework that dynamically adjusts source weights based on preceding word history. By integrating hierarchical smoothing, MAP estimation, and discriminative Minimum Bayes Risk (MBR) training, the proposed method achieves a 6% relative improvement in speech recognition accuracy on large-scale Mandarin broadcast tasks.
What Problem Does This Paper Address?
In ASR, Language Models (LMs) are typically constructed by blending multiple component models trained on diverse data sources, such as news transcripts, web data, or audio captions. Standard practice uses global interpolation weights—fixed values representing the overall utility of a source.
However, a source's usefulness is rarely constant; a model trained on political news may excel at formal jargon but fail on conversational fillers. Global weights ignore this local variability. Additionally, standard LMs are often optimized for perplexity rather than the actual metric of interest: the Character Error Rate (CER) in speech recognition.
Methods
The authors propose a framework for Context-Dependent Interpolation. Instead of a static weight $\lambda_m$ for each source, they utilize a weighting function $\phi_m(h)$ that adapts based on the word history ($h$).
Technical Components
- Hierarchical Smoothing: To mitigate data sparsity, the model uses a tree-structured hierarchy. If a specific tri-gram context lacks sufficient data, the model "backs off" to bi-gram or uni-gram contexts to estimate the weight.
- MAP Estimation: Maximum A-Posteriori (MAP) estimation updates weights during test-time adaptation, using training-data weights as a stable "prior."
- Weight Combination Schemes: The paper evaluates several methods to merge training-data weights with test-data weights. The most effective was Weighted Log-linear Composition and Union, which combines both sets of weights and incorporates new contexts discovered during test supervision.
- Discriminative Training: Beyond perplexity, the authors apply Minimum Bayes Risk (MBR) training to optimize weights specifically for error reduction.
Datasets and Experimental Setup
Experiments were conducted on a state-of-the-art Mandarin broadcast speech transcription task. The training data comprised 1.0 billion words from 10 diverse sources, including audio transcriptions (bcm, bnm), newswire (giga-xin, giga-cna), and web data from TV channels like CCTV and Phoenix. Evaluation was performed on three sets:
- bn06: 3.4 hours of broadcast news.
- bc05: 2.5 hours of broadcast conversation.
- eval06: 1.8-hour GALE 2006 evaluation set.
Key Results
The proposed methods yielded significant improvements across all metrics. Context-dependent weights reduced perplexity by 12% to 23% relative to the global-weight baseline. In terms of accuracy, the best-performing system (incorporating MBR and Log-linear Union adaptation) achieved an absolute CER reduction of 0.5% to 0.6%. This corresponds to a 6% relative improvement on the bn06 task, a substantial gain for a mature ASR system.
Key Insights and Ideas
- Locality of Utility: The "usefulness" of a data source is highly local. Allowing the model to favor different sources depending on specific word histories provides necessary flexibility.
- The Union Advantage: Including "uniquely observed" contexts from test data is critical. Even if a word history was absent in training, its appearance in the test set's initial hypothesis warrants a specific weight.
- Log-Linear Robustness: Log-linear combination proved more robust than standard linear interpolation, as it can effectively "reject" erroneous test-time supervision if it deviates too far from reliable training priors.
Limitations and Future Work
The current approach focuses on discrete word histories, which leads to exponential growth in the state space. While hierarchical back-off structures manage this, they remain rigid. The authors suggest that future research should explore continuous forms of history weighting functions, foreshadowing the neural attention mechanisms that have since become standard in the field. Additionally, the system's performance remains sensitive to the quality of the initial "first-pass" transcription used for adaptation supervision.
Summary updated