Tied-State Mixture Language Model for WFST-based Speech Recognition

Hitoshi Yamamoto, Paul R. Dixon, Shigeki Matsuda, Chiori Hori, Hideki KashiokaInterspeech 2012 · DOI

Abstract

This paper introduces a "tied-state mixture" approach for combining multiple language models (LMs) in speech recognition. By merging specific states within a Weighted Finite-State Transducer (WFST) rather than simply averaging global probabilities, the authors create a model that captures the strengths of diverse domains without the accuracy "dilution" typical of standard linear interpolation. Evaluation on Japanese travel conversation tasks demonstrates that this method effectively mitigates probability degradation, achieving a 1.1% absolute improvement in Word Error Rate (WER) over traditional interpolation techniques.

Introduction: The Problem of Probability Degradation

In Automatic Speech Recognition (ASR), a language model (LM) predicts the next word in a sequence based on historical context. To handle diverse topics, researchers often combine multiple LMs (e.g., a domain-specific "travel" model and a "general conversation" model).

The standard approach for combination is linear interpolation, which takes a weighted average of the probabilities from each component model. However, this method suffers from a significant flaw: if one model is "out-of-domain" for a specific phrase, it will output a very low "back-off" probability. When this low value is averaged with a high-confidence value from an "in-domain" model, the overall score drops. This "probability degradation" makes the ASR system less accurate when dealing with broad, open-domain speech.

The Tied-State Mixture Approach

The authors propose a two-step approach using Weighted Finite-State Transducers (WFSTs), the mathematical backbone of many modern ASR decoders.

  1. Union Operation: Component LMs are first connected in parallel. This "Union WFST" can recognize any sequence supported by the individual models, but the paths remain separate.
  2. State Merging (Tying): This is the core innovation. The system identifies states in different models that represent the same "history" (the same preceding $n$-grams) and merges them into a single tied state.
WFST state merging (tying) across parallel model paths. States 1, 2, and 3 represent common histories where the models are integrated.

By tying states, the model allows for "local" interpolation. If a phrase is common to both models, the probabilities are mixed at that specific state. If a phrase is unique to one model, it retains its original high probability without being dragged down by the other model's lack of knowledge. Furthermore, this topology allows the decoder to "transit" across different component models mid-sentence, choosing the path that provides the best fit.

Local Probability Mixing and Weighting

At each tied state, the model must decide how to combine the outgoing transition probabilities. The authors evaluated two primary strategies:

  • Linear Weighting: A weighted sum of probabilities at the state level.
  • Max Weighting: Taking the maximum probability among the models, which further protects against low back-off values from less relevant models.
Detailed mechanism of local probability mixing at a tied state $h_t$. Diagram (c) shows how inputs from model 1 (a) and model 2 (b) are integrated into a single state.

This mechanism effectively avoids the "back-off penalty." By only mixing probabilities for expressions common to both components, the model prevents a model that doesn't know a word from forcing the system into a low-probability back-off path.

Experimental Evaluation

The researchers evaluated their method on a Japanese travel conversation task using two primary corpora: the BTEC (Basic Travel Expression Corpus, 7.6M words) and VTlog (real-world user logs from the "VoiceTra" app, 1.6M words). The evaluation set consisted of 562 utterances sampled from VoiceTra, containing many out-of-domain expressions.

Key Results

The tied-state mixture model outperformed all standard baselines in terms of Word Error Rate (WER):

Language Model WER (%)
BTEC (Baseline) 25.0
VTlog (Baseline) 23.3
Linear Interpolation 22.9
WFST Union 22.7
Proposed Method (Max) 21.8

The proposed method achieved a 1.1% absolute improvement over standard linear interpolation. The results indicate that the model is particularly robust at recognizing utterances where linear interpolation fails due to low back-off scores.

Conclusion

The tied-state mixture approach offers a more surgical way to combine language models compared to global linear interpolation. By leveraging the structural properties of WFSTs to merge states with common histories, the model preserves high-confidence local predictions while allowing for flexible cross-model transitions. Future work may explore more sophisticated algorithms for state selection and weight optimization to further enhance performance in multi-domain ASR tasks.

Summary updated

BibTeX
@inproceedings{yamamoto2012tied,
  author = {Hitoshi Yamamoto and Paul R. Dixon and Shigeki Matsuda and Chiori Hori and Hideki Kashioka},
  title = {{Tied-State Mixture Language Model for WFST-based Speech Recognition}},
  booktitle = {Interspeech},
  year = {2012},
  doi = {10.21437/interspeech.2012-60},
}
Figure