TL;DR
This paper introduces a novel "Word Relevance Modeling" (WRM) framework that dynamically adapts language models for speech recognition. Unlike previous methods that rely on computationally expensive retrieval procedures, WRM leverages "word-word" co-occurrence information learned during training, making it significantly more efficient and practical for real-time ASR.
What Problem Does This Paper Address?
Automatic Speech Recognition (ASR) systems heavily rely on language models (LMs) to predict the most likely sequence of words. However, standard n-gram LMs are notoriously "brittle" – their performance degrades significantly when the domain or topic of the speech deviates from the text they were trained on. While various adaptation techniques exist, many, like traditional Relevance Modeling (RM), require computationally intensive retrieval processes to find relevant documents for a given search history. This makes them impractical for real-time ASR. The authors aim to bridge this gap by developing a more efficient and effective method for dynamic LM adaptation.
Methods
The core contribution is the Word Relevance Modeling (WRM) framework. Instead of relying on "word-document" relevance as in traditional RM, WRM focuses on "word-word" relevance.
- WRM: During training, WRM models the relationship between a target word and its surrounding words within a defined context window. This "word-word" co-occurrence information is learned offline. During recognition, the WRM model for a search history (sequence of words) can be efficiently composed from these pre-trained models.
- Topic-based Word Relevance Model (TWRM): This is an extension of WRM that further incorporates latent topic variables. Similar to Latent Dirichlet Allocation (LDA) or Probabilistic Latent Semantic Analysis (PLSA), TWRM uses topics to capture deeper semantic relationships. However, it still leverages the efficient WRM structure, avoiding the on-the-fly inference required by PLSA/LDA during recognition.
- Proximity Information: The authors also explore incorporating proximity information, where the influence of history words on the target word decays exponentially with their distance. This is modeled using a decay function.
- Language Model Adaptation: The WRM or TWRM models are combined with a background n-gram language model using a tunable weighting parameter to create an adaptive language model.
Datasets
- MATBN (Mandarin Across Taiwan Broadcast News) corpus: Used for acoustic training and evaluation.
- 25-hour speech data (November 2001 - December 2002) for acoustic bootstrapping.
- 3-hour speech data (collected in 2003) for development (1.5 hours) and testing (1.5 hours).
- Central News Agency (CNA) corpus: A background text corpus of 170 million Chinese characters (2001-2002) used to train the baseline trigram LM.
- MATBN corpus transcripts: A corpus of about one million Chinese characters from 3,643 news stories (excluding the test set) used for training the proposed WRM/TWRM models and other adaptation methods.
Key Insights and Ideas
- Efficiency is Key: The most significant insight is that by shifting from "word-document" relevance (requiring retrieval) to "word-word" relevance learned offline, WRM achieves substantial efficiency gains, making dynamic adaptation feasible for real-time ASR.
- "Word-Word" is Sufficient: The paper demonstrates that modeling local "word-word" co-occurrence patterns is effective for capturing relevance and adapting language models, even without explicit document retrieval.
- Complementarity of Approaches: Combining WRM/TWRM with topic models like LDA shows synergistic benefits, suggesting that different types of relevance information (local word co-occurrence vs. global topic distribution) can complement each other.
- Proximity's Limited Impact: Surprisingly, the proximity-based prior in WRM did not consistently outperform the uniform prior, suggesting that simple distance decay might not be the most effective way to incorporate proximity in this context, warranting further investigation.
Key Results
- WRM vs. Baseline: WRM achieved a relative Character Error Rate (CER) reduction of approximately 3.4% over the baseline trigram model.
- WRM vs. RM: WRM was more attractive than traditional RM due to its significantly higher efficiency, avoiding the time-consuming retrieval step.
- TWRM Performance: TWRM (WRM + topics) further reduced CER to 19.24% compared to WRM's 19.39% (under similar settings), representing a 4.2% relative CER reduction over the baseline.
- TWRM vs. PLSA/LDA/TRM: TWRM demonstrated comparable performance to PLSA and LDA but was much more efficient.
- Combined WRM/TWRM + LDA: Combining WRM with LDA achieved a CER of 19.09%, and TWRM with LDA achieved 19.06%, showing good complementarity.
Limitations and Future Work
- Proximity Information: The authors note that the proximity information did not show a clear gain and requires further investigation.
- Window Size: The optimal window size for WRM varied depending on whether a uniform or proximity-based prior was used, suggesting that this parameter needs careful tuning.
- Future Directions:
- Discriminative training for improved performance.
- Exploring different granularities of semantic context for relevance modeling.
- Leveraging WRM and TWRM for speech retrieval and summarization tasks.
Summary updated