Maximum entropy language model adaptation for mobile speech input

Tanel Alumäe, Kaarel KaljurandInterspeech 2012

TL;DR

This paper tackles the challenge of adapting language models (LMs) for mobile speech input, where user vocabulary and context vary wildly across applications. They introduce a novel hierarchical maximum entropy (ME) LM adaptation approach that leverages unsupervised document selection to build context-specific models, achieving significant improvements in speech recognition accuracy.

What Problem Does This Paper Address?

Mobile speech input applications, like dictating SMS messages or searching for locations, face a significant challenge: a single, static language model struggles to accurately predict user vocabulary and intent. The language used in a "grocery list" app is vastly different from that in a "news reader" app. Traditional approaches often rely on large, hand-transcribed datasets or complex interpolation schemes. This paper aims to improve speech recognition accuracy for mobile devices by developing an unsupervised method to adapt LMs to the specific context of different mobile applications, without requiring extensive manual annotation.

Methods

The core of the paper's contribution lies in its hierarchical maximum entropy (ME) language model adaptation strategy. Here's a breakdown:

  • Unsupervised Document Selection: Instead of relying on hand-labeled data, the authors use automatically transcribed speech hypotheses from the target domain. They employ a log-likelihood based criterion to select relevant documents from a large background LM training corpus. The key idea is to find documents that, when interpolated with the background model, improve the perplexity of the target domain data.
  • Hierarchical ME Model Building:
    • Parent Model: A global ME LM is first trained on the union of all documents selected across all target applications. This "parent" model captures commonalities.
  • Domain-Specific Models: For each individual mobile application (target domain), a specific ME LM is built. This adaptation uses the parent model as a prior, encouraging the domain-specific model to stay close to the global model unless there's strong evidence to deviate. This is achieved through L2 regularization during parameter optimization.
  • Interpolation: The adapted ME models are then used to rescore N-best speech recognition hypotheses, interpolating their scores with the background LM.

This approach differs from prior work by using automatic transcripts for document selection and by employing a hierarchical ME modeling strategy, which is found to be crucial for performance gains.

Datasets

  • Training Data: A large corpus of Estonian text data, comprising various sources like newspapers, news portals, scientific publications, magazines, fiction, broadcast conversations, blogs, and conference transcripts. This data totals approximately 299 million tokens.
  • Development and Test Data: Approximately 5000 mobile speech input requests (about 4 hours) collected over one week. 500 of these requests were randomly selected for hand-transcription to create development and test sets. The remaining data was used for unsupervised adaptation.
  • Benchmark: The paper implicitly sets up a benchmark by comparing their method against a baseline background LM and other adaptation techniques like "mix-per-app" and unigram scaling.

Key Insights and Ideas

  • Context is King for Mobile Speech: The paper strongly reinforces the idea that user vocabulary and language style are highly context-dependent in mobile applications. A one-size-fits-all LM is inherently suboptimal.
  • Unsupervised Adaptation is Viable: By cleverly using automatically transcribed hypotheses, the authors demonstrate that effective LM adaptation can be achieved without the costly process of manual transcription for every new domain.
  • Hierarchy Matters: The hierarchical approach to building ME models (global parent model adapted to specific domains) is a key innovation. It allows the model to leverage shared information across related domains while still specializing for individual contexts, leading to better performance than simply pooling all data or adapting independently.
  • Document Selection for Interpolation: The document selection criterion, which prioritizes documents that improve interpolation with the background model, is a nuanced and effective strategy for adaptation.

Key Results

The proposed hierarchical ME LM adaptation method achieved significant improvements:

  • Perplexity Reduction: A 27% reduction in LM perplexity compared to using a single background LM.
  • Word Error Rate (WER) Reduction: Over 10% relative WER reduction in Estonian mobile speech input experiments.
  • Specific Improvements: Table 3 shows a substantial reduction in both perplexity (up to 30%) and WER (up to 13%) compared to the baseline background LM. The adapted ME models outperformed other adaptation techniques like unigram scaling and simple ME models.
  • Application-Specific Gains: The relative improvement for the top 25 most popular applications was 12.6%, and for the "long tail" applications, it was 17%.

Limitations and Future Work

  • Offline Adaptation: The experiments presented were conducted offline. While the authors note that this approach can be ported to online settings, further work would be needed to demonstrate real-time performance.
  • Unsupervised Nature: While a strength, the reliance on automatic transcripts means that errors in the initial transcription can propagate. The accuracy of the multi-pass transcription system is crucial.
  • Future Work: The paper suggests that this approach can be extended to other languages and that further exploration of hierarchical modeling could yield even greater gains. The ability to port this to an online setting is also a natural next step.

Summary updated

BibTeX
@inproceedings{alumae2012maximum,
  author = {Tanel Alumäe and Kaarel Kaljurand},
  title = {{Maximum entropy language model adaptation for mobile speech input}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure