On-demand language model interpolation for mobile speech input

Brandon Ballinger, Cyril Allauzen, Alexander Gruenstein, Johan SchalkwykInterspeech 2010

TL;DR

This paper tackles the challenge of improving speech recognition accuracy on mobile devices by dynamically adapting language models to specific user contexts. They introduce an "on-demand" interpolation method that significantly reduces word error rates (WER) by blending multiple pre-trained language models based on the user's current activity, leading to an 11.2% relative improvement.

What Problem Does This Paper Address?

Mobile speech input, whether for search, dictation, or application APIs, faces a significant challenge: a vast diversity of user intents and speaking styles. A single, static language model struggles to accurately capture the nuances of short search queries, long dictated messages, or specialized application commands. This leads to suboptimal recognition accuracy. The authors aim to bridge this gap by creating a system that can intelligently combine information from various language models on the fly, tailoring the recognition process to the specific context of each utterance.

Methods

The core innovation is on-demand language model interpolation. Instead of pre-building a single, massive interpolated model or switching between a few pre-defined models, this approach dynamically computes the best blend of component language models for each incoming utterance.

Here's a breakdown:

  • Component Language Models: They start with several pre-trained language models, each optimized for a different domain (e.g., Google Search queries, SMS messages, emails, Twitter, transcribed speech from Android apps).
  • Contextual Interpolation Weights: For each specific user context (defined by the application and text field, like "SMS - Compose" or "Browser"), they train custom interpolation weights using a development set. This is done via an EM-based algorithm that iteratively adjusts weights to maximize the likelihood of the training data.
  • Compact Interpolated FST: To enable efficient on-demand interpolation, they represent the interpolated model as a "compact interpolated" finite state transducer (FST). This structure allows for dynamic computation of transition weights based on the learned interpolation vector for the current context. This is crucial for performance, as it avoids pre-computing and storing a separate FST for every possible context.
  • Back-off Strategy: To handle contexts with limited training data, they employ a back-off strategy, falling back to application-level or global interpolation weights if field-specific data is insufficient.

Datasets

The paper utilizes several datasets for training and evaluation:

  • Component Language Model Training Data:
    • Anonymized typed SMS and email messages.
  • Publicly available tweets.
  • Anonymized Google search queries (used in Google's search by voice service).
  • Transcribed utterances from the Android Speech API (from users of Google's speech API).
  • Broadcast News transcripts.
  • Development Set: 21,000 utterances used to train the mixture weights for different contexts.
  • Test Set: 42,227 transcribed utterances from various mobile voice tasks, including Google Search by Voice, Google Maps search, and Voice Input into text fields (specifically SMS body, Browser fields, Android Market search, and Gmail body).

Key Insights and Ideas

  • Context is King (for Speech): The diversity of mobile speech input is so vast that a one-size-fits-all language model is inherently suboptimal. Tailoring the model to the specific task (e.g., SMS vs. search) is critical.
  • On-Demand is Efficient: Pre-computing interpolated models for thousands of contexts would be computationally prohibitive. The "on-demand" approach, enabled by the compact FST, makes this feasible.
  • The "Long Tail" Benefits Most: The authors found that the greatest gains in accuracy came from both the most popular and the least popular (long tail) text fields. This suggests the interpolation method is particularly effective at improving recognition for both high-volume, distinct tasks and niche, infrequent ones.
  • Component Influence is Complex: A language model's individual performance doesn't perfectly predict its impact when part of a mixture. Even models with moderate individual accuracy can be crucial for overall system performance when removed.
  • Single-Pass Decoding is Key: Performing interpolation during the initial decoding pass (single-pass) yielded better results than a two-pass system (decode then rescore) in many cases, especially for tasks where a single component LM was already well-suited. This highlights the importance of having the right model from the start.

Key Results

  • Overall Improvement: The on-demand interpolation method achieved an 11.2% relative reduction in Word Error Rate (WER) compared to using a single, statically interpolated language model.
  • Per-Field Interpolation: Training interpolation weights specifically for each text field (e.g., "SMS - Compose") resulted in a 5.1% total relative improvement over a global mixture baseline when using a rescoring approach.
  • Single-Pass Advantage: Moving interpolation to the decoding pass ("Single Pass Mix per Field") further boosted accuracy, achieving the overall 11.2% gain.
  • Component LM Performance: All mixture techniques outperformed the best individual component LM by 20-29% relative.
  • Impact on Popularity: The most significant gains were observed for the most and least frequent text fields.

Limitations and Future Work

  • Real-time Performance: The "Single Pass" system currently operates at approximately 8x realtime, indicating that further optimizations are needed for true real-time deployment.
  • Acoustic Model: The paper focuses on language model interpolation. Improvements in the acoustic model could further enhance overall speech recognition accuracy.
  • Context Definition: While "application and text field" is a reasonable context, exploring more granular or dynamic context definitions could yield further improvements.
  • Future Work: The authors suggest further optimizations for real-time performance and exploring more sophisticated methods for context definition and weight optimization.

Summary updated

BibTeX
@inproceedings{ballinger2010demand,
  author = {Brandon Ballinger and Cyril Allauzen and Alexander Gruenstein and Johan Schalkwyk},
  title = {{On-demand language model interpolation for mobile speech input}},
  booktitle = {Interspeech},
  year = {2010},
}
Figure