Estimating word-stability during incremental speech recognition

Ian McGraw, Alexander GruensteinInterspeech 2012

TL;DR

This paper tackles the challenge of "instability" in incremental speech recognition, where the system might change its mind about recognized words as more audio comes in. They propose a novel framework to estimate a "stability statistic" for each word, allowing applications to intelligently decide how much of the recognized text to display, balancing real-time feedback with accuracy.

What Problem Does This Paper Address?

Modern speech user interfaces often try to provide real-time feedback by displaying recognized words as the user speaks, rather than waiting for them to finish. This is called "incremental speech recognition." However, a major hurdle is "instability": the system might change its mind about what words were said, especially for the most recently processed audio. This can lead to a confusing user experience where the displayed text contradicts itself. The paper aims to quantify this instability and provide a way for applications to manage it, for example, by only showing text that is likely to remain unchanged.

Methods

The core contribution is a framework for estimating a "stability statistic" for each word in an incremental recognition hypothesis. This statistic predicts the probability that a given prefix of the recognized text will remain unchanged in all future incremental hypotheses.

The authors frame this as a logistic regression problem, where features are extracted from the audio and the recognition process up to the word in question. They explore several feature types:

  • Duration-based features:
    • Right-context (cw): How close the word is to the end of the current incremental result.
  • Age (aw): How long the current prefix has survived on the best path without changing.
  • Lattice-based features:
    • Word posteriors (pw): The probability of a word being correct given the audio seen so far.
  • Search space size (sw): A measure of the complexity of the recognition hypotheses at that point.
  • Word-specific features: Indicator functions for common words or categories of words.

They also consider interaction terms between these features (e.g., age * word category). The "PAWS" feature set combines posteriors, age, words, and search statistics.

Datasets

  • Training Data: Anonymized logs from Google Voice Search and Android Voice Input. The training set contained approximately 444,328 individual words. They used a 4-gram language model with a vocabulary of about 1 million words.
  • Test Data: Separate, anonymized logs from the same two applications, sampled from a different time period.

The paper notes that the Voice Input utterances are primarily short dictations (like SMS), while Voice Search utterances are typically spoken queries.

Key Insights and Ideas

  • Stability vs. Lag Tradeoff: The fundamental insight is that there's an inherent tradeoff between how quickly you display recognized text (lag) and how stable that text is likely to be. Displaying text sooner means a shorter lag but potentially more changes later.
  • "Age" is a Powerful Predictor: The "age" of a prefix (how long it's been stable) emerged as a surprisingly strong predictor of future stability, even more so than how close a word is to the end of the current hypothesis.
  • Word-Specific Features Help: While simple features like "age" are effective, incorporating word-specific information and interactions (like the PAWS set) can further improve stability estimation, especially for Voice Search.
  • Client-Side Control: The proposed stability statistic empowers applications to decide how to use it. A simple threshold can be set to display only prefixes deemed stable enough, giving clients fine-grained control.

Key Results

  • The "age" feature alone provided significant improvements in stability estimation compared to just using "right-context."
  • The PAWS feature set, combining multiple feature types, achieved operating points closest to the "oracle" (ideal, but unachievable) stability for Voice Search.
  • For Voice Input, the gains from more complex feature sets were reduced, possibly due to shorter utterance lengths where longer dependencies are less critical.
  • The paper visualizes the stability/lag tradeoff, showing how increasing the acceptable lag allows for higher estimated stability.

Limitations and Future Work

  • Utterance Length: The authors suggest that features targeting longer dependencies might be needed to improve performance on longer utterances, particularly for Voice Input.
  • Lag Definition: The current analysis of lag is based on decoded frames. Future work could consider the impact of network and computational efficiency on perceived lag.
  • Sub-word Units: The framework could be extended to estimate stability for sub-word units, enabling even finer-grained incremental display.
  • Feature Exploration: The authors acknowledge that many other features could be explored.
  • Monotonicity Constraints: While they analyzed stability under monotonicity constraints (stabilities decreasing left-to-right), further exploration of how to handle spurious deletions around thresholds is mentioned.

Summary updated

BibTeX
@inproceedings{mcgraw2012estimating,
  author = {Ian McGraw and Alexander Gruenstein},
  title = {{Estimating word-stability during incremental speech recognition}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure