Towards empirical dialog-state modeling and its use in language modeling

Nigel G. Ward, Alejandro VegaInterspeech 2012

TL;DR

This paper proposes a novel method for improving language models by incorporating prosodic features that capture nuanced dialog and mental states. By using Principal Component Analysis (PCA) on a rich set of prosodic features over extended time windows, they discover latent dimensions that significantly reduce word prediction errors (perplexity), suggesting a more robust way to model spoken interaction.

What Problem Does This Paper Address?

Speech recognition systems ideally would understand a speaker's mental and dialog state in real-time. While dialog systems often use discrete "dialog states" (like "listening" or "answering"), this is a simplification. The authors argue that a richer, continuous representation of these states, grounded in empirical data, could greatly benefit language modeling. Previous attempts to use prosody for language modeling have shown promise but haven't fully leveraged the complexity of prosodic signals, partly due to feature redundancy and independence assumptions in modeling. This paper aims to overcome these limitations by creating a more comprehensive and empirically derived set of dialog state features.

Methods

The core of the paper's methodology involves:

  • Feature Extraction: They start with 76 prosodic features (speaking rate, volume, pitch height, pitch range) extracted from 6-second windows of speech, covering both past and future context relative to a word. This broad temporal window is a key departure from previous work.
  • Dimensionality Reduction (PCA): To handle the high dimensionality and redundancy of these prosodic features, they apply Principal Component Analysis (PCA). This transforms the correlated features into a set of orthogonal "principal components" (PCs) that capture the most variance in the data.
  • Language Model Integration: Each of the top principal components is used to create a separate language model. They then build a combined language model by integrating the information from the best-performing components. The integration involves multiplying the baseline language model probability by a "scaling factor" derived from the prosodic context, raised to a power k to control its influence.

Datasets

  • Switchboard Corpus: This is the primary dataset used for both training and evaluation. It's a large collection of spontaneous telephone conversations between strangers. The authors used approximately 2 hours of dialog for feature extraction and 90 hours for training the baseline language model. The test set consisted of 45 tracks (about 225 minutes) of spoken dialog.

Key Insights and Ideas

  • Prosody Reflects Deeper States: The discovered principal components often correlate with interpretable aspects of dialog state, cognitive processes (like lexical access), and even speaker attitude (e.g., claiming the floor vs. yielding it). This empirically validates the idea that prosody carries rich, non-lexical information.
  • Broad Temporal Windows Matter: Using prosodic features from 6-second windows, rather than just word-level or short intervals, proved more effective. This suggests that dialog states and mental processes unfold over longer durations.
  • PCA Uncovers Useful Latent Features: PCA successfully decorrelates redundant prosodic features and reveals latent dimensions that are more informative for language modeling than the original, raw features.
  • Not All Variance is Useful: Interestingly, the principal components that explained the most variance in the prosodic data were not always the ones that provided the greatest benefit to the language model. This highlights that statistical significance doesn't always equate to linguistic or pragmatic relevance.
  • Continuous Representation is Key: The approach models dialog states as continuous scales, which is more flexible and potentially more accurate than discrete state models.

Key Results

  • Significant Perplexity Reduction: The best language model, using the top 25 principal components with tuned weights, achieved a 26.8% reduction in perplexity compared to a baseline trigram model.
  • Improved Word Predictions: This final model improved word predictions for 66% of the words in the test set.
  • Interpretable Dimensions: Several of the top components were successfully interpreted as representing meaningful dialog dynamics, such as "claiming the floor vs. yielding the floor," "explaining/excusing oneself vs. blaming someone/something," and "personal experience vs. second-hand opinion."

Limitations and Future Work

  • Subjectivity in Interpretation: While the authors applied discipline, the interpretation of the principal components is inherently subjective, relying on examining extreme cases and correlating with other evidence.
  • Domain Specificity: The models were trained on the Switchboard corpus. Their effectiveness on other dialog domains or languages is not discussed.
  • Future Work: The authors suggest that this empirical, bottom-up approach to modeling dialog state is a promising direction for speech science and engineering, indicating that the goal of modeling speaker mental states in real-time is achievable. They also note that the technique is easy to apply to new corpora and domains.

Summary updated

BibTeX
@inproceedings{ward2012towards,
  author = {Nigel G. Ward and Alejandro Vega},
  title = {{Towards empirical dialog-state modeling and its use in language modeling}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure