Bayesian Language Model based on Mixture of Segmental Contexts for Spontaneous Utterances with Unexpected Words

Ryu Takeda, Kazunori KomataniCOLING 2016

TL;DR

This paper introduces a Bayesian language model that handles spontaneous speech by considering mixtures of "segmental contexts" – parts of the preceding word sequence. This approach effectively reduces the negative impact of unexpected words like fillers and hesitations, leading to improved word prediction accuracy, especially in noisy speech data.

What Problem Does This Paper Address?

Spontaneous human speech is messy. It's filled with "ums," "ahs," and other hesitations that don't fit neatly into traditional language models, like N-grams. These unexpected words, often treated as "out-of-vocabulary" (OOV) by ASR systems, can be misrecognized as similar-sounding words. This misrecognition then propagates, negatively impacting subsequent word predictions and even word segmentation. Existing N-gram models struggle with this because they rely on contiguous word sequences and are prone to data sparsity issues when encountering these rare or novel speech phenomena.

Methods

The core innovation is a Bayesian language model based on a mixture of segmental contexts. Instead of relying solely on a fixed-length contiguous sequence of previous words (like N-grams), this model considers probabilities derived from various parts of the preceding word sequence.

Here's a breakdown:

  • Segmental Contexts: The model defines a "segmental N-gram" as a contiguous subsequence of words within the preceding context (e.g., p(today | It's, fine, umm, too) could be influenced by p(today | It's, fine), p(today | fine, umm), or p(today | umm, too)).
  • Mixture Model: The probability of the next word is a weighted mixture of probabilities from these different segmental contexts. This allows the model to "ignore" or downplay the influence of unexpected words by relying on other, cleaner segmental contexts.
  • Bayesian Framework: The model uses a Bayesian approach, specifically building upon the Hierarchical Pitman-Yor Language Model (HPYLM) and Variable Order Pitman-Yor Language Model (VPYLM). This is crucial for handling data sparsity and automatically tuning model parameters.
  • Generative Process: The generative process combines the stick-breaking process (for deciding which segments to use) and the VPYLM process (for determining word probabilities within those segments).
  • Parameter Inference: Gibbs sampling is used to infer the model's parameters.

Datasets

The paper uses two main types of data for evaluation:

  1. Artificial Noisy Text:
    • English: "War and Peace" (Project Gutenberg) for training and testing.
  2. Japanese: Corpus of Spontaneous Japanese (CSJ) for training and testing.
  3. These datasets were artificially perturbed by adding random words to simulate noisy speech.
  4. Actual Hesitation Text (Japanese Only):
    • The raw CSJ Japanese transcription text, which includes natural hesitations and fillers, was used. This dataset was split into "clean" (hesitations removed) and "noisy" (hesitations included) test sets.

Key Insights and Ideas

  • Segmental Contexts are Key: The most significant insight is that by looking at parts of the preceding context, rather than just the whole contiguous block, the model can effectively sidestep the negative influence of fillers and hesitations.
  • Bayesian Approach for Robustness: The Bayesian framework is essential for handling the inherent uncertainty and sparsity in speech data, allowing for automatic parameter tuning and better generalization.
  • Mixture for Flexibility: The mixture approach provides flexibility, allowing the model to adapt to different types of noise and context by dynamically weighting the contributions of various segmental contexts.
  • Model-Matched vs. Real-World Performance: The model performs well on artificially noisy data, but its true value is demonstrated on actual hesitation data, where it shows significant improvements over traditional methods.

Key Results

The paper evaluates its model using perplexity (PP), where lower values indicate better performance.

  • Artificial Noisy Data:
    • For both English and Japanese clean text, the proposed model's perplexity was slightly worse than some other methods (like RNN), but comparable to or better than some Bayesian methods like HPYLM and VPYLM.
  • Crucially, for noisy text, the proposed model achieved the lowest perplexity among the Bayesian methods, outperforming HPYLM, VPYLM, WB, and KN. This demonstrates its effectiveness in handling unexpected words.
  • Actual Hesitation Data (Japanese):
    • The perplexity was generally higher for all models due to the inherent difficulty of real speech.
  • The proposed model again showed strong performance, with its perplexity being lower than or comparable to other Bayesian methods, indicating its effectiveness in real-world noisy speech.
  • The paper notes that RNN's performance was sometimes better, potentially due to its handling of closed vocabularies or its inherent ability to model sequences, but the proposed model's advantage lies in its interpretability and ability to handle OOV issues more gracefully.

Limitations and Future Work

  • Ignoring Neighbor Context: A significant limitation acknowledged is that the model completely ignores neighboring context that is not part of the selected segmental context. This can degrade performance on clean text where all preceding words are usually informative. The authors suggest that future work could incorporate this by allowing mixtures of probabilities that include non-contiguous words.
  • Application to Segmentation: While the model is designed for word prediction, its potential for improving word segmentation and lexicon acquisition from speech signals is highlighted as a key area for future research.
  • Raw Symbol Sequences: For true conversational utterances, the authors suggest that handling raw phoneme sequences directly, rather than relying on pre-defined vocabularies or OOV symbols, would be more effective. Integrating their model with OOV-free models is also proposed.
  • Real-World ASR Integration: Evaluating the model's performance when used with text obtained from automatic speech recognition (ASR) systems is a practical next step.

Summary updated

BibTeX
@inproceedings{takeda2016bayesian,
  author = {Ryu Takeda and Kazunori Komatani},
  title = {{Bayesian Language Model based on Mixture of Segmental Contexts for Spontaneous Utterances with Unexpected Words}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2016},
}
Figure