Using dialogue-based dynamic language models for improving speech recognition

Juan Manuel Lucas-Cuesta, Fernando Fernández, Javier FerreirosInterspeech 2009

TL;DR

This paper introduces a novel method for dynamically adapting language models (LMs) in spoken dialogue systems by leveraging both semantic concepts and inferred dialogue goals. By interpolating these dynamically generated LMs with a static background LM, the system achieves significant improvements in speech recognition accuracy, particularly when merging both concept and goal information.

What Problem Does This Paper Address?

Automatic Speech Recognition (ASR) systems often struggle in the context of spoken dialogue systems because the language used can shift dynamically based on the user's intent and the ongoing conversation. Traditional static LMs, trained on general or domain-specific data, fail to capture these nuances, leading to higher word error rates (WER). While adapting LMs based on dialogue context is an active research area, this paper specifically addresses the challenge of effectively integrating both the semantic meaning (concepts) and the user's underlying objectives (goals) inferred by the Dialogue Manager (DM) to create more accurate, context-aware LMs.

Methods

The core of the proposed approach lies in dynamically generating and interpolating language models. The system uses a Bayesian Network (BN)-based Dialogue Manager (DM) to infer dialogue goals and concepts from user utterances.

  • Dynamic LM Generation: For each identified dialogue item (concept or goal), a separate LM is estimated using sentences from a database that reference that specific item.
  • Online LM Selection: Based on the DM's confidence in inferred concepts and goals (posterior probabilities), only those items with probabilities exceeding predefined thresholds (Φc for concepts, ΦG for goals) are selected for interpolation.
  • Online LM Interpolation: A weighted interpolation scheme is used to combine a static "background" LM with the dynamically generated item-based LMs. The interpolation weights are derived from the posterior probabilities of the selected dialogue items.
  • Goal-based Interpolation: The LM is interpolated using only goal-based LMs, with weights determined by the posterior probability of each goal.
  • Concept-based Interpolation: Similarly, interpolation is performed using only concept-based LMs, with weights based on concept posterior probabilities.
  • Concept and Goal Merging: The most advanced approach merges both goal-based and concept-based LMs, creating a single dynamic LM where weights are influenced by both types of dialogue items.

Datasets

  • HIFI-MM1 Database: A proprietary database used for training and evaluation. It consists of 100 sentences spoken by 13 different speakers (7 male, 6 female), totaling 1300 sentences related to controlling a Hi-Fi audio system. Each sentence is manually labeled with its concepts and dialogue goals. The database is split into training, validation, and test sets using a k-fold approach.

Key Insights and Ideas

  • Dialogue Context is Crucial: The paper strongly supports the idea that incorporating dialogue-specific information (concepts and goals) significantly improves ASR performance compared to static LMs.
  • Goals Offer More Robust Information: The experiments suggest that using dialogue goals alone for LM adaptation yields better results than using concepts alone. This is attributed to the fact that goals are often inferred from multiple concepts, representing a more integrated and reliable piece of information.
  • Merging is Best: The most significant improvements are achieved when both concept and goal information are merged for LM interpolation, indicating that these two sources of information are complementary.
  • Threshold Tuning Matters: The selection of appropriate thresholds (Φc and ΦG) for including dialogue items in the interpolation process is critical for optimizing performance. These thresholds allow the system to prioritize more confident inferences.

Key Results

  • Baseline WER: The static background LM achieved a Word Error Rate (WER) of 5.33%.
  • Goal-based LM Interpolation: Achieved a WER of 4.73%, representing an 11.24% relative reduction in WER.
  • Concept-based LM Interpolation: Achieved a WER of 4.78%, showing a similar improvement to goal-based interpolation, though slightly less effective.
  • Concept and Goal Merging: Achieved the best performance with a WER of 4.55%, representing a 14.63% relative reduction in WER compared to the baseline.

Limitations and Future Work

  • Database Size and Labeling: The performance of item-based LMs is influenced by the size of the training data for each item. The relatively smaller number of goals compared to concepts, and the fact that the same sentences were used for training all item-based LMs, led to more reliable goal-based models.
  • Rescoring Current Utterance: The current approach focuses on rescoring the current utterance using information from that same utterance. The authors acknowledge that a more natural dialogue system would dynamically modify the LM to better recognize utterances using information from previous dialogue acts.
  • Clustering Approaches: Future work includes exploring clustering approaches for dialogue concepts to create fewer, more robust LMs for interpolation.
  • Real-time Adaptation: The ultimate goal is to develop a system that dynamically modifies LMs based on the entire dialogue history, not just the current utterance, to improve overall dialogue system response.

Summary updated

BibTeX
@inproceedings{lucascuesta2009using,
  author = {Juan Manuel Lucas-Cuesta and Fernando Fernández and Javier Ferreiros},
  title = {{Using dialogue-based dynamic language models for improving speech recognition}},
  booktitle = {Interspeech},
  year = {2009},
}
Figure