New technique to enhance the performance of spoken dialogue systems based on dialogue states-dependent language models and grammatical rules

Ramón López-Cózar, David GriolInterspeech 2010

TL;DR

This paper introduces a novel technique for spoken dialogue systems (SDS) that significantly improves performance by intelligently correcting Automatic Speech Recognition (ASR) errors. By combining dialogue state-dependent language models with grammatical rules, the system achieves better word accuracy, speech understanding, and task completion rates.

What Problem Does This Paper Address?

Spoken dialogue systems rely heavily on accurately transcribing user speech. Automatic Speech Recognition (ASR) systems, while advanced, are prone to errors. These errors can cascade, leading to misunderstandings, incorrect system actions, and ultimately, user frustration. Traditional error correction methods often require vast amounts of training data and struggle with the nuances of conversational context. This paper tackles the challenge of improving SDS performance by developing a more robust and context-aware ASR error correction mechanism that leverages both statistical and linguistic information.

Methods

The core of the proposed technique lies in a two-tiered error correction approach:

  1. Statistical Level Correction: This involves a "pattern matching" step where the recognized utterance is transformed into a sequence of "concepts" (semantic categories like FOOD, DRINK, AMOUNT). This pattern is then compared against a "syntactic-semantic model" (SSM) associated with the current dialogue state. If a match is found, the system proceeds. If not, it searches for similar patterns within the SSM or a broader domain model, using a similarity score based on edit distance. This helps identify the most likely intended semantic structure. A subsequent "pattern alignment" step then attempts to correct individual words within the recognized concepts, using lexical models that capture word confusion probabilities.
  2. Linguistic Level Correction: For errors not caught at the statistical level, or those that still affect semantics, grammatical rules are applied. These rules, defined as syntactic-semantic patterns with associated restrictions, are slid across the corrected utterance. If a rule's pattern is found and its restriction is violated, the system attempts to identify and correct the erroneous word using lexical models.

The key novelty is the use of dialogue state-dependent models (SSM and lexical models specific to the dialogue state) and the intelligent selection of the most appropriate model based on similarity scores and confidence thresholds.

Datasets

  • Saplen system dialogue corpus: A corpus of approximately 5,500 utterances and 2,000 unique words collected from students interacting with the Saplen system in a fast-food domain. This corpus was divided into training and testing sets.
  • Simulated dialogues: The authors created simulated dialogues for training the models and for evaluation purposes. They found that approximately 900 dialogues provided an optimal trade-off for model construction.
  • Scenarios: Two sets of scenarios (ScenariosA and ScenariosB) were used to generate dialogues for evaluation.

Key Insights and Ideas

  • Context is King: The most significant insight is that ASR error correction is far more effective when it considers the current dialogue state and the user's likely intent, rather than just isolated word probabilities.
  • Synergy of Models: Combining statistical models (lexical confusion probabilities) with structured linguistic knowledge (grammatical rules and semantic concepts) provides a powerful mechanism for error recovery.
  • Intelligent Model Selection: The paper highlights the importance of a sophisticated strategy for selecting the correct error correction model. Simply using a general domain model or a state-specific model isn't always optimal; a similarity-based approach that dynamically chooses the best model is crucial.
  • Confirmation Errors are Tricky: The technique proved particularly effective in correcting common confirmation errors (e.g., "yes" vs. "six"), which are often problematic for ASR systems.

Key Results

The proposed technique demonstrated substantial improvements over a baseline ASR system:

  • Word Accuracy (WA): Increased by 8.5% absolute (from 76.12% to 84.62%).
  • Speech Understanding (SU): Increased by 16.54% absolute (from 54.71% to 71.25%).
  • Task Completion (TC): Increased by 44.17% absolute (from 24.51% to 68.32%).

These improvements were achieved by successfully correcting various types of ASR errors, including misrecognized numbers, product names, and confirmations.

Limitations and Future Work

  • Limitations: The authors acknowledge that the technique can fail when an uttered word is substituted by another word that results in a semantically valid, albeit incorrect, utterance (e.g., "two green salads" recognized as "twelve green salads"). This occurs when there's no conceptual conflict.
  • Future Work: The authors propose incorporating additional information sources, such as domain-dependent knowledge, to further enhance error correction. For instance, knowing that ordering "twelve green salads" is unusual could help flag such utterances as potentially misrecognized. They also plan to investigate prompt-dependent similarity thresholds.

Summary updated

BibTeX
@inproceedings{lopezcozar2010new,
  author = {Ramón López-Cózar and David Griol},
  title = {{New technique to enhance the performance of spoken dialogue systems based on dialogue states-dependent language models and grammatical rules}},
  booktitle = {Interspeech},
  year = {2010},
}
Figure