TL;DR
This paper introduces a novel "frame-correction" technique to improve how spoken dialogue systems (SDSs) understand user input, even when automatic speech recognition (ASR) makes errors. By learning from past misunderstandings, the system can intelligently replace incorrectly generated semantic frames with more accurate ones, significantly boosting task completion and overall understanding.
What Problem Does This Paper Address?
Spoken dialogue systems (SDSs) are designed to interact with users through natural language. However, a major hurdle is the inherent unreliability of Automatic Speech Recognition (ASR) in real-world conditions. Factors like background noise, accents, and rapid speech can lead to ASR errors, which in turn cause the Speech Language Understanding (SLU) component of the SDS to generate incorrect semantic representations (frames). These incorrect frames can derail the entire dialogue, leading to user frustration and system failure. While previous work has focused on improving ASR or making dialogue management more flexible, this paper tackles the problem of correcting the understanding after ASR has made its best guess, by leveraging the dialogue context.
Methods
The core of the proposed technique is a "frame-correction module" that sits between the SLU component and the dialogue manager. This module aims to replace an incorrectly generated semantic frame with a more plausible one. The process involves:
- Initial Correction Model Creation: The system logs instances where the SLU component generates a frame (
fo) that doesn't match the expected reference frame (fr) for a given prompt type (T). These(T, fr, fo)tuples form an initial correction model. - Correction Model Enhancement: This crucial step refines the initial model:
- Compaction: Duplicate tuples are removed to streamline the model.
- Removal of Inadequate Tuples: This involves two key checks:
- Completeness: Using a
Σmodel (defining what constitutes a complete frame for a given prompt), frames deemed incomplete are discarded. - Agreement: Using a
Πmodel (defining expected prompt-frame type pairs), tuples where the generated frame type doesn't align with the prompt type are removed. This prevents correcting a frame that, while not matching the specific reference, is still valid for the prompt. - Generalization of Behaviour: A
Ymodel, which groups prompt types by expected response kind, is used to expand the correction model. This allows the system to handle misunderstandings for prompt types not explicitly seen during training, by generalizing from similar prompt types. - Frame Correction: When the SLU component produces a frame, the correction module uses the enhanced correction model and the current system prompt to decide if the generated frame should be replaced with a corrected one.
Datasets
The paper describes experiments conducted using two existing dialogue systems, Saplen (for fast food ordering) and Viajero (for bus travel booking). For training and testing, they utilized utterance corpora and scenarios collected from telephone calls of real users interacting with these systems. A user simulator was employed to interact with the systems using these test utterances. The corpora contained orthographic transcriptions and corresponding reference frames. The Saplen corpus had ~5,500 utterances, and Viajero had ~5,800 utterances, split 50/50 for training and testing. No new benchmarks were introduced; existing corpora and scenarios were adapted.
Key Insights and Ideas
- Leveraging Dialogue Context for Correction: The most significant insight is that understanding past failures (misunderstood frames) can be used to correct current ones. This mimics human ability to recover from misunderstandings.
- Correction Model as a Learned Component: The correction model isn't hardcoded but is learned from system interactions, making it adaptable to different domains and system behaviors.
- Importance of Frame Completeness and Agreement: The enhancement steps (completeness and agreement checks) are critical for ensuring that corrections are accurate and don't introduce new errors. This highlights the need for domain-specific knowledge (
ΣandΠmodels). - Generalization is Key: The generalization step is vital for making the system robust to novel misunderstandings, moving beyond simply memorizing past errors.
- Impact on Word Accuracy: Surprisingly, improving understanding also led to better word accuracy. This is attributed to the system needing fewer clarification turns when it correctly understands the user's intent, thus reducing opportunities for ASR errors in repeated utterances.
Key Results
The proposed frame-correction technique significantly improved system performance:
- Saplen:
- Sentence Understanding (SU) increased by 19.54% (with prompt-independent language model - PILM).
- Task Completion (TC) increased by 26.25% (with PILM).
- Word Accuracy (WA) increased by 7.53% (with PILM).
- Viajero:
- Sentence Understanding (SU) increased by 14.93% (with PILM).
- Task Completion (TC) increased by 18.06% (with PILM).
- Word Accuracy (WA) increased by 6.98% (with PILM).
The paper notes that performance was generally better with prompt-dependent language models (PDLMs) in the baseline, but the proposed technique showed substantial gains across both PDLMs and PILMs, particularly for the PILM which is more challenging.
Limitations and Future Work
- Limitations: The paper acknowledges that the correction model can become very large, and the current compaction step aims to reduce its size for processing speed. However, this might discard valuable information.
- Future Work: The authors suggest exploring methods to extract more information from the initial correction model. Specifically, they propose analyzing the frequency of duplicate tuples, which could indicate how often a particular prompt type is misunderstood. This information could be used to further enhance the frame replacement process. They also mention plans to study how systems can automatically select between prompt-dependent and prompt-independent language models to adapt to user experience and interaction success.
Summary updated