TL;DR
This paper introduces a novel, domain-independent approach to representing and resolving ambiguities in spoken dialogue systems. By separating raw data from derived candidates and employing a pragmatic scoring mechanism combined with explicit user correction, they significantly reduce errors in travel reservation dialogues.
What Problem Does This Paper Address?
Spoken dialogue systems constantly grapple with ambiguity. This can stem from various sources: the inherent vagueness of natural language (e.g., "next Saturday" could mean this week or next), errors from speech recognition systems (which are still quite prevalent), and even user corrections that might themselves be ambiguous. Effectively handling these ambiguities is crucial for a smooth and successful user interaction. The authors aim to create a system that can robustly understand and resolve these ambiguities in a way that is not tied to a specific application domain.
Methods
The core of their approach lies in a novel semantic representation and a multi-stage resolution strategy.
- Prototype Tree: A domain-independent tree structure representing the ontology of a domain (e.g., a travel agent's knowledge about flights, hotels, and cars). This tree defines concepts and their relationships, serving as the backbone for semantic representation.
- Notepad Tree: A dynamic tree that mirrors the prototype tree, used to store attribute-value (AV) pairs extracted from user utterances.
- Application Tree: Stores derived candidate values for attributes, allowing for multiple possibilities to be considered simultaneously.
- Context Tracking: Algorithms that match user input (partial attributes) with the current dialogue context to form complete attributes. This involves exact, interpolated, and overlap matches, with mechanisms to handle context changes.
- Scoring Mechanism: Each candidate value is assigned a score (confidence factor) between -1 and 1. These scores are updated based on various evidence sources: raw data, match type from context tracking, and pragmatic analysis. A candidate is selected only if its score exceeds a threshold and is sufficiently higher than the next best candidate.
- Pragmatic Analysis: The system analyzes user responses to prompts to infer confirmation or objections, adjusting candidate scores accordingly. This allows for implicit confirmation of data.
- Correction Mechanisms: Users can directly address attributes with commands like "clear," "freeze," or "change," providing explicit control over the dialogue.
Datasets
The experimental results are based on an analysis of 35 dialogues collected during the June 2000 DARPA Communicator evaluation. These dialogues were run through both their previous system and the new system to compare performance. No new benchmarks were introduced.
Key Insights and Ideas
- Domain Independence is Key: By abstracting domain knowledge into a parameterizable "prototype tree," the system's core ambiguity resolution mechanisms can be reused across different applications.
- Separation of Concerns: The clear distinction between raw data (notepad tree) and derived possibilities (application tree) simplifies the development and application of scoring and selection algorithms.
- Pragmatics as a Powerful Signal: Incorporating pragmatic analysis of user responses (beyond just direct answers) provides strong evidence for or against candidate values, significantly improving disambiguation.
- User Control is Essential: Explicit correction mechanisms empower users to override system decisions and guide the dialogue, making the system more robust to errors and user intent.
Key Results
The paper reports that their new ambiguity resolution strategy significantly improves dialogue interaction. Specifically, it eliminated about half of the errors in dialogues from a travel reservation task. In an analysis of 49 dialogue turns where systems diverged, their current system showed improvement in 25 cases, compared to only 3 cases where the older system was superior. These improvements were attributed to better parsing, the introduction of scoring and pragmatic analysis, and the interaction of these modules.
Limitations and Future Work
The authors acknowledge that the system's performance relies on careful tuning of scoring functions and parameters. They also mention that not all domain knowledge can be encoded in the prototype tree, with some specialized knowledge requiring task-dependent procedures. Future work includes further tuning of scoring algorithms and integrating confidence measures from other system modules (like speech recognition).
Summary updated