TL;DR
This paper proposes an example-based approach to understanding user intentions in spoken dialogue systems, particularly for in-car environments. By comparing input utterances to a corpus of example dialogues with pre-assigned intention tags, the system infers intent based on similarity in morphology and dependency structures, weighted by dialogue context. This method achieves 68.9% accuracy and forms the basis of a functional in-car restaurant retrieval system.
What Problem Does This Paper Address?
Spoken dialogue systems need to accurately understand user intentions to respond appropriately. Traditional rule-based systems for intention recognition are labor-intensive to create and difficult to maintain, especially given the variability and complexity of human speech. Existing dialogue act tagging schemes are often too general for direct task execution. The authors aim to bridge this gap by developing a more robust and adaptable method for inferring fine-grained utterance intentions, specifically tailored for the challenging domain of in-car dialogue where users are often distracted.
Methods
The core of the proposed method is an example-based learning approach.
- Corpus-driven Similarity: The system leverages a spoken dialogue corpus where each utterance is annotated with an "intention tag." When a new utterance is received, its intention is inferred by finding the most similar utterance in the corpus.
- Multi-faceted Similarity Calculation: Similarity is not just based on word overlap. It's calculated using a weighted combination of:
- Morphological Similarity: Comparing the morphemes (the smallest meaningful units of language) of the input utterance with corpus utterances.
- Dependency Similarity: Comparing the grammatical relationships (how words modify each other) between the input utterance and corpus utterances.
- Contextual Weighting: To further improve accuracy, the similarity scores are weighted based on the dialogue context. This is achieved by using "intentions N-gram probabilities," which capture the likelihood of a sequence of intentions occurring. If a particular intention sequence is less probable, the similarity score for that utterance is reduced.
- Extended Intention Tags: The authors define 78 kinds of intention tags, extending existing dialogue act schemes to be more task-specific and directly linkable to system operations.
Datasets
- CIAIR in-car spoken dialogue corpus: This corpus, containing 412 dialogues about restaurant search, was used for assigning intention tags.
- 174 dialogues: A subset of the CIAIR corpus was used for learning intentions 2-gram probabilities.
- 1,609 driver's utterances (from 238 dialogues): This larger dataset was used for evaluating the intention inference method and the prototype system. This data was split into training and testing sets for cross-validation.
Key Insights and Ideas
- Beyond Keywords: Structure Matters: Relying solely on keyword matching for intention recognition is insufficient. Incorporating dependency structures provides a deeper understanding of sentence meaning and intent.
- Context is King (Even in Short Utterances): In noisy environments like cars, utterances can be short and ambiguous. Leveraging the history of the conversation (dialogue context) significantly improves the accuracy of intention inference.
- Task-Specific Intentions are Crucial: Defining detailed intention tags that directly map to system actions (e.g., "parking lot question" vs. "parking lot search") makes the system more effective and easier to build.
- Example-Based Learning Adapts: This approach avoids the manual rule-writing bottleneck of traditional systems and can naturally handle the diversity of user utterances.
Key Results
- Intention Inference Accuracy: The proposed method achieved 68.9% accuracy in inferring utterance intentions.
- Effectiveness of Combined Features: Using both morphology and dependency information for similarity calculation was more effective than using either alone. The best precision was achieved with a weight coefficient $\lambda = 0.2$, and the best recall with $\lambda = 0.3$.
- Impact of Context: Incorporating intentions 2-gram probabilities (context) further improved performance, with a maximum precision and recall of 68.9% at a weight coefficient $w = 0.8$.
- Prototype System Performance: The prototype in-car restaurant retrieval system, based on this method, showed a correct action rate of 76.7% (including partial correctness) when using given intentions, and 60.0% when using inferred intentions. This demonstrates the system's ability to function appropriately when intentions are correctly inferred.
Limitations and Future Work
- Limitations:
- The paper notes that for some utterances, the system's performance was limited by issues like keyword processing failures and utterances falling outside the system's expectations.
- The accuracy of the system is dependent on the quality and coverage of the annotated corpus.
- Future Work:
- The authors suggest improving transfer rules to better handle keyword processing failures.
- They also propose enhancing the system's ability to respond gracefully to questions outside its scope (e.g., "I cannot answer the question. If the questions are about..., I can do that.").
- Further refinement of the context modeling and intention tag set could lead to even higher accuracy.
Summary updated