TL;DR
This paper introduces a novel dialogue management strategy for spoken document retrieval systems that optimizes response generation by minimizing a "Bayes risk." This approach robustly handles errors from Automatic Speech Recognition (ASR) and user input, leading to improved retrieval success rates and fewer conversational turns.
What Problem Does This Paper Address?
Spoken dialogue systems are increasingly being tasked with retrieving information from vast collections of general documents, moving beyond simpler domains like flight information. A key challenge in these systems is effectively managing the dialogue to ensure users get the information they need efficiently, especially when dealing with imperfect speech recognition and potentially vague user queries. Traditional approaches often tune dialogue management modules independently, which can lead to suboptimal overall system performance. The authors aim to address this by proposing an integrated approach to dialogue management that considers the trade-offs between providing correct information and avoiding unnecessary conversational turns.
Methods
The core of the proposed method is a dialogue management strategy that optimizes response generation by minimizing a "Bayes risk." This risk is defined as a unified criterion that balances the reward for presenting correct information with penalties for redundant turns and the cost of incorrect information.
The system operates by:
- Generating Search Queries: It constructs search queries using the N-best hypotheses from the ASR system, incorporating contextual information from previous turns when beneficial.
- Retrieving Candidate Documents: For each generated query, the system retrieves candidate documents from its knowledge base and estimates their likelihood.
- Generating Response Candidates: For each retrieved document, the system generates three types of response candidates:
- Presentation (Pres(d)): A summary of the document.
- Confirmation (Conf(d)): A confirmation before presenting the document.
- Answer (Ans(d)): A direct answer extracted from the document.
- Rejection: A fallback option.
- Calculating Bayes Risk: The Bayes risk is calculated for each response candidate. This calculation incorporates:
- The likelihood of document retrieval and answer extraction.
- Rewards for successful presentations and answers.
- Penalties for failures, including improper presentations, user corrections, and extraneous turns (approximated by the number of sentences before obtaining the correct information).
- Selecting Optimal Response: The system selects the response candidate with the minimal Bayes risk.
The authors also detail how they handle contextual information by concatenating keywords from previous utterances and how they use N-best ASR hypotheses to improve robustness against recognition errors.
Datasets
The system was developed and evaluated using data from the "Dialogue Navigator for Kyoto City" project. The knowledge base consists of Wikipedia documents about Kyoto and official tourist information, totaling 810 documents and 220K words. For evaluation, 1,416 in-domain utterances (1,084 queries and 332 questions) were transcribed and manually labeled with correct documents/Named Entities (NEs).
Key Insights and Ideas
- Unified Optimization Criterion: The concept of Bayes risk provides a principled way to balance competing objectives in dialogue management, such as accuracy and efficiency. This moves beyond module-by-module optimization.
- Robustness to ASR Errors: Leveraging N-best hypotheses and contextual information is crucial for handling the inherent inaccuracies of ASR in spoken dialogue systems.
- Contextual Information is a Double-Edged Sword: While essential for understanding anaphoric references, blindly concatenating context can lead to inappropriate responses if the user changes topics. The Bayes risk framework implicitly handles this by penalizing incorrect context.
- Confirmation as a Trade-off: Confirmations are necessary to avoid errors, but they add to conversational length. The Bayes risk calculation quantifies this trade-off, allowing the system to decide when confirmation is truly beneficial.
- Beyond Simple Retrieval: The system's ability to handle specific questions ("Who built this shrine?") using QA techniques alongside document retrieval demonstrates a more sophisticated information access capability.
Key Results
The proposed Bayes risk-based dialogue management method achieved a higher success rate and a reduced number of sentences for information access compared to baseline methods. Specifically:
- Success Rate: The proposed method achieved a success rate of 65.6%, an improvement of 6.4% over the baseline method 1 (59.2%) and 2.2% over method 2 (63.4%).
- Number of Sentences: The average number of sentences for presentation was reduced to 4.71, an improvement of 0.78 sentences over baseline method 1 (5.49) and 0.27 sentences over method 2 (4.98).
- Candidate Selection: A breakdown of selected candidates showed that the system often utilized hypotheses beyond the first ASR hypothesis and effectively incorporated contextual information for question answering.
Limitations and Future Work
- Limitations: The paper acknowledges that the confirmation strategy implemented is relatively simple, relying on retrieval likelihood. The authors suggest that more complex dialogue management, such as confirmations considering retrieval impact, could be beneficial. The penalty values for failures and rewards for success were determined through a 2-fold cross-validation, implying they are empirical and might require further tuning for different domains.
- Future Work: The authors suggest extending the current framework to handle more complex dialogue management scenarios, particularly in considering the impact of confirmations on retrieval. Further research into optimizing the reward and penalty parameters and exploring different contextual information incorporation strategies is also implied.
Summary updated