TL;DR
This paper tackles the challenge of building question-answering (QA) systems that can handle ambiguous user queries by generating clarifying questions. They propose novel methods for creating "pseudo-ambiguous" questions from existing clear ones and for generating effective clarifying questions using sentence structures and case frame dictionaries, demonstrating improved performance in recovering lost information.
What Problem Does This Paper Address?
Traditional question-answering (QA) systems often assume that user questions are well-formed and contain all the necessary information to find an answer within a given document. However, in real-world scenarios, users frequently ask ambiguous questions due to incomplete information, forgetfulness, or the inherent complexity of natural language. This ambiguity prevents QA systems from providing accurate answers. The paper addresses this by focusing on "clarifying question answering," where the system actively engages with the user to resolve ambiguity. The core challenges are twofold: (1) creating datasets of ambiguous questions and (2) developing methods to generate appropriate clarifying questions that help the system pinpoint the user's true intent.
Methods
The authors propose a two-pronged approach:
- Pseudo Ambiguous Question Generation: Instead of manually creating ambiguous questions, which is labor-intensive and prone to missing variations, they generate "pseudo-ambiguous" questions by modifying clear questions from existing QA datasets. This is achieved by leveraging sentence structures, specifically by eliminating parts of the sentence (like verb phrases or prepositional phrases) that provide clarifying details, thus introducing ambiguity. This method aims to cover a wider range of ambiguity variations.
- Clarifying Question Generation: This method focuses on generating questions that help disambiguate the user's intent. It utilizes the syntax and semantic information of the ambiguous question. Specifically, it identifies predicates (verbs) and their arguments, then uses a case frame dictionary to determine missing arguments or clarify existing ones. The system then generates a clarifying question by selecting an appropriate interrogative word based on the identified predicate and missing case. This approach aims to recover the lost information by asking targeted questions.
Datasets
- HotpotQA dataset: This dataset was used for both training the QA model and for generating pseudo-ambiguous questions and evaluating the clarifying question generation system. The development set of HotpotQA was used as the test set.
Key Insights and Ideas
- Leveraging Sentence Structure for Ambiguity: The paper highlights that sentence structure is key to controlling clarity. By strategically removing syntactic components (like modifiers), they can effectively introduce ambiguity into otherwise clear questions, creating a more realistic testbed for clarifying QA systems.
- Case Frame Dictionaries as a Bridge: The use of case frame dictionaries is crucial for generating targeted clarifying questions. These dictionaries encode the typical arguments and roles associated with verbs, allowing the system to identify what information is missing or needs clarification.
- Recovering Lost Information: The core idea is that even if a question becomes ambiguous, a well-crafted clarifying question can help recover the lost information, thereby restoring the QA system's performance. The experiments show that this interaction can recover a significant portion of the performance drop.
- Domain-Independent Approach: The methods rely on general syntactic and semantic information, making them applicable to various QA domains without requiring domain-specific annotations.
Key Results
- Pseudo Ambiguous Question Generation: The generated pseudo-ambiguous questions successfully introduced ambiguity, as evidenced by a drop in QA system accuracy. The "Mixed" strategy (removing both VP and PP) achieved a suitable naturalness score in human evaluation.
- Clarifying Question Generation: The clarifying question answering system was able to recover approximately 50% of the lost information caused by the ambiguity. The system achieved an F1 score of 57.28% when answering ambiguous questions with the help of clarifying questions and user replies.
- Qualitative Findings: Examples showed that the clarifying questions could successfully recover specific details (e.g., names like "Jerry Goldsmith") that were removed to create ambiguity. Recovering information related to verb phrases (VPs) was found to be more challenging than recovering prepositional phrases (PPs).
Limitations and Future Work
- Limitations:
- Recovering information related to verb phrases (VPs) was more difficult than PPs.
- In some cases, clarifying questions were not perfectly appropriate, although the system could still recover the answer.
- Users might misunderstand the task and ask a new question instead of answering the clarifying one, a behavior observed in some examples and potentially influenced by modern search interfaces.
- Future Work:
- Integrate the model with other generative models.
- Use pseudo-ambiguous questions as training data for QA modules (e.g., discriminative systems).
- Improve the model architecture to incorporate dialogue history.
Summary updated