TL;DR
Instead of overwhelming users by listing every possible answer to an ambiguous open-domain question, this paper proposes a conversational pipeline that asks targeted clarification questions. To support this, the authors introduce CAMBIGNQ, a new dataset of 5,653 ambiguous questions paired with clarification options, and demonstrate that users significantly prefer this interactive approach over static lists.
What Problem Does This Paper Address?
Open-domain question answering (ODQA) systems frequently encounter ambiguous queries. For instance, asking "Who played young Tom Riddle in Harry Potter?" could refer to the child version in Half-Blood Prince or the teenager version in Chamber of Secrets.
Prior state-of-the-art approaches handle this by generating multiple "disambiguated questions" (DQs) and listing all possible answers simultaneously. While effective computationally, this approach fails in real-world applications like voice assistants or smartwatches where screen real estate and user attention span are limited.
Methods
The authors propose a three-stage pipeline to handle ambiguous questions interactively:
- Ambiguity Detection: A binary classifier (built on BERT-base) determines if a question is ambiguous or unambiguous based on the query and retrieved passages.
- Clarification Question (CQ) Generation: If ambiguous, a Seq2Seq model (BART-large) generates a structured clarification question formatted as: "Which [category]: [option1], [option2], ..., or [optionn]?" (e.g., "Which version: young in series 2, child in series 6, or teenager in series 6?").
- Clarification-based QA: Once the user selects an option, the system appends this choice to the original question, reranks the relevant text passages using a cross-encoder, and feeds it to a reader model to extract the final precise answer.
To evaluate the generated options and answers accurately without requiring exact string alignment, the authors adapted the Hungarian algorithm to compute optimal one-to-one partial matches between predicted sets and ground-truth sets.
Datasets
The authors introduce CAMBIGNQ, a dataset containing 5,653 ambiguous questions. It was constructed by taking ambiguous questions from the existing AmbigQA dataset and transforming their multiple disambiguated questions into a single structured clarification question.
To build it efficiently, they used a human-in-the-loop approach: InstructGPT generated initial drafts of the CQs via few-shot prompting, which were then manually inspected and refined by human editors to fix formatting errors and ensure accurate option mapping.
Key Insights and Ideas
- Human Preference for Interaction: Human evaluation showed that 59% of users preferred receiving a clarification question over a massive list of all possible answers (which was preferred by only 33%). This preference holds strong regardless of how many interpretations a question has.
- The "Predicting Answers First" Fallacy: In prior DQ-based work, predicting all possible answers first helped generate better disambiguated questions. Surprisingly, the authors found that in a CQ framework, feeding predicted answers into the CQ generator does not meaningfully improve the quality of the clarification question in realistic settings.
- Model Collapse on Subtle Inputs: A major bottleneck is that standard reader models (BART) suffer from "collapse"—they struggle to generate distinct answers for different options. Even when given different clarification options for the same question, the models frequently outputted the exact same answer, showing a blind spot to subtle prompt variations.
Key Results
The authors established baseline results across all three pipeline tasks:
- Ambiguity Detection: Achieved an F1 score of 61.3 using a fine-tuned BERT-base model.
- CQ Generation: The BART-large model achieved an option-level F1 score of 25.1 when utilizing predicted answers.
- Clarification-based QA: A CQ-finetuned BART model achieved an F1 score of 40.5 on the final answer extraction task when using predicted CQs.
Limitations and Future Work
- Evaluation Metric Constraints: Current n-gram overlap metrics (like BLEU) and semantic metrics (like BERTScore) often penalize perfectly valid clarification questions simply because they phrase the options or categories differently than the ground truth.
- Reasoning Failures: Extrinsic QA models frequently fail to reason through the added clarification context, leading to the model collapse mentioned above.
- Future Directions: The authors call for better prompt engineering, exploring more powerful Large Language Models (LLMs) to handle the interactive dialogue naturally, and developing more robust evaluation metrics that measure semantic validity rather than strict text overlap.
Summary updated