Asking Clarification Questions to Handle Ambiguity in Open-Domain QA

Dongryeol Lee, Segwang Kim, Minwoo Lee, Hwanhee Lee, Joonsuk Park, Sang-Woo Lee, Kyomin Jung

TL;DR

Instead of overwhelming users by listing every possible answer to an ambiguous open-domain question, this paper proposes a conversational pipeline that asks targeted clarification questions. To support this, the authors introduce CAMBIGNQ, a new dataset of 5,653 ambiguous questions paired with clarification options, and demonstrate that users significantly prefer this interactive approach over static lists.

What Problem Does This Paper Address?

Open-domain question answering (ODQA) systems frequently encounter ambiguous queries. For instance, asking "Who played young Tom Riddle in Harry Potter?" could refer to the child version in Half-Blood Prince or the teenager version in Chamber of Secrets.

Prior state-of-the-art approaches handle this by generating multiple "disambiguated questions" (DQs) and listing all possible answers simultaneously. While effective computationally, this approach fails in real-world applications like voice assistants or smartwatches where screen real estate and user attention span are limited.

Figure 1: Two possible approaches to handling ambigu- ous questions (AQs) in open-domain question answer- ing (ODQA): (a) presenting disambiguated questions (DQs) with answers (following Min et al. (2020)), and (b) asking a clarification question (CQ) and displaying an answer based on the user’s response to the CQ (ours).

Methods

The authors propose a three-stage pipeline to handle ambiguous questions interactively:

  1. Ambiguity Detection: A binary classifier (built on BERT-base) determines if a question is ambiguous or unambiguous based on the query and retrieved passages.
  2. Clarification Question (CQ) Generation: If ambiguous, a Seq2Seq model (BART-large) generates a structured clarification question formatted as: "Which [category]: [option1], [option2], ..., or [optionn]?" (e.g., "Which version: young in series 2, child in series 6, or teenager in series 6?").
  3. Clarification-based QA: Once the user selects an option, the system appends this choice to the original question, reranks the relevant text passages using a cross-encoder, and feeds it to a reader model to extract the final precise answer.
Figure 2: Overview of our proposed approach to ODQA. Given a question, it is first checked for ambiguity ((a) ambiguity detection). If it is not ambiguous, it is processed in a normal QA setup ((b) QA; outside the scope of this work). Otherwise, an extra step of eliciting a response to a clarification question precedes QA ((c) CQ generation + clarification-based QA). Yellow blocks represent the user input, and blue blocks, the system output.

To evaluate the generated options and answers accurately without requiring exact string alignment, the authors adapted the Hungarian algorithm to compute optimal one-to-one partial matches between predicted sets and ground-truth sets.

Datasets

The authors introduce CAMBIGNQ, a dataset containing 5,653 ambiguous questions. It was constructed by taking ambiguous questions from the existing AmbigQA dataset and transforming their multiple disambiguated questions into a single structured clarification question.

Statistics of the CAMBIGNQ dataset across train, validation, and test splits.

To build it efficiently, they used a human-in-the-loop approach: InstructGPT generated initial drafts of the CQs via few-shot prompting, which were then manually inspected and refined by human editors to fix formatting errors and ensure accurate option mapping.

Examples of manual revisions made to InstructGPT-generated clarification questions.

Key Insights and Ideas

  • Human Preference for Interaction: Human evaluation showed that 59% of users preferred receiving a clarification question over a massive list of all possible answers (which was preferred by only 33%). This preference holds strong regardless of how many interpretations a question has.
  • The "Predicting Answers First" Fallacy: In prior DQ-based work, predicting all possible answers first helped generate better disambiguated questions. Surprisingly, the authors found that in a CQ framework, feeding predicted answers into the CQ generator does not meaningfully improve the quality of the clarification question in realistic settings.
  • Model Collapse on Subtle Inputs: A major bottleneck is that standard reader models (BART) suffer from "collapse"—they struggle to generate distinct answers for different options. Even when given different clarification options for the same question, the models frequently outputted the exact same answer, showing a blind spot to subtle prompt variations.

Key Results

The authors established baseline results across all three pipeline tasks:

  • Ambiguity Detection: Achieved an F1 score of 61.3 using a fine-tuned BERT-base model.
  • CQ Generation: The BART-large model achieved an option-level F1 score of 25.1 when utilizing predicted answers.
Table 4: Evaluation results for the Clarification Question (CQ) Generation task, where generated CQs are compared against the reference CQs. Each CQ was generated from an ambiguous question (AQ), relevant passages (RPs), and either No Answers, Predicted Answers, or Ground Truth Answers for the AQ. The ground truth answers case represents an ideal scenario in which the QA system perfectly identifies all possible answers for the AQ.
  • Clarification-based QA: A CQ-finetuned BART model achieved an F1 score of 40.5 on the final answer extraction task when using predicted CQs.
Evaluation results for the Clarification-based QA task across different reader models.

Limitations and Future Work

  • Evaluation Metric Constraints: Current n-gram overlap metrics (like BLEU) and semantic metrics (like BERTScore) often penalize perfectly valid clarification questions simply because they phrase the options or categories differently than the ground truth.
  • Reasoning Failures: Extrinsic QA models frequently fail to reason through the added clarification context, leading to the model collapse mentioned above.
  • Future Directions: The authors call for better prompt engineering, exploring more powerful Large Language Models (LLMs) to handle the interactive dialogue naturally, and developing more robust evaluation metrics that measure semantic validity rather than strict text overlap.

Summary updated

Abstract

Ambiguous questions persist in open-domain question answering, because formulating a precise question with a unique answer is often challenging. Previously, Min et al. (2020) have tackled this issue by generating disambiguated questions for all possible interpretations of the ambiguous question. This can be effective, but not ideal for providing an answer to the user. Instead, we propose to ask a clarification question, where the user's response will help identify the interpretation that best aligns with the user's intention. We first present CAMBIGNQ, a dataset consisting of 5,654 ambiguous questions, each with relevant passages, possible answers, and a clarification question. The clarification questions were efficiently created by generating them using InstructGPT and manually revising them as necessary. We then define a pipeline of tasks and design appropriate evaluation metrics. Lastly, we achieve 61.3 F1 on ambiguity detection and 40.5 F1 on clarification-based QA, providing strong baselines for future work.
BibTeX
@inproceedings{lee2023asking,
  author = {Dongryeol Lee and Segwang Kim and Minwoo Lee and Hwanhee Lee and Joonsuk Park and Sang-Woo Lee and Kyomin Jung},
  title = {{Asking Clarification Questions to Handle Ambiguity in Open-Domain QA}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.findings-emnlp.772},
}
Figure