Asking Clarifying Questions in Open-Domain Information-Seeking Conversations

Mohammad Aliannejadi, Hamed Zamani, Fabio Crestani, W. Bruce Croft

Abstract

Users often fail to formulate their complex information needs in a single query, leading to frustration. Proactively asking clarifying questions can significantly improve user satisfaction, especially in conversational systems limited to returning a single result. This paper formulates the task of proactive clarification in open-domain search, introduces the Qulac dataset of 10K+ question-answer pairs, and proposes a neural retrieval framework. Experiments show that a single well-chosen question can improve retrieval performance (P@1) by over 170%.

Introduction

Standard web search interfaces allow users to browse multiple results, but conversational interfaces like voice assistants often operate under a "one-result" constraint. When a user provides a vague query (e.g., "dinosaur"), the system faces ambiguity: does the user want species lists, children's pictures, or museum exhibits? While current systems often guess or require manual reformulation, this research focuses on proactive clarification—identifying ambiguity and selecting the most effective question to refine the search intent.

Methodology: A Framework for Proactive Clarification

The authors propose a three-component retrieval framework designed to handle the iterative nature of conversational search.

The proposed conversational search framework for proactive clarification.
  1. Question Retrieval (BERT-LeaQuR): Traditional keyword matching is often insufficient for short, context-heavy clarifying questions. This model uses BERT to learn high-dimensional representations of queries and questions, enabling the retrieval of relevant candidates from a large pool.
  2. Question Selection (NeuQS): This core neural model ranks candidate questions by analyzing the original query, conversation history, and Query Performance Prediction (QPP) signals. The goal is to select the question that maximizes the precision of the final document ranking.
  3. Document Retrieval: After receiving a user's answer, the system employs a KL-divergence retrieval model. The answer acts as a query expansion, allowing the system to re-rank documents based on the refined information need.

The Qulac Dataset

To support this research, the authors introduced Qulac (Questions for Lack of Clarity), the first major dataset for clarifying questions in Information Retrieval (IR). Built on TREC Web Track (2009-2012) data, it includes 198 topics and 762 specific "facets" (intents). Over 10,000 question-answer pairs were collected via crowdsourcing, where workers generated questions based on search engine context and others provided answers based on assigned facets.

Experimental Results and Insights

The proposed NeuQS model significantly outperformed competitive Learning to Rank (LTR) baselines like LambdaMART and RankNet across metrics including MRR, P@1, and nDCG.

Retrieval performance gains (Δ MRR) across different topic types, facet types, and query lengths.

Several key insights emerged from the evaluation:

  • The Impact of Ambiguity: Clarifying questions are most effective for ambiguous queries (those with multiple distinct interpretations) compared to faceted queries (underspecified aspects of a single topic).
  • Brevity and Clarification: There is a clear negative correlation between query length and the benefit of clarification. Shorter queries, being naturally more ambiguous, see the highest performance gains from proactive questioning.
  • Robustness: NeuQS effectively incorporates conversation history, showing consistent improvements as the number of turns increases without losing track of the original intent.

Conclusion

This work demonstrates the immense potential of proactive clarification in conversational search, with oracle experiments showing up to a 170% improvement in P@1. While the current study uses simulated multi-turn conversations and fixed turn counts, it provides a robust foundation for future work. Future directions include developing "stopping criteria" to determine when enough information has been gathered and utilizing generative models to synthesize clarifying questions dynamically for open-domain topics.

Summary updated

Abstract

Users often fail to formulate their complex information needs in a single query. As a consequence, they may need to scan multiple result pages or reformulate their queries, which may be a frustrating experience. Alternatively, systems can improve user satisfaction by proactively asking questions of the users to clarify their information needs. Asking clarifying questions is especially important in conversational systems since they can only return a limited number of (often only one) result(s). In this paper, we formulate the task of asking clarifying questions in open-domain information-seeking conversational systems. To this end, we propose an offline evaluation methodology for the task and collect a dataset, called Qulac, through crowdsourcing. Our dataset is built on top of the TREC Web Track 2009-2012 data and consists of over 10K question-answer pairs for 198 TREC topics with 762 facets. Our experiments on an oracle model demonstrate that asking only one good question leads to over 170% retrieval performance improvement in terms of P@1, which clearly demonstrates the potential impact of the task. We further propose a retrieval framework consisting of three components: question retrieval, question selection, and document retrieval. In particular, our question selection model takes into account the original query and previous question-answer interactions while selecting the next question. Our model significantly outperforms competitive baselines. To foster research in this area, we have made Qulac publicly available.
BibTeX
@inproceedings{aliannejadi2019asking,
  author = {Mohammad Aliannejadi and Hamed Zamani and Fabio Crestani and W. Bruce Croft},
  title = {{Asking Clarifying Questions in Open-Domain Information-Seeking Conversations}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2019},
  url = {https://openreview.net/forum?id=vpRIG2tONMS},
}
Figure