Asking Clarifying Questions in Open-Domain Information-Seeking Conversations

Mohammad Aliannejadi, Hamed Zamani, Fabio Crestani, W. Bruce Croft

TL;DR

This paper introduces a framework for conversational search systems to proactively ask clarifying questions when user queries are ambiguous or underspecified. The authors contribute Qulac, a new dataset of over 10,000 clarifying question-answer pairs, and propose NeuQS, a neural model that significantly improves retrieval performance by selecting the most effective questions to narrow down a user's intent.

What Problem Does This Paper Address?

Standard web search allows users to browse a list of results, but conversational interfaces (like voice assistants) often have a "one-result" constraint. If a user provides a vague query like "dinosaur," the system doesn't know if they want a list of species, pictures for kids, or a specific museum's exhibit.

Currently, most systems either guess the intent or force the user to reformulate the query. This paper tackles the challenge of proactive clarification: how can a system identify that a query is ambiguous, retrieve a set of potential clarifying questions, and select the one that will most improve the final search results?

Methods

The authors propose a three-component retrieval framework:

  1. Question Retrieval (BERT-LeaQuR): Since clarifying questions are often short and context-dependent, traditional keyword matching fails. The authors use a BERT-based model to learn high-dimensional representations of queries and questions, allowing the system to retrieve relevant candidate questions from a large pool.
  2. Question Selection (NeuQS): This is the core neural model. It ranks candidate questions by considering the original query, the conversation history (previous Q&A turns), and "Query Performance Prediction" (QPP) signals. It aims to pick the question that maximizes the precision of the final document ranking.
  3. Document Retrieval: Once the user answers the clarifying question, the system uses a KL-divergence retrieval model. It treats the user's answer as a way to expand and refine the original query, re-ranking documents based on the newly acquired information.

Datasets

The authors introduced Qulac (Questions for Lack of Clarity), the first major dataset for clarifying questions in IR.

  • Source: Built on top of TREC Web Track (2009-2012) data.
  • Scale: 198 TREC topics, 762 specific "facets" (intents), and over 10,000 question-answer pairs.
  • Collection: Data was gathered via Amazon Mechanical Turk, where workers generated questions based on search engine snippets and auto-complete suggestions, and others provided answers based on specific assigned facets.

Key Insights and Ideas

  • The Power of One Question: An "oracle" experiment (choosing the best possible question) showed that asking just one good clarifying question can improve retrieval performance (P@1) by over 170%.
  • Brevity Demands Clarification: There is a negative correlation between query length and the benefit of clarification. Shorter queries are naturally more ambiguous and see the highest gains from proactive questioning.
  • Ambiguity vs. Facets: The system distinguishes between ambiguous queries (multiple distinct interpretations) and faceted queries (underspecified with different aspects). Clarifying questions were found to be significantly more effective for ambiguous queries.
  • Beyond Keywords: The success of the BERT-based retrieval model highlights that understanding the intent behind a question is more important than matching words between the query and the question.

Key Results

  • NeuQS Performance: The proposed NeuQS model outperformed competitive "Learning to Rank" (LTR) baselines like LambdaMART and RankNet across all metrics (MRR, P@1, nDCG).
  • Robustness: NeuQS showed consistent improvements even as the number of conversation turns increased, suggesting it effectively incorporates conversation history without "forgetting" the original intent.
  • Facet Coverage: The model improved retrieval for 45% of the facets in the dataset, while only hurting performance in 19% of cases (often due to "negative" answers that didn't provide new keywords).

Limitations and Future Work

  • Simulated Multi-turn: While the authors simulated multi-turn conversations by combining questions, the dataset wasn't collected from organic, back-and-forth human dialogues.
  • Fixed Turn Count: The current study assumes a fixed number of turns. A more advanced system would need a "stopping criterion" to decide when it has enough information to stop asking and start showing results.
  • Open-Domain Generation: The framework currently selects questions from a pre-existing bank. Future work could involve using generative models (like GPT variants) to synthesize clarifying questions on the fly for entirely new topics.

Summary updated

Abstract

Users often fail to formulate their complex information needs in a single query. As a consequence, they may need to scan multiple result pages or reformulate their queries, which may be a frustrating experience. Alternatively, systems can improve user satisfaction by proactively asking questions of the users to clarify their information needs. Asking clarifying questions is especially important in conversational systems since they can only return a limited number of (often only one) result(s). In this paper, we formulate the task of asking clarifying questions in open-domain information-seeking conversational systems. To this end, we propose an offline evaluation methodology for the task and collect a dataset, called Qulac, through crowdsourcing. Our dataset is built on top of the TREC Web Track 2009-2012 data and consists of over 10K question-answer pairs for 198 TREC topics with 762 facets. Our experiments on an oracle model demonstrate that asking only one good question leads to over 170% retrieval performance improvement in terms of [email protected], which clearly demonstrates the potential impact of the task. We further propose a retrieval framework consisting of three components: question retrieval, question selection, and document retrieval. In particular, our question selection model takes into account the original query and previous question-answer interactions while selecting the next question. Our model significantly outperforms competitive baselines. To foster research in this area, we have made Qulac publicly available.
BibTeX
@inproceedings{aliannejadi2019asking,
  author = {Mohammad Aliannejadi and Hamed Zamani and Fabio Crestani and W. Bruce Croft},
  title = {{Asking Clarifying Questions in Open-Domain Information-Seeking Conversations}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2019},
  url = {https://openreview.net/forum?id=vpRIG2tONMS},
}

Loading related content...

Figure