TL;DR
This paper introduces a framework for conversational search systems to proactively ask clarifying questions when user queries are ambiguous or underspecified. The authors contribute Qulac, a new dataset of over 10,000 clarifying question-answer pairs, and propose NeuQS, a neural model that significantly improves retrieval performance by selecting the most effective questions to narrow down a user's intent.
What Problem Does This Paper Address?
Standard web search allows users to browse a list of results, but conversational interfaces (like voice assistants) often have a "one-result" constraint. If a user provides a vague query like "dinosaur," the system doesn't know if they want a list of species, pictures for kids, or a specific museum's exhibit.
Currently, most systems either guess the intent or force the user to reformulate the query. This paper tackles the challenge of proactive clarification: how can a system identify that a query is ambiguous, retrieve a set of potential clarifying questions, and select the one that will most improve the final search results?
Methods
The authors propose a three-component retrieval framework:
- Question Retrieval (BERT-LeaQuR): Since clarifying questions are often short and context-dependent, traditional keyword matching fails. The authors use a BERT-based model to learn high-dimensional representations of queries and questions, allowing the system to retrieve relevant candidate questions from a large pool.
- Question Selection (NeuQS): This is the core neural model. It ranks candidate questions by considering the original query, the conversation history (previous Q&A turns), and "Query Performance Prediction" (QPP) signals. It aims to pick the question that maximizes the precision of the final document ranking.
- Document Retrieval: Once the user answers the clarifying question, the system uses a KL-divergence retrieval model. It treats the user's answer as a way to expand and refine the original query, re-ranking documents based on the newly acquired information.
Datasets
The authors introduced Qulac (Questions for Lack of Clarity), the first major dataset for clarifying questions in IR.
- Source: Built on top of TREC Web Track (2009-2012) data.
- Scale: 198 TREC topics, 762 specific "facets" (intents), and over 10,000 question-answer pairs.
- Collection: Data was gathered via Amazon Mechanical Turk, where workers generated questions based on search engine snippets and auto-complete suggestions, and others provided answers based on specific assigned facets.
Key Insights and Ideas
- The Power of One Question: An "oracle" experiment (choosing the best possible question) showed that asking just one good clarifying question can improve retrieval performance (P@1) by over 170%.
- Brevity Demands Clarification: There is a negative correlation between query length and the benefit of clarification. Shorter queries are naturally more ambiguous and see the highest gains from proactive questioning.
- Ambiguity vs. Facets: The system distinguishes between ambiguous queries (multiple distinct interpretations) and faceted queries (underspecified with different aspects). Clarifying questions were found to be significantly more effective for ambiguous queries.
- Beyond Keywords: The success of the BERT-based retrieval model highlights that understanding the intent behind a question is more important than matching words between the query and the question.
Key Results
- NeuQS Performance: The proposed NeuQS model outperformed competitive "Learning to Rank" (LTR) baselines like LambdaMART and RankNet across all metrics (MRR, P@1, nDCG).
- Robustness: NeuQS showed consistent improvements even as the number of conversation turns increased, suggesting it effectively incorporates conversation history without "forgetting" the original intent.
- Facet Coverage: The model improved retrieval for 45% of the facets in the dataset, while only hurting performance in 19% of cases (often due to "negative" answers that didn't provide new keywords).
Limitations and Future Work
- Simulated Multi-turn: While the authors simulated multi-turn conversations by combining questions, the dataset wasn't collected from organic, back-and-forth human dialogues.
- Fixed Turn Count: The current study assumes a fixed number of turns. A more advanced system would need a "stopping criterion" to decide when it has enough information to stop asking and start showing results.
- Open-Domain Generation: The framework currently selects questions from a pre-existing bank. Future work could involve using generative models (like GPT variants) to synthesize clarifying questions on the fly for entirely new topics.
Summary updated