Conversational Search with Mixed-Initiative - Asking Good Clarification Questions backed-up by Passage Retrieval

Yosi Mass, Doron Cohen, Asaf Yehudai, David Konopnicki

Abstract

We deal with the scenario of conversational search, where user queries are under-specified or ambiguous. This calls for a mixed-initiative setup. User-asks (queries) and system-answers, as well as system-asks (clarification questions) and user response, in order to clarify her information needs. We focus on the task of selecting the next clarification question, given the conversation context. Our method leverages passage retrieval from a background content to fine-tune two deep-learning models for ranking candidate clarification questions. We evaluated our method on two different use-cases. The first is an open domain conversational search in a large web collection. The second is a task-oriented customer-support setup. We show that our method performs well on both use-cases.

Introduction

In conversational search, user queries are often underspecified or ambiguous. A query like "all men are created equal" could refer to the history of the Declaration of Independence or biographical details about Thomas Jefferson. To address this, systems must adopt a mixed-initiative approach, asking clarification questions to resolve ambiguity and better understand the user's latent intent.

An example of a vague user query being clarified by a bot using a retrieved passage as context.

Methodology

The researchers propose a multi-stage pipeline that uses retrieved documents as a proxy for user intent. Instead of generating questions from scratch—which can lead to grammatical errors or irrelevance—the system selects the best question from a pre-defined pool.

The multi-stage pipeline architecture, featuring passage retrieval, clarification retrieval, and BERT-based re-ranking.

The pipeline consists of four primary stages:

  1. Passage Retrieval: The conversation history is treated as a verbose query to retrieve the top-k passages from a document index. A specialized weighting scheme favors more recent utterances.
  2. Question Retrieval: For every retrieved passage, the system searches an index of candidate clarification questions.
  3. Deep Re-ranking: Two BERT-based models are fine-tuned to rank these candidates:
  4. BERT-C-cq: Learns associations between the conversation context and the question.
  5. BERT-C-P-cq: Learns associations between the context, the retrieved passage, and the question.
  6. Score Fusion: Scores from both models are combined via CombSUM to select the final question.

Datasets

The approach was evaluated on two distinct scenarios:

  • ClariQ: An open-domain web search dataset containing 2.7 million documents and approximately 4,000 clarification questions.
  • Support: A task-oriented dataset of technical customer support logs. This dataset is noisier, requiring specific rules to filter human "chit-chat" from actual clarification questions.

Key Insights

  • Passages as Intent Proxies: Retrieved passages bridge the gap between vague queries and specific questions. If a passage mentions a specific entity, it validates questions related to that entity.
  • Complementary Models: While BERT-C-P-cq excels at identifying latent features in text, BERT-C-cq acts as a safety net when retrieved passages are noisy or slightly off-topic.
  • Context Weighting: Utterance-biased weighting (prioritizing recent turns) significantly improves the performance of the initial passage retrieval.

Results

The "BERT-fusion" approach consistently outperformed standard Information Retrieval (IR) baselines. On the Support dataset, it achieved a Recall@30 of 0.551, representing an 82% improvement over the base IR system (0.294). On the ClariQ dataset, the system achieved a Recall@30 of 0.791, ranking 4th overall in the official challenge.

Limitations and Future Work

Current limitations include the reliance on a fixed question pool and the lack of a "trigger" mechanism to decide if clarification is needed versus providing a direct answer. Additionally, BERT's 512-token limit can lead to context loss in long conversations. Future work will explore generative models and dynamic triggering mechanisms.

Summary updated

Abstract

We deal with the scenario of conversational search, where user queries are under-specified or ambiguous. This calls for a mixed-initiative setup. User-asks (queries) and system-answers, as well as system-asks (clarification questions) and user response, in order to clarify her information needs. We focus on the task of selecting the next clarification question, given the conversation context. Our method leverages passage retrieval from a background content to fine-tune two deep-learning models for ranking candidate clarification questions. We evaluated our method on two different use-cases. The first is an open domain conversational search in a large web collection. The second is a task-oriented customer-support setup. We show that our method performs well on both use-cases.
BibTeX
@inproceedings{mass2022conversational,
  author = {Yosi Mass and Doron Cohen and Asaf Yehudai and David Konopnicki},
  title = {{Conversational Search with Mixed-Initiative - Asking Good Clarification Questions backed-up by Passage Retrieval}},
  booktitle = {DialDoc @ ACL},
  year = {2022},
  url = {https://openreview.net/forum?id=gyeUg6lpYn},
}
Figure