Improving search and recommendation by asking clarifying questions

Jie ZouSIGIR 2021 · OpenReview

Abstract

Recently, there has been renewed interest in interactive search and recommender systems through the development of systems that ask clarifying questions (CQs) to users to better understand their information needs. Although the significance and effectiveness of CQs have been recognized, research on modelling CQs to improve search and recommender systems is limited compared to traditional systems. Furthermore, understanding users' willingness or ability to answer these CQs, and the extent to which they impact user behavior, remains relatively unexplored. This work provides a series of models and online user studies for search and recommender systems that construct and use CQs. We propose novel algorithms for document search (SBSTAR), product search (QSBPS), and general recommendation (Qrec), demonstrating that they outperform state-of-the-art baselines. Additionally, we conduct large-scale user studies to evaluate user perception and interaction quality in real-world scenarios.

A word cloud visualizing the key themes and algorithmic frameworks of the thesis, such as SBSTAR, QSBPS, and Qrec.

Introduction: The Semantic Gap in Retrieval

Traditional search and recommendation systems often suffer from a "semantic gap." Users frequently provide short, ambiguous, or faceted queries that do not fully capture their underlying intent. In specialized fields like Technology-Assisted Review (TAR) for medical systematic reviews, standard active learning methods often reach a performance plateau—finding 90% of relevant documents easily but requiring exhaustive manual effort to identify the final 10%.

Furthermore, recommender systems struggle with "cold-start" scenarios where no historical data exists for a new user or item. While asking clarifying questions (CQs) is an intuitive solution, the field has lacked robust algorithms to select the most informative questions, automated methods to determine when to stop asking, and empirical data on whether users are actually willing or able to answer these questions accurately.

Algorithmic Frameworks for Clarification

This research proposes three primary algorithmic frameworks to address these challenges:

  1. SBSTAR (Sequential Bayesian Search for TAR): Designed for high-recall document search, this method uses Generalized Binary Search (GBS) over entities to select questions that best "dichotomize" the probability mass of document relevance. It includes noise-tolerant variants and an extension (SBSTARext) that uses a trained classifier to automatically decide when to stop asking questions.
  2. QSBPS (Question-based Sequential Bayesian Product Search): This framework introduces "duet training," which learns both product relevance and the "reward" (informativeness) of potential questions from historical data. It balances information gain with the likelihood that a user can provide a correct answer.
  3. Qrec (Question-based Recommendation): This model integrates CQs with Matrix Factorization (MF). It initializes user and item latent factors offline and uses Alternating Least Squares (ALS) to update those factors in real-time based on user answers, effectively "steering" the recommendation toward the user's current need.

Experimental Setup and Datasets

The proposed models were evaluated using several key datasets and real-world studies:

  • CLEF 2017 e-Health Lab: A collection of ~267k MEDLINE abstracts used to test high-recall document search (TAR).
  • Amazon Product Dataset: Used for product search and recommendation experiments across categories such as "Home & Kitchen" and "Clothing, Shoes & Jewelry."
  • User Interaction Data: The author conducted online user studies with over 150 participants to move beyond simulations and observe real human-computer interaction dynamics.

Key Insights and Results

  • Efficiency in the "Last Mile": CQs are most powerful when standard ranking plateaus. In TAR, SBSTAR reduced the effort required to achieve total recall by 18% to 98% across various stopping points compared to baseline methods.
  • The Quality Threshold: A striking finding from the user studies was that while high-quality CQs improve satisfaction, low or mid-quality CQs are actively harmful, frustrating users more than having no clarification system at all.
  • User Noise and Fatigue: Real users provide incorrect answers approximately 17% of the time, necessitating noise-tolerant algorithms. On average, users are willing to answer about 11 questions before fatigue sets in, though 84% of participants rated the CQ-based system as helpful.
  • Contextual Advantages: Users are more willing to engage with CQs on mobile devices (where scrolling is tedious) and when they perceive the search task to be difficult.

Limitations and Future Work

While the thesis demonstrates the effectiveness of CQ-based systems, several areas for future exploration remain. Current systems rely on explicit entity mentions; future work should explore semantic correlations (e.g., linking a question about "durability" to a product described as "sturdy"). Additionally, moving from template-based questions toward free-form natural language generation could make interactions feel more human. Finally, future models must account for "intent shift" during conversations and integrate privacy-preserving techniques to handle user data ethically.

This paper is based on the doctoral thesis of Jie Zou, awarded by the University of Amsterdam on 6 September 2021, supervised by Evangelos Kanoulas and Maarten de Rijke.

Summary updated

Abstract

Recently, there has been renewed interest in interactive search and recommender systems through the development of systems that ask clarifying questions (CQs) to users to better understand their information needs. Although the significance and effectiveness of CQs have been recongnized, research on modelling CQs to improve search and recommender systems is limited, compared to traditional search and recommender systems. Also, understanding users' willingness or ability to answer these CQs, and the extent to which CQs impact users' behavior and their ability to identify relevant information remains relatively unexplored. The work in this thesis provides a series of models and online user studies for search and recommender systems that construct and use CQs. In the first part of the thesis, we focus on CQ-based search and recommendation algorithms. We first propose a novel CQ-based document search algorithm, Sequential Bayesian Search based method for Technology-Assisted Review (SBSTAR) and its extension, to efficiently retrieve the last few, but significant, relevant documents [Zou et al., 2018; Zou and Kanoulas, 2020]. The algorithm sequentially selects and directly asks CQs to users about the presence or absence of an entity in the missing relevant documents. Next, we propose a novel CQ-based product search algorithm, Question-based Sequential Bayesian Product Search (QSBPS), to effectively locate products that users are looking for [Zou and Kanoulas, 2019]. The model is based on a duet learning framework that learns product relevance and the reward of the potential CQ to be asked to the user. Furthermore, we propose a novel CQ-based recommendation algorithm, Question-based recommendation (Qrec), to assist users to find items interactively [Zou et al., 2020a]. The model is first trained offline by a novel matrix factorization algorithm, and then iteratively updates the user and item latent factors online based on the user's answers. We experimentally test the performance of the proposed models and demonstrate that they outperform state-of-the-art baselines. In the second part of the thesis, we focus on the evaluation of CQ-based search and recommendation. We first conduct an online user study by deploying a CQ-based system in the domain of online retail, to understand to what extent users can answer CQs. We explore the user willingness, ability, and user perception to provide answers to CQs [Zou et al., 2020b]. Next, we conduct a large user study on web search to understand the impact of CQ interactions on user search behavior and satisfaction, and how users interact with different levels of quality of CQs under different user perceptions and conditions. Awarded by: University of Amsterdam, Amsterdam, The Netherlands on 6 September 2021. Supervised by: Evangelos Kanoulas, Maarten de Rijke. Available at: https://hdl.handle.net/11245.1/3c46aa78-98af-4997-aef7-44db7ab2fa3d.
BibTeX
@inproceedings{zou2021improving,
  author = {Jie Zou},
  title = {{Improving search and recommendation by asking clarifying questions}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2021},
  url = {https://openreview.net/forum?id=JE1chxm_2IQ},
}
Figure