Abstract
Recently, there has been renewed interest in interactive search and recommender systems through the development of systems that ask clarifying questions (CQs) to users to better understand their information needs. Although the significance and effectiveness of CQs have been recognized, research on modelling CQs to improve search and recommender systems is limited compared to traditional systems. Furthermore, understanding users' willingness or ability to answer these CQs, and the extent to which they impact user behavior, remains relatively unexplored. This work provides a series of models and online user studies for search and recommender systems that construct and use CQs. We propose novel algorithms for document search (SBSTAR), product search (QSBPS), and general recommendation (Qrec), demonstrating that they outperform state-of-the-art baselines. Additionally, we conduct large-scale user studies to evaluate user perception and interaction quality in real-world scenarios.
Introduction: The Semantic Gap in Retrieval
Traditional search and recommendation systems often suffer from a "semantic gap." Users frequently provide short, ambiguous, or faceted queries that do not fully capture their underlying intent. In specialized fields like Technology-Assisted Review (TAR) for medical systematic reviews, standard active learning methods often reach a performance plateau—finding 90% of relevant documents easily but requiring exhaustive manual effort to identify the final 10%.
Furthermore, recommender systems struggle with "cold-start" scenarios where no historical data exists for a new user or item. While asking clarifying questions (CQs) is an intuitive solution, the field has lacked robust algorithms to select the most informative questions, automated methods to determine when to stop asking, and empirical data on whether users are actually willing or able to answer these questions accurately.
Algorithmic Frameworks for Clarification
This research proposes three primary algorithmic frameworks to address these challenges:
- SBSTAR (Sequential Bayesian Search for TAR): Designed for high-recall document search, this method uses Generalized Binary Search (GBS) over entities to select questions that best "dichotomize" the probability mass of document relevance. It includes noise-tolerant variants and an extension (SBSTARext) that uses a trained classifier to automatically decide when to stop asking questions.
- QSBPS (Question-based Sequential Bayesian Product Search): This framework introduces "duet training," which learns both product relevance and the "reward" (informativeness) of potential questions from historical data. It balances information gain with the likelihood that a user can provide a correct answer.
- Qrec (Question-based Recommendation): This model integrates CQs with Matrix Factorization (MF). It initializes user and item latent factors offline and uses Alternating Least Squares (ALS) to update those factors in real-time based on user answers, effectively "steering" the recommendation toward the user's current need.
Experimental Setup and Datasets
The proposed models were evaluated using several key datasets and real-world studies:
- CLEF 2017 e-Health Lab: A collection of ~267k MEDLINE abstracts used to test high-recall document search (TAR).
- Amazon Product Dataset: Used for product search and recommendation experiments across categories such as "Home & Kitchen" and "Clothing, Shoes & Jewelry."
- User Interaction Data: The author conducted online user studies with over 150 participants to move beyond simulations and observe real human-computer interaction dynamics.
Key Insights and Results
- Efficiency in the "Last Mile": CQs are most powerful when standard ranking plateaus. In TAR, SBSTAR reduced the effort required to achieve total recall by 18% to 98% across various stopping points compared to baseline methods.
- The Quality Threshold: A striking finding from the user studies was that while high-quality CQs improve satisfaction, low or mid-quality CQs are actively harmful, frustrating users more than having no clarification system at all.
- User Noise and Fatigue: Real users provide incorrect answers approximately 17% of the time, necessitating noise-tolerant algorithms. On average, users are willing to answer about 11 questions before fatigue sets in, though 84% of participants rated the CQ-based system as helpful.
- Contextual Advantages: Users are more willing to engage with CQs on mobile devices (where scrolling is tedious) and when they perceive the search task to be difficult.
Limitations and Future Work
While the thesis demonstrates the effectiveness of CQ-based systems, several areas for future exploration remain. Current systems rely on explicit entity mentions; future work should explore semantic correlations (e.g., linking a question about "durability" to a product described as "sturdy"). Additionally, moving from template-based questions toward free-form natural language generation could make interactions feel more human. Finally, future models must account for "intent shift" during conversations and integrate privacy-preserving techniques to handle user data ethically.
This paper is based on the doctoral thesis of Jie Zou, awarded by the University of Amsterdam on 6 September 2021, supervised by Evangelos Kanoulas and Maarten de Rijke.
Summary updated