A First Look at Selection Bias in Preference Elicitation for Recommendation

Shashank Gupta, Harrie Oosterhuis, Maarten de RijkearXiv 2024 · DOI · arXiv

Abstract

Preference elicitation (PE)—asking users directly about their interests—is a common strategy to mitigate the cold-start problem in recommender systems. However, this paper identifies a critical flaw: selection bias. Users typically provide feedback only on topics they already know or like, leading to skewed data. We demonstrate that standard recommendation models trained on such biased PE data underperform. By applying Inverse Propensity Scoring (IPS), we can effectively debias these systems. Our experiments on real-world and synthetic datasets show that the proposed MF-IPS approach significantly improves recommendation accuracy and ranking performance.

The Hidden Bias in Preference Elicitation

In recommender systems, the "cold-start" problem arises when a system lacks sufficient data to make accurate predictions for new users. Preference Elicitation (PE) addresses this by proactively asking users about their preferences (e.g., "Do you like Sci-Fi?"). While intuitive, PE data is inherently biased. Users are more likely to respond to prompts about popular topics or genres they are familiar with, a phenomenon known as selection bias.

Visualization of selection bias in rating and genre distributions.

As shown in the figure above, there is a clear mismatch between the distribution of ratings users choose to provide (Train) and a truly random sample of their preferences (Test). This bias propagates from the item level to the topic level, potentially trapping the recommender system in an "echo chamber" that ignores the long tail of user interests.

Methodology: Simulation and Debiasing

Since real-world datasets capturing human-system elicitation interactions are scarce, this research introduces a PE Simulator. This tool aggregates item-level ratings from static datasets into "topic" ratings to mimic user responses to category-level questions. For datasets without explicit genre labels, the authors employ Bipartite Network Embedding (BINE) and Gaussian Mixture Models (GMM) to cluster items into synthetic topics.

To counter selection bias, the authors adapt Inverse Propensity Scoring (IPS). IPS weights observed interactions by the inverse of their observation probability. By giving more weight to rare or unpopular interactions, IPS creates an unbiased estimator of the ideal loss function, allowing the model to learn a more balanced representation of user preferences.

Experimental Evaluation and Results

The study evaluated three models: standard Matrix Factorization (MF), a generative model (ExpoMF), and the debiased MF-IPS. Tests were conducted on the Yahoo! R3 music dataset, the Coat dataset, and a fully synthetic dataset designed to control bias levels ($\alpha$).

Performance metrics on the Yahoo! R3 dataset comparing MF, ExpoMF, and MF-IPS.

On the Yahoo! R3 dataset, the MF-IPS approach demonstrated substantial improvements. As shown in the table above, for 25 clusters, MF-IPS reduced the Mean Absolute Error (MAE) from 1.3041 to 0.8327 and the Mean Squared Error (MSE) from 2.5634 to 1.0832. Furthermore, MF-IPS consistently achieved higher NDCG@3 scores, indicating superior ranking quality. In synthetic experiments, the performance gap between MF-IPS and naive models widened as the degree of selection bias increased, highlighting the robustness of the IPS approach.

Conclusion

This work provides a foundational look at selection bias within the context of preference elicitation. The results confirm that ignoring such bias leads to significant performance degradation. The proposed IPS-based debiasing framework offers a powerful solution, particularly in the sparse data environments typical of conversational AI. Future research will focus on joint debiasing methods for both elicitation and recommendation stages and validating these findings in live conversational systems with real users.

Summary updated

Abstract

Preference elicitation explicitly asks users what kind of recommendations they would like to receive. It is a popular technique for conversational recommender systems to deal with cold-starts. Previous work has studied selection bias in implicit feedback, e.g., clicks, and in some forms of explicit feedback, i.e., ratings on items. Despite the fact that the extreme sparsity of preference elicitation interactions make them severely more prone to selection bias than natural interactions, the effect of selection bias in preference elicitation on the resulting recommendations has not been studied yet. To address this gap, we take a first look at the effects of selection bias in preference elicitation and how they may be further investigated in the future. We find that a big hurdle is the current lack of any publicly available dataset that has preference elicitation interactions. As a solution, we propose a simulation of a topic-based preference elicitation process. The results from our simulation-based experiments indicate (i) that ignoring the effect of selection bias early in preference elicitation can lead to an exacerbation of overrepresentation in subsequent item recommendations, and (ii) that debiasing methods can alleviate this effect, which leads to significant improvements in subsequent item recommendation performance. Our aim is for the proposed simulator and initial results to provide a starting point and motivation for future research into this important but overlooked problem setting.
BibTeX
@misc{gupta2024first,
  author = {Shashank Gupta and Harrie Oosterhuis and Maarten de Rijke},
  title = {{A First Look at Selection Bias in Preference Elicitation for Recommendation}},
  year = {2024},
  eprint = {2405.00554},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2405.00554},
}
Figure