Abstract
Preference elicitation (PE)—asking users directly about their interests—is a common strategy to mitigate the cold-start problem in recommender systems. However, this paper identifies a critical flaw: selection bias. Users typically provide feedback only on topics they already know or like, leading to skewed data. We demonstrate that standard recommendation models trained on such biased PE data underperform. By applying Inverse Propensity Scoring (IPS), we can effectively debias these systems. Our experiments on real-world and synthetic datasets show that the proposed MF-IPS approach significantly improves recommendation accuracy and ranking performance.
The Hidden Bias in Preference Elicitation
In recommender systems, the "cold-start" problem arises when a system lacks sufficient data to make accurate predictions for new users. Preference Elicitation (PE) addresses this by proactively asking users about their preferences (e.g., "Do you like Sci-Fi?"). While intuitive, PE data is inherently biased. Users are more likely to respond to prompts about popular topics or genres they are familiar with, a phenomenon known as selection bias.
As shown in the figure above, there is a clear mismatch between the distribution of ratings users choose to provide (Train) and a truly random sample of their preferences (Test). This bias propagates from the item level to the topic level, potentially trapping the recommender system in an "echo chamber" that ignores the long tail of user interests.
Methodology: Simulation and Debiasing
Since real-world datasets capturing human-system elicitation interactions are scarce, this research introduces a PE Simulator. This tool aggregates item-level ratings from static datasets into "topic" ratings to mimic user responses to category-level questions. For datasets without explicit genre labels, the authors employ Bipartite Network Embedding (BINE) and Gaussian Mixture Models (GMM) to cluster items into synthetic topics.
To counter selection bias, the authors adapt Inverse Propensity Scoring (IPS). IPS weights observed interactions by the inverse of their observation probability. By giving more weight to rare or unpopular interactions, IPS creates an unbiased estimator of the ideal loss function, allowing the model to learn a more balanced representation of user preferences.
Experimental Evaluation and Results
The study evaluated three models: standard Matrix Factorization (MF), a generative model (ExpoMF), and the debiased MF-IPS. Tests were conducted on the Yahoo! R3 music dataset, the Coat dataset, and a fully synthetic dataset designed to control bias levels ($\alpha$).
On the Yahoo! R3 dataset, the MF-IPS approach demonstrated substantial improvements. As shown in the table above, for 25 clusters, MF-IPS reduced the Mean Absolute Error (MAE) from 1.3041 to 0.8327 and the Mean Squared Error (MSE) from 2.5634 to 1.0832. Furthermore, MF-IPS consistently achieved higher NDCG@3 scores, indicating superior ranking quality. In synthetic experiments, the performance gap between MF-IPS and naive models widened as the degree of selection bias increased, highlighting the robustness of the IPS approach.
Conclusion
This work provides a foundational look at selection bias within the context of preference elicitation. The results confirm that ignoring such bias leads to significant performance degradation. The proposed IPS-based debiasing framework offers a powerful solution, particularly in the sparse data environments typical of conversational AI. Future research will focus on joint debiasing methods for both elicitation and recommendation stages and validating these findings in live conversational systems with real users.
Summary updated