Recommender Systems, Missing Data and Statistical Model Estimation

Benjamin M. Marlin, Richard S. Zemel, Sam T. Roweis, Malcolm Slaney

Introduction

Collaborative filtering systems typically operate on sparse user-item rating matrices. A common but often unexamined assumption in these systems is that data is Missing at Random (MAR). Under MAR, the probability that a rating is missing is independent of the rating's value. However, empirical evidence suggests that users do not rate items randomly; they primarily rate items they have chosen to consume based on prior preferences. This leads to a Missing Not at Random (MNAR) scenario, where selection bias significantly skews the observed data.

Methodology: Modeling the Observation Process

To address the MNAR problem, the authors propose incorporating an explicit model of the missing data process into the recommendation framework.

1. The Data Model

The base model is a Multinomial Mixture Model (MMM), which assumes users belong to latent classes with distinct preference distributions.

2. Missing Data Models

Two approaches are introduced to model the probability of an item being rated:

  • CPT-v (Conditional Probability Table): This models the probability of a rating existing as a function of the rating value itself. It captures the "apathy effect," where users are more likely to rate items they feel strongly about (highly positive or highly negative) and less likely to rate neutral items.
  • Logit-vd: This extends the model using a logistic function that incorporates both the rating value and item-specific popularity factors, accounting for the fact that famous items are more frequently rated regardless of individual preference.

Empirical Analysis of Selection Bias

The authors analyze several datasets to demonstrate the prevalence of selection bias. In datasets like EachMovie, MovieLens, and Netflix, ratings are heavily skewed toward higher values.

YouTube Rating Distribution showing extreme positive bias.

As shown in the YouTube dataset, nearly 90% of ratings are five stars. This suggests that users almost exclusively rate content they enjoy, leaving the system with little information about what they dislike. To validate their models, the authors utilized a unique Yahoo! Music dataset containing both user-selected ratings and ratings for items selected uniformly at random.

Yahoo! Random Rating Distribution showing the true underlying preference for random items.

The distribution of random ratings reveals a starkly different reality: when forced to rate random items, users provide far more low ratings. Standard models trained on biased user-selected data fail to generalize to this "ground truth" distribution because they assume the missing data is representative of the whole.

Results and Discussion

The proposed MNAR-aware models (MM/Logit-vd and MM/CPT-v) were compared against standard baselines such as Matrix Factorization and Item-based K-Nearest Neighbors.

  • Prediction Accuracy: On the unbiased Yahoo! Random test set, the MNAR models significantly outperformed standard MAR-based models. Standard models tended to over-predict high ratings, failing to account for the fact that unrated items are likely to be disliked.
  • Ranking Performance: The MNAR models achieved higher Normalized Discounted Cumulative Gain (NDCG), effectively placing preferred items higher in recommendation lists by leveraging the "absence of evidence" as evidence of low preference.

Conclusion

This research highlights that the MAR assumption is a dangerous simplification for recommender systems. By explicitly modeling the missingness mechanism, we can mitigate selection bias and provide more robust, personalized recommendations. Future work includes exploring dependencies between missing ratings and incorporating auxiliary data like content features or social signals.

Summary updated

Abstract

The goal of rating-based recommender systems is to make personalized predictions and recommendations for individual users by leveraging the preferences of a community of users with respect to a collection of items like songs or movies. Recommender systems are often based on intricate statistical models that are estimated from data sets containing a very high proportion of missing ratings. This work describes evidence of a basic incompatibility between the properties of recommender system data sets and the assumptions required for valid estimation and evaluation of statistical models in the presence of missing data. We discuss the implications of this problem and describe extended modelling and evaluation frameworks that attempt to circumvent it. We present prediction and ranking results showing that models developed and tested under these extended frameworks can significantly outperform standard models.
BibTeX
@inproceedings{marlin2011recommender,
  author = {Benjamin M. Marlin and Richard S. Zemel and Sam T. Roweis and Malcolm Slaney},
  title = {{Recommender Systems, Missing Data and Statistical Model Estimation}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2011},
  doi = {10.5591/978-1-57735-516-8/ijcai11-447},
}
Figure