Robust Bayesian Matrix Factorisation

Balaji Lakshminarayanan, Guillaume Bouchard, Cédric ArchambeauAISTATS 2011 · OpenReview

Abstract

Standard Probabilistic Matrix Factorisation (PMF) assumes constant error variance across all users and items. This paper proposes a Robust Bayesian Matrix Factorisation framework that accounts for heteroscedasticity—varying noise levels per user and item—and utilizes Student-t priors to handle outliers in latent features. Using variational inference, the authors demonstrate that this approach significantly improves predictive accuracy and likelihood on real-world recommendation datasets.

Introduction

Most recommendation systems rely on Probabilistic Matrix Factorisation (PMF), which assumes that the difference between a predicted rating and an actual rating follows a standard Gaussian distribution with a constant variance. However, in real-world scenarios, this "one-size-fits-all" noise model is fundamentally flawed. Some users are consistent and easy to predict, while others are erratic. Similarly, some items, like "cult classics," may polarize audiences more than mainstream blockbusters. This phenomenon is known as heteroscedasticity, where the variance of noise changes across different rows (users) and columns (items) of the data matrix.

Methodology

The authors propose a hierarchical Bayesian model to address these limitations through two main innovations:

  1. Heteroscedastic Noise Model: Instead of a single global noise parameter, the model employs a Gaussian Scale Mixture. It introduces two scaling factors for every observation: one for the user ($\alpha_n$) and one for the item ($\beta_m$). This allows the model to learn which users are "noisy" and which items are "unpredictable."
  2. Student-t Priors: To handle outliers in the latent feature space, standard Gaussian priors are replaced with Student-t distributions. This makes the model more robust to atypical users or items that deviate from general population trends.
Graphical model of the proposed framework.

To ensure computational feasibility for large datasets, the authors derive Variational Inference algorithms (specifically a "Structured Variational" approach). This provides a deterministic approximation of the full Bayesian posterior, offering a faster alternative to traditional MCMC sampling.

Empirical Evidence and Insights

A key discovery of this research is the empirical proof that standard PMF fails to capture the true distribution of variances. By plotting histograms of residuals, the authors demonstrated that a homoscedastic model significantly underestimates the spread of real-world noise.

Observed vs. predicted residual variance.

Several key insights emerged from the study:

  • Synergistic Scaling: Modeling variance at both the user and item levels simultaneously provides a much more accurate fit than modeling either in isolation.
  • Inference Strategy: "Structured" variational inference, which preserves certain dependencies between variables, outperforms simpler "Mean Field" approaches when dealing with complex heteroscedastic noise.
  • Nature of Robustness: While Student-t distributions are typically used to ignore outliers, in this context, they better represent the inherent "peakedness" and heavy tails of user preference distributions.

Experimental Results

The models were evaluated on MovieLens (100k, 1M, 10M) and Jester Joke datasets. Four model variants were compared: GG (Gaussian noise/priors), RG (Robust noise/Gaussian priors), GR (Gaussian noise/Robust priors), and RR (Robust noise/Robust priors).

  • Predictive Accuracy: On MovieLens datasets, heteroscedastic models (RG and RR) consistently achieved lower Root Mean Squared Error (RMSE) and Mean Absolute Error (MAE) compared to the standard GG model.
  • Likelihood: The Ordinal Log-Likelihood (OLL) improved significantly, indicating that robust models are superior at estimating the probability of specific star ratings.
  • Dataset Sensitivity: Student-t priors were particularly beneficial for the Jester dataset, which features continuous ratings, suggesting that the requirement for robustness varies by data collection method.

Conclusion

Robust Bayesian Matrix Factorisation offers a more realistic framework for collaborative filtering by accounting for individual user and item variance. While the model introduces additional hyperparameters and computational complexity (scaling at $O(K^3)$ for latent factors), the gains in predictive accuracy and likelihood estimation make it a compelling alternative to standard PMF. Future work may explore dynamic noise models where user consistency evolves over time.

Summary updated

BibTeX
@inproceedings{lakshminarayanan2011robust,
  author = {Balaji Lakshminarayanan and Guillaume Bouchard and Cédric Archambeau},
  title = {{Robust Bayesian Matrix Factorisation}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2011},
  url = {https://openreview.net/forum?id=SMZiGXkyQq},
}
Figure