Scalable Collaborative Bayesian Preference Learning

Mohammad Emtiyaz Khan, Young-Jun Ko, Matthias W. SeegerAISTATS 2014 · OpenReview

Abstract

Traditional Bayesian preference learning models often struggle with scalability due to the complex dependencies inherent in pairwise comparison data. This paper presents a scalable Bayesian framework that integrates matrix factorization with Gaussian Processes (GP) to learn user preferences effectively. By employing a novel Expectation-Maximization (EM) algorithm powered by parallelized Expectation Propagation (EP), the proposed GPVU model achieves high accuracy and scales to datasets with millions of observations, bridging the gap between sophisticated Bayesian modeling and large-scale recommendation tasks.

Introduction

In recommendation systems, implicit feedback—such as a user choosing one item over another—is often more abundant and reflective of true utility than explicit ratings. However, learning from these pairwise preferences is computationally demanding. Standard Bayesian inference becomes expensive because each observation couples two items, creating intricate dependencies. This work addresses the "scalability wall" by introducing a hybrid model and an efficient inference algorithm capable of handling large-scale real-world datasets like MovieLens 10M.

Methodology: The GPVU Framework

The authors propose the GPVU (Gaussian Process + VU) model, which decomposes a user's utility for an item into three components:

  1. Matrix Factorization (VU): Captures latent collaborative patterns between users and items.
  2. Gaussian Process (GP): Leverages item features (metadata) to predict utility, providing a robust "safety net" for cold-start scenarios where interaction data is sparse.
  3. Item Bias ($\mu$): Accounts for global item popularity.

To make this model scalable, a specialized EM algorithm is derived. The E-Step uses parallelized Expectation Propagation (EP) to approximate the non-Gaussian posterior, scaling linearly with the number of users. The M-Step employs a closed-form update inspired by Probabilistic PCA, using iterative matrix-vector multiplications to avoid the $O(M^3)$ cost of full matrix inversions, making it feasible for millions of items.

Key Results and Scalability

The GPVU model consistently outperforms standalone Matrix Factorization (VU) and GP models.

Performance comparison of GPVU vs VU showing error reduction over time and sensitivity to hyperparameters.

As shown in Figure 1(a), GPVU achieves lower error rates significantly faster than the baseline VU model. The algorithm's efficiency is further highlighted by its performance on the MovieLens 10M dataset: it processed 3 million pairs in approximately one hour on a 4-core workstation, and just 10 minutes when scaled to 32 cores.

Convergence of error rates across multiple passes for various dataset scales.

Figure 3 demonstrates the model's convergence behavior across various dataset sizes. The proposed EM algorithm exhibits faster and more reliable convergence compared to prior non-parametric methods, which often fail on larger scales.

Discussion

A significant advantage of this Bayesian approach is its ability to quantify uncertainty. As more preference data is collected, the model's confidence increases, which is crucial for risk-aware recommendation strategies. The hybrid nature of GPVU allows it to excel by combining the collaborative strengths of matrix factorization with the feature-aware flexibility of Gaussian Processes.

Conclusion

This work demonstrates that Bayesian preference learning can be scaled to massive datasets without sacrificing accuracy. By decoupling user and item updates and leveraging parallelized inference, the GPVU framework provides a practical solution for large-scale preference-based recommendation systems.

Summary updated

BibTeX
@inproceedings{khan2014scalable,
  author = {Mohammad Emtiyaz Khan and Young-Jun Ko and Matthias W. Seeger},
  title = {{Scalable Collaborative Bayesian Preference Learning}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2014},
  url = {https://openreview.net/forum?id=4AIn1UFHqpk},
}
Figure