A Bayesian Matrix Factorization Model for Relational Data

Ajit Paul Singh, Geoffrey J. GordonUAI 2010 · OpenReview

TL;DR

This paper introduces a hierarchical Bayesian matrix factorization model for relational data, enabling the integration of multiple correlated data sources to improve predictive accuracy. They tackle the computational challenges of full Bayesian inference by developing an efficient Hessian-based Metropolis-Hastings sampler, demonstrating significant improvements in predicting brain responses by augmenting fMRI data with word co-occurrence statistics.

What Problem Does This Paper Address?

In many real-world scenarios, we have multiple, related datasets that offer complementary information. For instance, in neuroscience, researchers want to predict brain activity (measured by fMRI) in response to stimuli. This fMRI data can be augmented with "side information" about the stimuli themselves, such as how often the stimulus word co-occurs with other words in large text corpora. The challenge is to effectively combine these diverse data sources to build more accurate predictive models. Traditional matrix factorization methods often struggle with this, either by making restrictive assumptions about data types (e.g., all Gaussian) or by being computationally prohibitive when trying to capture the full uncertainty in a Bayesian framework. This paper aims to bridge this gap by proposing a flexible, hierarchical Bayesian model that can handle different data types and efficiently learn from multiple relational datasets.

Methods

The core of the paper lies in developing a Hierarchical Bayesian Collective Matrix Factorization (HB-CMF) model. Here's a breakdown of their approach:

  • Collective Matrix Factorization (CMF): They start by representing each relation (e.g., fMRI response, word co-occurrence) as a matrix. These matrices are then jointly factored into low-rank latent representations, allowing shared information between entities (like stimuli) to be captured. This is a generalization of standard matrix factorization.
  • Hierarchical Priors: To allow for more flexible modeling and to learn relationships between different data sources more effectively, they introduce hierarchical priors on the model's parameters. This means the model itself learns the "hyperparameters" of the priors, rather than them being fixed.
  • Bayesian Inference: Instead of just finding a single "best" set of parameters (like in Maximum A Posteriori - MAP estimation), they aim to capture the full posterior distribution of the parameters. This is crucial because it allows for a more robust estimation of uncertainty, especially when dealing with limited data or new entities.
  • Hessian Metropolis-Hastings (HMH): The main computational hurdle is performing full Bayesian inference. Standard random-walk Metropolis-Hastings samplers are too slow. The authors cleverly exploit the bilinear structure of matrix factorization and the local structure of the likelihood to develop an adaptive block Metropolis-Hastings sampler. This sampler uses the gradient and Hessian of the target distribution to dynamically tune its proposal distribution, making the sampling process much more efficient and practical. This is a key technical contribution.

Datasets

The primary dataset used for evaluation is a combination of:

  • fMRI data: Measuring brain responses (voxel activation) to word-picture stimuli. This is represented as the Response relation.
  • Word co-occurrence data: Derived from the Google Tera-word corpus, indicating whether a stimulus word co-occurs with other common words. This is represented as the Co-occurs relation.

They use data from 9 subjects, with 60 stimuli each. The fMRI data is pre-processed to select the 500 most stable voxels.

Key Insights and Ideas

  • The Power of Augmentation: Simply adding side information (word co-occurrence) to the fMRI data significantly improves predictive accuracy for brain responses, even when using a relatively simple baseline model.
  • Bayesian Inference is Crucial for Hierarchy: While hierarchical models offer theoretical advantages, their benefits are only fully realized when using full Bayesian inference (sampling from the posterior). Point estimates (like MAP) fail to capture the necessary uncertainty and don't yield significant improvements.
  • Non-Gaussian Response Types Matter: The ability to model different types of response data (e.g., Bernoulli for co-occurrence, Gaussian for fMRI) is critical for achieving high predictive accuracy. Forcing all data into a Gaussian model degrades performance.
  • Efficient Sampling is Key: The computational cost of full Bayesian inference for matrix factorization is a major bottleneck. The development of the Hessian-based Metropolis-Hastings sampler is a significant step towards making these powerful models practical.

Key Results

  • Significant Predictive Improvement: The HB-CMF model, using both fMRI and word co-occurrence data, achieved a statistically significant improvement in predicting voxel responses compared to using only fMRI data.
  • Hold-out vs. Fold-in: The model performed well on both hold-out (predicting known entities) and fold-in (predicting new entities) tasks. Notably, it achieved high prediction accuracy in fold-in scenarios, even for voxels not seen during training.
  • MAP vs. Bayesian: The fully Bayesian approach (HB-CMF) significantly outperformed the MAP-based hierarchical model (H-CMF) and the baseline CMF, especially in the challenging fold-in scenario.
  • Computational Efficiency: While HB-CMF is computationally more intensive than H-CMF, the developed Hessian-based sampler makes it practical, with training taking on average 7.2s per epoch on a multi-processor setup.

Limitations and Future Work

  • Computational Cost: While their sampler is efficient, full Bayesian inference for large-scale relational data can still be computationally demanding.
  • Model Complexity: The paper focuses on a specific type of hierarchical Bayesian matrix factorization. Extending this to more complex relational structures or different types of priors could be explored.
  • Future Work: The authors suggest that their approach could be applied to other domains where relational data is available. They also highlight the potential for shedding light on the brain's internal representation of concepts by analyzing the learned latent factors.

Summary updated

Abstract

Kubet - Link vào trang chủ nhà cái KU BET mới nhất 2024. Đăng ký ngay Kubet11 nhận 88k khuyến mãi hội viên Kubet mới của chúng tôi.
BibTeX
@inproceedings{singh2010bayesian,
  author = {Ajit Paul Singh and Geoffrey J. Gordon},
  title = {{A Bayesian Matrix Factorization Model for Relational Data}},
  booktitle = {Conference on Uncertainty in Artificial Intelligence},
  year = {2010},
  url = {https://openreview.net/forum?id=PYod3n4Vo7},
}
Figure