Bayesian Mean-parameterized Nonnegative Binary Matrix Factorization.

Alberto Lumbreras, Louis Filstroff, Cédric FévottearXiv 2018 · OpenReview

TL;DR

This paper introduces a new framework for Bayesian Nonnegative Matrix Factorization (NMF) specifically designed for binary data. By enforcing a "mean-parameterized" model, they ensure that the learned factors are directly interpretable as probabilities, overcoming limitations of previous methods that used link functions. They also develop efficient inference algorithms and a nonparametric extension that automatically determines the number of latent components.

What Problem Does This Paper Address?

Analyzing binary data, like social networks or gene expression, often involves approximating a binary matrix $V$ with the product of two smaller nonnegative matrices, $W$ and $H$ ($V \approx WH$). This is the core idea of Nonnegative Matrix Factorization (NMF). When modeling binary data probabilistically, a common approach is to assume a Bernoulli distribution for the data, where the probability of observing a '1' is determined by the product $WH$. However, to ensure this product stays within the valid probability range $[0, 1]$, researchers often use "link functions" (like the logistic function) to map $WH$ to this range. The problem is that these link functions make the direct interpretation of $W$ and $H$ as probabilities difficult, hindering model interpretability. This paper addresses this by proposing "mean-parameterized" NMF models where the expected value of the binary observation is directly the product $WH$, thus maintaining interpretability without sacrificing the probabilistic framework.

Methods

The authors propose a unified framework for Bayesian Mean-parameterized Nonnegative Binary Matrix Factorization (NBMF). They introduce three specific models by imposing different constraints on the factors $W$ and $H$ to ensure their product remains a valid probability. For inference, they develop novel collapsed Gibbs samplers and collapsed variational inference algorithms. These methods are "collapsed" because they marginalize out some latent variables, leading to more efficient and better-mixing Markov chains for Gibbs sampling and potentially faster convergence for variational inference. A key contribution is the extension of these models to a nonparametric setting, where the number of latent dimensions ($K$) is not fixed beforehand but is automatically inferred from the data. This is achieved by using techniques inspired by the Chinese Restaurant Process.

Datasets

The paper evaluates their proposed methods on five public datasets:

  • Animals: Binary attributes for animals.
  • Last.fm: Binarized user-artist listening data.
  • Paleontological data (paleo): Fossil presence in locations.
  • Catalan parliament (parliament): Twitter follower relationships between members.
  • UN votes (unvotes): Country voting records in the UN General Assembly.

No new benchmarks were introduced, but the datasets were preprocessed to fit the binary matrix factorization context.

Key Insights and Ideas

  • Interpretability through Mean-Parameterization: The core idea is that by directly modeling $E[V|WH] = WH$, the factors $W$ and $H$ themselves can be interpreted as probabilities, which is crucial for understanding the underlying structure in binary data.
  • Collapsed Inference for Efficiency: Developing collapsed Gibbs and variational inference algorithms significantly improves the efficiency and mixing properties of the inference process, especially for large datasets.
  • Automatic Rank Selection (Nonparametric Extension): The nonparametric extension is a major advantage, as it removes the need to pre-specify the number of latent components ($K$), which is often a difficult hyperparameter to tune. This is achieved through a mechanism that allows components to "prune" themselves if they are not supported by the data.
  • Model Flexibility vs. Interpretability Trade-off: The paper highlights how more flexible models like logPCA (which uses unconstrained factors and a link function) can achieve better data approximation (lower perplexity) but at the cost of interpretability, whereas their mean-parameterized models offer a better balance.

Key Results

  • The proposed NBMF methods (Beta-Dir GS, Beta-Dir VB, Dir-Dir GS) achieve competitive or superior performance compared to state-of-the-art methods like logPCA and bICA on various tasks, including dictionary learning and missing data prediction.
  • In dictionary learning experiments, the mean-parameterized models (especially Beta-Dir GS and VB) produced more interpretable dictionaries that clearly reflected underlying structures (e.g., political parties, voting blocks) compared to logPCA.
  • The nonparametric extension demonstrated its ability to automatically detect the relevant number of components, performing comparably to parametric methods that required manual tuning of $K$.
  • The collapsed inference algorithms showed stable convergence and good prediction performance across different datasets.

Limitations and Future Work

  • Computational Cost: While efficient, the collapsed algorithms still have a computational cost of $O(FKN)$, and unlike some Poisson models, zeros in binary data cannot be ignored as they represent a distinct category.
  • Dir-Dir Model Complexity: The paper notes that obtaining a collapsed variational algorithm for the Dir-Dir model is not straightforward and is left for future work.
  • Nonparametric Challenges: While the nonparametric extension is promising, the paper mentions that achieving exact inference in the truly nonparametric model with an infinite number of components is challenging and might require more advanced techniques like the Chinese Restaurant Franchise process.
  • Further Model Extensions: The paper suggests that their framework could be extended to more complex generative models, such as deep generative networks, for tasks like recommendation systems.

Summary updated

Abstract

Binary data matrices can represent many types of data such as social networks, votes, or gene expression. In some cases, the analysis of binary matrices can be tackled with nonnegative matrix factorization (NMF), where the observed data matrix is approximated by the product of two smaller nonnegative matrices. In this context, probabilistic NMF assumes a generative model where the data is usually Bernoulli-distributed. Often, a link function is used to map the factorization to the $[0,1]$ range, ensuring a valid Bernoulli mean parameter. However, link functions have the potential disadvantage to lead to uninterpretable models. Mean-parameterized NMF, on the contrary, overcomes this problem. We propose a unified framework for Bayesian mean-parameterized nonnegative binary matrix factorization models (NBMF). We analyze three models which correspond to three possible constraints that respect the mean-parametrization without the need for link functions. Furthermore, we derive a novel collapsed Gibbs sampler and a collapsed variational algorithm to infer the posterior distribution of the factors. Next, we extend the proposed models to a nonparametric setting where the number of used latent dimensions is automatically driven by the observed data. We analyze the performance of our NBMF methods in multiple datasets for different tasks such as dictionary learning and prediction of missing data. Experiments show that our methods provide similar or superior results than the state of the art, while automatically detecting the number of relevant components.
BibTeX
@misc{lumbreras2018bayesian,
  author = {Alberto Lumbreras and Louis Filstroff and Cédric Févotte},
  title = {{Bayesian Mean-parameterized Nonnegative Binary Matrix Factorization.}},
  year = {2018},
  eprint = {1812.06866},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1812.06866},
}
Figure