Polya-gamma augmentations for factor models.

Arto KlamiACML 2014 · OpenReview

Abstract

Bayesian inference for latent factor models, such as principal component and canonical correlation analysis, is straightforward for Gaussian likelihoods. In particular, full conjugacy makes both Gibbs sampling and variational inference efficient. However, for non-Gaussian data—such as binary or count data—the lack of conjugacy typically necessitates slow sampling methods or complex, potentially inaccurate approximations. This paper introduces an efficient inference framework for non-Gaussian latent factor models by leveraging Pòlya-Gamma (PG) augmentation. By introducing auxiliary variables, we transform difficult non-conjugate likelihoods into a form that restores conjugacy, allowing for fast, closed-form updates that scale to larger datasets.

Introduction

Latent factor models, including Principal Component Analysis (PCA) and Canonical Correlation Analysis (CCA), are workhorses of modern data science. In a Bayesian context, these models are highly effective when data is continuous and Gaussian because the mathematical properties of conjugacy allow for simple, closed-form parameter updates.

However, real-world data is frequently discrete, taking the form of binary indicators (e.g., yes/no) or counts. For these data types, researchers typically employ logistic or negative binomial likelihoods. These choices break the conjugacy of the model, forcing a difficult choice between two suboptimal paths: slow sampling algorithms like Hamiltonian Monte Carlo (HMC) that struggle with high-dimensional latent spaces, or complex variational bounds (like the Bohning bound) that can be inaccurate for extreme probabilities.

The Pòlya-Gamma Augmentation Trick

The core innovation of this work is the application of the Pòlya-Gamma (PG) augmentation to the matrix factorization framework. The PG trick involves introducing an auxiliary random variable ($\omega$) into the model. When conditioned on this new variable, the previously intractable logistic or negative binomial likelihood takes on a Gaussian form relative to the latent factors. This "restores" conjugacy, allowing researchers to utilize the same efficient machinery developed for standard Gaussian PCA while maintaining the appropriate likelihood for discrete data.

Architectural Approaches to Factorization

The paper explores two distinct structural approaches to implementing this augmentation:

  1. The Direct Approach: This model ties the observed data directly to the low-rank factors. While highly accurate, it can be computationally demanding because it requires the inversion of a $K \times K$ matrix (where $K$ is the number of factors) for every data point in the matrix.
  2. The Explicit Noise Approach: This model introduces an intermediate latent variable with Gaussian noise between the factors and the data. This allows for faster individual iterations but may require more iterations to reach convergence because the coupling between the data and the factors is less direct.

Inference Flavors: Gibbs vs. Variational Bayes

The framework supports both Gibbs samplers for high-fidelity posterior estimation and Variational Bayes (VB) for rapid approximation. To address the computational bottleneck of matrix inversions in large-scale settings, the author proposes a "fully factorized" VB approach. By utilizing gradient-based updates, this method avoids repeated inversions, providing a significant speedup for high-dimensional matrices.

Empirical Evaluation

The methods were evaluated using synthetic datasets to allow for comparison against a known "ground truth."

  • Binary Data: The models were tested across "easy" (balanced) and "hard" (imbalanced or extreme probability) scenarios. The PG-based methods significantly outperformed the state-of-the-art Bohning bound in imbalanced cases, reducing mean error from 0.49 to 0.32.
  • Count Data: Using Poisson and Negative Binomial distributions, the PG-Direct variants proved robust, particularly in high-count scenarios where traditional Gaussian approximations often fail.

In terms of speed, the proposed factorized variational method (PG-Direct-F) reached high accuracy in seconds, whereas traditional samplers required minutes or hours to achieve comparable results.

Conclusion and Future Work

Pòlya-Gamma augmentation effectively bridges the gap between the flexibility of non-Gaussian likelihoods and the efficiency of Gaussian inference. By regaining conjugacy, we can treat binary and count factor analysis with nearly the same computational ease as standard PCA.

While the "Direct" Gibbs sampler remains computationally expensive for massive datasets due to the inversion bottleneck, the factorized variational approach offers a viable path forward for large-scale applications. Future research may extend these PG techniques to more complex structures, such as tensor factorizations for multi-dimensional arrays or models incorporating missing covariates.

Summary updated

Abstract

Bayesian inference for latent factor models, such as principal component and canonical correlation analysis, is easy for Gaussian likelihoods. In particular, full conjugacy makes both Gibbs sampler...
BibTeX
@inproceedings{klami2014polya,
  author = {Arto Klami},
  title = {{Polya-gamma augmentations for factor models.}},
  booktitle = {ACML},
  year = {2014},
  url = {https://openreview.net/forum?id=O62PPYJLLu},
}
Figure