Beta-Negative Binomial Process and Poisson Factor Analysis

Mingyuan Zhou, Lauren Hannah, David B. Dunson, Lawrence CarinAISTATS 2012 · OpenReview

Abstract

This paper introduces the Beta-Negative Binomial (BNB) process, a novel nonparametric Bayesian prior designed for multivariate count data. By generalizing the Indian Buffet Process (IBP) to allow for "multiple scoops" of each dish, the authors develop a flexible Poisson Factor Analysis (PFA) model. This framework effectively handles the overdispersion common in real-world count datasets and automatically learns the optimal number of latent topics. Experimental results on text corpora demonstrate that the BNB process achieves superior predictive performance and robustness compared to traditional models like Latent Dirichlet Allocation (LDA).

Introduction

Count data, such as word frequencies in documents or event counts in clinical records, is ubiquitous in data science. However, standard factor analysis tools often rely on Gaussian distributions, which are ill-suited for discrete, non-negative, and frequently "overdispersed" data—where the variance significantly exceeds the mean.

While the Indian Buffet Process (IBP) provides a powerful nonparametric framework for latent feature modeling, it is inherently binary: a feature is either present or absent. In many applications, the intensity of a feature is as critical as its presence. The BNB process addresses this gap by extending the "buffet" analogy to a multi-scoop setting, providing a more nuanced representation of latent structures in count data.

The Beta-Negative Binomial Process

The core innovation of this work is the "multi-scoop" buffet analogy. In a standard IBP, customers (data points) choose dishes (features) in a binary fashion. In the BNB process, each customer takes a variable number of scoops of each dish.

Technically, this is modeled using a Negative Binomial distribution. Unlike the Poisson distribution, which constrains the mean and variance to be equal, the Negative Binomial distribution possesses two parameters, allowing the model to decouple mean and variance. This flexibility is essential for capturing the "bursty" nature of real-world counts. The authors formulate this as a beta-gamma-gamma-Poisson hierarchical structure, creating a robust framework for Poisson Factor Analysis (PFA).

Poisson Factor Analysis and Inference

In the PFA setting, a count matrix $X$ is factorized into a factor loading matrix $\Phi$ (representing topics) and a factor score matrix $\Theta$ (representing document-specific topic intensities). The BNB process serves as a prior for $\Theta$.

To perform inference, the authors utilize a finite approximation of the underlying Lévy random measure and employ Markov Chain Monte Carlo (MCMC) sampling. By using clever data augmentation and marginalization techniques, the model remains computationally feasible, allowing for efficient learning of both the latent features and the appropriate model complexity.

Experimental Evaluation

The model was evaluated on two distinct text datasets:

  1. JACM: 536 abstracts from the Journal of the ACM, including common "stop words" to test robustness.
  2. PsyRev: 1,281 abstracts from Psychological Review, with stop words removed.

Key Results: Perplexity and Sparsity

The BNB-based model (denoted as $\beta\gamma\Gamma^-$ in the figures) consistently outperformed Latent Dirichlet Allocation (LDA) and Non-negative Matrix Factorization (NMF) in terms of held-out perplexity, a standard metric for predictive accuracy in topic modeling.

Perplexity comparison across different models on JACM and PsyReview datasets.

A significant advantage of the BNB process is its ability to automatically determine the number of active factors. As shown in the comparison against $K$, the BNB model remains stable and achieves lower perplexity regardless of the initial maximum number of factors, whereas LDA is highly sensitive to the choice of $K$.

Perplexity as a function of the number of factors K, showing the stability and automatic complexity control of the BNB-based model.

Robustness to Noise

One of the most striking findings was the model's handling of the JACM dataset. Despite the presence of stop words, the BNB process naturally assigned common words like "the" and "and" to a few high-variance factors. This effectively "filtered" the noise, allowing the remaining factors to capture specific, meaningful research topics without manual preprocessing.

Conclusion

The Beta-Negative Binomial process offers a powerful and flexible alternative to binary nonparametric priors for count data. By accounting for overdispersion and providing automatic complexity control, it simplifies the modeling of complex datasets. Future work may extend this "marked" beta process to other data types, potentially broadening the scope of feature-based Bayesian modeling.

Summary updated

Abstract

A beta-negative binomial (BNB) process is proposed, leading to a beta-gamma-Poisson process, which may be viewed as a “multi-scoop” generalization of the beta-Bernoulli process. The BNB process is ...
BibTeX
@inproceedings{zhou2012beta,
  author = {Mingyuan Zhou and Lauren Hannah and David B. Dunson and Lawrence Carin},
  title = {{Beta-Negative Binomial Process and Poisson Factor Analysis}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2012},
  url = {https://openreview.net/forum?id=0jdGK3jcnQE},
}
Figure