Nonparametric Bayesian Factor Analysis for Dynamic Count Matrices

Ayan Acharya, Joydeep Ghosh, Mingyuan ZhouAISTATS 2015 · OpenReview

Abstract

A gamma process dynamic Poisson factor analysis model is proposed to factorize a dynamic count matrix, whose columns are sequentially observed count vectors. The model builds a novel Markov chain that sends the latent gamma random variables at time (t-1) as the shape parameters of those at time t, which are linked to observed or latent counts under the Poisson likelihood. The significant challenge of inferring the gamma shape parameters is fully addressed, using unique data augmentation and marginalization techniques for the negative binomial distribution. The same nonparametric Bayesian model also applies to the factorization of a dynamic binary matrix, via a Bernoulli-Poisson link that connects a binary observation to a latent count, with closed-form conditional posteriors for the latent counts and efficient computation for sparse observations. We apply the model to text and music analysis, with state-of-the-art results.

Introduction

In domains such as text analysis, social network modeling, and music recommendation, data often manifests as a sequence of count matrices—for instance, the frequency of words in documents over successive years. Handling the temporal evolution of such data typically relies on Gaussian-based models like the Kalman filter.

However, Gaussian assumptions are often ill-suited for count data, which are discrete, non-negative, and frequently exhibit overdispersion (where variance significantly exceeds the mean). While log-normal or logistic-normal links can bridge this gap, they often incur high computational costs and obscure the interpretability of latent factors. This research introduces a model that remains "native" to the discrete nature of count data.

Methods: The GP-DPFA Model

The proposed Gamma Process Dynamic Poisson Factor Analysis (GP-DPFA) model introduces several key innovations:

  1. The Gamma Markov Chain: Instead of adding Gaussian noise, the model employs a Gamma distribution where the value at time $t-1$ serves as the shape parameter for the value at time $t$. This ensures latent rates remain positive and evolve smoothly.
  2. Poisson Factor Analysis (PFA): Observed counts are modeled via a Poisson distribution, driven by a "dictionary" matrix of factors and a time-varying "factor score" matrix.
  3. The Bernoulli-Poisson (BePo) Link: For binary data, a link is established that treats a '1' as an indication of at least one latent count. This is highly efficient for sparse matrices as it bypasses zeros during key computations.
  4. Inference via Data Augmentation: To overcome the difficulty of inferring Gamma shape parameters, the authors utilize data augmentation with the Chinese Restaurant Table (CRT) distribution, enabling closed-form Gibbs sampling.
Illustration of the inference process in the GP-DPFA model, showing the transition from the original model (a) through negative binomial (b) and CRT (c) augmentations to the final marginalized state (d).

Experimental Evaluation

The model was evaluated across several diverse datasets:

  • Synthetic Data: 1D time-series including exponential bumps and sinusoids.
  • Coalmine Disasters: A historical 1D time-series of British coal mine accidents (1851–1962).
  • State-of-the-Union (STU): Transcripts of 225 US Presidential addresses (1790–2014).
  • Conference Abstracts: DBLP abstracts for specific authors over 14 years.
  • Polyphonic Music: Binary datasets representing piano sequences (Nottingham, Piano.midi, etc.).

GP-DPFA consistently achieved lower Mean Squared Error (MSE) on synthetic data compared to Sigmoidal Gaussian Cox Processes. In text analysis, it successfully tracked topic evolution over decades, outperforming the Dynamic Rank Factor Model (DRFM) in predictive precision and recall. For music, the latent space revealed structured patterns of progression even from noisy binary data.

Key Insights

  • Shape as History: Using the previous state as a shape parameter enforces temporal smoothness within the Gamma-Poisson framework and naturally handles mean-variance scaling.
  • Nonparametric Flexibility: The Gamma Process allows the model to automatically determine the number of factors ($K$) by shrinking unnecessary factors toward zero.
  • Computational Efficiency: The BePo link makes the model exceptionally scalable for massive, sparse binary matrices by only sampling latent counts for non-zero entries.
  • Interpretability: Strictly non-negative factors are more easily interpreted as "topics" or "musical motifs" compared to Gaussian latent models.

Limitations and Future Work

Despite its strengths, the model faces some challenges. In very long time-series, the variance of the Gamma Markov chain can lead to numerical precision issues. Furthermore, while Gibbs sampling is precise, scaling to "Big Data" may require transition to a stochastic variational inference framework. Finally, the current first-order Markov chain could be extended to higher-order dynamics to capture more complex long-term trends.

Plots of variance over time for different parameter settings. Discrepancies between simulated and theoretical values in long sequences highlight potential numerical precision issues.

Summary updated

Abstract

A gamma process dynamic Poisson factor analysis model is proposed to factorize a dynamic count matrix, whose columns are sequentially observed count vectors. The model builds a novel Markov chain that sends the latent gamma random variables at time (t-1) as the shape parameters of those at time t, which are linked to observed or latent counts under the Poisson likelihood. The significant challenge of inferring the gamma shape parameters is fully addressed, using unique data augmentation and marginalization techniques for the negative binomial distribution. The same nonparametric Bayesian model also applies to the factorization of a dynamic binary matrix, via a Bernoulli-Poisson link that connects a binary observation to a latent count, with closed-form conditional posteriors for the latent counts and efficient computation for sparse observations. We apply the model to text and music analysis, with state-of-the-art results.
BibTeX
@inproceedings{acharya2015nonparametric,
  author = {Ayan Acharya and Joydeep Ghosh and Mingyuan Zhou},
  title = {{Nonparametric Bayesian Factor Analysis for Dynamic Count Matrices}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2015},
  url = {https://openreview.net/forum?id=namcQIe4wQx},
}
Figure