Poisson-Randomized Gamma Dynamical Systems

Aaron Schein, Scott W. Linderman, Mingyuan Zhou, David M. Blei, Hanna M. Wallach

TL;DR

This paper introduces the Poisson-Randomized Gamma Dynamical System (PRGDS), a novel Bayesian model for analyzing sequential count data (like event logs or neural spikes). PRGDS uses a unique alternating chain of discrete Poisson and continuous gamma latent states to efficiently capture complex temporal patterns like "burstiness" and sparsity, outperforming existing methods on several real-world datasets.

What Problem Does This Paper Address?

Analyzing sequential count data, such as international events between countries or neural spike trains, presents significant challenges. These datasets are often "bursty" (events occur in clusters) and "sparse" (many entries are zero). Traditional statistical methods struggle with these properties, and existing advanced models like Poisson factorization or Poisson-Gamma Dynamical Systems (PGDS) have limitations. PGDS, for instance, is expressive but computationally intractable, requiring complex data augmentation schemes. The authors aim to bridge this gap by developing a model that is both expressive enough to capture complex temporal dynamics and computationally tractable, allowing for efficient inference.

Methods

The core innovation of PRGDS is a new modeling motif: an alternating chain of discrete Poisson and continuous gamma latent states. This structure breaks the intractable dependencies found in purely continuous state-space models (like the original PGDS). This "Poisson-randomization" allows for closed-form complete conditionals for most variables, making inference much simpler. Specifically, the model leverages the Bessel distribution and a newly introduced shifted confluent hypergeometric (SCH) distribution for posterior inference. The paper also introduces a sparse variant of the PRGDS by allowing gamma latent states to be exactly zero, which encodes a strong inductive bias towards sparsity and burstiness.

Datasets

The authors evaluate their model on several real-world datasets:

  • Text data: NeurIPS papers, DBLP abstracts, and State of the Union speeches.
  • International event data: GDELT and ICEWS datasets.
  • Neural spike train data: Recordings from macaque monkey motor cortexes.

These datasets are used to compare the PRGDS against baselines like Bayesian Poisson Tensor Factorization (BPTF) and the original Poisson-Gamma Dynamical System (PGDS).

Key Insights and Ideas

  • Poisson-randomization is key to tractability: Introducing discrete Poisson states between continuous gamma states is a clever way to break complex dependencies, enabling closed-form posteriors without resorting to computationally expensive data augmentation.
  • Sparsity as a feature, not a bug: The sparse variant of PRGDS (where latent states can be exactly zero) is not only more efficient but also demonstrably better at capturing bursty and temporally localized structures. This suggests that explicitly modeling sparsity can lead to more interpretable and accurate latent representations.
  • Bridging the gap between expressiveness and tractability: The PRGDS successfully navigates the trade-off between complex model expressiveness and practical computational feasibility, a common challenge in Bayesian modeling.

Key Results

  • Improved Predictive Performance: The PRGDS, particularly its sparse variant, consistently outperforms or matches the performance of the PGDS and other baselines across smoothing and forecasting tasks on various datasets.
  • Enhanced Latent Structure Inference: The sparse PRGDS is uniquely capable of inferring latent structures that are highly localized in time, effectively capturing "bursty" temporal dynamics that other models miss. This is visualized in Figure 5, showing components that are active only for short durations.
  • Computational Efficiency: The closed-form conditionals lead to efficient inference, making the PRGDS practical for large-scale datasets.

Limitations and Future Work

  • Forecasting vs. Smoothing: While the PRGDS performs well in smoothing, the paper notes that the original PGDS sometimes shows a slight edge in forecasting. The authors conjecture this is due to an additive term in the PRGDS's expectation that grows with forecast horizon, which is absent in the PGDS.
  • Hyperparameter Sensitivity: The paper mentions setting hyperparameters for empirical studies but doesn't delve deeply into their sensitivity or automatic selection.
  • Future Work: The authors suggest exploring the PRGDS for other applications and potentially extending it to handle more complex data structures or dynamics. The ability to infer temporally localized structures opens avenues for understanding event sequences with fine-grained temporal dependencies.

Summary updated

Abstract

This paper presents the Poisson-randomized gamma dynamical system (PRGDS), a model for sequentially observed count tensors that encodes a strong inductive bias toward sparsity and burstiness. The PRGDS is based on a new motif in Bayesian latent variable modeling, an alternating chain of discrete Poisson and continuous gamma latent states that is analytically convenient and computationally tractable. This motif yields closed-form complete conditionals for all variables by way of the Bessel distribution and a novel discrete distribution that we call the shifted confluent hypergeometric distribution. We draw connections to closely related models and compare the PRGDS to these models in studies of real-world count data sets of text, international events, and neural spike trains. We find that a sparse variant of the PRGDS, which allows the continuous gamma latent states to take values of exactly zero, often obtains better predictive performance than other models and is uniquely capable of inferring latent structures that are highly localized in time.
BibTeX
@inproceedings{schein2019poisson,
  author = {Aaron Schein and Scott W. Linderman and Mingyuan Zhou and David M. Blei and Hanna M. Wallach},
  title = {{Poisson-Randomized Gamma Dynamical Systems}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
}
Figure