Poisson-Randomized Gamma Dynamical Systems

Aaron Schein, Scott W. Linderman, Mingyuan Zhou, David M. Blei, Hanna M. Wallach

Introduction

The Poisson-Randomized Gamma Dynamical System (PRGDS) is a novel Bayesian model designed to track how count-based data—such as international events, word frequencies, or neural spikes—evolves over time. By cleverly alternating between discrete Poisson and continuous Gamma latent states, the PRGDS achieves a mathematically tractable structure (with closed-form updates) while uniquely capturing the sparse, "bursty" nature of real-world events.

The Challenge of Modeling Count Data

Many real-world datasets consist of "event counts" recorded over time, often represented as multi-dimensional arrays or count tensors. Modeling this data presents two major challenges:

  1. Burstiness and Sparsity: Events frequently occur in sudden clusters (bursts) followed by long periods of inactivity (sparsity). Traditional models tend to "smooth" these variations out, losing crucial signal.
  2. Mathematical Intractability: To model temporal dynamics, researchers chain variables together. When chaining Gamma variables—the standard choice for count modeling—the math becomes highly complex. This non-conjugacy usually forces researchers to rely on slow approximations or restrictive assumptions that limit model flexibility.

The Poisson-Randomized Gamma Motif

The core innovation of this paper is the Poisson-randomized gamma motif.

In previous models like the Poisson-Gamma Dynamical System (PGDS), continuous Gamma latent states were linked directly to one another. This created intractable, non-conjugate dependencies that made calculating the "next step" computationally painful.

Intractable PGDS Model

The PRGDS solves this by breaking the continuous chain. It inserts a discrete Poisson latent state between every continuous Gamma state, creating a tractable $\text{Gamma} \to \text{Poisson} \to \text{Gamma}$ sequence.

Tractable PRGDS Model

This structural shift restores mathematical conjugacy, allowing for "closed-form complete conditionals." By leveraging the little-known Bessel distribution and a newly derived Shifted Confluent Hypergeometric (SCH) distribution, the authors enable highly efficient Gibbs sampling without the complex "data augmentation" hacks required by previous models.

Furthermore, the authors introduce a sparse variant of the model. By setting a specific hyperparameter ($\epsilon_0^{(\theta)}$) to zero, latent states are allowed to drop to exactly zero. This acts as a "sparsity switch," letting the model completely turn off inactive components to perfectly match bursty data.

Key Insights and Results

The authors evaluated the PRGDS across 16 different studies using international event data (GDELT & ICEWS), text data (NeurIPS, DBLP, State of the Union), and neuroscience data (macaque motor cortex).

  • The Power of the "Sparsity Switch": The ability for latent states to hit exactly zero proved critical. For example, in the ICEWS political dataset, the model identified a "South Sudan" component. Because South Sudan did not exist until 2011, the sparse PRGDS correctly kept that component at zero for the first 94% of the timeline before letting it "burst" into existence. Other models incorrectly smeared this activity across the entire timeline.
  • Superior Localized Tracking: The PRGDS was uniquely capable of finding highly localized structures. In neural data, it isolated specific neurons that fired only during brief windows of a trial, whereas baseline models produced blurry, less precise signals.
  • Smoothing vs. Forecasting: In all 16 studies, dynamic models significantly outperformed non-dynamic baselines. The PRGDS excelled at "smoothing" (interpolating missing data within a timeline). However, the older PGDS was occasionally better at "forecasting" future events, likely because its continuous structure enforces a baseline level of activity that aids long-term prediction.

Limitations and Future Work

While highly effective, the PRGDS has a few trade-offs. The aggressive "turning off" of components in the sparse variant can make it slightly less robust for long-term forecasting compared to models that maintain a low baseline of activity. Additionally, while the model scales linearly with the number of non-zero entries (making it efficient for sparse tensors), extremely dense, high-frequency data could still pose computational challenges. Future work may also explore the model's sensitivity across a wider range of hyperparameters.

Summary updated

Abstract

This paper presents the Poisson-randomized gamma dynamical system (PRGDS), a model for sequentially observed count tensors that encodes a strong inductive bias toward sparsity and burstiness. The PRGDS is based on a new motif in Bayesian latent variable modeling, an alternating chain of discrete Poisson and continuous gamma latent states that is analytically convenient and computationally tractable. This motif yields closed-form complete conditionals for all variables by way of the Bessel distribution and a novel discrete distribution that we call the shifted confluent hypergeometric distribution. We draw connections to closely related models and compare the PRGDS to these models in studies of real-world count data sets of text, international events, and neural spike trains. We find that a sparse variant of the PRGDS, which allows the continuous gamma latent states to take values of exactly zero, often obtains better predictive performance than other models and is uniquely capable of inferring latent structures that are highly localized in time.
BibTeX
@inproceedings{schein2019poisson,
  author = {Aaron Schein and Scott W. Linderman and Mingyuan Zhou and David M. Blei and Hanna M. Wallach},
  title = {{Poisson-Randomized Gamma Dynamical Systems}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
}
Figure