Switching Poisson Gamma Dynamical Systems

Wenchao Chen, Bo Chen, Yicheng Liu, Qianru Zhao, Mingyuan ZhouIJCAI 2020 · OpenReview

Abstract

We propose Switching Poisson gamma dynamical systems (SPGDS) to model sequentially observed multivariate count data. Different from previous models, SPGDS assigns its latent variables into mixture of gamma distributed parameters to model complex sequences and describe the nonlinear dynamics, meanwhile, capture various temporal dependencies. For efficient inference, we develop a scalable hybrid stochastic gradient-MCMC and switching recurrent autoencoding variational inference, which is scalable to large scale sequences and fast in out-of-sample prediction. Experiments on both unsupervised and supervised tasks demonstrate that the proposed model not only has excellent fitting and prediction performance on complex dynamic sequences, but also separates different dynamical patterns within them.

Introduction

Modeling multivariate count data sequences—such as word counts in news archives, social network interactions, or international relations events—presents significant challenges. These datasets are typically high-dimensional, sparse, and exhibit overdispersion, where the variance significantly exceeds the mean. Traditional modeling approaches often fall short: Linear Dynamical Systems (LDS) assume Gaussian distributions that poorly fit discrete counts; Hidden Markov Models (HMMs) struggle with high-dimensional dependencies; and standard Poisson-Gamma Dynamical Systems (PGDS) assume rigid, time-invariant dynamics, failing to capture "regime shifts" where underlying system behavior changes.

The SPGDS Framework

To address these limitations, we introduce the Switching Poisson Gamma Dynamical System (SPGDS). This framework incorporates a "switching" mechanism into the Poisson-Gamma architecture, allowing the model to transition between different dynamical modes.

Generative Model

Instead of a single transition matrix, SPGDS utilizes a mixture of gamma distributions. At each time step $t$, a discrete switching variable $z_t$ selects which dynamical pattern to follow. The latent state $\theta_t$ is then drawn from a gamma distribution conditioned on the previous state $\theta_{t-1}$ and the selected transition parameters. Finally, the observed multivariate counts $x_t$ are generated from a Poisson distribution parameterized by these latent states.

Graphical representation of the SPGDS generative model and inference network.

Inference and Scalability

To enable practical application on large-scale datasets, we developed a Switching Recurrent Variational Inference network. This approach addresses the challenges of discrete latent variables and non-conjugate distributions:

  • Gumbel-Softmax: Facilitates gradient-based optimization for the discrete switching variables $z_t$.
  • Weibull Distribution: Used to approximate the posterior of the latent gamma variables, offering mathematical convenience and a similar shape to the Gamma distribution.
  • Hybrid Inference: By combining Stochastic Gradient MCMC for global parameters with Autoencoding Variational Inference (AVI) for local latent variables, the model achieves scalability and rapid out-of-sample prediction.

Experimental Results and Insights

The SPGDS was evaluated across synthetic benchmarks, real-world text/event data (GDELT, ICEWS, SOTU, NIPS), and supervised classification tasks (sMNIST).

Dynamic Segmentation

A core strength of SPGDS is its ability to perform dynamic segmentation, effectively "color-coding" a timeline by identifying when a system enters a new state. For instance, in the NIPS dataset, the model successfully distinguished the shift from "neuron" research in the late 1980s to "Bayesian learning" in the mid-1990s as distinct dynamical regimes.

Dynamic segmentation and reconstruction of non-linear sequences.

Performance Metrics

  • Prediction Accuracy: On synthetic data, SPGDS achieved significantly lower Mean Square Error (MSE) and Prediction MSE (PMSE) compared to HMM, LDS, and standard PGDS baselines. On text datasets like NIPS and GDELT, it consistently outperformed others in "Precision at Top-M."
  • Supervised Learning: In sequential MNIST classification, the model reached 98.4% accuracy, surpassing standard RNNs and LSTMs. This indicates that the latent features captured by SPGDS are highly discriminative.

Conclusion

SPGDS provides a flexible and scalable framework for modeling complex count data sequences. By integrating switching mechanisms with gamma-distributed latent variables, it captures non-linear dynamics and regime shifts that traditional models miss. While the hybrid inference scheme adds complexity and requires careful hyperparameter tuning, the gains in predictive performance and interpretability are substantial. Future work will explore deeper hierarchical structures to capture even more abstract temporal dependencies.

Summary updated

Abstract

We propose Switching Poisson gamma dynamical systems (SPGDS) to model sequentially observed multivariate count data. Different from previous models, SPGDS assigns its latent variables into mixture of gamma distributed parameters to model complex sequences and describe the nonlinear dynamics, meanwhile, capture various temporal dependencies. For efficient inference, we develop a scalable hybrid stochastic gradient-MCMC and switching recurrent autoencoding variational inference, which is scalable to large scale sequences and fast in out-of-sample prediction. Experiments on both unsupervised and supervised tasks demonstrate that the proposed model not only has excellent fitting and prediction performance on complex dynamic sequences, but also separates different dynamical patterns within them.
BibTeX
@inproceedings{chen2020switching,
  author = {Wenchao Chen and Bo Chen and Yicheng Liu and Qianru Zhao and Mingyuan Zhou},
  title = {{Switching Poisson Gamma Dynamical Systems}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2020},
  url = {https://openreview.net/forum?id=MB983CWhRHQ},
}
Figure