Deep Poisson gamma dynamical systems

Dandan Guo, Bo Chen, Hao Zhang, Mingyuan Zhou

Abstract

We develop deep Poisson-gamma dynamical systems (DPGDS) to model sequentially observed multivariate count data, improving previously proposed models by not only mining deep hierarchical latent structure from the data, but also capturing both first-order and long-range temporal dependencies. Using sophisticated but simple-to-implement data augmentation techniques, we derived closed-form Gibbs sampling update equations by first backward and upward propagating auxiliary latent counts, and then forward and downward sampling latent variables. Moreover, we develop stochastic gradient MCMC inference that is scalable to very long multivariate count time series. Experiments on both synthetic and a variety of real-world data demonstrate that the proposed model not only has excellent predictive performance, but also provides highly interpretable multilayer latent structure to represent hierarchical and temporal information propagation.

Introduction

Multivariate count time series are prevalent in various domains, from document word frequencies to international event logs. Modeling such data is inherently difficult due to its sparse and over-dispersed nature. Traditional Linear Dynamical Systems (LDS) rely on Gaussian assumptions, which are ill-suited for discrete counts. While the Poisson-Gamma Dynamical System (PGDS) offers a more appropriate likelihood, its shallow Markovian structure limits its ability to capture long-range temporal dependencies—the "rhythm" of a sequence that extends beyond immediate state transitions.

Deep Poisson Gamma Dynamical Systems

To address these limitations, we propose the Deep Poisson Gamma Dynamical System (DPGDS). The DPGDS extends the shallow PGDS into a deep hierarchical architecture. In this model, the observed count vector $x_t^{(1)}$ at time $t$ is factorized through multiple layers of Gamma-distributed hidden units $\theta_t^{(l)}$.

Figure 1: (a) Graphical representation of the DPGDS architecture showing temporal and hierarchical dependencies. (b) Latent variable trajectories across layers for the ICEWS dataset, illustrating how broad themes in higher layers influence specific events in lower layers.

As shown in Figure 1(a), the state of a hidden unit at layer $l$ and time $t$ depends on both its own past state $\theta_{t-1}^{(l)}$ (temporal dependency) and the current state of the layer above it $\theta_t^{(l+1)}$ (hierarchical dependency). This structure allows the model to decouple fast-changing, specific information in the lower layers from slow-evolving, general themes in the higher layers. Effectively, the higher layers act as a long-term memory, breaking the Markov bottleneck of shallow models.

Inference and Scalability

Inference in deep hierarchical models with non-conjugate distributions is typically challenging. We overcome this by utilizing data augmentation techniques involving the Chinese Restaurant Table (CRT) and Sum-Log distributions. This allows for the derivation of a Backward-Upward-Forward-Downward (BUFD) Gibbs sampler with closed-form update equations.

For large-scale datasets, such as the GDELT dataset containing over 120,000 time points, we developed a Stochastic Gradient MCMC (SGMCMC) approach. This ensures the model remains scalable to very long time series that would otherwise be computationally prohibitive for standard batch-processing methods.

Experimental Results

The DPGDS was evaluated on diverse datasets, including synthetic "Bouncing Ball" videos and real-world corpora like ICEWS, GDELT, and State of the Union (SOTU) addresses.

  • Predictive Performance: DPGDS consistently outperformed state-of-the-art models, including Deep Temporal Sigmoid Belief Networks (DTSBN) and shallow PGDS, in terms of precision and recall across all text datasets.
  • Non-linear Dynamics: In the bouncing ball task, DPGDS was uniquely capable of accurately predicting trajectories post-collision, demonstrating its ability to model complex, non-linear physical dynamics.
  • Interpretability: A key advantage of DPGDS is its interpretable latent space. As illustrated in Figure 1(b) for the ICEWS dataset, the model learns a hierarchy of topics. While layer 1 captures specific bilateral interactions (e.g., "Israel-Palestine"), layer 3 identifies broader global trends that evolve over much longer timescales.

Conclusion

The Deep Poisson Gamma Dynamical System represents a significant advancement in modeling multivariate count time series. By combining a deep hierarchical structure with temporal dependencies, it captures both immediate fluctuations and long-term rhythms in data. While currently focused on Poisson and Bernoulli likelihoods, future work may extend this framework to continuous-valued data or explore non-parametric methods to automatically determine optimal model depth.

Summary updated

Abstract

We develop deep Poisson-gamma dynamical systems (DPGDS) to model sequentially observed multivariate count data, improving previously proposed models by not only mining deep hierarchical latent structure from the data, but also capturing both first-order and long-range temporal dependencies. Using sophisticated but simple-to-implement data augmentation techniques, we derived closed-form Gibbs sampling update equations by first backward and upward propagating auxiliary latent counts, and then forward and downward sampling latent variables. Moreover, we develop stochastic gradient MCMC inference that is scalable to very long multivariate count time series. Experiments on both synthetic and a variety of real-world data demonstrate that the proposed model not only has excellent predictive performance, but also provides highly interpretable multilayer latent structure to represent hierarchical and temporal information propagation.
BibTeX
@inproceedings{guo2018deep,
  author = {Dandan Guo and Bo Chen and Hao Zhang and Mingyuan Zhou},
  title = {{Deep Poisson gamma dynamical systems}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2018},
  url = {https://openreview.net/forum?id=rkZDqPWuZB},
}
Figure