TL;DR
This paper introduces the Poisson-Randomized Gamma Dynamical System (PRGDS), a novel Bayesian model for analyzing sequential count data (like event logs or neural spikes). PRGDS uses a unique alternating chain of discrete Poisson and continuous gamma latent states to efficiently capture complex temporal patterns like "burstiness" and sparsity, outperforming existing methods on several real-world datasets.
What Problem Does This Paper Address?
Analyzing sequential count data, such as international events between countries or neural spike trains, presents significant challenges. These datasets are often "bursty" (events occur in clusters) and "sparse" (many entries are zero). Traditional statistical methods struggle with these properties, and existing advanced models like Poisson factorization or Poisson-Gamma Dynamical Systems (PGDS) have limitations. PGDS, for instance, is expressive but computationally intractable, requiring complex data augmentation schemes. The authors aim to bridge this gap by developing a model that is both expressive enough to capture complex temporal dynamics and computationally tractable, allowing for efficient inference.
Methods
The core innovation of PRGDS is a new modeling motif: an alternating chain of discrete Poisson and continuous gamma latent states. This structure breaks the intractable dependencies found in purely continuous state-space models (like the original PGDS). This "Poisson-randomization" allows for closed-form complete conditionals for most variables, making inference much simpler. Specifically, the model leverages the Bessel distribution and a newly introduced shifted confluent hypergeometric (SCH) distribution for posterior inference. The paper also introduces a sparse variant of the PRGDS by allowing gamma latent states to be exactly zero, which encodes a strong inductive bias towards sparsity and burstiness.
Datasets
The authors evaluate their model on several real-world datasets:
- Text data: NeurIPS papers, DBLP abstracts, and State of the Union speeches.
- International event data: GDELT and ICEWS datasets.
- Neural spike train data: Recordings from macaque monkey motor cortexes.
These datasets are used to compare the PRGDS against baselines like Bayesian Poisson Tensor Factorization (BPTF) and the original Poisson-Gamma Dynamical System (PGDS).
Key Insights and Ideas
- Poisson-randomization is key to tractability: Introducing discrete Poisson states between continuous gamma states is a clever way to break complex dependencies, enabling closed-form posteriors without resorting to computationally expensive data augmentation.
- Sparsity as a feature, not a bug: The sparse variant of PRGDS (where latent states can be exactly zero) is not only more efficient but also demonstrably better at capturing bursty and temporally localized structures. This suggests that explicitly modeling sparsity can lead to more interpretable and accurate latent representations.
- Bridging the gap between expressiveness and tractability: The PRGDS successfully navigates the trade-off between complex model expressiveness and practical computational feasibility, a common challenge in Bayesian modeling.
Key Results
- Improved Predictive Performance: The PRGDS, particularly its sparse variant, consistently outperforms or matches the performance of the PGDS and other baselines across smoothing and forecasting tasks on various datasets.
- Enhanced Latent Structure Inference: The sparse PRGDS is uniquely capable of inferring latent structures that are highly localized in time, effectively capturing "bursty" temporal dynamics that other models miss. This is visualized in Figure 5, showing components that are active only for short durations.
- Computational Efficiency: The closed-form conditionals lead to efficient inference, making the PRGDS practical for large-scale datasets.
Limitations and Future Work
- Forecasting vs. Smoothing: While the PRGDS performs well in smoothing, the paper notes that the original PGDS sometimes shows a slight edge in forecasting. The authors conjecture this is due to an additive term in the PRGDS's expectation that grows with forecast horizon, which is absent in the PGDS.
- Hyperparameter Sensitivity: The paper mentions setting hyperparameters for empirical studies but doesn't delve deeply into their sensitivity or automatic selection.
- Future Work: The authors suggest exploring the PRGDS for other applications and potentially extending it to handle more complex data structures or dynamics. The ability to infer temporally localized structures opens avenues for understanding event sequences with fine-grained temporal dependencies.
Summary updated