Variational Inference for Neyman-Scott Processes

Chengkuan Hong, Christian R. Shelton

Abstract

Neyman-Scott processes (NSPs) have been applied across a range of fields to model points or temporal events with a hierarchy of clusters. Markov chain Monte Carlo (MCMC) is typically used for posterior sampling in the model. However, MCMC’s mixing time can cause the resulting inference to be slow, and thereby slow down model learning and prediction. We develop the first variational inference (VI) algorithm for NSPs, and give two examples of suitable variational posterior point process distributions. Our method minimizes the inclusive Kullback-Leibler (KL) divergence for VI to obtain the variational parameters. We generate samples from the approximate posterior point processes much faster than MCMC, as we can directly estimate the approximate posterior point processes without any MCMC steps or gradient descent. We include synthetic and real-world data experiments that demonstrate our VI algorithm achieves better prediction performance than MCMC when computational time is limited.

Introduction

Neyman-Scott Processes (NSPs) are a class of hierarchical models used to describe clustered events, such as an initial earthquake (parent) triggering a cluster of aftershocks (children). Despite their power in modeling "parent-child" event hierarchies, performing inference on these models is notoriously difficult because the "parent" events are often latent.

Traditionally, researchers rely on Markov chain Monte Carlo (MCMC) to sample the posterior distribution of these hidden points. While MCMC is theoretically exact given infinite time, it suffers from long mixing times. In practical applications like real-time disaster prediction or high-frequency social media monitoring, the latency of MCMC convergence is prohibitive. This paper introduces the first variational inference (VI) framework for NSPs, providing a faster, approximate inference method that maintains high-quality predictions under strict time budgets.

Methodology

The core innovation is a pivot from the traditional top-down generative approach of NSPs to an upward-generative variational posterior. Instead of attempting to find parents that could have caused the observed children, the model directly "generates" the likely hidden layers above the observations.

Inclusive KL Divergence

Unlike standard Variational Inference which minimizes the exclusive KL divergence (often leading to underestimated uncertainty), this method minimizes the inclusive KL divergence ($KL(p \parallel q)$). This approach ensures the approximation covers the true posterior more broadly, which is critical for capturing the multi-modal nature of point process clusters.

Variational Architectures

The authors propose two specific architectures:

  1. UNSP (Upward NSP): A functional form that mirrors the original NSP but operates in reverse (upward).
  2. USAP (Upward Self-Attention Process): A flexible model utilizing self-attention (Transformers) to encode dependencies between events. This allows the model to leverage information from both the past and the future (relative to a specific point) to infer hidden states.

By using a generative point process as the variational posterior, the authors can employ inversion sampling to generate posterior samples almost instantly, bypassing the iterative steps required by MCMC.

Experimental Evaluation

The framework was evaluated on both synthetic and real-world datasets:

  • Synthetic Datasets: 1-hidden and 2-hidden layer NSPs to establish ground-truth performance.
  • Retweets: Sequences of tweet types categorized by follower counts.
  • Earthquakes: Data from the Northern California earthquake catalog.
  • Homicides: Spatio-temporal data of homicide events in Chicago.

Results and Discussion

The proposed VI methods, particularly the USAP model, demonstrate a significant "upward" advantage. By designing the variational distribution to flow from observations to hidden layers, the authors create a "one-pass" inference mechanism that is fundamentally more efficient for prediction than the global search performed by MCMC.

Comparison of intensity functions for UNSP and USAP against MCMC ground truth.

As shown in the figure above, the USAP model fits the "spiky" nature of the true posterior much more closely than the simpler UNSP. Quantitatively, the VI methods generate samples orders of magnitude faster than MCMC because they avoid the mixing phase. On the Retweet dataset, the USAP model achieved a time-prediction RMSE several orders of magnitude lower than MCMC when both were limited to the same computational budget. Furthermore, the proposed DNSP-USAP consistently outperformed other state-of-the-art neural point processes, such as the Transformer Hawkes Process (THP) and Neural Hawkes Process (NHP).

Conclusion

This work provides a scalable alternative to MCMC for hierarchical point processes. While MCMC may eventually reach lower error given massive computational resources, the VI methods (especially USAP) provide high-accuracy results much faster. Future work includes extending the Self-Attention (USAP) model to high-dimensional spatial data and exploring the theoretical convergence of variational parameters in unbounded dimensions.

Summary updated

Abstract

Neyman-Scott processes (NSPs) have been applied across a range of fields to model points or temporal events with a hierarchy of clusters. Markov chain Monte Carlo (MCMC) is typically used for posterior sampling in the model. However, MCMC’s mixing time can cause the resulting inference to be slow, and thereby slow down model learning and prediction. We develop the first variational inference (VI) algorithm for NSPs, and give two examples of suitable variational posterior point process distributions. Our method minimizes the inclusive Kullback-Leibler (KL) divergence for VI to obtain the variational parameters. We generate samples from the approximate posterior point processes much faster than MCMC, as we can directly estimate the approximate posterior point processes without any MCMC steps or gradient descent. We include synthetic and real-world data experiments that demonstrate our VI algorithm achieves better prediction performance than MCMC when computational time is limited.
BibTeX
@inproceedings{hong2023variational,
  author = {Chengkuan Hong and Christian R. Shelton},
  title = {{Variational Inference for Neyman-Scott Processes}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2023},
  url = {https://openreview.net/forum?id=0jgk4iB7YL},
}
Figure