Learning Temporal Point Processes with Intermittent Observations

Vinayak Gupta, Srikanta Bedathur, Sourangshu Bhattacharya, Abir DeAISTATS 2021 · OpenReview

Abstract

Marked temporal point processes (MTPP) have emerged as a powerful framework to model the underlying generative mechanism of asynchronous events localized in continuous time. Most existing models and inference methods in the MTPP framework consider only the complete observation scenario, an ideal setting barely encountered in practice. A recent line of work which considers missing events uses supervised learning techniques which require a missing or observed label for each event. In this work, we provide a novel unsupervised model and inference method for MTPPs in the presence of missing events. We first model the generative processes of observed events and missing events using two MTPPs, where the missing events are represented as latent random variables. Then we devise an unsupervised training method that jointly learns both the MTPPs by means of variational inference. Experiments with real datasets show that our modeling and inference frameworks can effectively impute the missing data among the observed events, which in turn enhances its predictive prowess.

Introduction

Marked Temporal Point Processes (MTPPs) are the standard tool for modeling sequences of events where both the time and the type (mark) of the event matter—such as stock trades, patient hospital visits, or social media posts. However, most MTPP models assume a "complete observation" scenario. In reality, data is often "leaky" due to privacy settings or technical limits. For instance, a social media crawler might miss posts due to rate limits, or a patient might miss a check-up.

Existing solutions to this "missing data" problem are often flawed: they either require manual labels of what was missed (supervised learning), use slow sampling methods like MCMC, or employ bi-directional neural networks that must re-process the entire history every time a new event arrives, leading to quadratic computational complexity.

The IMTPP Framework

The authors introduce IMTPP (Intermittently-observed Marked Temporal Point Processes), a new unsupervised framework that treats missing events as hidden variables, learning to "fill in the blanks" while simultaneously improving predictions for future events.

Overview of the IMTPP framework: (a) setup with observed and missing events, (b) MTPP for observations, and (c) posterior MTPP for missing events.

IMTPP models the world using two coupled processes: one for the events we see and one for the events we miss. The framework relies on several key innovations:

  1. Latent Variable Modeling: Missing events are treated as latent random variables. The model assumes the data is "Missing Not At Random" (MNAR), meaning the probability of an event being missing depends on the history of both observed and missing events.
  2. Variational Inference: To train the model without knowing where the missing events are, the authors use an Evidence Lower Bound (ELBO) approach. They define a "prior" MTPP for missing events and a "posterior" MTPP that updates its belief once a new observed event arrives.
  3. Log-Normal Arrival Times: Unlike traditional MTPPs that use complex intensity functions, IMTPP uses a log-normal distribution to model the time between events. This facilitates training via the re-parameterization trick and allows for accurate closed-form predictions.
  4. Forward-Only RNNs: Unlike previous state-of-the-art models that use bi-directional RNNs, IMTPP only computes in the forward direction. This keeps the computational cost linear ($O(N)$) relative to the number of events, making it suitable for streaming data.

Experimental Evaluation

The researchers evaluated IMTPP on one synthetic dataset and six diverse real-world datasets, including Amazon reviews, NYC Taxi data, Retweets, Stack Overflow badges, and Foursquare check-ins.

IMTPP consistently outperformed eight baselines (including Hawkes Processes and Transformer-based models) across almost all datasets. For example, on the Amazon Movies dataset, it achieved a 10.12% improvement in Mean Absolute Error (MAE) for time prediction compared to the next best model. Furthermore, runtime analysis confirmed that while competing models scale quadratically, IMTPP scales linearly, offering significant efficiency gains.

Comparison of true event sequences versus sequences predicted/imputed by the IMTPP model.

Key Insights

  • Missing Events as Physical Latents: In IMTPP, latent variables represent actual events that happened but weren't recorded. This physical grounding makes the model more interpretable than standard VAEs.
  • The Power of Forward Dynamics: By designing the model to update incrementally, the authors proved that "looking back" at the entire history is not necessary to impute missing data, solving a major scalability bottleneck.
  • Accounting for Hidden Influence: Even if missing events are not the primary focus, modeling them improves the prediction of observed events. These missing events carry "hidden" influence that explains sudden bursts or lulls in the visible data.

Conclusion

IMTPP provides a robust, unsupervised, and efficient framework for handling intermittent observations in event sequences. By treating missing events as latent variables and employing variational inference with forward-only dynamics, it achieves state-of-the-art accuracy while maintaining linear scalability. Future work may extend this to handle "partial missingness" (e.g., knowing the time but not the mark) and continuous mark spaces.

Summary updated

Abstract

Marked temporal point processes (MTPP) have emerged as a powerful framework to model the underlying generative mechanism of asynchronous events localized in continuous time. Most existing models and inference methods in MTPP framework consider only the complete observation scenario i.e. the event sequence being modeled is completely observed with no missing events – an ideal setting barely encountered in practice. A recent line of work which considers missing events uses supervised learning techniques which require a missing or observed label for each event. In this work, we provide a novel unsupervised model and inference method for MTPPs in presence of missing events. We first model the generative processes of observed events and missing events using two MTPPs, where the missing events are represented as latent random variables. Then we devise an unsupervised training method that jointly learns both the MTPPs by means of variational inference. Experiments with real datasets show that our modeling and inference frameworks can effectively impute the missing data among the observed events, which in turn enhances its predictive prowess.
BibTeX
@inproceedings{gupta2021learning,
  author = {Vinayak Gupta and Srikanta Bedathur and Sourangshu Bhattacharya and Abir De},
  title = {{Learning Temporal Point Processes with Intermittent Observations}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2021},
  url = {https://openreview.net/forum?id=tCICsvJZb8d},
}
Figure