Abstract
Temporal Point Processes (TPPs) are essential for modeling asynchronous event sequences. While Neural TPPs have gained popularity for their flexibility, the lack of standardized evaluation has hindered progress. This paper presents a large-scale, unified study of Neural TPPs across 15 real-world datasets. We evaluate the impact of architectural choices—event encoders, history encoders, and decoders—on predictive accuracy and probabilistic calibration. Our findings suggest that RNN-based models often outperform Transformers, long-term history provides marginal gains in many real-world scenarios, and current models struggle with mark prediction calibration.
Introduction
Temporal Point Processes (TPPs) provide a mathematical framework for modeling "event streams"—sequences of discrete events occurring at irregular intervals. Applications range from predicting user interactions on digital platforms to modeling seismic activity or high-frequency financial transactions. Traditional models, such as the Hawkes process, offer interpretability but often lack the capacity to capture the complexity of modern datasets.
The emergence of "Neural TPPs" has addressed these limitations by integrating deep learning architectures. However, the rapid expansion of the field has led to inconsistent experimental setups, varying datasets, and disparate evaluation metrics. This lack of standardization makes it difficult to discern which architectural innovations truly drive performance. This study addresses this gap by providing a rigorous, modular comparison of state-of-the-art Neural TPPs.
Modular Architecture and Methodology
To systematically evaluate Neural TPPs, the authors decompose the architecture into three primary components:
- Event Encoder: This component represents the features of the most recent event, including its timing and "mark" (type). The study compares various representations, including raw time, log-transformed time, and vectorial embeddings like sinusoidal encodings.
- History Encoder: This module aggregates past events into a representative state vector. The researchers evaluated Gated Recurrent Units (GRUs), Self-Attention mechanisms (Transformers), and a "Constant" baseline that ignores history to establish a performance floor.
- Decoder: The decoder models the conditional intensity function or the probability distribution of the next event. The study tested 11 decoders, including the Log-Normal Mixture (LNM), the Neural Hawkes (NH) process, and the Recurrent Marked Temporal Point Process (RMTPP).
Beyond standard Negative Log-Likelihood (NLL) metrics, the study emphasizes probabilistic calibration—measuring the reliability of a model's uncertainty estimates. For instance, determining if an event predicted with 90% confidence actually occurs 90% of the time.
Data and Experimental Setup
The evaluation spans 15 real-world datasets, categorized into "Marked" (containing both time and event type) and "Unmarked" (time only).
- Marked Datasets: LastFM, MOOC, Wikipedia, MIMIC2, Github, Stack Overflow, and Retweets.
- Unmarked Datasets: Twitter, PUBG, Yelp, Reddit, and Taxi pickups.
The authors noted that several popular benchmarks, such as MIMIC2 and Stack Overflow, may be "too easy," as simple baseline models often match the performance of complex neural architectures on these specific datasets.
Key Insights
The "Short Memory" Phenomenon
A significant finding of this study is that many Neural TPPs do not effectively utilize long-term history. In several datasets, models using only the last 2 to 5 events performed as well as those utilizing the entire history. This suggests that many real-world event streams are effectively Markovian, or that current history encoders fail to capture long-range dependencies in continuous time.
RNNs vs. Transformers
Despite the dominance of Transformers in natural language processing, GRU-based encoders generally proved more stable and effective for TPP tasks. Self-attention mechanisms were found to be highly sensitive to the specific method of time encoding. While Transformers can perform well, they require sophisticated vectorial time encodings (like TEM or LE) to remain competitive, whereas RNNs perform adequately with simple log-transformations.
The Calibration Gap
The study reveals a discrepancy between temporal and categorical predictions. While models are generally well-calibrated when predicting when an event will occur, they are frequently overconfident and miscalibrated when predicting what the event will be (the mark). Interestingly, the classical Hawkes process often outperformed complex neural models in mark prediction accuracy and calibration.
Performance and Efficiency
- Temporal Prediction: The Log-Normal Mixture (LNM) decoder emerged as the most robust choice for arrival time prediction across the majority of datasets.
- Mark Prediction: RMTPP and the classical Hawkes process provided the best balance of accuracy and calibration for event types.
- Computational Overhead: The Neural Hawkes (NH) decoder was identified as highly inefficient, with training times up to 200 times slower than simpler alternatives, often without a corresponding gain in accuracy.
Conclusion
This comprehensive study clarifies the current state of Neural TPP research, highlighting that architectural complexity does not always translate to better performance. Future research should prioritize improving mark calibration and developing encoders that can better leverage long-term dependencies. Furthermore, the field must move toward more challenging benchmarks to truly test the limits of neural modeling in continuous-time event data.
Summary updated