Abstract
The neural Hawkes process is a generative model of irregularly spaced sequences of discrete events. To handle complex domains with many event types, previous work considered a setting in which each event updates a deductive database of facts; future events are then conditioned on the database contents. We modify both models, replacing their recurrent LSTM-based architectures with flatter attention-based architectures, which are simpler and more parallelizable. This does not appear to hurt our accuracy, which is comparable to or better than that of the original models as well as previous attention-based methods.
Introduction
Many real-world data streams—like high-frequency trading, patient hospital visits, or social media interactions—consist of "irregularly spaced events." Unlike words in a sentence, these events happen at specific, arbitrary timestamps in continuous time. The previous gold standard for modeling these sequences was the Neural Hawkes Process (NHP), which uses an LSTM to summarize the past into a fixed-size vector. However, LSTMs suffer from a memory bottleneck and are inherently sequential, making them slow to train on long histories.
Methodology: A-NHP and A-NDTT
We propose two attention-based architectures to overcome these limitations: the Attentive Neural Hawkes Process (A-NHP) and the Attentive Neural Datalog Through Time (A-NDTT).
Continuous-Time Attention
Instead of discrete positions, we use sinusoidal functions of actual timestamps as positional encodings. A-NHP calculates the intensity of a future event by having that potential event attend to all previous events. Unlike LSTMs, it queries the entire past directly, allowing the model to capture long-range dependencies without a fixed-size bottleneck.
Neuro-Symbolic Integration
A-NDTT incorporates domain knowledge via Datalog rules. These rules define which past events are relevant to a specific future event (e.g., a user's behavior is influenced by emails they received). These symbolic rules act as a selective attention mask, providing a strong inductive bias that helps in complex domains with many event types, such as robot soccer matches.
Experimental Evaluation
We evaluated our models on several benchmarks, including clinical records (MIMIC-II), user badge history (StackOverflow), and robot soccer actions (RoboCup).
Key Results
- Accuracy: A-NHP matched or outperformed the original NHP and other attention-based models like THP and SAHP on MIMIC-II and StackOverflow across multiple metrics including negative log likelihood and error rate.
- Symbolic Power: On the RoboCup dataset (528 event types), A-NDTT significantly outperformed unstructured models by leveraging logical constraints to guide the attention mechanism.
- Efficiency: A-NHP achieved a roughly 4x speedup in training over the LSTM-based NHP for long sequences due to its parallelizable Transformer-style architecture.
Discussion and Future Work
Our models demonstrate that attention-based architectures can effectively replace LSTMs for continuous-time event modeling without sacrificing accuracy. The influence of past events "drifts" naturally through time-based attention rather than requiring manually stipulated decay functions.
While A-NHP is highly parallelizable on GPUs, A-NDTT currently faces bottlenecks due to the dynamic nature of symbolic databases. Future work will explore "structure learning" to automatically discover symbolic rules and further optimize the computational cost of intensity calculations for large event vocabularies.
Summary updated