Transformer Embeddings of Irregularly Spaced Events and Their Participants

Chenghao Yang, Hongyuan Mei, Jason Eisner

Abstract

The neural Hawkes process is a generative model of irregularly spaced sequences of discrete events. To handle complex domains with many event types, previous work considered a setting in which each event updates a deductive database of facts; future events are then conditioned on the database contents. We modify both models, replacing their recurrent LSTM-based architectures with flatter attention-based architectures, which are simpler and more parallelizable. This does not appear to hurt our accuracy, which is comparable to or better than that of the original models as well as previous attention-based methods.

Introduction

Many real-world data streams—like high-frequency trading, patient hospital visits, or social media interactions—consist of "irregularly spaced events." Unlike words in a sentence, these events happen at specific, arbitrary timestamps in continuous time. The previous gold standard for modeling these sequences was the Neural Hawkes Process (NHP), which uses an LSTM to summarize the past into a fixed-size vector. However, LSTMs suffer from a memory bottleneck and are inherently sequential, making them slow to train on long histories.

Methodology: A-NHP and A-NDTT

We propose two attention-based architectures to overcome these limitations: the Attentive Neural Hawkes Process (A-NHP) and the Attentive Neural Datalog Through Time (A-NDTT).

Comparison of A-NHP and A-NDTT architectures. A-NHP uses global attention over history, while A-NDTT uses symbolic logic to mask attention.

Continuous-Time Attention

Instead of discrete positions, we use sinusoidal functions of actual timestamps as positional encodings. A-NHP calculates the intensity of a future event by having that potential event attend to all previous events. Unlike LSTMs, it queries the entire past directly, allowing the model to capture long-range dependencies without a fixed-size bottleneck.

Neuro-Symbolic Integration

A-NDTT incorporates domain knowledge via Datalog rules. These rules define which past events are relevant to a specific future event (e.g., a user's behavior is influenced by emails they received). These symbolic rules act as a selective attention mask, providing a strong inductive bias that helps in complex domains with many event types, such as robot soccer matches.

Experimental Evaluation

We evaluated our models on several benchmarks, including clinical records (MIMIC-II), user badge history (StackOverflow), and robot soccer actions (RoboCup).

Performance comparison on MIMIC-II and StackOverflow datasets across negative log likelihood, RMSE, and error rate.

Key Results

  • Accuracy: A-NHP matched or outperformed the original NHP and other attention-based models like THP and SAHP on MIMIC-II and StackOverflow across multiple metrics including negative log likelihood and error rate.
  • Symbolic Power: On the RoboCup dataset (528 event types), A-NDTT significantly outperformed unstructured models by leveraging logical constraints to guide the attention mechanism.
  • Efficiency: A-NHP achieved a roughly 4x speedup in training over the LSTM-based NHP for long sequences due to its parallelizable Transformer-style architecture.

Discussion and Future Work

Our models demonstrate that attention-based architectures can effectively replace LSTMs for continuous-time event modeling without sacrificing accuracy. The influence of past events "drifts" naturally through time-based attention rather than requiring manually stipulated decay functions.

While A-NHP is highly parallelizable on GPUs, A-NDTT currently faces bottlenecks due to the dynamic nature of symbolic databases. Future work will explore "structure learning" to automatically discover symbolic rules and further optimize the computational cost of intensity calculations for large event vocabularies.

Summary updated

Abstract

The neural Hawkes process (Mei & Eisner, 2017) is a generative model of irregularly spaced sequences of discrete events. To handle complex domains with many event types, Mei et al. (2020a) further consider a setting in which each event in the sequence updates a deductive database of facts (via domain-specific pattern-matching rules); future events are then conditioned on the database contents. They show how to convert such a symbolic system into a neuro-symbolic continuous-time generative model, in which each database fact and the possible event has a time-varying embedding that is derived from its symbolic provenance. In this paper, we modify both models, replacing their recurrent LSTM-based architectures with flatter attention-based architectures (Vaswani et al., 2017), which are simpler and more parallelizable. This does not appear to hurt our accuracy, which is comparable to or better than that of the original models as well as (where applicable) previous attention-based methods (Zuo et al., 2020; Zhang et al., 2020a).
BibTeX
@misc{yang2021transformer,
  author = {Chenghao Yang and Hongyuan Mei and Jason Eisner},
  title = {{Transformer Embeddings of Irregularly Spaced Events and Their Participants}},
  year = {2021},
  eprint = {2201.00044},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2201.00044},
}
Figure