Transformer Embeddings of Irregularly Spaced Events and Their Participants

Chenghao Yang, Hongyuan Mei, Jason Eisner

Abstract

The neural Hawkes process (Mei & Eisner, 2017) is a generative model of irregularly spaced sequences of discrete events. To handle complex domains with many event types, Mei et al. (2020a) further consider a setting in which each event in the sequence updates a deductive database of facts (via domain-specific pattern-matching rules); future events are then conditioned on the database contents. They show how to convert such a symbolic system into a neuro-symbolic continuous-time generative model, in which each database fact and the possible event has a time-varying embedding that is derived from its symbolic provenance. In this paper, we modify both models, replacing their recurrent LSTM-based architectures with flatter attention-based architectures (Vaswani et al., 2017), which are simpler and more parallelizable. This does not appear to hurt our accuracy, which is comparable to or better than that of the original models as well as (where applicable) previous attention-based methods (Zuo et al., 2020; Zhang et al., 2020a).

Introduction

Many real-world data streams—from electronic health records and financial transactions to social media logs—consist of "events" happening at unpredictable intervals. Modeling these sequences requires predicting both what will happen next and when it will happen. Traditionally, the Neural Hawkes Process (NHP) utilized Long Short-Term Memory (LSTM) networks to summarize the history of events into a fixed-size vector. However, LSTMs suffer from two major drawbacks: a sequential bottleneck that hinders parallelization during training, and memory compression issues that can cause the model to forget important events from the distant past. Furthermore, in complex domains such as robot soccer, events involve many participants and follow specific logic rules that standard neural models often struggle to learn from data alone.

Methods

We propose two primary architectures to address these challenges: the Attentive Neural Hawkes Process (A-NHP) and the Attentive Neural Datalog Through Time (A-NDTT).

Attentive Neural Hawkes Process (A-NHP)

A-NHP replaces the LSTM with a Transformer architecture. To handle continuous time, it uses continuous-time positional encodings—sinusoidal functions of the timestamp $t$. Unlike standard Transformers that attend to a fixed grid of tokens, A-NHP computes an embedding for a potential event at any time $t$ by attending to all previous events. The attention weights vary smoothly as time $t$ advances, allowing the model to predict intensities (instantaneous probabilities) that evolve continuously.

Attentive Neural Datalog Through Time (A-NDTT)

A-NDTT is a neuro-symbolic extension that uses a logic programming language (Datalog) to maintain a database of facts about the world. It incorporates two types of symbolic flow:

  • Horizontal Flow: Symbolic "add" rules define which past events the model should attend to.
  • Vertical Flow: Symbolic "deduction" rules define how current facts combine to create new embeddings.

This hybrid approach allows a human expert to guide the attention mechanism, restricting focus to logically relevant facts and improving data efficiency in complex, multi-agent environments.

Architectural comparison of A-NHP and A-NDTT showing horizontal and vertical information flow.

Experiments and Results

The models were evaluated on a mix of synthetic and real-world benchmarks, including MIMIC-II (clinical visit records), StackOverflow (user badge awards), and RoboCup (robot soccer logs).

Performance Comparison

A-NHP matched or outperformed the original LSTM-based NHP and other recent Transformer variants (like THP and SAHP) on the MIMIC-II and StackOverflow datasets. The attention mechanism's ability to examine the entire history at every step proved crucial for capturing long-range influences that a fixed-dimensional LSTM state might drop.

Neuro-Symbolic Advantage

On the RoboCup dataset, the A-NDTT model drastically reduced the error rate compared to unstructured models. By using symbolic rules as an inductive bias, A-NDTT achieved an error rate of approximately 35%, whereas unstructured models were stuck at 90% or higher.

Quantitative results on MIMIC-II and StackOverflow datasets comparing A-NHP against baseline models.

Discussion and Conclusion

Our findings suggest that "time as a query" is a powerful framework for temporal modeling. As prediction time $t$ moves forward, shifting attention weights naturally model decaying or fluctuating relevance of past events. While training is highly parallelizable, inference remains computationally expensive due to the need to recompute embeddings for each time point $t$. Future work will focus on developing GPU-parallelized neuro-symbolic engines and exploring methods to automatically learn symbolic rules from data.

Summary updated

Abstract

The neural Hawkes process (Mei & Eisner, 2017) is a generative model of irregularly spaced sequences of discrete events. To handle complex domains with many event types, Mei et al. (2020a) further consider a setting in which each event in the sequence updates a deductive database of facts (via domain-specific pattern-matching rules); future events are then conditioned on the database contents. They show how to convert such a symbolic system into a neuro-symbolic continuous-time generative model, in which each database fact and the possible event has a time-varying embedding that is derived from its symbolic provenance. In this paper, we modify both models, replacing their recurrent LSTM-based architectures with flatter attention-based architectures (Vaswani et al., 2017), which are simpler and more parallelizable. This does not appear to hurt our accuracy, which is comparable to or better than that of the original models as well as (where applicable) previous attention-based methods (Zuo et al., 2020; Zhang et al., 2020a).
BibTeX
@misc{yang2021transformer,
  author = {Chenghao Yang and Hongyuan Mei and Jason Eisner},
  title = {{Transformer Embeddings of Irregularly Spaced Events and Their Participants}},
  year = {2021},
  eprint = {2201.00044},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2201.00044},
}
Figure