Self-Adaptable Point Processes with Nonparametric Time Decays

Zhimeng Pan, Zheng Wang, Jeff M. Phillips, Shandian ZheNeurIPS 2021 · OpenReview

SPRITE (Self-adaptable Point pRocess wIth nonparametric Time dEcays) is a novel temporal point process framework designed to model multi-type event data with unprecedented flexibility and interpretability. By combining Gaussian Processes with neural embeddings, it captures both excitation and inhibition effects while guaranteeing that influence strength decays over time, outperforming both traditional Hawkes processes and modern RNN-based models.

The Flexibility-Interpretability Trade-off

In many real-world systems—like social media retweets, financial trades, or healthcare hospital visits—events do not happen in isolation. One event often triggers or inhibits another. To model this, researchers typically use Temporal Point Processes (TPPs). However, existing TPPs face a strict trade-off between flexibility and interpretability:

  1. Hawkes Processes (HP): These are highly interpretable but rigid. They usually assume that the influence of an event decays exponentially over time. If the real-world decay is linear or quadratic, HPs fail. They also struggle to model "inhibition" (where one event makes another less likely).
  2. RNN-based Models: These are highly expressive and can model complex dependencies, but they act as "black boxes." It is nearly impossible to extract meaningful knowledge, such as exactly how much a "Type A" event influences a "Type B" event over a specific time window.

SPRITE bridges this gap by providing the expressiveness of neural networks alongside the clear, decoupled influence structures of classical point processes.

The SPRITE Methodology

SPRITE introduces several clever mathematical components to achieve its goals:

  • Neural Embeddings: Each event type is represented by a low-dimensional vector (embedding). The influence between two types is modeled as a function of their embeddings and the time elapsed since the first event. This allows the model to scale to many event types ($K$) without needing $K^2$ separate functions.
  • Nonparametric Decay Construction: The core innovation is a general mathematical form for the influence function that guarantees the magnitude of influence decreases over time (monotonicity) without forcing it into a specific shape like an exponential curve.
  • Gaussian Process (GP) Priors: To keep the model nonparametric, GP priors are placed over the latent functions. This allows the model to "learn" the shape of the decay (e.g., linear, power-law, or custom) directly from the data.
  • Scalable Inference: To avoid the $O(N^3)$ complexity typical of GPs, the authors use weight-space augmentation, approximating the GP using a finite set of basis functions generated by a neural network.
  • Gauss-Legendre Quadrature: Because the rate function involves complex integrals lacking closed-form solutions, this numerical integration technique is used to maintain efficiency and accuracy during training.

Datasets and Evaluation

SPRITE was evaluated on both synthetic and real-world datasets:

  • Synthetic Data: Designed to test if the model could recover Quadratic, Linear, and Exponential decay patterns, as well as mixed excitation/inhibition.
  • Retweets: 24,000 sequences from Twitter, categorized by user "influence" levels.
  • MIMIC: Clinical visit sequences for 650 patients, involving 75 different diagnosis types.
  • Stack Overflow (SO): 6,633 sequences of users receiving different types of badges/awards (22 types).

Key Insights and Results

  • Enforced but Flexible Decay: The derivation of a construction that satisfies the constraint ensuring influence strength always fades over time provides a helpful inductive bias. It prevents the model from overfitting while remaining more flexible than an exponential assumption.
  • Decoupling Influences: Unlike RNNs, which mash all past events into a single hidden state, SPRITE keeps influences separate. This allows researchers to see the specific interaction between any two event types.
  • Synthetic Recovery: SPRITE was the only model that accurately recovered ground-truth rate functions for non-exponential (linear and quadratic) decays. Traditional Hawkes processes and neural variants showed significant deviations.
SPRITE accurately recovers ground-truth rate functions across various decay types (Quadratic, Linear, Exponential), outperforming baselines like HP and NeuralHP.
  • Predictive Accuracy: On real-world data, SPRITE consistently outperformed state-of-the-art models like NeuralHP and SAHP (Self-Attentive Hawkes Process) in predicting both the time of the next event (RMSE) and the type of the next event (Accuracy).
  • Social Insights: In the Retweet dataset, SPRITE discovered that "large" users (influencers) have a massive, long-lasting excitatory effect on "small" users, whereas "small" users have almost zero influence on "large" users.

Limitations and Future Work

While SPRITE enforces that influence magnitude must decay over time, some real-world processes might have "delayed" influences (where the peak effect happens some time after the trigger). SPRITE, in its current form, cannot model these non-monotonic patterns. Additionally, although weight-space augmentation scales the GPs, the model is still more computationally intensive than a simple Hawkes process due to the numerical integration steps. Future directions include exploring more complex embedding structures or applying the nonparametric decay framework to other types of stochastic processes.

Summary updated

Abstract

Many applications involve multi-type event data. Understanding the complex influences of the events on each other is critical to discover useful knowledge and to predict future events and their types. Existing methods either ignore or partially account for these influences. Recent works use recurrent neural networks to model the event rate. While being highly expressive, they couple all the temporal dependencies in a black-box and can hardly extract meaningful knowledge. More important, most methods assume an exponential time decay of the influence strength, which is over-simplified and can miss many important strength varying patterns. To overcome these limitations, we propose SPRITE, a $\underline{S}$elf-adaptable $\underline{P}$oint p$\underline{R}$ocess w$\underline{I}$th nonparametric $\underline{T}$ime d$\underline{E}$cays, which can decouple the influences between every pair of the events and capture various time decays of the influence strengths. Specifically, we use an embedding to represent each event type and model the event influence as an unknown function of the embeddings and time span. We derive a general construction that can cover all possible time decaying functions. By placing Gaussian process (GP) priors over the latent functions and using Gauss-Legendre quadrature to obtain the integral in the construction, we can flexibly estimate all kinds of time-decaying influences, without restricting to any specific form or imposing derivative constraints that bring learning difficulties. We then use weight space augmentation of GPs to develop an efficient stochastic variational learning algorithm. We show the advantages of our approach in both the ablation study and real-world applications.
BibTeX
@inproceedings{pan2021self,
  author = {Zhimeng Pan and Zheng Wang and Jeff M. Phillips and Shandian Zhe},
  title = {{Self-Adaptable Point Processes with Nonparametric Time Decays}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=aFQHBNeABb},
}
Figure