Abstract
Event forecasting involves predicting the type and timing of future occurrences based on historical sequences. Traditional methods like Hawkes processes and Recurrent Neural Network (RNN)-based point processes often struggle with long-range dependencies or lack interpretability. We present Attentive Neural Point Processes (ANPP), a framework that utilizes a time-aware self-attention mechanism to model event sequences. By explicitly calculating influences between all pairs of historical events and employing a novel logarithmic time-bucket embedding, ANPP achieves superior accuracy in both marker and timestamp prediction while providing clear visual explanations for its forecasts.
Introduction: The Challenge of Event Forecasting
Predicting "what" will happen next and "when" is critical across domains, from e-commerce purchase cycles to clinical diagnosis sequences. Historically, two main paradigms dominated this field:
- Hawkes Processes: These mathematical models assume a fixed, often decaying influence of past events. While interpretable, they lack the flexibility to capture complex, non-linear patterns in real-world data.
- Neural Point Processes: Models like RMTPP leverage RNNs to encode history into hidden states. However, RNNs suffer from a "bottleneck" effect, where the entire history is compressed into a single vector, making it difficult to capture long-term dependencies and offering no transparency into the model's decision-making process.
ANPP addresses these limitations by replacing the recurrent bottleneck with a self-attention mechanism, allowing the model to maintain the distinct identity of every past event.
Methodology: Time-Aware Self-Attention
The core of ANPP is the adaptation of self-attention—originally designed for discrete natural language sequences—to the continuous-time domain.
Key Components
- Time-Aware Self-Attention: Unlike RNNs that process events sequentially, ANPP looks at the entire history simultaneously. It calculates attention scores between every pair of events, weighing how much a specific past action (e.g., a purchase weeks ago) influences the current prediction.
- Inter-event Duration Bucket Embedding: To handle continuous time, the authors discretize intervals between events into $L$ logarithmic buckets. This allows the model to learn semantic embeddings for different time scales, treating duration as a first-class feature.
- Hybrid Prediction Layer: The model generates a context vector that feeds into two specialized heads:
- Marker Prediction: A standard softmax layer predicts the event type (marker).
- Time Prediction: A conditional intensity function $\lambda^*(t)$ represents the instantaneous probability of an event. The most likely timestamp is then derived through numerical integration.
Interpretability: Visualizing the "Why"
A significant advantage of the attention-based approach is its inherent interpretability. By examining the attention weights, researchers can identify which historical events were most influential for a specific prediction.
For instance, in e-commerce scenarios, the model demonstrates the ability to "attend" to relevant past purchases. When predicting a "Nail Polish" purchase, the model places higher weights on previous "Nail Polish" and "Oils" entries rather than more recent but unrelated items like "Makeup Brushes." This proves the model learns meaningful cross-item relationships over time.
Experimental Results
ANPP was evaluated against five diverse datasets, including high-frequency financial transactions (NYSE), medical records (MIMIC), and e-commerce sequences (Amazon Beauty & Clothes).
- Accuracy Gains: ANPP consistently outperformed state-of-the-art baselines like RMTPP and Neural Hawkes. In marker prediction, the model saw an average accuracy improvement of 38.1%, reaching nearly 78% on the Amazon Clothes dataset.
- Temporal Precision: Time prediction error (MAE) was reduced by an average of 14.9%, with a remarkable 50% improvement in the volatile Financial dataset.
- Robustness: Unlike RNN-based models, which often degrade as historical sequences grow longer, ANPP maintained high performance across varying sequence lengths due to its global attention mechanism.
Conclusion and Future Directions
ANPP represents a significant step forward in making neural point processes both more accurate and more transparent. While the current model uses a fixed history length $T$ with $O(T^2)$ complexity, future iterations could incorporate sparse attention mechanisms to handle even longer sequences. Additionally, exploring dynamic bucket intervals or more complex non-parametric intensity functions could further refine the precision of temporal forecasts.
Summary updated