TL;DR
GLFormer challenges the assumption that self-attention is necessary for dynamic graph learning by replacing it with a lightweight, attention-free token mixer. By combining local time-and-order-aware aggregation with a hierarchical structure, it achieves state-of-the-art link prediction accuracy while slashing computational complexity from quadratic to linear.
What Problem Does This Paper Address?
Dynamic graphs model relationships that evolve over time, such as user interactions on social networks, financial transactions, or traffic flows. To predict future connections (dynamic link prediction), state-of-the-art models heavily rely on Transformer architectures.
However, standard self-attention scales quadratically ($O(N^2)$) with the sequence length of historical interactions. This high computational overhead makes standard Transformers poorly scalable for high-frequency or large-scale real-world graphs. Furthermore, global self-attention indiscriminately pairs all historical events, which can amplify background noise and lead to over-smoothing.
The authors of this paper ask a fundamental question: Is self-attention actually necessary for dynamic graph learning, or can we achieve the same representational power using a simpler, more efficient architecture?
Methods
The authors introduce GLFormer (Global-Lens Transformer), an attention-free, Transformer-style framework designed specifically for dynamic graphs. Its architecture consists of three main components:
- Adaptive Token Aggregation Module: Instead of computing all-to-all pairwise attention, GLFormer performs local aggregation over a sliding window of the $M$ most recent neighbors. It computes aggregation weights by fusing two critical factors: a learnable parameter vector that captures the sequential order of interactions, and a softmax-normalized factor based on the relative time intervals between interactions.
- Hierarchical Aggregation Mechanism: To avoid losing long-term dependencies due to local mixing, GLFormer stacks multiple layers with progressively expanding receptive fields (inspired by dilated causal convolutions). As the network goes deeper, the token mixer considers wider, gapped intervals of past interactions, allowing it to capture long-range temporal patterns efficiently.
- Channel Aggregation Module: A standard channel-wise feed-forward network (FFN) that processes features across channels, maintaining the structural benefits of the overall Transformer design.
Datasets
The model was evaluated on six widely used continuous-time dynamic graph benchmarks:
- Wikipedia and Reddit (user-page and user-post interaction networks)
- MOOC (user-course interaction actions)
- LastFM (online music listening behaviors)
- SocialEvo (proximal mobile phone interactions)
- Enron (email communication networks)
No new datasets were introduced, ensuring direct and fair comparison against existing baselines.
Key Insights and Ideas
- The MetaFormer Hypothesis Holds for Graphs: The authors conducted controlled experiments replacing self-attention with simple pooling or MLPs across five Transformer baselines. Surprisingly, these attention-free variants often matched or exceeded vanilla Transformer performance, proving that the Transformer's macro-architecture (residual connections, layer normalization, FFN) is doing the heavy lifting, not the attention mechanism itself.
- Local & Temporal Context Trumps Global Matching: In dynamic graphs, the most recent interactions and their exact timing carry the strongest predictive signals. Global self-attention dilutes this signal by mixing in irrelevant historical noise.
- Linear Complexity ($O(N)$): By restricting token mixing to local windows and expanding them hierarchically across layers, GLFormer drops the per-layer time complexity to linear relative to sequence length, making it highly scalable.
Key Results
- Superior Accuracy: GLFormer consistently outperformed or matched vanilla Transformer baselines (such as TGAT and DyGFormer) across all six datasets in both Average Precision (AP) and AUC-ROC. For example, on the MOOC dataset using a TGN backbone, GLFormer achieved an AP of 91.14 compared to the vanilla version's 89.15.
- Faster Inference: Due to its linear complexity, GLFormer delivers significantly faster evaluation times compared to attention-heavy baselines, demonstrating an excellent trade-off between speed and accuracy.
Limitations and Future Work
- Transductive Focus: The paper primarily focuses on the transductive link prediction setting (predicting future links between nodes seen during training). Evaluating its performance in inductive settings (handling completely unseen nodes) remains an open direction.
- Hyperparameter Sensitivity: The optimal number of layers and window sizes requires tuning based on the dataset's sequence characteristics. For instance, sequence-based backbones like TCL and DyGFormer required deeper 3-layer configurations, while graph-attention models like TGAT worked best with 2 layers.
- Task Expansion: Future work could explore extending this attention-free token mixing approach to other dynamic graph tasks, such as dynamic node classification or graph-level property prediction.
Summary updated