Global-Lens Transformers: Adaptive Token Mixing for Dynamic Link Prediction

Tao Zou, Chengfeng Wu, Tianxi Liao, Junchen Ye, Bowen Du

TL;DR

GLFormer challenges the assumption that self-attention is necessary for dynamic graph learning by replacing it with a lightweight, attention-free token mixer. By combining local time-and-order-aware aggregation with a hierarchical structure, it achieves state-of-the-art link prediction accuracy while slashing computational complexity from quadratic to linear.

What Problem Does This Paper Address?

Dynamic graphs model relationships that evolve over time, such as user interactions on social networks, financial transactions, or traffic flows. To predict future connections (dynamic link prediction), state-of-the-art models heavily rely on Transformer architectures.

However, standard self-attention scales quadratically ($O(N^2)$) with the sequence length of historical interactions. This high computational overhead makes standard Transformers poorly scalable for high-frequency or large-scale real-world graphs. Furthermore, global self-attention indiscriminately pairs all historical events, which can amplify background noise and lead to over-smoothing.

The authors of this paper ask a fundamental question: Is self-attention actually necessary for dynamic graph learning, or can we achieve the same representational power using a simpler, more efficient architecture?

Methods

The authors introduce GLFormer (Global-Lens Transformer), an attention-free, Transformer-style framework designed specifically for dynamic graphs. Its architecture consists of three main components:

  1. Adaptive Token Aggregation Module: Instead of computing all-to-all pairwise attention, GLFormer performs local aggregation over a sliding window of the $M$ most recent neighbors. It computes aggregation weights by fusing two critical factors: a learnable parameter vector that captures the sequential order of interactions, and a softmax-normalized factor based on the relative time intervals between interactions.
  2. Hierarchical Aggregation Mechanism: To avoid losing long-term dependencies due to local mixing, GLFormer stacks multiple layers with progressively expanding receptive fields (inspired by dilated causal convolutions). As the network goes deeper, the token mixer considers wider, gapped intervals of past interactions, allowing it to capture long-range temporal patterns efficiently.
  3. Channel Aggregation Module: A standard channel-wise feed-forward network (FFN) that processes features across channels, maintaining the structural benefits of the overall Transformer design.
Figure 3: Framework of GLFormer.

Datasets

The model was evaluated on six widely used continuous-time dynamic graph benchmarks:

  • Wikipedia and Reddit (user-page and user-post interaction networks)
  • MOOC (user-course interaction actions)
  • LastFM (online music listening behaviors)
  • SocialEvo (proximal mobile phone interactions)
  • Enron (email communication networks)

No new datasets were introduced, ensuring direct and fair comparison against existing baselines.

Key Insights and Ideas

  • The MetaFormer Hypothesis Holds for Graphs: The authors conducted controlled experiments replacing self-attention with simple pooling or MLPs across five Transformer baselines. Surprisingly, these attention-free variants often matched or exceeded vanilla Transformer performance, proving that the Transformer's macro-architecture (residual connections, layer normalization, FFN) is doing the heavy lifting, not the attention mechanism itself.
Figure 2: We show the average precision results for dynamic link prediction on four datasets with three types of token aggregation mechanisms.
  • Local & Temporal Context Trumps Global Matching: In dynamic graphs, the most recent interactions and their exact timing carry the strongest predictive signals. Global self-attention dilutes this signal by mixing in irrelevant historical noise.
  • Linear Complexity ($O(N)$): By restricting token mixing to local windows and expanding them hierarchically across layers, GLFormer drops the per-layer time complexity to linear relative to sequence length, making it highly scalable.

Key Results

  • Superior Accuracy: GLFormer consistently outperformed or matched vanilla Transformer baselines (such as TGAT and DyGFormer) across all six datasets in both Average Precision (AP) and AUC-ROC. For example, on the MOOC dataset using a TGN backbone, GLFormer achieved an AP of 91.14 compared to the vanilla version's 89.15.
  • Faster Inference: Due to its linear complexity, GLFormer delivers significantly faster evaluation times compared to attention-heavy baselines, demonstrating an excellent trade-off between speed and accuracy.

Limitations and Future Work

  • Transductive Focus: The paper primarily focuses on the transductive link prediction setting (predicting future links between nodes seen during training). Evaluating its performance in inductive settings (handling completely unseen nodes) remains an open direction.
  • Hyperparameter Sensitivity: The optimal number of layers and window sizes requires tuning based on the dataset's sequence characteristics. For instance, sequence-based backbones like TCL and DyGFormer required deeper 3-layer configurations, while graph-attention models like TGAT worked best with 2 layers.
  • Task Expansion: Future work could explore extending this attention-free token mixing approach to other dynamic graph tasks, such as dynamic node classification or graph-level property prediction.

Summary updated

Abstract

Dynamic graph learning plays a pivotal role in modeling evolving relationships over time, especially for temporal link prediction tasks in domains such as traffic systems, social networks, and recommendation platforms. While Transformer-based models have demonstrated strong performance by capturing long-range temporal dependencies, their reliance on self-attention results in quadratic complexity with respect to sequence length, limiting scalability on high-frequency or large-scale graphs. In this work, we revisit the necessity of self-attention in dynamic graph modeling. Inspired by recent findings that attribute the success of Transformers more to their architectural design than attention itself, we propose GLFormer, a novel attention-free Transformer-style framework for dynamic graphs. GLFormer introduces an adaptive token mixer that performs context-aware local aggregation based on interaction order and time intervals. To capture long-term dependencies, we further design a hierarchical aggregation module that expands the temporal receptive field by stacking local token mixers across layers. Experiments on six widely-used dynamic graph benchmarks show that GLFormer achieves SOTA performance, which reveals that attention-free architectures can match or surpass Transformer baselines in dynamic graph settings with significantly improved efficiency.
BibTeX
@misc{zou2025global,
  author = {Tao Zou and Chengfeng Wu and Tianxi Liao and Junchen Ye and Bowen Du},
  title = {{Global-Lens Transformers: Adaptive Token Mixing for Dynamic Link Prediction}},
  year = {2025},
  eprint = {2511.12442},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.12442},
}
Figure