Abstract
Modeling temporal networks requires capturing complex dependencies across time while accounting for sparse dynamics and diverse contextual factors. We present the Transformer-style Relational Reasoning Network (TRRN), which integrates multi-head self-attention with a dynamic memory system. TRRN utilizes a policy network with differentiable binary routers, optimized via Gumbel-Softmax, to selectively update relevant memory states. This approach enables efficient relational reasoning across distant time steps. Evaluations on four real-world datasets demonstrate that TRRN significantly outperforms existing temporal models in node classification and link prediction tasks.
Introduction
Most real-world networks, from social media interactions to brain activity, are dynamic. However, standard Graph Neural Networks (GNNs) are often designed for static snapshots. While some temporal models exist, they face three major hurdles: complex temporal dependencies across distant time steps, diverse contextual factors (topology vs. attributes), and sparse dynamics where only a small subset of the system changes at any given moment.
The authors argue that "relational reasoning"—the ability to compare and contrast information from different time points—is the missing piece for successful temporal network modeling. TRRN addresses these challenges by combining Transformer-style self-attention with a dynamic memory system that selectively updates only the most relevant information at each time step.
Methods
TRRN introduces several key technical innovations to facilitate efficient relational reasoning:
Transformer-Style Relational Reasoning
Instead of a single hidden state, TRRN maintains a set of memories. It applies multi-head self-attention across these memories, creating "shortcuts" for information to flow from past observations to the current state. This allows the model to treat memories as interchangeable objects that can "talk" to each other, compartmentalizing different historical patterns.
Dynamic Memory Updating via Policy Networks
Not all memories are useful at every time step. TRRN employs a Policy Network to estimate the probability of each memory being "activated." To make the discrete "activate/inactivate" decision-making process trainable via backpropagation, the authors use the Gumbel-Softmax reparameterization trick.
Spatio-Temporal Integration
The model extracts network factors (e.g., topology via Random Walk with Restart) and node attributes, feeding them into a gated recurrent structure (LSTM-based) that incorporates the updated memories to produce final node representations. By explicitly penalizing the activation of too many memories using a sparsity constraint, the model learns to focus on the most salient changes.
Evaluation
The authors evaluated TRRN on four diverse real-world temporal network datasets: Task-fMRI (brain activity), DBLP5 (co-authorship), Epinions (social trust), and Reddit (keyword-based interaction).
TRRN was tested on Node Classification and Link Prediction tasks, consistently outperforming state-of-the-art competitors like DySAT, STAR, and DANE. On the DBLP5 dataset for node classification, TRRN achieved 88.9% accuracy, a significant jump over the previous best temporal model, STAR (80.3%). In link prediction tasks, TRRN showed much more stable performance across time steps compared to models like GAT, which tended to fluctuate as the network evolved.
Ablation studies confirmed that the dynamic selective memory was crucial; versions of the model that updated all memories or a fixed number of memories performed worse than the selective TRRN.
Conclusion
TRRN offers a robust framework for temporal network modeling by treating memory as a dynamic workspace. Its ability to make explicit decisions about which memories to activate provides both performance gains and interpretability, revealing when the model relies on history versus current attributes. Future work may address scalability for massive graphs and the explicit modeling of higher-order graph motifs.
Summary updated