Transformer-Style Relational Reasoning with Dynamic Memory Updating for Temporal Network Modeling

Dongkuan Xu, Junjie Liang, Wei Cheng, Hua Wei, Haifeng Chen, Xiang Zhang

Abstract

Modeling temporal networks requires capturing complex dependencies across time while accounting for sparse dynamics and diverse contextual factors. We present the Transformer-style Relational Reasoning Network (TRRN), which integrates multi-head self-attention with a dynamic memory system. TRRN utilizes a policy network with differentiable binary routers, optimized via Gumbel-Softmax, to selectively update relevant memory states. This approach enables efficient relational reasoning across distant time steps. Evaluations on four real-world datasets demonstrate that TRRN significantly outperforms existing temporal models in node classification and link prediction tasks.

Introduction

Most real-world networks, from social media interactions to brain activity, are dynamic. However, standard Graph Neural Networks (GNNs) are often designed for static snapshots. While some temporal models exist, they face three major hurdles: complex temporal dependencies across distant time steps, diverse contextual factors (topology vs. attributes), and sparse dynamics where only a small subset of the system changes at any given moment.

Comparison of TRRN architecture with standard Graph NNs and Gated RNNs.

The authors argue that "relational reasoning"—the ability to compare and contrast information from different time points—is the missing piece for successful temporal network modeling. TRRN addresses these challenges by combining Transformer-style self-attention with a dynamic memory system that selectively updates only the most relevant information at each time step.

Methods

TRRN introduces several key technical innovations to facilitate efficient relational reasoning:

Transformer-Style Relational Reasoning

Instead of a single hidden state, TRRN maintains a set of memories. It applies multi-head self-attention across these memories, creating "shortcuts" for information to flow from past observations to the current state. This allows the model to treat memories as interchangeable objects that can "talk" to each other, compartmentalizing different historical patterns.

Dynamic Memory Updating via Policy Networks

Not all memories are useful at every time step. TRRN employs a Policy Network to estimate the probability of each memory being "activated." To make the discrete "activate/inactivate" decision-making process trainable via backpropagation, the authors use the Gumbel-Softmax reparameterization trick.

The dynamic memory updating mechanism using a policy network and Gumbel-Softmax reparameterization.

Spatio-Temporal Integration

The model extracts network factors (e.g., topology via Random Walk with Restart) and node attributes, feeding them into a gated recurrent structure (LSTM-based) that incorporates the updated memories to produce final node representations. By explicitly penalizing the activation of too many memories using a sparsity constraint, the model learns to focus on the most salient changes.

Evaluation

The authors evaluated TRRN on four diverse real-world temporal network datasets: Task-fMRI (brain activity), DBLP5 (co-authorship), Epinions (social trust), and Reddit (keyword-based interaction).

TRRN was tested on Node Classification and Link Prediction tasks, consistently outperforming state-of-the-art competitors like DySAT, STAR, and DANE. On the DBLP5 dataset for node classification, TRRN achieved 88.9% accuracy, a significant jump over the previous best temporal model, STAR (80.3%). In link prediction tasks, TRRN showed much more stable performance across time steps compared to models like GAT, which tended to fluctuate as the network evolved.

Ablation studies confirmed that the dynamic selective memory was crucial; versions of the model that updated all memories or a fixed number of memories performed worse than the selective TRRN.

Conclusion

TRRN offers a robust framework for temporal network modeling by treating memory as a dynamic workspace. Its ability to make explicit decisions about which memories to activate provides both performance gains and interpretability, revealing when the model relies on history versus current attributes. Future work may address scalability for massive graphs and the explicit modeling of higher-order graph motifs.

Summary updated

Abstract

Network modeling aims to learn the latent representations of nodes such that the representations preserve both network structures and node attribute information. This problem is fundamental due to its prevalence in numerous domains. However, existing approaches either target the static networks or struggle to capture the complicated temporal dependency, while most real-world networks evolve over time and the success of network modeling hinges on the understanding of how entities are temporally connected. In this paper, we present TRRN, a transformer-style relational reasoning network with dynamic memory updating, to deal with the above challenges. TRRN employs multi-head self-attention to reason over a set of memories, which provides a multitude of shortcut paths for information to flow from past observations to the current latent representations. By utilizing the policy networks augmented with differentiable binary routers, TRRN estimates the possibility of each memory being activated and dynamically updates the memories at the time steps when they are most relevant. We evaluate TRRN with the tasks of node classification and link prediction on four real temporal network datasets. Experimental results demonstrate the consistent performance gains for TRRN over the leading competitors.
BibTeX
@inproceedings{xu2021transformer,
  author = {Dongkuan Xu and Junjie Liang and Wei Cheng and Hua Wei and Haifeng Chen and Xiang Zhang},
  title = {{Transformer-Style Relational Reasoning with Dynamic Memory Updating for Temporal Network Modeling}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i5.16583},
}
Figure