Temporal Graph Representation Learning with Adaptive Augmentation Contrastive

Hongjiang Chen, Pengfei Jiao, Huijun Tang, Huaming Wu

TL;DR

This paper introduces TGAC, a novel approach for temporal graph representation learning that tackles noise by adaptively augmenting graphs with both structural and temporal information. By combining pruning, novel augmentation strategies, and a contrastive learning framework, TGAC learns more robust node embeddings for predicting future interactions in dynamic networks.

What Problem Does This Paper Address?

Temporal graphs are increasingly used to model real-world systems like social networks and recommendation engines, where interactions evolve over time. However, these dynamic networks are susceptible to time-varying noise, which can significantly degrade the performance of representation learning models. Existing methods often focus on capturing fine-grained details, inadvertently learning noise, or struggle to handle time-dependent noise effectively. While contrastive learning has shown promise for static graphs, its application to temporal graphs, especially with time-dependent noise, remains a challenge. This paper aims to bridge this gap by developing a robust temporal graph representation learning method that can effectively mitigate noise and capture essential temporal dynamics.

Methods

The core of the proposed method, TGAC, lies in its adaptive augmentation strategy and contrastive learning framework.

  • Temporal Graph Pruning: Before augmentation, TGAC prunes noisy edges from the temporal graph. This is achieved by calculating an "edge centrality" that considers both node centrality (using degree, eigenvector, or PageRank) and the temporal information of the edge. Edges with lower centrality are removed, aiming to retain more informative connections.
  • Adaptive Augmentation: TGAC generates two distinct "views" of the pruned temporal graph for contrastive learning. Unlike static graph augmentation, TGAC's augmentation strategies are designed to incorporate temporal information. The paper proposes three specific strategies that modify topological features to reduce noise while preserving vital temporal dynamics.
  • Temporal Graph Encoder: The model utilizes a Temporal Graph Neural Network (TGNN) encoder, inspired by TGN, which maintains a memory vector for each node to store its interaction history. This encoder updates node embeddings by considering neighborhood information and memory states.
  • Contrastive Learning Framework: TGAC employs a contrastive loss function to maximize the agreement between node representations from the two augmented views. This involves both intra-view contrast (comparing nodes within the same view) and inter-view contrast (comparing representations of the same node across different views). The objective is to pull positive pairs (representations of the same node) closer while pushing negative pairs (representations of different nodes) farther apart.
  • Task Loss: In addition to the contrastive loss, a link prediction task loss is incorporated to ensure the learned representations are useful for downstream tasks.

Datasets

The paper evaluates TGAC on four public temporal graph datasets:

  • Wikipedia: A citation network.
  • Reddit: A social interaction network.
  • MOOC: A course enrollment network.
  • CollegeMsg: A messaging network.

These datasets are used for both temporal link prediction and dynamic node classification tasks.

Key Insights and Ideas

  • Noise is a critical issue in temporal graphs: The paper strongly emphasizes that time-varying noise can significantly hinder the performance of temporal graph representation learning, and simply capturing fine-grained details can be detrimental.
  • Temporal information is key for augmentation: Static graph augmentation methods are insufficient for temporal graphs. TGAC's novelty lies in its ability to adaptively augment temporal graphs by explicitly considering temporal dynamics, not just structural changes.
  • Pruning noisy edges is beneficial: Pre-processing the temporal graph by removing less important edges based on a combined structural and temporal centrality measure helps improve the quality of learned representations.
  • Contrastive learning can be effectively adapted for temporal graphs: By generating multiple temporal views and applying contrastive objectives, TGAC learns more robust and generalizable node embeddings.

Key Results

TGAC demonstrates superior performance across various temporal graph tasks:

  • Temporal Link Prediction: TGAC consistently outperforms state-of-the-art methods on both transductive and inductive link prediction tasks across the four datasets. For instance, on the Wikipedia dataset for transductive link prediction, TGAC-PR achieves an AUC of 98.85±0.0 and AP of 98.90±0.0, often outperforming other methods by a significant margin.
  • Dynamic Node Classification: TGAC also shows strong performance in dynamic node classification, outperforming existing methods on Wikipedia, Reddit, and Mooc datasets.
  • Ablation Studies: The paper's ablation studies confirm the effectiveness of both the pruning mechanism and the contrastive learning component. Removing either aspect leads to a notable decline in performance, highlighting their importance.
  • Parameter Sensitivity: The experiments show that TGAC is relatively robust to hyperparameter choices, achieving good results with settings like c=0.05, λ=0.1, and pe=0.4.

Limitations and Future Work

  • Computational Cost: While TGAC improves efficiency compared to some methods like CAWs, the paper notes that CAWs, despite its effectiveness, requires significant processing time due to its online time random walk sampling. TGAC aims to improve efficiency, but further optimization might be explored.
  • Dataset Specificity: The paper evaluates on four public datasets. Future work could explore TGAC's performance on a wider range of temporal graph types and sizes.
  • Adaptive Augmentation Strategies: While three adaptive augmentation strategies are proposed, exploring more sophisticated or dynamic augmentation techniques could further enhance robustness.
  • Hyperparameter Tuning: Although parameter sensitivity is analyzed, further research into automated hyperparameter optimization for TGAC could be beneficial.

Summary updated

Abstract

Temporal graph representation learning aims to generate low-dimensional dynamic node embeddings to capture temporal information as well as structural and property information. Current representation learning methods for temporal networks often focus on capturing fine-grained information, which may lead to the model capturing random noise instead of essential semantic information. While graph contrastive learning has shown promise in dealing with noise, it only applies to static graphs or snapshots and may not be suitable for handling time-dependent noise. To alleviate the above challenge, we propose a novel Temporal Graph representation learning with Adaptive augmentation Contrastive (TGAC) model. The adaptive augmentation on the temporal graph is made by combining prior knowledge with temporal information, and the contrastive objective function is constructed by defining the augmented inter-view contrast and intra-view contrast. To complement TGAC, we propose three adaptive augmentation strategies that modify topological features to reduce noise from the network. Our extensive experiments on various real networks demonstrate that the proposed model outperforms other temporal graph representation learning methods.
BibTeX
@inproceedings{chen2023temporal,
  author = {Hongjiang Chen and Pengfei Jiao and Huijun Tang and Huaming Wu},
  title = {{Temporal Graph Representation Learning with Adaptive Augmentation Contrastive}},
  booktitle = {ECML/PKDD},
  year = {2023},
  url = {https://openreview.net/forum?id=RSNzapp318},
}
Figure