Preserving Node Distinctness in Graph Autoencoders via Similarity Distillation

Ge Chen, Yulan Hu, Sheng Ouyang, Yong Liu, Cuicui LuoarXiv 2024 · DOI · OpenReview

TL;DR

This paper introduces "ClearGAE," a novel method to improve Graph Autoencoders (GAEs) by preventing node representations from becoming too similar during reconstruction. By employing a knowledge distillation approach with a KL divergence constraint, ClearGAE effectively preserves node distinctness, leading to better performance across various graph tasks.

What Problem Does This Paper Address?

Graph Autoencoders (GAEs) are powerful tools for learning representations of graph-structured data through self-supervised learning. A common approach is to train GAEs by reconstructing the input graph, often using metrics like Mean Squared Error (MSE). However, the authors observe that this reliance on a single reconstruction criterion can lead to a phenomenon called "oversmoothing," where node representations become too similar, losing their unique characteristics. This "collapse" of distinctiveness can hinder the GAE's performance on downstream tasks like node classification and link prediction. The paper aims to address this by preserving the inherent distinctness of nodes present in the original graph throughout the reconstruction process.

Methods

The core of the proposed method, dubbed "ClearGAE," is inspired by knowledge distillation. The authors view the GAE's encoder-decoder architecture as a teacher-student setup. The "teacher" is the raw input graph, which possesses rich node distinctness, and the "student" is the reconstructed graph, which might lose this distinctness.

ClearGAE introduces a KL divergence constraint to transfer the "knowledge of distinctness" from the raw graph to the reconstructed graph. Specifically:

  1. Teacher Signal: For each node in the raw graph, they compute its similarity with its immediate neighbors. This forms a set of similarity scores representing the local structure and distinctness.
  2. Student Signal: Similarly, for each node in the reconstructed graph, they compute the similarity with its neighbors.
  3. Distillation: A KL divergence loss is applied to minimize the difference between these two sets of similarity scores. This encourages the reconstructed graph's node similarities to mimic those of the original graph, thereby preserving distinctness.

This KL divergence loss is optimized alongside the standard reconstruction loss (e.g., MSE), forming the final objective function. The authors emphasize that this is a "plug-and-play" solution that can be easily integrated into existing GAE frameworks.

Datasets

The paper evaluates ClearGAE on a variety of datasets for different tasks:

  • Node Classification & Link Prediction: Cora, Citeseer, PubMed (citation networks), A-Photo (co-purchase graph), A-CS, C-Physics (co-authorship graphs).
  • Graph Classification: IMDB-B, IMDB-M, PROTEINS, COLLAB, MUTAG, NCI1 (a mix of bioinformatics and social network datasets).

Key Insights and Ideas

  • Oversmoothing is a critical issue in GAEs: The paper clearly demonstrates how standard reconstruction losses can lead to node representations becoming overly similar, losing valuable information.
  • Knowledge distillation for graph distinctness: Applying knowledge distillation principles, not just for feature reconstruction but specifically for preserving pairwise node similarity, is a novel approach.
  • KL divergence as a simple yet effective constraint: The KL divergence loss is shown to be a straightforward way to enforce the desired similarity structure.
  • Plug-and-play applicability: The method's ease of integration into existing GAEs makes it highly practical.
  • Distinctness matters for different tasks: The paper shows that preserving node distinctness is beneficial not only for node classification but also for link prediction and graph classification, especially on social network datasets.

Key Results

ClearGAE consistently outperforms baseline methods across various tasks:

  • Node Classification: ClearGAE achieves state-of-the-art or competitive results on most datasets, demonstrating improved accuracy. For example, on Cora, it achieves 85.35±0.35, outperforming many baselines.
  • Link Prediction: While the primary focus is on feature reconstruction, ClearGAE shows competitive performance, often outperforming GraphMAE, its base model.
  • Graph Classification: ClearGAE shows superior performance on graph classification tasks, particularly on social network datasets, highlighting the importance of node distinctness in these scenarios.
  • Pairwise Similarity Analysis: Table V shows that ClearGAE's decoder produces significantly lower pairwise similarity scores compared to a standard GAE (Decoder v1 vs. Decoder v2), indicating preserved distinctness.

Limitations and Future Work

  • C-Physics performance: The paper notes that the performance enhancement for C-Physics was not as satisfactory. They hypothesize that solely preserving differences might not always be optimal and that a balance between similarity and dissimilarity is crucial.
  • Feature reconstruction focus: While ClearGAE improves link prediction, its primary strength lies in feature reconstruction. The paper suggests that further work could explore how to better leverage this distinctness for topological tasks.
  • Hyperparameter tuning: The effectiveness of the KL divergence constraint might depend on the temperature parameter ($\tau$) and the coefficient ($\alpha$), which require careful tuning.

Summary updated

Abstract

Graph autoencoders (GAEs), as a kind of generative self-supervised learning approach, have shown great potential in recent years. GAEs typically rely on distance-based criteria, such as mean-square-error (MSE), to reconstruct the input graph. However, relying solely on a single reconstruction criterion may lead to a loss of distinctiveness in the reconstructed graph, causing nodes to collapse into similar representations and resulting in sub-optimal performance. To address this issue, we have developed a simple yet effective strategy to preserve the necessary distinctness in the reconstructed graph. Inspired by the knowledge distillation technique, we found that the dual encoder-decoder architecture of GAEs can be viewed as a teacher-student relationship. Therefore, we propose transferring the knowledge of distinctness from the raw graph to the reconstructed graph, achieved through a simple KL constraint. Specifically, we compute pairwise node similarity scores in the raw graph and reconstructed graph. During the training process, the KL constraint is optimized alongside the reconstruction criterion. We conducted extensive experiments across three types of graph tasks, demonstrating the effectiveness and generality of our strategy. This indicates that the proposed approach can be employed as a plug-and-play method to avoid vague reconstructions and enhance overall performance.
BibTeX
@misc{chen2024preserving,
  author = {Ge Chen and Yulan Hu and Sheng Ouyang and Yong Liu and Cuicui Luo},
  title = {{Preserving Node Distinctness in Graph Autoencoders via Similarity Distillation}},
  year = {2024},
  eprint = {2406.17517},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.17517},
}
Figure