Tackling Oversmoothing of GNNs with Contrastive Learning

Lecheng Zheng, Dongqi Fu, Jingrui He

Introduction

Graph Neural Networks (GNNs) have revolutionized representation learning on non-Euclidean data. However, a persistent challenge in the field is the "shallow" nature of most successful GNN architectures, which typically utilize only 2 or 3 layers. While sufficient for simple graphs, real-world data often requires capturing long-range dependencies. Deepening GNNs traditionally leads to vanishing gradients, over-smoothing (where node representations become indistinguishable), and the "shading neighbors" effect, where distant information inappropriately drowns out local context.

The Deeper-GXX Framework

To overcome these limitations, we introduce Deeper-GXX, a model-agnostic framework designed to scale GNNs to dozens of layers. The framework integrates two primary innovations into arbitrary GNN backbones:

  1. Weight-Decaying Graph Residual Connection (WDG-ResNet): Standard residual connections can amplify distant noise. WDG-ResNet employs a decaying factor ($e^{-l/\lambda}$) to preserve the hierarchical importance of local neighborhoods while still allowing access to deep information.
  2. Topology-Guided Graph Contrastive Loss (TGCL): This contrastive objective mitigates over-smoothing by pulling 1-hop neighbors together in the feature space and pushing non-neighbors apart, ensuring discriminative features are maintained at depth.
The Deeper-GXX framework architecture, featuring Weight-Decaying Graph Residual Connections (WDG-ResNet) and Topology-Guided Graph Contrastive Loss (TGCL).

Experimental Evaluation

Deeper-GXX was evaluated on standard benchmarks including citation networks (Cora, CiteSeer, PubMed), social networks (Reddit), and large-scale graphs (OGB-Arxiv).

Key Results

  • Stability at Depth: On the Cora dataset, a 60-layer Deeper-GXX model achieved 80.59% accuracy, whereas a vanilla GCN collapsed to 29.62% due to over-smoothing.
  • Competitive Performance: Deeper-GXX consistently outperformed state-of-the-art deep GNN baselines, including GCNII, DGN, and PairNorm.
  • Robustness: The model demonstrates significant resilience to missing data. By leveraging its depth, it can "repair" features by borrowing information from distant, similar nodes when local features are masked.
Deeper-GXX performance compared to state-of-the-art deep GNN baselines like PairNorm, GCNII, and DropEdge.

Conclusion and Future Work

Deeper-GXX successfully addresses the classic pitfalls of deep GNNs. A key insight from this work is the connection between the optimal decay hyperparameter and the graph diameter, providing a physical heuristic for model tuning. Future research will focus on optimizing the computational cost of the contrastive loss for massive-scale graphs and exploring multi-hop community-aware sampling strategies.

Summary updated

BibTeX
@inproceedings{zheng2022tackling,
  author = {Lecheng Zheng and Dongqi Fu and Jingrui He},
  title = {{Tackling Oversmoothing of GNNs with Contrastive Learning}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=kQMXLDF_z20},
}
Figure