Tackling Oversmoothing of GNNs with Contrastive Learning

Lecheng Zheng, Dongqi Fu, Jingrui He

TL;DR

Stacking multiple layers in Graph Neural Networks (GNNs) typically causes oversmoothing, where node representations collapse into indistinguishable vectors. This paper introduces the Topology-guided Graph Contrastive Layer (TGCL), a model-agnostic layer that uses topological distance between node adjacency profiles to calibrate a contrastive loss. By penalizing representation similarity based on structural graph topology, TGCL enables deep GNN architectures to retain discriminative representations—particularly shining when node features are heavily corrupted or missing.

What Problem Does This Paper Address?

Graph Neural Networks update node representations through iterative neighborhood aggregation (message passing). While deeper architectures are routinely used in vision and NLP to expand receptive fields and extract hierarchical features, deep GNNs suffer from two major hurdles:

  1. Vanishing gradients and overfitting, and
  2. Oversmoothing, where repeated Laplacian smoothing drives all node embeddings toward a uniform representation, obliterating discriminative power for classification and link prediction.

In scenarios where node features are incomplete or missing, GNNs must go deeper to pull latent signals from high-order neighbors ($K$-hop neighborhoods). While recent de-oversmoothing strategies (such as PairNorm, DropEdge, and DGN) attempt to maintain variance across embeddings, they often lack a consistent, easy-to-determine divergence metric or cannot be seamlessly plugged into arbitrary architectures.

Methods

The authors propose the Topology-guided Graph Contrastive Layer (TGCL), which can be inserted before the final prediction layer of any standard GNN.

  1. Reinterpreting PairNorm as Contrastive Learning: The authors show via Jensen’s inequality that PairNorm's regularization objective is mathematically a lower bound and special case of a contrastive InfoNCE-like objective.
  2. Topology-Guided Weights: PairNorm uniformly penalizes all non-connected pairs, ignoring the fact that remote nodes (e.g., 2-hop neighbors) might legitimately share cluster identity or structural roles. TGCL addresses this by weighting positive and negative pairs using the Hamming distance between their adjacency vectors ($\mathbf{A}_i, \mathbf{A}_j$): $$\sigma_{ij} = 1 - \frac{\text{dist}(\mathbf{A}_i, \mathbf{A}_j)}{n}, \quad \gamma_{ik} = 1 + \frac{\text{dist}(\mathbf{A}_i, \mathbf{A}_k)}{n}$$
  3. Contrastive Objective: The resulting loss encourages neighbors with overlapping local structures to pull closer, while non-neighbors are pushed apart proportionally to how dissimilar their neighborhood connectivity profiles are: $$\mathcal{L}_{\text{TGCL}} = -\mathbb{E}_{v_i \sim \mathcal{V}, v_j \in \mathcal{N}_i} \left[ \log \frac{\sigma_{ij} f(z_i, z_j)}{\sigma_{ij} f(z_i, z_j) + \sum_{v_k \in \bar{\mathcal{N}}_i} \gamma_{ik} f(z_i, z_k)} \right]$$
  4. Joint Training: The overall loss balances the task loss with TGCL: $\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{agnostic}} + \alpha \mathcal{L}_{\text{TGCL}}$.

Datasets

The authors evaluate TGCL on four standard benchmark datasets under a low-label regime (3% training, 10% validation, 87% testing):

  • Cora (Citation network: 2,708 nodes, 5,429 edges, 7 classes)
  • CiteSeer (Citation network: 3,327 nodes, 9,228 edges, 6 classes)
  • PubMed (Diabetes citation network: 19,717 nodes, 88,651 edges, 3 classes)
  • Reddit (Social post network: 4,584 nodes, 19,460 edges)

No new datasets were introduced.

Key Insights and Ideas

  • Oversmoothing mitigation via Mutual Information: The paper proves theoretically that minimizing $\mathcal{L}_{\text{TGCL}}$ maximizes a lower bound on the mutual information $I(z_i, z_j)$ between representations of connected node pairs. The bound is especially tight for sparser graphs.
  • Not all non-neighbors are equally "negative": Standard negative sampling in graphs treats all disconnected pairs identically. Weighting contrastive negatives by topological divergence prevents the model from aggressively repelling nodes that share identical 2-hop neighborhood structures or community labels.
  • Deep GNNs excel when data is missing: Deep GNNs are often dismissed as unnecessary because 2-to-4 layer networks saturate performance on standard benchmarks. However, the authors demonstrate that as feature missingness grows (up to 75%), the optimal model depth shifts from 3–4 layers to 15–30 layers, proving deep GNNs are genuinely needed when local node signals are scarce.

Key Results

  • 10-Layer Benchmark: Under a 10-layer GCN backbone, TGCL outperforms PairNorm, DropEdge, GCNII, and DGN on Cora (0.7199), PubMed (0.8090), and is competitive on CiteSeer (0.5783) and Reddit (0.7556). Combining TGCL with residual connections (GCN+ResNet) achieves state-of-the-art performance across datasets (e.g., 0.7699 on Cora, 0.8192 on PubMed).
  • Extreme Depth (60 Layers): On Cora at 60 hidden layers, vanilla GCN performance collapses, but TGCL maintains a >7.5% improvement over GCN, and TGCL+ResNet beats GCN+ResNet by >4%.
  • Feature Missingness Scenario: When 75% of node features are randomly masked, TGCL+ResNet achieves 0.805 accuracy on PubMed at layer depth 20, outperforming GCN+ResNet (0.737 at 8 layers) by 6.8%.
  • Efficiency: TGCL runtime scales linearly with network depth.

Limitations and Future Work

  • Computational Complexity of Negatives: Computing adjacency vector distances over all non-neighbor pairs ($\bar{\mathcal{N}}_i$) can become a bottleneck on massive, dense graphs without aggressive mini-batch negative sampling.
  • Homophily Assumption: The topological weighting heuristic assumes that structural similarity in adjacency profiles implies semantic similarity. In highly heterophilic graphs (where connected nodes differ), this assumption may degrade performance.
  • Evaluation Scope: The paper mainly benchmarks citation networks and Reddit; evaluating on large-scale benchmarks (e.g., OGB) was left for future exploration.

Summary updated

Abstract

Graph neural networks (GNNs) integrate the comprehensive relation of graph data and the representation learning capability of neural networks, which is one of the most popular deep learning methods and achieves state-of-the-art performance in many applications, such as natural language processing and computer vision. In real-world scenarios, increasing the depth (i.e., the number of layers) of GNNs is sometimes necessary to capture more latent knowledge of the input data to mitigate the uncertainty caused by missing values. However, involving more complex structures and more parameters will decrease the performance of GNN models. One reason called oversmoothing is recently proposed, whose research still remains nascent. In general, oversmoothing makes the final representations of nodes indiscriminative to hurt the node classification and link prediction performance. In this paper, we first survey the current de-oversmoothing methods and propose three major metrics to evaluate a de-oversmoothing method, i.e., constant divergence indicator, easy-to-determine divergence indicator, and model-agnostic strategy. Then, we propose the Topology-guided Graph Contrastive Layer, named TGCL, which is the first de-oversmoothing method maintaining the three mentioned metrics. With the contrastive learning manner, we provide the theoretical analysis of the effectiveness of the proposed method. Last but not least, we design extensive experiments to illustrate the empirical performance of TGCL comparing with state-of-the-art baselines.
BibTeX
@inproceedings{zheng2022tackling,
  author = {Lecheng Zheng and Dongqi Fu and Jingrui He},
  title = {{Tackling Oversmoothing of GNNs with Contrastive Learning}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=kQMXLDF_z20},
}

Loading related content...

Figure