On Understanding and Mitigating the Dimensional Collapse of Graph Contrastive Learning: a Non-Maximum Removal Approach

Jiawei Sun, Ruoxin Chen, Jie Li, Chentao Wu, Yue Ding, Junchi YanarXiv 2022 · DOI · arXiv

Abstract

Graph Contrastive Learning (GCL) has shown promising performance in graph representation learning (GRL) without the supervision of manual annotations. GCL can generate graph-level embeddings by maximizing the Mutual Information (MI) between different augmented views of the same graph (positive pairs). However, the GCL is limited by dimensional collapse, i.e., embedding vectors only occupy a low-dimensional subspace. In this paper, we show that the smoothing effect of the graph pooling and the implicit regularization of the graph convolution are two causes of the dimensional collapse in GCL. To mitigate the above issue, we propose a non-maximum removal graph contrastive learning approach (nmrGCL), which removes "prominent" dimensions (i.e., contribute most in similarity measurement) for positive pair in the pre-text task. Comprehensive experiments on various benchmark datasets are conducted to demonstrate the effectiveness of nmrGCL, and the results show that our model outperforms the state-of-the-art methods.

Introduction

Graph Contrastive Learning (GCL) has become a cornerstone of self-supervised learning on graph-structured data, such as social networks and molecular structures. The fundamental objective is to train a model such that different augmented "views" of the same graph (e.g., node dropping or edge perturbation) result in similar embeddings, while views of distinct graphs are pushed apart in the representation space.

Despite its success, GCL often falls victim to dimensional collapse. This phenomenon occurs when the learned embedding vectors cluster into a narrow, low-dimensional subspace rather than utilizing the full high-dimensional space available. This "information concentration" limits the discriminative power of the embeddings, making it difficult for downstream classifiers to distinguish between complex graph structures.

Theoretical Analysis of Dimensional Collapse

The authors identify two primary mathematical drivers of collapse specific to graph neural networks (GNNs):

  1. Smoothing via Pooling: To generate a single vector for an entire graph, models typically use global pooling (e.g., mean or sum pooling). This operation naturally smooths out the variance between augmented views, making them excessively similar even before the contrastive objective is applied.
  2. Implicit Regularization: The paper provides a theoretical proof that multi-layer graph convolutions act as an implicit regularizer, tending toward low-rank weight matrices. This rank deficiency in the weights directly constrains the rank of the resulting embedding space, especially as the depth of the GNN increases.

The nmrGCL Methodology

To mitigate these issues, the researchers propose nmrGCL (Non-Maximum Removal Graph Contrastive Learning). The core of this approach is an adversarial "erasing operation" inspired by non-maximum suppression in computer vision.

Conceptual overview of the nmrGCL framework, illustrating the progression from collapsed (left) to discriminative (right) embedding spaces.

In the nmrGCL framework, for every positive pair of views, the model identifies the "prominent" dimensions in the first view—those that contribute most significantly to the similarity measurement. It then masks these specific dimensions to zero in the second view. By intentionally removing these "shortcuts," the model is forced to maximize similarity using the remaining, less dominant dimensions. This encourages the network to explore and utilize the entire embedding space, learning complementary features rather than redundant ones.

Experimental Evaluation

The effectiveness of nmrGCL was tested across a diverse set of benchmarks, including bioinformatics (PTC-MR, PROTEINS, NCI1, DD) and social networks (COLLAB, IMDB, REDDIT).

Key Results

  • State-of-the-Art Performance: nmrGCL outperformed existing leaders like GraphCL and AD-GCL on eight out of nine graph classification datasets. On the IMDB-B dataset, it achieved a notable 73.83% accuracy, a 2.69% improvement over GraphCL.
  • Robust Transfer Learning: When pre-trained on the massive ZINC 2M dataset and fine-tuned on molecular property tasks (BBBP, ToxCast, etc.), nmrGCL achieved the best performance on three out of four datasets, indicating that the features learned are highly generalizable.
  • Ablation Studies: The authors confirmed that removing the largest (prominent) dimensions is significantly more effective than removing random or smallest dimensions, validating the hypothesis that dominant dimensions are the primary drivers of collapse.
Visualization of dimension activations; nmrGCL prevents the "information concentration" effect where a few dimensions dominate the similarity measurement.

Discussion and Conclusion

nmrGCL addresses the "lazy" nature of standard contrastive learning. By preventing the model from relying on a few high-variance dimensions, the approach ensures a more holistic representation of the graph. While the method shows some sensitivity to the threshold parameter (δ) used to define "prominent" dimensions, it provides a robust and theoretically grounded solution to the dimensional collapse problem in graph-level tasks. Future work may explore learnable masking strategies to further automate this process across varying graph topologies.

Summary updated

Abstract

Graph Contrastive Learning (GCL) has shown promising performance in graph representation learning (GRL) without the supervision of manual annotations. GCL can generate graph-level embeddings by maximizing the Mutual Information (MI) between different augmented views of the same graph (positive pairs). However, the GCL is limited by dimensional collapse, i.e., embedding vectors only occupy a low-dimensional subspace. In this paper, we show that the smoothing effect of the graph pooling and the implicit regularization of the graph convolution are two causes of the dimensional collapse in GCL. To mitigate the above issue, we propose a non-maximum removal graph contrastive learning approach (nmrGCL), which removes "prominent'' dimensions (i.e., contribute most in similarity measurement) for positive pair in the pre-text task. Comprehensive experiments on various benchmark datasets are conducted to demonstrate the effectiveness of nmrGCL, and the results show that our model outperforms the state-of-the-art methods. Source code will be made publicly available.
BibTeX
@misc{sun2022understanding,
  author = {Jiawei Sun and Ruoxin Chen and Jie Li and Chentao Wu and Yue Ding and Junchi Yan},
  title = {{On Understanding and Mitigating the Dimensional Collapse of Graph Contrastive Learning: a Non-Maximum Removal Approach}},
  year = {2022},
  eprint = {2203.12821},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2203.12821},
}
Figure