Spectral Augmentations for Graph Contrastive Learning

Amur Ghose, Yingxue Zhang, Jianye Hao, Mark Coates

Abstract

Contrastive learning has emerged as a premier method for learning representations with or without supervision. Recent studies have shown its utility in graph representation learning for pre-training. However, standard graph augmentations often rely on heuristics that may destroy structural semantics. This paper introduces SGCL (Spectral Graph Contrastive Learning), a framework utilizing spectral graph theory to create semantically-preserving augmentations. By leveraging the graph Laplacian spectrum, SGCL enables robust out-of-domain transfer, allowing models to learn universal structural patterns that generalize across disparate graph domains.

Introduction

In contrastive learning, models are trained to pull together representations of "positive pairs" (augmented views of the same data) while pushing apart "negative pairs." While computer vision benefits from intuitive augmentations like cropping and color jittering that preserve semantic meaning, graph augmentations are significantly more challenging. Standard practices, such as randomly dropping nodes or edges, are ad-hoc heuristics that often destroy the graph's underlying topological properties—the very "semantics" the model needs to learn.

This issue is particularly acute in out-of-domain transfer. When pre-training a Graph Neural Network (GNN) on a social network to later evaluate it on a molecular graph, the model cannot rely on node attributes, which differ entirely between domains. It must rely solely on universal structural patterns. Existing heuristic augmentations fail to preserve these clues, leading to poor generalization.

Methods: The SGCL Framework

The authors propose SGCL, a framework that replaces random heuristics with mathematically motivated transformations derived from the graph's Laplacian spectrum (the eigenvalues and eigenvectors describing connectivity and frequencies).

The SGCL framework pipeline, illustrating the sequence of spectral augmentations and alignment steps.

Key Spectral Augmentations

  1. Spectral Graph Cropping: Instead of random node deletion, SGCL utilizes the Fiedler vector (the second smallest eigenvector of the graph Laplacian) to identify and select a contiguous, structurally meaningful subgraph. The authors provide a theoretical proof that for grid graphs, this method reduces to standard image cropping.
  2. Graph Frequency Component Reordering: By factorizing the graph Laplacian, the model creates a structural positional embedding. SGCL permutes these components, a process mathematically analogous to swapping RGB color channels in an image to teach the model invariance to specific frequency orderings.
  3. Local-Global Embedding Alignment: To prevent misalignment between local views generated via random walks, SGCL uses a global graph embedding as a bridge, aligning local views through Wasserstein-Procrustes alignment.
  4. Augmentation Filtering: A quality-control mechanism measures the similarity between generated views, rejecting candidate augmentations that are too noisy to maintain semantic identity.

Key Results and Discussion

SGCL was evaluated across a wide range of datasets, including DBLP, IMDB, and Reddit, focusing on the challenging "frozen encoder" setting for out-of-domain transfer. In this setting, the pre-trained model is tested on a new domain without any fine-tuning, forcing it to rely entirely on the structural features learned during pre-training.

Comparison of SGCL against state-of-the-art baselines, showing SGCL consistently achieving the highest rank across evaluation tasks.

Performance Gains

  • Superior Transferability: SGCL consistently outperformed state-of-the-art baselines like GCC, GraphCL, and MVGRL. For instance, on the REDDIT-BINARY classification task, SGCL achieved 90.6% accuracy, surpassing GCC (89.8%) and GraphCL (88.7%).
  • Ablation Studies: Replacing spectral cropping with standard random cropping led to a performance drop (e.g., from 73.0% to 71.8% on IMDB-B), validating the importance of the spectral approach.
  • Efficiency: Despite the mathematical complexity, SGCL adds minimal overhead. On the DBLP dataset, SGCL took approximately 41 seconds per epoch, compared to 33 seconds for the baseline GCC.

Conclusion

SGCL demonstrates that mathematically grounded spectral augmentations are superior to random heuristics for graph contrastive learning. By preserving the core topological identity of graphs, SGCL enables GNNs to learn universal structural concepts that transfer effectively across domains. Future work may involve integrating node attributes for domain-specific tasks and exploring even more diverse pre-training corpora to further enhance the model's "generalist" capabilities.

Summary updated

Abstract

Contrastive learning has emerged as a premier method for learning representations with or without supervision. Recent studies have shown its utility in graph representation learning for pre-trainin...
BibTeX
@inproceedings{ghose2023spectral,
  author = {Amur Ghose and Yingxue Zhang and Jianye Hao and Mark Coates},
  title = {{Spectral Augmentations for Graph Contrastive Learning}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2023},
  url = {https://openreview.net/forum?id=iRSacLRCsp},
}
Figure