Spectral Augmentations for Graph Contrastive Learning

Amur Ghose, Yingxue Zhang, Jianye Hao, Mark Coates

Abstract

Contrastive learning has emerged as a premier method for learning representations with or without supervision. Recent studies have shown its utility in graph representation learning for pre-training. Despite successes, the understanding of how to design effective graph augmentations that can capture structural properties common to many different types of downstream graphs remains incomplete. We propose a set of well-motivated graph transformation operations derived via graph spectral analysis to provide a bank of candidates when constructing augmentations for a graph contrastive objective, enabling contrastive learning to capture useful structural representation from pre-training graph datasets. We first present a spectral graph cropping augmentation that involves filtering nodes by applying thresholds to the eigenvalues of the leading Laplacian eigenvectors. Our second novel augmentation reorders the graph frequency components in a structural Laplacian-derived position graph embedding. Further, we introduce a method that leads to improved views of local subgraphs by performing alignment via global random walk embeddings. Our experimental results indicate consistent improvements in out-of-domain graph data transfer compared to state-of-the-art graph contrastive learning methods, shedding light on how to design a graph learner that is able to learn structural properties common to diverse graph types.

Introduction

Contrastive learning has significantly advanced representation learning by enabling models to learn from unlabeled data. In domains like computer vision, principled augmentations such as cropping and color jittering preserve semantic identity while varying appearance. However, graph representation learning has largely relied on heuristic augmentations—such as random edge dropping or node masking—which are often "blind" to the graph's underlying geometry and can inadvertently destroy critical structural semantics. Furthermore, many existing methods depend on domain-specific node attributes, limiting their generalizability across diverse domains like social networks and molecular structures.

This paper introduces SGCL (Spectral Graph Contrastive Learning), a framework that leverages graph spectral analysis to develop principled, structure-only augmentations. By treating a graph's eigenvectors and eigenvalues as structural "axes" and "frequencies," SGCL enables Graph Neural Networks (GNNs) to learn universal topological properties that transfer effectively across different graph types.

Methodology

The SGCL framework moves beyond heuristics by proposing three novel techniques based on the Graph Laplacian:

  1. Spectral Graph Cropping: Analogous to image cropping, this technique uses the first two non-zero eigenvectors (Fiedler vectors) to define structural axes. Thresholding these values allows for the extraction of contiguous, structurally significant subgraphs.
  2. Frequency Component Reordering: By treating eigenvectors as different "frequencies," the authors propose reordering these components within positional embeddings. This alters the "texture" of the representation without changing the underlying connectivity.
  3. Local-Global Embedding Alignment: To ensure stable contrastive learning, SGCL aligns local views generated via random walks using a global embedding (e.g., Node2vec) as an anchor. This step corrects for potential rotations or flips in local coordinate systems.
Overview of the SGCL pipeline, illustrating the flow from egonet extraction to spectral augmentations and embedding alignment.

Additionally, an Augmentation Filtering step is employed to reject noisy views that deviate too significantly from the original ego-network, ensuring high-quality contrastive pairs.

Experimental Setup

The authors evaluate SGCL in a transfer learning context, pre-training on large-scale networks (e.g., DBLP, Academia, IMDB, Facebook, and LiveJournal) and testing on smaller, out-of-domain tasks. Evaluation benchmarks include:

  • Graph Classification: IMDB-B, IMDB-M, COLLAB, RDT-B, RDT-M.
  • Node Classification: US-Airport and H-index.
  • Similarity Search: Academic graphs from KDD, SIGIR, and SIGMOD.

Results and Key Insights

SGCL demonstrates superior performance in out-of-domain transfer, consistently outperforming state-of-the-art methods like GCC and GraphCL. For instance, on the IMDB-B classification task, SGCL achieved 73.4% accuracy, surpassing GCC's 72.0%.

Comparison of SGCL against state-of-the-art graph contrastive learning methods, showing SGCL consistently achieving the top rank across benchmarks.

Key Insights

  • The Image-Graph Analogy: Graphs can be treated like images through their spectra. Eigenvectors provide a "grid" for operations like cropping that were previously considered domain-specific to Euclidean data.
  • Principled Robustness: Math-driven spectral augmentations are more robust than random heuristics, as they preserve the "semantic essence" of the graph.
  • Alignment is Critical: Because eigenvectors are defined up to a sign flip or rotation, aligning local views to a global structural anchor is essential for effective contrastive learning.

Despite its mathematical depth, SGCL remains computationally efficient, with a runtime of 0.084 seconds per mini-batch, making it competitive with simpler heuristic methods and significantly faster than diffusion-based alternatives.

Conclusion

SGCL establishes a new paradigm for graph contrastive learning by deriving augmentations from spectral properties. While the current focus is on structure-only transfer, future work could integrate these spectral techniques with attribute-aware models to further enhance performance on feature-rich datasets. Additionally, scaling these methods to massive, monolithic graphs remains an open area for exploration.

Summary updated

Abstract

Contrastive learning has emerged as a premier method for learning representations with or without supervision. Recent studies have shown its utility in graph representation learning for pre-training. Despite successes, the understanding of how to design effective graph augmentations that can capture structural properties common to many different types of downstream graphs remains incomplete. We propose a set of well-motivated graph transformation operations derived via graph spectral analysis to provide a bank of candidates when constructing augmentations for a graph contrastive objective, enabling contrastive learning to capture useful structural representation from pre-training graph datasets. We first present a spectral graph cropping augmentation that involves filtering nodes by applying thresholds to the eigenvalues of the leading Laplacian eigenvectors. Our second novel augmentation reorders the graph frequency components in a structural Laplacian-derived position graph embedding. Further, we introduce a method that leads to improved views of local subgraphs by performing alignment via global random walk embeddings. Our experimental results indicate consistent improvements in out-of-domain graph data transfer compared to state-of-the-art graph contrastive learning methods, shedding light on how to design a graph learner that is able to learn structural properties common to diverse graph types.
BibTeX
@inproceedings{ghose2023spectral,
  author = {Amur Ghose and Yingxue Zhang and Jianye Hao and Mark Coates},
  title = {{Spectral Augmentations for Graph Contrastive Learning}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2023},
  url = {https://openreview.net/forum?id=iRSacLRCsp},
}
Figure