Abstract
Graph Contrastive Learning (GCL) has recently drawn much research interest for learning generalizable node representations in a self-supervised manner. In general, the contrastive learning process in GCL is performed on top of the representations learned by a graph neural network (GNN) backbone, which transforms and propagates the node contextual information based on its local neighborhoods. However, nodes sharing similar characteristics may not always be geographically close, which poses a great challenge for unsupervised GCL efforts due to their inherent limitations in capturing such global graph knowledge. In this work, we address these limitations by proposing a simple yet effective framework: Simple Neural Networks with Structural and Semantic Contrastive Learning ($S^3$-CL). Notably, by virtue of the proposed structural and semantic contrastive learning algorithms, even a simple neural network can learn expressive node representations that preserve valuable global structural and semantic patterns. Our experiments demonstrate that the node representations learned by $S^3$-CL achieve superior performance on different downstream tasks compared with the state-of-the-art unsupervised GCL methods.
Introduction
Most Graph Contrastive Learning (GCL) methods rely on shallow GNN backbones that utilize message-passing schemes restricted to immediate neighborhoods. This architectural choice leads to two primary challenges: structural blindness and semantic ignorance.
- Structural Blindness: Nodes with similar characteristics are often distant in the graph. Capturing these relationships typically requires deep GNNs, which are prone to over-smoothing—where node representations become indistinguishable—and high computational costs.
- Semantic Ignorance: Standard GCL focuses on instance-level contrast, often ignoring the broader community or cluster structures inherent in the data.
To address these issues, we introduce $S^3$-CL, a framework designed to elicit global knowledge using simple neural network architectures. By decoupling feature propagation from the network transformation, $S^3$-CL enables shallow models to capture long-range dependencies and global semantic patterns effectively.
The $S^3$-CL Framework
The $S^3$-CL architecture consists of two specialized contrastive learning branches integrated with a simple encoder, such as a 1-layer MLP.
Structural Contrastive Learning
Instead of relying on deep GNN layers, $S^3$-CL employs multi-scale feature propagation. Feature matrices are pre-calculated for various "hops" (e.g., 1-hop, 2-hop, etc.). The model learns by maximizing the agreement between a node’s local view and these high-order global views using an InfoNCE loss. This approach captures long-range structural information without the risk of over-smoothing.
Semantic Contrastive Learning
To capture global semantic clusters, the model iteratively groups nodes into prototypes. Since the number of classes is typically unknown in unsupervised settings, we utilize Bayesian non-parametric prototype inference (specifically a Dirichlet Process Mixture Model). This allows the model to dynamically discover the optimal number of clusters during training. The optimization follows an Expectation-Maximization (EM) approach: the E-step infers prototypes and assignments, while the M-step updates encoder weights to enhance intra-cluster compactness in the latent space.
Experimental Results
We evaluated $S^3$-CL on six benchmarks, including citation networks (Cora, Citeseer, Pubmed, ogbn-arxiv) and co-purchase/co-authorship graphs.
Robustness to Over-smoothing
A key advantage of $S^3$-CL is its stability across different propagation scales. As shown in the figure below, while baseline GCL methods experience a sharp decline in accuracy as the number of propagation steps $L$ increases, $S^3$-CL maintains consistently high performance.
Performance and Efficiency
$S^3$-CL consistently outperforms state-of-the-art unsupervised methods like MVGRL and MERIT. On the Cora dataset, it achieved an accuracy of 84.5%, surpassing several supervised baselines. Furthermore, the model exhibits superior clustering performance, with a 2.2% improvement in Normalized Mutual Information (NMI) on Cora. Due to its simple backbone, $S^3$-CL is significantly more parameter-efficient and faster than deep GNN alternatives. It also demonstrates high robustness in low-resource settings, maintaining a lead even when only 1% of labels are available for fine-tuning.
Conclusion
$S^3$-CL demonstrates that complex GNN architectures are not strictly necessary for state-of-the-art graph representation learning if global structural and semantic knowledge is properly elicited. By decoupling propagation from transformation and introducing automated clustering, the framework provides a scalable and effective solution for unsupervised graph learning. Future work will focus on refining prototype inference and extending the method to heterogeneous graph types.
Summary updated