Abstract
This paper studies learning node representations with graph neural networks (GNNs) for unsupervised scenario. Specifically, we derive a theoretical analysis and provide an empirical demonstration about the non-steady performance of GNNs over different graph datasets, when the supervision signals are not appropriately defined. The performance of GNNs depends on both the node feature smoothness and the locality of graph structure. To smooth the discrepancy of node proximity measured by graph topology and node feature, we proposed SAIL - a novel \underline{S}elf-\underline{A}ugmented graph contrast\underline{i}ve \underline{L}earning framework, with two complementary self-distilling regularization modules, \emph{i.e.}, intra- and inter-graph knowledge distillation. We demonstrate the competitive performance of SAIL on a variety of graph applications. Even with a single GNN layer, SAIL has consistently competitive or even better performance on various benchmark datasets, comparing with state-of-the-art baselines.
Introduction
Graph Neural Networks (GNNs) have emerged as the standard for modeling graph-structured data. However, their practical application is often hindered by two significant challenges. First, high-performing GNNs typically require large amounts of labeled data, which is frequently scarce or expensive to acquire. Second, GNNs are prone to "oversmoothing," where stacking multiple layers causes node representations to converge and lose their discriminative power, destroying the model's predictive utility.
Existing unsupervised approaches often rely on specific pretext tasks, such as link prediction, which can introduce unwanted biases. This paper introduces SAIL (Self-Augmented graph contrastive Learning), a framework designed to enhance GNN expressivity using only the information inherent in the graph itself, effectively bypassing the need for deep architectures and external labels.
The SAIL Framework
The core of SAIL is a self-distillation mechanism that leverages the model's own internal states to refine its representations. The framework consists of two primary modules:
- Intra-distilling Module: This module enforces consistency between raw input features and learned node embeddings. By maximizing the agreement between a node's representation and the features of its neighbors through a contrastive loss, SAIL effectively denoises low-level features by aligning them with high-level representations.
- Inter-distilling Module: This module employs a teacher-student paradigm where the model learns from its own "future" states. A teacher model (a copy of the current target) provides targets for a student model (the target model with added noise). This iterative process allows a shallow GNN to capture the smoothing benefits of deeper networks without the associated risk of oversmoothing.
Experimental Evaluation
We evaluated SAIL across six benchmark datasets: citation networks (Cora, Citeseer, Pubmed), product co-purchase networks (Amazon Computers, Amazon Photo), and a co-author network (CS). These datasets represent a wide spectrum of graph properties, specifically varying in structural locality and feature smoothness. This diversity provides a robust testing ground for the framework's versatility across different graph topologies.
Results and Discussion
SAIL demonstrates state-of-the-art performance across multiple tasks. In node classification, SAIL consistently outperforms both unsupervised baselines (e.g., DGI, GMI, MVGRL) and standard supervised models like GCN and GAT. For instance, on the Cora dataset, SAIL achieves 84.6% accuracy, surpassing the supervised GCN's 81.5%.
A key finding is the efficacy of shallow architectures; a single-layer GNN within the SAIL framework can match or exceed the performance of multi-layer supervised models. This suggests that the primary bottleneck in GNN performance is not layer depth, but the efficient utilization of graph-inherent information. Furthermore, SAIL exhibits significant robustness; it maintains high performance even when a substantial portion of the graph's edges are removed, indicating that the self-distillation process learns highly resilient representations.
Conclusion
SAIL presents a robust and efficient framework for unsupervised graph representation learning. By integrating intra- and inter-graph distillation, it overcomes the limitations of oversmoothing and label dependency. The success of single-layer SAIL models highlights a promising direction for developing lightweight yet powerful GNNs. While highly effective, future research could explore SAIL's resilience against adversarial attacks and its potential for transfer learning across different graph domains or few-shot classification scenarios.
Summary updated