Abstract
Graph neural networks (GNN) such as GCN, GAT, MoNet have achieved state-of-the-art results on semi-supervised learning on graphs. However, when the number of labeled nodes is very small, the performances of GNNs downgrade dramatically. Self-training has proved to be effective for resolving this issue, however, the performance of self-trained GCN is still inferior to that of G2G and DGI for many settings. Moreover, additional model complexity make it more difficult to tune the hyper-parameters and do model selection. We argue that the power of self-training is still not fully explored for the node classification task. In this paper, we propose a unified end-to-end self-training framework called Dynamic Self-training, which generalizes and simplifies prior work. A simple instantiation of the framework based on GCN is provided and empirical results show that our framework outperforms all previous methods including GNNs, embedding based method and self-trained GCNs by a noticeable margin. Moreover, compared with standard self-training, hyper-parameter tuning for our framework is easier.
Introduction
Semi-supervised node classification is a fundamental task in graph machine learning, typically used for applications like predicting user interests in social networks. While GNNs excel with sufficient labels, they suffer from severe performance degradation in "extreme" scenarios where only one labeled node per class is available. This failure stems from two primary issues: high training instability and imbalanced pseudo-labels. In self-training regimes, models often fall into a "winner-takes-all" trap, where majority-class predictions create a self-reinforcing feedback loop of errors.
The Stabilized Self-Training (SST) Framework
To address these challenges, we introduce the Stabilized Self-Training (SST) framework. SST is designed as a modular enhancement that can be integrated into existing GNN architectures. It consists of three core components:
- Stabilized Pseudo-Labeling: To prevent class dominance, we apply a weighting factor of $1/(N_i + 1)$, where $N_i$ is the count of nodes predicted for class $i$. This stabilizer balances the influence of different classes during backpropagation.
- Dynamic Thresholding ($\beta$): Unlike static self-training, SST re-evaluates pseudo-labels every epoch. Nodes are only retained if their prediction confidence exceeds a threshold $\beta$, allowing the model to correct its "beliefs" as training progresses.
- Negative Sampling Regularization: We incorporate a signal from unsupervised learning by ensuring that nodes distant in the graph maintain distinct label distributions, improving node representations without additional ground-truth labels.
Experimental Analysis
The framework was rigorously evaluated on standard benchmarks including Cora, Citeseer, PubMed, Cora-full, and the large-scale Ogbn-arxiv dataset. We focused on "few-labeled" settings, ranging from 1 to 20 labels per class.
A critical finding was the reduction in training variance. Standard GNNs exhibit massive variance in early epochs when labels are scarce, leading to unpredictable results. As shown in the comparison below, the SST framework significantly flattens this variance, creating a more reliable training trajectory.
Key Results
The SST framework demonstrates its greatest value in the most challenging low-label environments:
- Accuracy Gains: On the Cora dataset with 1 label per class, SSTGCN achieved 62.5% accuracy, representing a 17.9% improvement over standard GCN (44.6%).
- Model Enhancement: Applying SST to the DAGNN architecture (SSTDA) resulted in a 6.6% performance boost on Cora-1.
- Consistency and Efficiency: SST-enhanced models consistently outperformed 10 state-of-the-art competitors across all label counts. Furthermore, the framework remains computationally efficient; for instance, SSTDA converges on the Ogbn-arxiv dataset in approximately 119 seconds.
Conclusion
The SST framework effectively stabilizes GNN training in extreme low-label regimes by addressing gradient discrepancies and class imbalance. While the framework is general, future work will explore its application to a broader range of backbones like GraphSAGE and GAT, and investigate additional unsupervised techniques to further refine the negative sampling component.
Summary updated