Introduction
Graph Neural Networks (GNNs) have demonstrated remarkable success in node classification tasks across various domains, such as citation networks and social graphs. However, their performance is significantly hindered by the presence of noisy labels in real-world datasets. GNNs are particularly prone to overfitting on these incorrect labels, leading to poor generalization. While existing methods attempt to mitigate this through single-model cleaning or multi-teacher distillation, they often suffer from error accumulation or rely on static, manually tuned teacher weights that fail to adapt during the training process.
To address these challenges, we present BO-NNC (Bi-level Optimization Based Multi-Teacher Distillation), a framework designed to robustly train GNNs under high label noise. BO-NNC leverages diverse self-supervised teacher models and a dynamic bi-level optimization strategy to effectively distill knowledge into a student model while filtering out noise.
The BO-NNC Framework
The BO-NNC framework is structured into three integrated modules that work together to refine the learning process:
- Multi-Teacher Construction: To avoid the pitfalls of identical models, BO-NNC employs diverse self-supervised learning (SSL) techniques (e.g., DGI and GCA) to train multiple teacher models. Since these teachers learn from the graph's intrinsic structure without relying on noisy labels, they provide robust and varied perspectives on node classification.
- Bi-level Optimization: This core module treats training as a nested optimization problem. In the lower level, the student model learns to mimic "soft labels" generated from a weighted combination of teacher predictions. In the upper level, the system automatically optimizes the weights assigned to each teacher based on the student's performance on a small set of identified "clean" nodes. This feedback loop ensures the student prioritizes the most helpful teacher advice at any given stage of training.
- Label Improvement: The framework iteratively cleans the dataset by filtering out high-loss nodes (likely noise) and incorporating high-confidence "pseudo-labels" into the training set, gradually improving the quality of the supervision signal.
Experiments and Results
Experimental Setup
BO-NNC was evaluated on five benchmark datasets: Cora, Citeseer, DBLP, Amazon Computers, and Amazon Photo. To simulate real-world conditions, artificial noise was injected using two patterns: Uniform noise (random label changes) and Pair noise (swapping labels between similar classes), with noise rates ranging from 20% to 60%.
Performance and Sensitivity
BO-NNC consistently outperformed state-of-the-art baselines like MTS-GNN and NRGNN. The advantage was most pronounced at high noise levels; for instance, on the Cora dataset with 60% uniform noise, BO-NNC achieved 73.16% accuracy, significantly higher than the 63.34% achieved by the next best method.
Sensitivity analysis regarding the hyperparameters $r$ (filtering threshold) and $\rho$ (pseudo-label selection threshold) shows that the model maintains stable performance across a wide range of values, though optimal settings vary slightly by dataset.
Conclusion
BO-NNC introduces a robust approach to node classification in the presence of label noise by combining self-supervised multi-teacher distillation with bi-level optimization. By dynamically weighting teacher advice and iteratively cleaning labels, the framework effectively prevents error accumulation. Future research will focus on reducing the computational overhead of bi-level optimization to scale the method to massive graphs and extending the framework to other graph-based tasks like link prediction.
Summary updated