Noisy Node Classification by Bi-level Optimization Based Multi-Teacher Distillation

Yujing Liu, Zongqian Wu, Zhengyu Lu, Ci Nie, Guoqiu Wen, Yonghua Zhu, Xiaofeng Zhu

Introduction

Graph Neural Networks (GNNs) have demonstrated remarkable success in node classification tasks across various domains, such as citation networks and social graphs. However, their performance is significantly hindered by the presence of noisy labels in real-world datasets. GNNs are particularly prone to overfitting on these incorrect labels, leading to poor generalization. While existing methods attempt to mitigate this through single-model cleaning or multi-teacher distillation, they often suffer from error accumulation or rely on static, manually tuned teacher weights that fail to adapt during the training process.

To address these challenges, we present BO-NNC (Bi-level Optimization Based Multi-Teacher Distillation), a framework designed to robustly train GNNs under high label noise. BO-NNC leverages diverse self-supervised teacher models and a dynamic bi-level optimization strategy to effectively distill knowledge into a student model while filtering out noise.

The BO-NNC Framework

The BO-NNC framework is structured into three integrated modules that work together to refine the learning process:

  1. Multi-Teacher Construction: To avoid the pitfalls of identical models, BO-NNC employs diverse self-supervised learning (SSL) techniques (e.g., DGI and GCA) to train multiple teacher models. Since these teachers learn from the graph's intrinsic structure without relying on noisy labels, they provide robust and varied perspectives on node classification.
  2. Bi-level Optimization: This core module treats training as a nested optimization problem. In the lower level, the student model learns to mimic "soft labels" generated from a weighted combination of teacher predictions. In the upper level, the system automatically optimizes the weights assigned to each teacher based on the student's performance on a small set of identified "clean" nodes. This feedback loop ensures the student prioritizes the most helpful teacher advice at any given stage of training.
  3. Label Improvement: The framework iteratively cleans the dataset by filtering out high-loss nodes (likely noise) and incorporating high-confidence "pseudo-labels" into the training set, gradually improving the quality of the supervision signal.
Overview of the BO-NNC framework, illustrating multi-teacher construction, bi-level optimization for distillation, and the label improvement process.

Experiments and Results

Experimental Setup

BO-NNC was evaluated on five benchmark datasets: Cora, Citeseer, DBLP, Amazon Computers, and Amazon Photo. To simulate real-world conditions, artificial noise was injected using two patterns: Uniform noise (random label changes) and Pair noise (swapping labels between similar classes), with noise rates ranging from 20% to 60%.

Performance and Sensitivity

BO-NNC consistently outperformed state-of-the-art baselines like MTS-GNN and NRGNN. The advantage was most pronounced at high noise levels; for instance, on the Cora dataset with 60% uniform noise, BO-NNC achieved 73.16% accuracy, significantly higher than the 63.34% achieved by the next best method.

Sensitivity analysis regarding the hyperparameters $r$ (filtering threshold) and $\rho$ (pseudo-label selection threshold) shows that the model maintains stable performance across a wide range of values, though optimal settings vary slightly by dataset.

Sensitivity analysis of accuracy across five datasets with respect to hyperparameters r and ρ.

Conclusion

BO-NNC introduces a robust approach to node classification in the presence of label noise by combining self-supervised multi-teacher distillation with bi-level optimization. By dynamically weighting teacher advice and iteratively cleaning labels, the framework effectively prevents error accumulation. Future research will focus on reducing the computational overhead of bi-level optimization to scale the method to massive graphs and extending the framework to other graph-based tasks like link prediction.

Summary updated

Abstract

Previous graph neural networks (GNNs) usually assume that the graph data is with clean labels for representation learning, but it is not true in real applications. In this paper, we propose a new multi-teacher distillation method based on bi-level optimization (namely BO-NNC), to conduct noisy node classification on the graph data. Specifically, we first employ multiple self-supervised learning methods to train diverse teacher models, and then aggregate their predictions through a teacher weight matrix. Furthermore, we design a new bi-level optimization strategy to dynamically adjust the teacher weight matrix based on the training progress of the student model. Finally, we design a label improvement module to improve the label quality. Extensive experimental results on real datasets show that our method achieves the best results compared to state-of-the-art methods.
BibTeX
@inproceedings{liu2025noisy,
  author = {Yujing Liu and Zongqian Wu and Zhengyu Lu and Ci Nie and Guoqiu Wen and Yonghua Zhu and Xiaofeng Zhu},
  title = {{Noisy Node Classification by Bi-level Optimization Based Multi-Teacher Distillation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i18.34095},
}
Figure