Abstract
Semi-supervised learning (SSL) typically assumes that unlabeled data shares the same distribution as labeled data. In practice, unlabeled sets often contain "unseen" classes, leading to confirmation bias and performance degradation. We present SAFE-STUDENT, a teacher-student framework that utilizes a novel Energy-Discrepancy (ED) scoring function to identify unseen classes. By applying Confirmation Bias Elimination (CBE) and Unseen-Class Label Distribution Learning (UCDL), SAFE-STUDENT effectively filters out harmful samples and leverages unseen data as a regularizer. Experimental results across multiple benchmarks demonstrate that SAFE-STUDENT significantly outperforms existing SSL methods in scenarios with high mismatch ratios.
Introduction
Deep Semi-Supervised Learning (SSL) has achieved remarkable success by leveraging large amounts of unlabeled data. However, the "closed-set" assumption—that unlabeled data belongs to the same class set as labeled data—is frequently violated in real-world applications. When "unseen-class" instances are present, standard SSL algorithms often misclassify them into known categories with high confidence, a phenomenon known as confirmation bias. This challenge, termed Safe Deep semi-supervised learning with Unseen-class unlabeled data (SDU), requires robust mechanisms to distinguish between seen and unseen classes.
Methodology: The SAFE-STUDENT Framework
SAFE-STUDENT addresses the SDU problem through a specialized teacher-student architecture designed for iterative refinement and robust out-of-distribution (OOD) detection.
Energy-Discrepancy (ED) Scoring
Traditional OOD detection methods often rely on energy scores dominated by the maximum logit. SAFE-STUDENT introduces the Energy-Discrepancy (ED) score, which measures the difference between the total energy and the energy calculated after excluding the most confident class. This approach effectively highlights uncertainty in non-maximum logits, providing a more reliable signal for identifying unseen classes than standard confidence metrics.
Learning Objectives
- Confirmation Bias Elimination (CBE) Loss: For samples identified as "seen," the student model is trained to match the teacher's distribution. This loss is specifically formulated to mitigate overconfidence, preventing the student from inheriting errors from the teacher.
- Unseen-Class Label Distribution Learning (UCDL): Rather than discarding unseen-class samples, SAFE-STUDENT uses them for regularization. The student's predictions for these samples are forced toward a uniform distribution using KL divergence, teaching the model to maintain high uncertainty for unknown inputs.
Iterative Optimization
The framework employs a cyclical optimization process. A teacher model identifies reliable seen and unseen sets; the student learns from these sets; and the teacher is then updated via an Exponential Moving Average (EMA) of the student's parameters. This loop creates a virtuous cycle where improved classification leads to better OOD detection.
Experimental Evaluation
The authors evaluated SAFE-STUDENT on MNIST, CIFAR-10, CIFAR-100, and TinyImageNet, introducing varying "mismatch ratios" to simulate polluted data environments.
Key Results
- Robustness to Pollution: SAFE-STUDENT consistently outperformed baselines like FixMatch and DS3L. On CIFAR-10 with a 60% mismatch ratio, it achieved 83.8% accuracy, a 6.9% improvement over the previous state-of-the-art.
- Effective OOD Detection: The ED score consistently yielded higher Area Under the Curve (AUC) values compared to standard energy or confidence scores, particularly as training progressed.
- Regularization Benefits: The inclusion of unseen data via UCDL proved superior to simple exclusion, confirming that "negative signals" help calibrate the model's knowledge boundaries.
Discussion and Conclusion
While SAFE-STUDENT demonstrates significant gains in SDU scenarios, it relies on manual thresholding for class separation, which may require prior knowledge of the mismatch ratio. Additionally, like most deep learning models, it remains potentially vulnerable to adversarial attacks. Future research could focus on automated threshold selection and extending the framework to open-world environments where the number of seen classes is not fixed.
In conclusion, SAFE-STUDENT provides a robust solution for safe SSL by rethinking how models handle uncertainty and out-of-distribution data, turning potentially harmful "unseen" samples into valuable regularization signals.
Summary updated