Introduction
Semi-supervised learning (SSL) has emerged as a powerful paradigm for training deep neural networks by leveraging a small set of labeled data alongside a much larger pool of unlabeled samples. The standard mechanism for SSL is pseudo-labeling, where a model generates labels for unlabeled data and incorporates them into the training set if the model's prediction confidence exceeds a certain threshold.
However, traditional SSL frameworks often assume a balanced class distribution—a condition rarely met in real-world scenarios. In imbalanced or "long-tailed" datasets, standard models become biased toward majority (head) classes, leading to high-confidence but incorrect predictions for minority (tail) classes. This paper introduces InPL (Inlier Pseudo-Labeling), a novel approach that replaces traditional softmax confidence thresholds with energy-based scores to improve the reliability of pseudo-labels in imbalanced settings.
The Challenge: Softmax Bias and Overconfidence
In imbalanced SSL, the reliance on softmax confidence scores creates a "vicious cycle." Because the model is exposed to more head-class samples, it produces higher confidence scores for those classes, even when the input belongs to a tail class or is entirely out-of-distribution (OOD).
Deep networks are notoriously overconfident, assigning high softmax scores to samples that do not resemble the training data. In a long-tailed distribution, this results in the model "hallucinating" head-class labels for tail-class unlabeled data, further entrenching the class imbalance and degrading overall performance.
The InPL Framework
InPL shifts the criterion for pseudo-labeling from model confidence to data density. The core insight is to treat pseudo-labeling as an evolving in-distribution detection task. Instead of asking "What class does the model think this is?", InPL asks "Does this sample look like the data we have already learned?"
1. Energy-Based Selection
Instead of softmax probabilities, InPL utilizes the energy score, a metric borrowed from OOD detection literature. Theoretically, a lower energy score corresponds to a higher likelihood that a sample follows the current training data distribution. This provides a more robust measure of whether the model truly "knows" a sample.
2. The "Inliers First" Strategy
At each training iteration, the model calculates energy scores for all unlabeled samples. Only "inliers"—samples with energy scores below a specific threshold ($\tau_e$)—are pseudo-labeled and added to the training process. Initially, only the small set of ground-truth labeled data is considered "in-distribution." As the model learns, its understanding of the distribution expands, allowing it to gradually incorporate more complex unlabeled samples.
3. Adaptive Margin Loss (AML)
To further combat class bias, the authors introduce an optional Adaptive Margin Loss. This component adjusts the decision boundaries based on the moving average of the model's predictions, providing more "breathing room" for minority classes and preventing the head classes from dominating the feature space.
Experimental Results
The researchers evaluated InPL across several challenging benchmarks, including CIFAR10-LT, CIFAR100-LT, and the large-scale ImageNet-127.
State-of-the-Art Performance
InPL demonstrated significant accuracy gains over existing SSL frameworks like FixMatch. On CIFAR10-LT, InPL improved absolute accuracy by approximately 3%. When integrated with the ABC (Auxiliary Balanced Classifier) framework in highly imbalanced scenarios, it achieved a 6.9% improvement over previous state-of-the-art methods. On the large-scale ImageNet-127, InPL outperformed the vanilla FixMatch baseline by nearly 3% (54.82% vs 51.96%).
Precision and Recall
Analysis of the model's internal behavior showed that InPL roughly doubled the recall for tail classes compared to FixMatch. Crucially, it achieved this boost in minority class recognition without sacrificing the precision of the majority classes.
Robustness to OOD Noise
InPL proved remarkably resilient to out-of-distribution noise. In experiments where SVHN images were mixed into a CIFAR10 unlabeled pool, InPL outperformed UDA and FixMatch by over 4%. Because energy scores are better at identifying truly "foreign" data, InPL is less likely to assign labels to irrelevant images.
Conclusion
InPL represents a significant shift in how we approach semi-supervised learning for real-world, imbalanced data. By reframing pseudo-labeling as an inlier detection task and utilizing energy scores, the method avoids the pitfalls of softmax overconfidence. While the unbounded nature of energy scores presents some challenges for hyperparameter tuning, the substantial gains in tail-class recall and OOD robustness make InPL a powerful tool for building more equitable and reliable machine learning models.
Summary updated