Introduction
Semi-supervised learning (SSL) typically relies on pseudo-labeling: a model predicts labels for unlabeled data, and if it is "confident" enough, that prediction becomes a training target. Traditionally, this confidence is measured using a softmax score threshold (e.g., > 0.95).
However, this approach breaks down in imbalanced (long-tailed) settings. Deep networks are notoriously biased toward majority "head" classes and can produce arbitrarily high softmax confidence scores for samples that are actually out-of-distribution (OOD) or belong to minority "tail" classes. This leads to two major failures:
- Low Recall: The model rarely feels confident about tail classes, so they are largely ignored during pseudo-labeling.
- Poor Precision: The model mistakenly assigns high-confidence majority labels to tail-class samples or OOD noise, reinforcing its own bias.
To address this, researchers have introduced InPL (Inlier Pseudo-Labeling), a novel approach that ditches traditional softmax confidence thresholds in favor of energy scores. By prioritizing "in-distribution" samples (inliers) for pseudo-labeling, the method significantly improves accuracy for minority classes and provides robust protection against out-of-distribution noise.
The InPL Approach: Energy Over Softmax
InPL reframes pseudo-labeling as an evolving in-distribution vs. out-of-distribution classification problem.
Instead of looking at the relative probability (softmax), InPL uses the energy score. Derived from the model's logits, the energy score is theoretically aligned with the probability density of the data. A lower energy score indicates a sample is an "inlier"—it closely resembles the data the model has already seen. Softmax scores are normalized, meaning a model can be "99% sure" about a sample simply because one logit is slightly higher than the others, even if the sample is far from the training manifold. Energy scores provide an absolute measure of "fit."
The InPL Workflow: 1. Energy Calculation: For an unlabeled sample, calculate its energy score from the model's logits. 2. Thresholding: If the energy is below a specific threshold ($\tau_e$), the sample is considered an inlier. 3. Pseudo-labeling: Only these inliers are pseudo-labeled using the model's current prediction. 4. Expansion: Initially, only samples very close to the small labeled set are deemed inliers. As training progresses, the "in-distribution" boundary naturally expands, gradually pulling in more complex samples and tail-class instances.
By labeling what the model actually knows first, InPL builds a more reliable foundation. This prevents the "rich get richer" cycle where majority classes dominate the pseudo-labeling process early on. The authors also integrate an optional Adaptive Margin Loss (AML) to further penalize majority-class bias by adjusting the logit margins based on the moving average of model predictions.
Key Results and Performance
The method was rigorously evaluated on several benchmarks, including long-tailed versions of CIFAR-10/100, ImageNet-127 (a large-scale imbalanced dataset with an imbalance ratio of 286), and specific OOD robustness tests.
InPL demonstrated state-of-the-art performance across the board:
- Accuracy Boost: On CIFAR-10-LT, InPL improved absolute accuracy by 3% over the standard FixMatch baseline. When combined with other long-tail frameworks, it achieved a 6.9% improvement over the previous best competitor in the most challenging scenarios.
- Tail Class Recovery: Analysis showed that InPL doubled the recall for tail classes compared to FixMatch without significantly hurting precision.
- OOD Robustness: In an experiment where SVHN outliers were mixed into CIFAR-10 unlabeled data, InPL outperformed UDA and FixMatch by over 4%, proving it is highly effective at ignoring irrelevant data.
- Large Scale: On ImageNet-127, InPL maintained a significant lead (~3% improvement), showing the method scales well to complex, real-world data.
Limitations and Future Directions
While highly effective, InPL does have limitations. Energy scores are not as intuitive as probabilities; they are unbounded and their scale changes depending on the number of classes and the model's state. This makes selecting the energy threshold ($\tau_e$) harder than picking a simple 0.95 softmax threshold, often requiring cross-validation and tuning for different datasets.
Future work may explore more interpretable versions of the energy score and investigate how this "inlier-first" logic might apply to other domains, such as natural language processing or imbalanced object detection tasks.
Summary updated