InPL: Pseudo-labeling the Inliers First for Imbalanced Semi-supervised Learning

Zhuoran Yu, Yin Li, Yong Jae Lee

Introduction

Semi-supervised learning (SSL) typically relies on pseudo-labeling: a model predicts labels for unlabeled data, and if it is "confident" enough, that prediction becomes a training target. Traditionally, this confidence is measured using a softmax score threshold (e.g., > 0.95).

However, this approach breaks down in imbalanced (long-tailed) settings. Deep networks are notoriously biased toward majority "head" classes and can produce arbitrarily high softmax confidence scores for samples that are actually out-of-distribution (OOD) or belong to minority "tail" classes. This leads to two major failures:

  1. Low Recall: The model rarely feels confident about tail classes, so they are largely ignored during pseudo-labeling.
  2. Poor Precision: The model mistakenly assigns high-confidence majority labels to tail-class samples or OOD noise, reinforcing its own bias.

To address this, researchers have introduced InPL (Inlier Pseudo-Labeling), a novel approach that ditches traditional softmax confidence thresholds in favor of energy scores. By prioritizing "in-distribution" samples (inliers) for pseudo-labeling, the method significantly improves accuracy for minority classes and provides robust protection against out-of-distribution noise.

The InPL Approach: Energy Over Softmax

InPL reframes pseudo-labeling as an evolving in-distribution vs. out-of-distribution classification problem.

Instead of looking at the relative probability (softmax), InPL uses the energy score. Derived from the model's logits, the energy score is theoretically aligned with the probability density of the data. A lower energy score indicates a sample is an "inlier"—it closely resembles the data the model has already seen. Softmax scores are normalized, meaning a model can be "99% sure" about a sample simply because one logit is slightly higher than the others, even if the sample is far from the training manifold. Energy scores provide an absolute measure of "fit."

Comparison of confidence-based pseudo-labeling and InPL.

The InPL Workflow: 1. Energy Calculation: For an unlabeled sample, calculate its energy score from the model's logits. 2. Thresholding: If the energy is below a specific threshold ($\tau_e$), the sample is considered an inlier. 3. Pseudo-labeling: Only these inliers are pseudo-labeled using the model's current prediction. 4. Expansion: Initially, only samples very close to the small labeled set are deemed inliers. As training progresses, the "in-distribution" boundary naturally expands, gradually pulling in more complex samples and tail-class instances.

The expansion of the in-distribution boundary during InPL training.

By labeling what the model actually knows first, InPL builds a more reliable foundation. This prevents the "rich get richer" cycle where majority classes dominate the pseudo-labeling process early on. The authors also integrate an optional Adaptive Margin Loss (AML) to further penalize majority-class bias by adjusting the logit margins based on the moving average of model predictions.

Key Results and Performance

The method was rigorously evaluated on several benchmarks, including long-tailed versions of CIFAR-10/100, ImageNet-127 (a large-scale imbalanced dataset with an imbalance ratio of 286), and specific OOD robustness tests.

InPL demonstrated state-of-the-art performance across the board:

  • Accuracy Boost: On CIFAR-10-LT, InPL improved absolute accuracy by 3% over the standard FixMatch baseline. When combined with other long-tail frameworks, it achieved a 6.9% improvement over the previous best competitor in the most challenging scenarios.
  • Tail Class Recovery: Analysis showed that InPL doubled the recall for tail classes compared to FixMatch without significantly hurting precision.
  • OOD Robustness: In an experiment where SVHN outliers were mixed into CIFAR-10 unlabeled data, InPL outperformed UDA and FixMatch by over 4%, proving it is highly effective at ignoring irrelevant data.
  • Large Scale: On ImageNet-127, InPL maintained a significant lead (~3% improvement), showing the method scales well to complex, real-world data.

Limitations and Future Directions

While highly effective, InPL does have limitations. Energy scores are not as intuitive as probabilities; they are unbounded and their scale changes depending on the number of classes and the model's state. This makes selecting the energy threshold ($\tau_e$) harder than picking a simple 0.95 softmax threshold, often requiring cross-validation and tuning for different datasets.

Future work may explore more interpretable versions of the energy score and investigate how this "inlier-first" logic might apply to other domains, such as natural language processing or imbalanced object detection tasks.

Summary updated

Abstract

Recent state-of-the-art methods in imbalanced semi-supervised learning (SSL) rely on confidence-based pseudo-labeling with consistency regularization. To obtain high-quality pseudo-labels, a high confidence threshold is typically adopted. However, it has been shown that softmax-based confidence scores in deep networks can be arbitrarily high for samples far from the training data, and thus, the pseudo-labels for even high-confidence unlabeled samples may still be unreliable. In this work, we present a new perspective of pseudo-labeling for imbalanced SSL. Without relying on model confidence, we propose to measure whether an unlabeled sample is likely to be "in-distribution''; i.e., close to the current training data. To decide whether an unlabeled sample is "in-distribution'' or "out-of-distribution'', we adopt the energy score from out-of-distribution detection literature. As training progresses and more unlabeled samples become in-distribution and contribute to training, the combined labeled and pseudo-labeled data can better approximate the true class distribution to improve the model. Experiments demonstrate that our energy-based pseudo-labeling method, InPL, albeit conceptually simple, significantly outperforms confidence-based methods on imbalanced SSL benchmarks. For example, it produces a 4-6% absolute accuracy improvement on CIFAR10-LT when the imbalance ratio is higher than 50. When combined with state-of-the-art long-tailed SSL methods, further improvements are attained. In particular, in one of the most challenging scenarios, InPL achieves a 6.9% accuracy improvement over the best competitor.
BibTeX
@inproceedings{yu2023inpl,
  author = {Zhuoran Yu and Yin Li and Yong Jae Lee},
  title = {{InPL: Pseudo-labeling the Inliers First for Imbalanced Semi-supervised Learning}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=m6ahb1mpwwX},
}
Figure