Introduction
Semi-supervised learning (SSL) is a powerful tool for training deep neural networks when labeled data is scarce but unlabeled data is plentiful. Most state-of-the-art SSL methods rely on "pseudo-labeling": the model predicts labels for unlabeled data and then treats those predictions as ground truth for further training.
However, standard SSL algorithms typically assume that training class distributions are balanced. In the real world, data is often "long-tailed," meaning a few classes have many samples while most have very few. When trained on imbalanced data, standard SSL algorithms generate pseudo-labels that are heavily biased toward majority classes, leading to poor generalization on balanced testing criteria.
To address this, researchers introduce DARP (Distribution Aligning Refinery of Pseudo-label), a simple yet powerful optimization-based plugin that refines these biased labels to match the true class distribution, significantly boosting performance on minority classes.
The Problem: Bias Amplification
When a model is trained on imbalanced data, it naturally becomes biased toward majority classes. In an SSL context, this bias creates a vicious cycle: the model generates biased pseudo-labels, which then reinforce the model's bias during the next training iteration.
The most striking insight from the researchers is that SSL algorithms don't just reflect data imbalance; they amplify it. For instance, if the true imbalance ratio is 150, a standard algorithm like MixMatch might produce pseudo-labels with an effective imbalance ratio of over 1,000. Consequently, while standard SSL improves accuracy for majority classes, it often performs worse than simple supervised learning on minority classes, leading to a total collapse in performance for rare categories.
The DARP Solution
The authors propose DARP as a post-processing step that "refines" biased pseudo-labels before they are used for training. DARP is not a standalone SSL algorithm but a "refinery" compatible with almost any existing SSL framework (like MixMatch, ReMixMatch, or FixMatch).
Key components of the DARP methodology include:
- Convex Optimization: Instead of accepting the model's raw output, DARP formulates a convex optimization problem. It seeks new pseudo-labels that remain as close as possible to the original predictions (measured via KL-divergence) while strictly satisfying a constraint: the sum of all pseudo-labels for a specific class must match the expected number of samples for that class in the unlabeled set.
- Confidence Weighting: The optimization gives more weight to samples where the model is already confident (low entropy), ensuring that high-quality predictions are preserved.
- Noisy Entry Removal: To prevent the model from "hallucinating" minority class labels for samples that clearly belong to a majority class, DARP uses a pre-processing step to zero out small, noisy entries in the original prediction matrix.
- Efficient Solver: Solving this optimization for millions of samples could be a bottleneck. The authors developed a Dual Coordinate Ascent algorithm that solves the problem provably and efficiently, adding less than 20% computational overhead to the total training time.
- Distribution Estimation: If the true distribution of the unlabeled data is unknown, DARP estimates it using a confusion matrix derived from a small held-out portion of the labeled data.
Experimental Results
The researchers evaluated DARP on synthetically long-tailed versions of standard benchmarks, including CIFAR-10, CIFAR-100, and STL-10.
DARP consistently improved the performance of all baseline SSL algorithms across all tested datasets. By forcing the aggregate distribution of pseudo-labels to match the expected distribution, the model is forced to "re-think" its most uncertain predictions, correctly re-assigning many to minority classes.
DARP proved particularly effective in "distribution mismatch" scenarios—where the labeled data is biased toward one set of classes while the unlabeled data is biased toward another. For example, on CIFAR-10 with a severe mismatch ratio ($\gamma_u = 1$), DARP achieved massive relative reductions in balanced test error:
- 77.2% reduction for MixMatch
- 31.4% reduction for ReMixMatch
- 53.1% reduction for FixMatch
Crucially, DARP-enabled models successfully recover minority class performance, gaining significant accuracy on rare classes by effectively leveraging the unlabeled data rather than being poisoned by it.
Limitations and Future Work
While highly effective, DARP has a few limitations:
- Knowledge of Distribution: DARP performs best when the true class distribution of the unlabeled data is known. While the estimation method works, its accuracy depends on the labeled and unlabeled data coming from the same underlying source.
- Hyperparameter Sensitivity: The "noisy entry removal" step relies on a hyperparameter ($\delta$). While a value of 2 worked well across most experiments, it may require tuning for vastly different datasets.
- Out-of-Distribution (OOD) Data: The current formulation assumes all unlabeled data belongs to one of the known classes. Future work could explore how DARP handles "open-set" scenarios where unlabeled data contains entirely unknown categories.
Summary updated