Balanced Pseudo Labeling for Robust Self Training

This direction is no longer present in the current build. Its last version stays readable and citable here.

Version 1 of 1121 papers151 conceptsfrozen 2026-10-04

In semi-supervised learning, a model acts as its own teacher. It takes a small amount of labeled data, makes predictions on a large pool of unlabeled data, and then treats its most confident predictions as "pseudo-labels" to train itself further.

This process, known as self-training, suffers from a fundamental flaw: confirmation bias. If a model is slightly better at recognizing "Dogs" than "Cats," it will produce more high-confidence pseudo-labels for dogs. During the next round of training, the model sees even more dogs, becoming even more biased. In real-world datasets—which are almost always imbalanced—this creates a "rich get richer" feedback loop where majority classes dominate the training signal and minority classes are eventually ignored entirely.

To break this loop, researchers have converged on a shared strategy: Balanced Pseudo-Labeling. Instead of trusting the model's raw confidence, these methods force the model to select a representative distribution of samples across all classes.

The Failure of Global Thresholds

The standard approach to self-training uses a global confidence threshold. For example, a model might only accept pseudo-labels where its confidence is above 95%.

However, as Yixin Chen notes in SL: Stable Learning in Source-Free Domain Adaption for Medical Image Segmentation, global thresholds are "traps" for imbalanced data. In medical imaging or remote sensing, easy-to-identify features (like "background" or "water") will naturally reach high confidence scores quickly, while difficult or rare features (like "tumors" or "buildings") may never cross the 95% bar.

If we use a global threshold, the model effectively deletes the minority classes from its own training set. Nicholas Botzer and colleagues demonstrate in TK-KNN that this leads to a collapse where the model develops a representative understanding of only a fraction of the label space.

Strategy 1: Per-Class Quotas and Top-K Selection

The most direct way to fix this is to move from a global threshold to a per-class quota. Instead of asking "Is this prediction 95% confident?", the model asks "What are the top 100 most confident samples for this specific class?"

This approach, used by Yadan Luo et al. in Source-Free Progressive Graph Learning, ensures that every class—no matter how difficult—is represented in every training iteration. By selecting a fixed ratio of the most confident samples within each individual category, the model is forced to learn the manifolds of difficult classes.

Similarly, Ali Mottaghi and the team behind AdaEmbed use a $k$-nearest neighbor strategy to generate balanced pseudo-labels. By enforcing class parity during selection, they prevent the model from developing biases even when target labels are extremely scarce.

Strategy 2: Adaptive Thresholding

A more flexible approach is to dynamically adjust the "bar" for each class. If a model is struggling with a minority class, we should lower the confidence threshold for that class to allow more samples in.

The Adsh (Adaptive Thresholding) framework formalizes this by equalizing the proportion of selected pseudo-labels across classes. If the model is over-predicting a majority class, the threshold for that class rises; if it's under-predicting a minority class, the threshold falls. This prevents the common failure mode where minority samples are misclassified as majority ones simply because the majority class has a lower "entry barrier" for confidence.

Strategy 3: Distribution-Aware Re-sampling

When we cannot easily change the threshold, we can change the sampling rate. In CReST, Chen Wei and Kihyuk Sohn exploit a key observation: even when a model has low recall on a minority class, its high-confidence predictions for that class are often still precise.

CReST addresses imbalance by adding pseudo-labeled samples to the training set at a rate inversely proportional to the class frequency in the labeled data. This effectively "oversamples" the minority classes from the unlabeled pool. Shao-Yuan Li et al. apply a similar logic in DASC, using distribution-aware selection to prevent the imbalance ratio from escalating during iterative training.

Why It Works: Decoupling Confidence from Frequency

The intuition behind all these methods is that model confidence is not a neutral metric. In an imbalanced setting, confidence is a confounded variable—it reflects both the model's certainty and the class's prevalence in the training data.

By using mechanisms like the "Pseudo-Class Balance" (PCB) proposed by Jihwan Bang and Jae-Gil Lee, researchers decouple these two factors. Whether it is through Haohua Dong’s "pseudo-balancing" for fairness in Face Gender Classification or Abdullah Alchihabi’s Uni-GNN for graph learning, the goal is the same:

Force the model to see a balanced world, even when the data it is given is skewed.

When the training signal is balanced, the model is forced to learn the unique features of every category rather than relying on the statistical shortcut of simply predicting the most frequent class. This leads to better generalization, higher fairness, and more robust performance in "open-set" or "drifting" environments where the distribution of data is constantly changing.

Papers in this version

Showing 60 of 121 papers.

History & citationVersion 1 · 2026-10-04

No longer present in the current build · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_balanced_pseudo_labeling_for_robust_self_f98c59c4,
  title = {Balanced Pseudo Labeling for Robust Self Training},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/balanced-pseudo-labeling-for-robust-self-training/rdr_f98c59c4242f4fb789857ea04a6b7edb},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_balanced_pseudo_labeling_for_robust_self_f98c59c4_v1,
  title = {Balanced Pseudo Labeling for Robust Self Training},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/balanced-pseudo-labeling-for-robust-self-training/rdr_f98c59c4242f4fb789857ea04a6b7edb/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}