When labeled data is scarce or expensive to obtain, self-training offers a compelling path forward. The core idea is elegant: take a model trained on a small labeled dataset, use it to generate predictions on a massive pool of unlabeled data, and treat those predictions as "pseudo-labels" to further train the model.
However, this process contains a dangerous flaw. If the model makes an incorrect prediction and subsequently trains on that prediction, it reinforces its own mistake. This phenomenon—known as confirmation bias or error propagation—can cause the model's performance to rapidly degrade. As noted by Jacob Kahn et al. in Self-Training for End-to-End Speech Recognition, aggressive filtering of generated labels is a critical requirement to prevent a model from learning from its own decoding errors.
Confidence-based pseudo-label filtering solves this problem by introducing a simple gatekeeper: the model's own predictive certainty.
Confidence as a Proxy for Quality
How do we break the feedback loop of error propagation without access to ground-truth labels? The foundational insight of this approach is that a model's internal certainty often correlates strongly with its actual accuracy.
As demonstrated by Chaoqun Liu et al. in Zero-to-Strong Generalization: Eliciting Strong Capabilities of Large Language Models Iteratively without Gold Labels, internal metrics such as token probability or reasoning self-consistency serve as highly effective proxies for accuracy. By setting a high threshold on these confidence scores, we can filter out the vast majority of incorrect predictions before they ever touch the training loss.
[Unlabeled Data] ---> [Model Prediction] ---> [Confidence Score]
|
Is Score > Threshold?
/ \
(Yes) (No)
/ \
[Accept as Pseudo-Label] [Discard / Reject]
This thresholding strategy intentionally trades recall for precision. In Unsupervised Adaptation for Deep Stereo, Alessio Tonioni et al. show that setting a high confidence threshold creates a sparse but highly accurate set of pseudo-labels. This sparse supervision is sufficient to guide fine-tuning while successfully preventing the network from inheriting systematic errors from a noisy teacher.
Why It Works: The Automated Curriculum
It might seem counterintuitive that throwing away large portions of unlabeled data improves performance. The reason this mechanism succeeds is that it acts as an automated, self-pacing curriculum.
- Early Training: The model is highly uncertain about complex, noisy, or out-of-distribution samples. It only generates high confidence scores for "easy" samples or those that closely align with its initial training.
- Representation Alignment: By updating its weights using only these high-confidence pseudo-labels, the model refines its underlying feature representations on the target domain without being poisoned by noise.
- Expansion: As the model's representations improve, its decision boundaries stabilize. In subsequent iterations, it naturally becomes more accurate—and consequently more confident—on previously ambiguous or harder samples.
This bootstrapping process allows the model to gradually expand its mastery over the dataset. For instance, Muhammad Khalifa demonstrates in Self-Training Pre-Trained Language Models for Zero- and Few-Shot Multi-Dialectal Arabic Sequence Labeling that filtering model predictions based on fixed probability thresholds or top-$N$ selection allows a model to successfully adapt to regional linguistic variations without error propagation.
Overcoming the Pitfalls of Naive Thresholding
While a single, global confidence threshold is simple to implement, it introduces severe biases when applied to complex real-world tasks. Modern frameworks employ more sophisticated filtering architectures to handle these edge cases.
Class Imbalance and Dominant Biases
Easy-to-learn or dominant classes naturally yield higher confidence scores early in training. If a single global threshold is used, the model will exclusively generate pseudo-labels for these easy classes, completely ignoring minority or difficult classes. To counteract this, Jiaxing Huang et al. in Cross-View Regularization for Domain Adaptive Panoptic Segmentation utilize class-balanced confidence thresholds. By dynamically adjusting the threshold per class, the system ensures that the model remains unbiased and continues to receive supervision signals across all categories.
Tiered Confidence and Multi-Thresholding
Discarding every sample below a high threshold throws away valuable structural information contained in "uncertain" samples. To extract value from these borderline cases, the authors of Diverse Learner: Exploring Diverse Supervision for Semi-supervised Object Detection (Linfeng Li et al.) propose a dual-threshold strategy. By separating pseudo-labels into distinct confidence tiers (high-confidence certainties vs. medium-confidence uncertainties), the system can apply different loss functions to each group, maximizing the utility of the unlabeled data without introducing destabilizing noise.
Alternative Dimensions of Confidence
Confidence does not always have to be derived from raw softmax probabilities. Depending on the task modality, researchers leverage structural or cross-modal consistency as a proxy for reliability:
- Cross-Modal Alignment: In Enhancing Cross Text-Molecule Learning by Self-Augmentation, Yinuo Jiang et al. combine generation likelihoods with cross-modal similarity metrics to filter out noisy text-molecule pairs.
- Adversarial Validation: In Semi-Supervised Semantic Segmentation With High- and Low-Level Consistency, Sudhanshu Mittal et al. employ an adversarial discriminator as a quality metric. The discriminator evaluates whether a pseudo-labeled region forms a coherent, realistic shape, providing a spatial filtering mechanism that goes beyond pixel-level probabilities.