In all LikelihoodS: How to Reliably Select Pseudo-Labeled Data for Self-Training in Semi-Supervised Learning

Julian Rodemann, Christoph Jansen, Georg Schollmeyer, Thomas AugustinarXiv 2023 · DOI · arXiv

Abstract

Semi-supervised learning (SSL) leverages unlabeled data to augment small labeled datasets, primarily through self-training or pseudo-labeling. However, standard self-training is prone to error accumulation, where a model's initial biases are reinforced through confident but incorrect predictions. This paper introduces a robust framework for Pseudo-Label Selection (PLS) that moves beyond single-model confidence scores. By applying formal decision theory and evaluating "all likelihoods" across multiple models and potential labels, the proposed methods significantly improve the reliability of self-training pipelines.

Introduction

In semi-supervised learning, the scarcity of labeled data often necessitates the use of unlabeled samples to improve model performance. Self-training is a widely adopted approach where a model is trained on labeled data, used to predict labels for unlabeled data, and then retrained on the most "confident" of these predictions.

The critical vulnerability of this process lies in the selection step. If the initial model is biased or overconfident in its errors, those errors are incorporated into the training set, creating a feedback loop of misinformation known as error accumulation. Most current PLS methods are brittle because they rely on the internal confidence of a single model. This paper addresses this fragility by reframing pseudo-label selection as a problem of robust decision-making under uncertainty.

Robust Selection Framework

The authors propose that the logic used to select data should be distinct from the logic used to label it. While a complex model might be best for prediction, a more robust, multi-model "committee" or a simpler model hierarchy may be better suited for determining which labels are trustworthy.

1. Multi-Model Likelihood (Pseudo Posterior Predictive)

Instead of trusting the output of a single model, the researchers propose evaluating the likelihood of a pseudo-labeled point across a variety of models. This approach, termed Pseudo Posterior Predictive (PPP), uses an analytical approximation (via the Fisher information matrix) to integrate over model uncertainty, ensuring that selected points are consistent across the model space.

2. Reversed Occam’s Razor

In a novel application of nested model hierarchies, the authors suggest that the most reliable data points are those that can be explained well even by the simplest models. If a simple linear model and a complex neural network both agree on a point's likelihood, that point is less likely to be an artifact of the complex model's overfitting and more likely to represent a genuine pattern in the data.

3. Multi-Label Utility

Standard PLS often looks only at the probability of the most likely class. This framework extends selection to consider the entire predictive distribution. By weighting the selection by the plausibility of all possible labels, the system becomes more resilient to cases where the model is "sure" about a label but the overall distribution is unstable.

4. Credal Sets and α-cuts

For scenarios involving extreme uncertainty regarding model choice or priors, the paper employs "imprecise probabilities." By using an α-cut updating rule, the framework can "hedge" against worst-case scenarios, only accepting pseudo-labels that remain plausible across a set of reasonably likely models (a credal set).

Experimental Evaluation

The proposed methods were evaluated using both simulated environments and real-world data.

Simulated Data

Using binomially distributed data designed to simulate covariate shift and model mismatch, the multi-model PPP approach demonstrated significant robustness. In these controlled tests, the robust selection criteria achieved accuracy gains of up to 15 percentage points over standard self-training methods.

UCI Banknote Authentication

The framework was also tested on the UCI Banknote Authentication dataset, which involves distinguishing genuine from counterfeit banknotes based on wavelet transform features. With 80% of the data unlabeled, the multi-model PPP consistently outperformed standard PLS criteria, such as simple probability thresholds or predictive variance, proving its utility in practical classification tasks.

Key Insights

  • Selection vs. Prediction: The research demonstrates that the selection mechanism is orthogonal to the prediction mechanism. High-performance models can generate labels, but robust, simpler, or multi-model ensembles should decide which labels to keep.
  • Model Robustness is Paramount: Among various types of uncertainty (covariate shift, error accumulation, and model choice), the authors found that addressing robustness toward model choice yielded the most significant performance improvements.
  • Utility in Early Training: The multi-label extension was most effective when initial supervised accuracy was lower. This suggests that robust selection is most valuable during the early stages of self-training when the "seed" model is still struggling to find the decision boundary.

Limitations and Future Work

While theoretically sound, the calculation of the Pseudo Posterior Predictive involves complex integrals. Although the authors provide an analytical approximation, scaling this to massive deep learning architectures remains a computational challenge. Additionally, while the α-cut method offers the highest theoretical robustness, its implementation complexity suggests a need for further empirical study to determine its cost-benefit ratio compared to simpler multi-model ensembles. Future research may also explore the interaction effects between model selection and covariate shift when addressed simultaneously.

Summary updated

Abstract

Self-training is a simple yet effective method within semi-supervised learning. The idea is to iteratively enhance training data by adding pseudo-labeled data. Its generalization performance heavily depends on the selection of these pseudo-labeled data (PLS). In this paper, we aim at rendering PLS more robust towards the involved modeling assumptions. To this end, we propose to select pseudo-labeled data that maximize a multi-objective utility function. The latter is constructed to account for different sources of uncertainty, three of which we discuss in more detail: model selection, accumulation of errors and covariate shift. In the absence of second-order information on such uncertainties, we furthermore consider the generic approach of the generalized Bayesian alpha-cut updating rule for credal sets. As a practical proof of concept, we spotlight the application of three of our robust extensions on simulated and real-world data. Results suggest that in particular robustness w.r.t. model choice can lead to substantial accuracy gains.
BibTeX
@misc{rodemann2023all,
  author = {Julian Rodemann and Christoph Jansen and Georg Schollmeyer and Thomas Augustin},
  title = {{In all LikelihoodS: How to Reliably Select Pseudo-Labeled Data for Self-Training in Semi-Supervised Learning}},
  year = {2023},
  eprint = {2303.01117},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2303.01117},
}
Figure