Abstract
Self-training is a simple yet effective method within semi-supervised learning. Self-training’s rationale is to iteratively enhance training data by adding pseudo-labeled data. Its generalization performance heavily depends on the selection of these pseudo-labeled data (PLS). In this paper, we render PLS more robust towards the involved modeling assumptions. To this end, we treat PLS as a decision problem, which allows us to introduce a generalized utility function. The idea is to select pseudo-labeled data that maximize a multi-objective utility function. We demonstrate that the latter can be constructed to account for different sources of uncertainty and explore three examples: model selection, accumulation of errors and covariate shift. In the absence of second-order information on such uncertainties, we furthermore consider the generic approach of the generalized Bayesian $\alpha$-cut updating rule for credal sets. We spotlight the application of three of our robust extensions on both simulated and three real-world data sets. In a benchmarking study, we compare these extensions to traditional PLS methods. Results suggest that robustness with regard to model choice can lead to substantial accuracy gains.
Introduction
In the landscape of semi-supervised learning (SSL), self-training stands out for its simplicity: a model trained on a small labeled dataset predicts labels for unlabeled data, and the most confident of these "pseudo-labels" are added to the training set for the next iteration. However, this simplicity masks a significant risk known as confirmation bias. If the initial model is overconfident in an incorrect prediction, it incorporates that error into its own training data, leading to a feedback loop of reinforced mistakes.
This paper addresses the fragility of pseudo-label selection (PLS) by reframing it as a formal statistical decision problem. By moving beyond simple thresholding and toward robust utility functions, the authors provide a framework that accounts for model misspecification, error accumulation, and covariate shift.
Reframing Selection as a Decision
Standard self-training relies on a single model's point estimate of confidence. The authors argue that this is insufficient because it ignores the uncertainty inherent in the model choice itself. Instead, they propose a utility-based approach where the goal is to maximize the "plausibility" of a data point being generated jointly with the existing training data.
Key Methodological Innovations
- Multi-Model Likelihood Utility: Rather than trusting a single "best" model, the framework evaluates pseudo-labels across a variety of models. This ensures that a selected point is not just an artifact of one model's specific bias.
- Reversed Occam’s Razor: This is perhaps the paper's most intuitive contribution. While standard model selection uses Occam’s Razor to pick the simplest model for a dataset, the authors apply it in reverse for data selection. They prioritize unlabeled instances that can be explained well by simpler models within a hierarchy. If a simple linear model "understands" a data point as well as a complex neural network, that point is a safer, more robust candidate for pseudo-labeling.
- $\alpha$-cut Updating: For scenarios where sources of uncertainty are difficult to parameterize, the authors use "credal sets" (sets of probability distributions). They apply a soft revision rule that discards priors with low marginal likelihood, providing a regret-based guarantee for the selection process.
- Computational Efficiency: To make these Bayesian methods practical, the authors utilize Laplace-style approximations for the pseudo posterior predictive (PPP) based on the Fisher information matrix, avoiding the prohibitive cost of MCMC sampling.
Experimental Evaluation
The researchers benchmarked their robust extensions against traditional PLS methods using both simulated data and three real-world datasets from the UCI repository: Banknote Authentication, Car Evaluation, and Mushroom Classification.
Key Findings
- Significant Accuracy Gains: On simulated datasets designed to test uncertainty, the multi-model approach achieved accuracy gains of up to 15 percentage points over standard methods.
- The Power of Model Robustness: The experiments revealed that being robust to which model is used is far more effective than being robust to which label might be correct. In fact, "multi-label" utility functions (which consider all possible labels for a point) often performed poorly, sometimes trailing behind simple supervised baselines when initial accuracy was already high.
- Consistency in Real-World Tasks: On the Banknote and Mushroom datasets, the multi-model extensions consistently ranked as top performers, demonstrating that the "simpler is safer" heuristic translates well to practical classification tasks.
Discussion and Future Work
The success of the "Reversed Occam’s Razor" suggests that self-training benefits significantly from a cautious approach to data augmentation. A data point that requires a highly complex, high-variance model to be classified is a risky candidate that likely contributes to confirmation bias.
While the computational overhead of fitting multiple models remains a consideration for massive deep learning architectures, the use of analytical approximations provides a viable path forward. Future research will explore "fractional Bayesian updating" to adaptively adjust learning rates based on accumulated expected errors and further test these generic $\alpha$-cut generalizations in deep learning contexts.
Summary updated