Semi-supervised learning trains a classifier on a few labeled examples plus many unlabeled ones. Adding unlabeled data should help, yet in practice accuracy often plateaus, or even falls, as the unlabeled pool grows. A clear symptom comes from Pruning the Unlabeled Data to Improve Semi-Supervised Learning. That work throws away about 40% of the unlabeled images, specifically the ambiguous ones near the decision boundary. This improves ten different methods, and on CIFAR-100 with 300 labels it lifts SoftMatch from roughly 43% to 46%. So the ambiguous points are treated as poison. We propose a different reading. The points are not the problem. The problem is the loss applied to them.
Standard methods ask the network to predict a single hard label for each unlabeled image, and that label is the model's own best guess. For an image the model believes is "80% leopard, 18% cheetah," this forces a commitment the model cannot back up, and roughly a fifth of those commitments are wrong. The same image does contain reliable information: it is almost certainly not one of the other 98 classes. We propose training on exactly that information. For each unlabeled image, the model builds the smallest set of classes that covers most of its predicted probability. The augmented copy of the image is then trained to put its mass inside that set, and the ratio within the set is left for the labeled data to decide. Our claim is that this single change removes most of the benefit of pruning. It should also make accuracy keep rising as more unlabeled data is added, including contaminated data that currently causes degradation.
Core Claim
The research file describes two literatures that rarely talk to each other. The first is the objective-mismatch literature. It shows that fitting a probability distribution well is not the same as classifying well. Improved topic classification over maximum entropy model using k-norm based new objectives documents that likelihood training gains mostly by making easy examples more confident while hard examples stay wrong. Label Distribution Learning Machine shows that a smaller distribution-fitting error can still give the wrong top-1 class. SHOT-VAE finds that a better generative bound stops improving, or even hurts, semi-supervised accuracy.
The second is the saturation literature. It shows that unlabeled data stops helping and can start hurting. Evidence includes the out-of-domain degradation past 500k samples in Industry Scale Semi-Supervised Learning for Natural Language Understanding, the pruning result above, and the information-theoretic error floor in How Does Pseudo-Labeling Affect the Generalization Error of the Semi-Supervised Gibbs Algorithm?.
Our connecting claim is that the unlabeled term in modern semi-supervised learning is itself a mismatched objective. It is a likelihood fit to self-generated labels, and it demands more information than the model has. The degradation that pruning fixes is a consequence of this mismatch, not a property of the data.
Concretely, let $p(\cdot \mid x)$ be the model's class probabilities on a weakly augmented unlabeled image $x$. Let $S(x)$ be the smallest set of classes whose total probability is at least $\rho$, capped at $k_{\max}$ classes. The unlabeled loss on a strongly augmented view $A(x)$ is
$$\ell_u(x) = -\log \sum_{c \in S(x)} p(c \mid A(x)).$$
When $S(x)$ contains one class, this is exactly the FixMatch pseudo-label loss with confidence threshold $\rho$. When the model is unsure, the loss still eliminates implausible classes but makes no choice among the plausible ones. Every unlabeled image contributes something, and no image is forced into a guess. If the true class falls outside the set, which is the only way this loss injects a wrong signal, the image is simply discarded for that step. We measure how often that happens directly.
Relation to Prior Work
Several lines of work take the other side of our claim, and we test against them directly. PruneSSL removes boundary points offline using self-supervised embeddings; its own stated weakness is that a poor embedding invalidates the pruning. SoftMatch replaces the hard threshold with a soft confidence weight. It recovers many of the correct pseudo-labels that FixMatch's 0.95 threshold discards, which the file puts at up to 71%. It still assigns every weighted image a single class. Both methods treat ambiguity as a reason to trust an image less. We treat it as a reason to ask the image a coarser question.
Other work supplies the idea we reuse. The margin objective in Label Distribution Learning Machine adds an adaptive margin over the runner-up classes, and the $K$-norm objectives focus training on classification error rather than aggregate likelihood. Both show that the useful signal lives in the separation between plausible and implausible classes, not in exact probability values. We carry that insight from supervised label-distribution and topic classification over to self-generated labels on unlabeled data.
The closest cousin in the file is the transductive SVM in Combining Labelled and Unlabelled Data. It places margins on unlabeled points and nearly matches full supervision with 24% of the labels. Related to it is the conditional-entropy regularizer in Semi-supervised training of Gaussian mixture models by conditional entropy minimization. Both push every unlabeled point away from the boundary toward one class. That is precisely the forced commitment we avoid.
The remaining papers constrain our design. From Theory to Decision Rule: Calibrating the Noisy-Label Crossover reports that noise-robust losses (Generalized Cross Entropy, Co-teaching) and student models spanning an 11× range in parameter count do not move the point at which weak labels start hurting. A loss change that only reweights samples is therefore unlikely to fix saturation. Our change alters what is being asserted about each sample, not merely how much each sample counts. The Gibbs analysis motivates a secondary hypothesis. Set-valued targets depend less sharply on the errors of the labeled set than top-1 pseudo-labels do. If so, they should raise the floor, not just the early slope.
Research Program
We will build the set-valued loss as a drop-in replacement for the unlabeled term in FixMatch and SoftMatch. Everything else stays the same: augmentations, labeled loss, and schedule. SoftMatch's per-sample weight and its uniform-alignment step are applied on top in the SoftMatch variant. The set threshold $\rho$ and cap $k_{\max}$ are the only new hyperparameters. We will sweep them jointly and report sensitivity, since a method that needs per-dataset tuning of $\rho$ would undercut the practical case against pruning.
The analysis centers on one diagnostic, which we call the commitment error. For each confidence bin of unlabeled images, we compare two quantities: how often the top-1 pseudo-label is wrong, and how often the true label falls outside $S(x)$. The labels come from the held-back ground truth of the benchmark's unlabeled split. Our mechanism predicts two things. In the low-confidence bins that PruneSSL removes, top-1 error should be high, while set error stays low. Set error is the one error rate that should track downstream harm.
We will also track how the gradient on the unlabeled data divides across confidence bins over training. This tests the objective-mismatch story directly. Under hard pseudo-labeling, wrong-label boundary images should account for a disproportionate share of the gradient that pushes toward incorrect classes. Under set targets, that share should collapse.
Baselines
The naive reference points are a supervised-only model trained on the labeled split and plain FixMatch. The FixMatch run corresponds to our own loss with $k_{\max}=1$. It tells us how much a standard pipeline gains from unlabeled data at all.
The strongest existing competitor is SoftMatch combined with PruneSSL pruning. This is the exact pairing for which PruneSSL reports its headline gain, from about 43% to about 46% on CIFAR-100 with 300 labels. We will run it with the published 60% retention rate and with retention retuned on our validation split.
For context on STL-10 with 40 labels, SoftMatch reports 21.42% error, against 29.15% for FlexMatch. No pruned-SoftMatch number on STL-10 was found on Lacuna, so we will produce it ourselves.
We add two further competitors aimed at the mechanism. The first is Generalized Cross Entropy applied to the hard pseudo-labels. This bounded loss naturally downweights uncertain images, so it acts as a soft form of pruning. The second is soft-target consistency, where the strong view is trained to match the full weak-view distribution. This also avoids forcing a single class.
Two ablations isolate the key ingredient. The first is a hybrid run. It keeps hard pseudo-labels for confident images and applies set targets only to the images PruneSSL would discard, so any gain can be traced to those images. The second is a random-set control. It replaces $S(x)$ with a random class set of the same size, which checks that the gain does not come from a generic regularization effect.
All runs share one codebase, one backbone, and one iteration budget. Each method gets the same number of hyperparameter trials, and each uses three seeds. We count the method as beating the strongest baseline only if two conditions hold. SoftMatch with set targets and no pruning must match or exceed pruned SoftMatch on CIFAR-100 at 300 and 400 labels. And pruning applied on top of set targets must add less than a third of the gain it adds to SoftMatch.
Why not just…?
Why not just prune the ambiguous images, as PruneSSL does? Pruning is the direct threat, and it may win on clean benchmarks where the self-supervised embedding is good. Two considerations keep our approach in contention. PruneSSL depends on a separate embedding and clustering stage that can fail under domain shift, and it discards images that still carry useful information about which classes they are not. The prune-on-top experiment settles the question. If pruning still adds its full gain on top of set targets, ambiguity really is poison and our mechanism is wrong.
Why not just use a noise-robust loss such as Generalized Cross Entropy? The crossover study found that such losses soften the damage from bad labels but do not move the point where damage begins (From Theory to Decision Rule). That study used vision-language pseudo-labels, not self-training, so we test the question in our setting rather than assume the answer. Our argument is that robust losses still commit each image to one class and only trust uncertain images less. The contamination sweep described below separates the two approaches: a reweighting fix should slow the decline, while a change to what each image asserts should remove it.
Why not just use SoftMatch's soft weighting, or train on the full soft distribution? This is the most serious alternative, because soft-target training also avoids hard commitment. The difference is what gets fit. Soft targets ask the strong view to reproduce the model's uncertain 80/18 ratio, which is exactly the distribution-fitting that Label Distribution Learning Machine shows can diverge from top-1 correctness. Set targets leave that ratio for the labeled data to decide. If soft-target consistency matches our method under contamination, we will report that the benefit comes from avoiding hard labels in general, not from the set structure specifically.
Why not just use entropy minimization or a transductive-SVM-style margin on unlabeled points? These objectives push every point away from the boundary into one class. That is the forced commitment we hypothesize causes degradation. The file's own summary of the conditional-entropy work flags confirmation bias and collapse under out-of-domain noise as a known weakness. We include entropy minimization in the contamination sweep as a method we expect to degrade fastest.
Proposed Experiments
The primary benchmarks are CIFAR-100 with 300, 400, and 2,500 labels, CIFAR-10 with 40 and 250 labels, and STL-10 with 40 labels. STL-10 matters most because its 100,000 unlabeled images come from a broader distribution than its labeled set. We report top-1 accuracy and commitment error by confidence bin.
The decisive saturation experiment varies both the amount and the purity of unlabeled data. We subsample each unlabeled pool at 10%, 25%, 50%, and 100%. For CIFAR-100, we additionally mix in CIFAR-10 images, whose classes do not match CIFAR-100's, at contamination ratios from 0% to 50%. The measured outcome is the shape of the accuracy curve against unlabeled volume. The hypothesis predicts roughly monotone improvement for set targets. It predicts a plateau or decline for FixMatch, SoftMatch, entropy minimization, and robust-loss baselines at high contamination. It also predicts that PruneSSL's advantage grows with contamination for hard-label methods but shrinks toward zero for ours.
A secondary transfer test repeats the design of the crossover study on ISIC 2019, where that study places the crossover at 20–50 gold labels. The zero-shot BiomedCLIP labeler's outputs are turned into candidate sets rather than top-1 labels. We then ask whether the crossover moves. If it does, this would be the first loss-level intervention to shift a crossover that robust losses and larger students leave unchanged.
Three outcomes would falsify the proposal. The first is that pruning still adds its full benefit on top of set targets. The second is that set error in low-confidence bins turns out to be as high as top-1 error. The third is that the random-set control recovers most of the gain.
References
- Pruning the Unlabeled Data to Improve Semi-Supervised Learning
- Improved topic classification over maximum entropy model using k-norm based new objectives
- Label Distribution Learning Machine
- SHOT-VAE: Semi-supervised Deep Generative Models With Label-aware ELBO Approximations
- Industry Scale Semi-Supervised Learning for Natural Language Understanding
- How Does Pseudo-Labeling Affect the Generalization Error of the Semi-Supervised Gibbs Algorithm?
- SoftMatch: Addressing the Quantity-Quality Tradeoff in Semi-supervised Learning
- Combining Labelled and Unlabelled Data: A Case Study on Fisher Kernels and Transductive Inference for Biological Entity Recognition
- Semi-supervised training of Gaussian mixture models by conditional entropy minimization
- From Theory to Decision Rule: Calibrating the Noisy-Label Crossover for VLM Weak Supervision Across Three Medical-Imaging Benchmarks