A model trained with only a few dozen labels and a large pile of unlabeled images can reach high accuracy. FixMatch reports 88.61% on CIFAR-10 with 40 labels. But such a model still makes a confident guess on every test input, and its mistakes are of a particular kind. The model partly teaches itself from its own guesses, so its errors tend to be self-reinforced: wrong answers that became confident late in training. We propose to let semi-supervised models abstain ("I don't know") and to score each input by pooling the model's confidence across the snapshots and augmented views that semi-supervised training already produces. The pooling uses a single tunable exponent that slides between "average the evidence" and "any doubtful snapshot vetoes the prediction."
The core bet is that the right amount of strictness is not fixed. It should grow stricter as labels get scarcer, because self-reinforced errors are exactly the ones a strict "veto" rule catches and an average hides. The family has one parameter, so both that parameter and the abstention threshold can be chosen and statistically certified from a labeled validation set of a few hundred examples. That matters in a regime where labels are the scarce resource.
Core Claim
Selective prediction is the practice of letting a classifier decline inputs, trading coverage (the fraction answered) against risk (the error rate on answered inputs). It has been developed almost entirely for fully supervised models. Semi-supervised learning (SSL), meanwhile, uses a confidence gate during training, FixMatch's fixed threshold of 0.95, but forces a prediction on every input at test time. Our claim concerns how an SSL model's confidence evolves over training. Its errors on the test set are disproportionately inputs whose predicted class had low confidence at some earlier point in training and was later "talked into" confidence by pseudo-labels.
A confidence score built from the whole trajectory should therefore rank errors better than the final model's confidence. The pooling operator should also be conjunctive, meaning closer to a minimum than to a mean. Concretely, let $\hat y$ be the final model's predicted class for input $x$, and let $p_1, \dots, p_T$ be the probabilities assigned to $\hat y$ by $T$ saved teacher snapshots and weak-augmentation views. The score is the power mean
$$s_\alpha(x) = \left(\frac{1}{T}\sum_{t=1}^{T} p_t^{\alpha}\right)^{1/\alpha},$$
where $\alpha$ is the strictness exponent. Setting $\alpha = 1$ gives the plain average, $\alpha \to 0$ gives the geometric mean, and $\alpha \to -\infty$ gives the minimum (a veto). The input is answered when $s_\alpha(x) \ge \theta$ for a threshold $\theta$.
The testable prediction is that the risk-minimizing $\alpha$ moves toward the minimum as the label budget shrinks (4 labels per class versus 25 or more). A second prediction is that at the smallest budgets the gain over the final-model confidence is larger than it is for a supervised model on the same data.
Relation to Prior Work
The selective-prediction side supplies both method and constraint. Selective Classification for Deep Neural Networks introduced Selection with Guaranteed Risk (SGR), a binary search for a threshold using binomial bounds on a labeled validation set. It also showed that the maximum softmax probability, called Softmax Response (SR), is a strong ranking signal. We reuse SGR for certification, with one change: we certify over a small grid of $\alpha$ values rather than a single score.
Towards Better Selective Classification constrains the design. It showed that dedicated abstention heads gain their benefit by regularizing the classifier, not through the head itself, and that SR on a well-trained model wins. We therefore add no selection head and keep SR as the zero-cost reference. That paper borrows entropy minimization from SSL but applies it only to labeled data; we instead study models genuinely trained with unlabeled data.
Selective Prediction via Training Dynamics (SPTD) is the closest method and the main threat. It scores inputs by how often checkpoint predictions disagree with the final one, using a fixed linear weighting that penalizes late disagreement more heavily. It studies only supervised training. We differ in three ways: the pooling strictness is a parameter rather than a fixed linear sum; the trajectories come from SSL training, where confirmation bias makes their shape different; and the parameter is chosen under an explicit validation-label budget.
Calibrated Selective Classification trains a separate selector network with a calibration objective. It is supervised and needs enough labels to fit that network, which is exactly what the low-label regime lacks.
On the aggregation side, Power mean pyramid scores for summarization evaluation supplies the operator and the motivating evidence. There, tuning a power mean beat the plain linear combination (Spearman 0.2417 versus 0.169 on TAC-08), but the best $\alpha$ shifted across tasks, which is why we treat $\alpha$ as something to select rather than fix. Reducing t-norms and augmenting t-conorms offers a richer strictness family, but it proves those operators can be non-associative, so their output would depend on snapshot order. We use the power mean because it is symmetric in its inputs and order-free.
On the SSL side, FixMatch supplies the training pipeline. Unsupervised Selective Labeling (USL) supplies an alternative seed-label strategy that picks representative examples before labeling, raising FixMatch from 82.9% to 93.5% on CIFAR-10 with 40 labels. USL deliberately avoids cluster-boundary points when choosing what to label. We use it to ask whether abstention then concentrates on those same boundary regions.
Research Program
We will train FixMatch and, as a second SSL recipe, an exponential-moving-average (EMA) teacher–student setup of the kind used in Semi-ViT. During training we save $T$ teacher snapshots at log-spaced steps, and at test time we add a few weak-augmentation views of each input. From these we compute $s_\alpha$ over a grid of about ten $\alpha$ values from $-\infty$ to 1.
For certification, we split a labeled validation budget $V$ (100, 300, 1000 examples, disjoint from the training labels). SGR is run on each $\alpha$, with the confidence level corrected for the grid size $K$ by dividing $\delta$ by $K$. This costs only a $\log K$ term in the bound; a learned selector gets no comparable cheap bound. We report the $\alpha$ with the largest certified coverage.
The analysis half characterizes the trajectories of test errors: when their predicted-class confidence first crossed FixMatch's 0.95 gate, and how many snapshots show a dip. This tests whether a veto rule is the right mechanism, not merely a lucky one. As a secondary, clearly separated experiment, we replace FixMatch's training-time gate with the same $s_\alpha$ computed over recent EMA snapshots. This addresses the tension that FixMatch trusts high softmax confidence while the selective-calibration literature documents confident errors. The main claim does not depend on this experiment.
Baselines
The naive reference is the FixMatch model forced to answer everything, and the same model with SR abstention thresholded by SGR. SR costs nothing and, per Towards Better Selective Classification, is hard to beat. Any method that fails to lower the area under the risk–coverage curve relative to SR on the identical SSL model has failed.
The strongest existing method is SPTD applied unchanged to the same saved SSL snapshots, with its published linear weighting. This is the strongest competitor because it uses the same information our method does. SPTD reports no numeric result on Lacuna for CIFAR-10 or CIFAR-100 (no reported result found on Lacuna for SPTD's risk–coverage values). The closest reported selective number is SGR's 1% error at 78% coverage on CIFAR-10 for a supervised model (Selective Classification for Deep Neural Networks); no reported result found on Lacuna for selective risk of any SSL model. Beating SPTD means a lower area under the risk–coverage curve and higher certified coverage at a target risk. The comparison uses the same snapshots, the same number of test-time forward passes, and the same validation budget $V$. SPTD's own weighting is left untuned, which is its intended use, and we additionally give it a tuned version with the same grid size $K$.
A deep ensemble of five independently trained FixMatch models is the compute-heavy upper reference. It is matched in test-time forward passes to $T = 5$, and we report its fivefold training cost alongside it. The calibrated selector of Calibrated Selective Classification is trained on the validation budget $V$ plus AugMix perturbations, to test whether a learned selector can compete when its labels are scarce.
Two ablations isolate our ingredients. Fixing $\alpha = 1$ (plain averaging of the same inputs) isolates strictness. Computing $s_\alpha$ over snapshots of a supervised model trained on the same 40 or 250 labels isolates whether SSL trajectories are special. We also compare snapshots only against augmentation views only.
Why not just…?
Why not just use Softmax Response on the final model? It is free and was the winner in the supervised comparison of Towards Better Selective Classification. Our hypothesis is that SSL errors are specifically confident ones created by self-training, which SR by construction cannot flag. Calibrated Selective Classification documents confident errors under shift, but not in SSL, so the file offers no direct evidence either way. The SR-versus-$s_\alpha$ comparison at 40 labels is the direct test. If SR wins, the proposal's premise is wrong, and we would report that.
Why not just run SPTD on SSL checkpoints? This is the real threat. If SPTD matches us, the contribution shrinks to "SPTD also works for SSL." That is still useful, but it is not our claim. Two things would distinguish the approaches. The first is whether the selected $\alpha$ moves systematically with label budget, which SPTD's fixed weighting cannot express. The second is whether tuned SPTD, given the same grid-selection budget, closes the gap. If the optimal $\alpha$ stays near 1 at all budgets, the strictness idea adds nothing, and we would say so.
Why not just average the views, or learn a selector network? Averaging is the $\alpha = 1$ ablation. The power-mean result on TAC-08 suggests fixed linear pooling is rarely optimal, but that is a different domain, so the ablation must decide. A learned selector has many parameters, and with a few hundred validation labels it must spend some on fitting and some on certification. Our one-parameter family pays only $\log K$. This predicts that our method should beat the selector most clearly at $V = 100$ and least clearly at $V = 1000$, which the validation-budget sweep will show.
Proposed Experiments
We will use CIFAR-10 with 40 and 250 labels, CIFAR-100 with 400 and 2,500 labels, STL-10, and the ImageNet 10% split. For ImageNet, FixMatch reports 28.54% top-1 error and the CIFAR-10 references are 88.61% and 94.93%, so our runs should first reproduce these within a point. We will also use CIFAR-10-C to test whether a threshold certified in-distribution still holds under corruption. Each label budget is run with both random seed labels and USL-T seed labels, over five seeds.
The metrics are area under the risk–coverage curve, selective risk at 80% and 90% coverage, SGR-certified coverage at target risks of 1% and 2% with $\delta = 0.001$ as a function of $V$, and the gap between violations of the risk target and $\delta$ on CIFAR-10-C.
Three decisive checks follow. First, the selected $\alpha$ should decrease monotonically as labels per class go from 4 to 25 and beyond, and should be less strict for the supervised-trajectory ablation at matched accuracy. Second, at 40 CIFAR-10 labels, $s_\alpha$ should beat both SR and tuned SPTD in area under the risk–coverage curve by more than the seed-to-seed standard deviation, at equal forward passes. Third, the errors that $s_\alpha$ rejects but SR accepts should show a confidence dip in some snapshot followed by late crossing of the 0.95 pseudo-label gate. That pattern would tie the gain to the confirmation-bias mechanism rather than to generic ensembling.
For USL seeds, we additionally measure whether abstentions fall disproportionately in the low-density, cluster-boundary regions that USL avoided labeling. That would connect choosing what to label with choosing what to decline.
References
- FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence
- Selective Classification for Deep Neural Networks
- Towards Better Selective Classification
- Selective Prediction via Training Dynamics
- Calibrated Selective Classification
- Power mean pyramid scores for summarization evaluation
- Reducing t-norms and augmenting t-conorms
- Unsupervised Selective Labeling for More Effective Semi-supervised Learning
- Semi-supervised Vision Transformers at Scale