Semi-supervised learning trains a classifier from a handful of labeled examples and a large pool of unlabeled ones. The model guesses labels for the unlabeled pool (pseudo-labels) and then trains on its own guesses. When a guess is wrong but confident, the model learns the mistake and becomes more sure of it. Today's filters decide which guesses to trust by looking at the model's own confidence, through a fixed cutoff, an adaptive cutoff, or a mixture model fitted to confidence scores. A model that is confidently wrong passes all of them. Error detection research offers a different tool: a separate, small "meta-classifier" trained to predict whether the main model's output is correct from signals such as logits, margins, and hidden activations. The obvious way to use it here fails for a simple reason. With 4 labels per class, the main model memorizes every labeled example, so the target task contains almost no observed errors to learn from.
We propose getting that supervision from somewhere else. We would run ordinary self-training on datasets where the true labels of the "unlabeled" pool are known, record for every pseudo-label a set of features that do not depend on the class vocabulary, and train an auditor to predict whether each pseudo-label is wrong. On a new dataset, the auditor is frozen and replaces the confidence threshold, so it needs no target labels. The core bet is that confident mistakes in self-training share signatures across datasets, much as correctness prediction transfers across language models (Generalized Correctness Models). Candidate signatures include a prediction that locks in late, flips between classes, disagrees between augmented views, or lands on a class that is absorbing more than its share of the pool.
Core Claim
A class-agnostic pseudo-label error detector, trained on the full ground-truth history of self-training runs on source datasets, will rank a target run's pseudo-labels by error risk better than the model's own confidence. That advantage should be largest on the high-confidence mistakes that current filters admit. Using the detector to gate the loss should reduce final error at extreme label scarcity. The claim can fail in two separable places. Detection may not transfer across datasets, or better detection may not turn into better final accuracy. The experiments test each point separately.
Why this is the right cut through the two literatures
Supervised error detection is well established outside training loops. Secondary classifiers verify extraction and translation outputs (Culotta et al., Blatz et al.). Runner-up class scores expose confusion that the top probability hides (Yamasaki et al.). Raw logit energy and top-two margin carry more signal than softmax values (Suitability Filter). These papers motivate our feature choice, but all of them assume labeled examples of the model's failures, drawn from the deployment distribution. In semi-supervised learning that assumption fails, and this is the gap we target.
The closest prior work is the quality filter of Auto-Annotation Quality Prediction for Semi-Supervised Learning with Ensembles. It trains a small CNN on a labeled subset of Cityscapes to mask untrustworthy pixels produced by an 18-model ensemble, and it matches fully supervised mIoU of 77.0% with 30% of the labels. It depends on a labeled subset large enough to contain errors, an expensive ensemble, and an offline pass. Our auditor is trained on other datasets, reads a single model's cheap per-sample statistics, and runs online at every step. Generalized Correctness Models supplies the transfer idea: correctness predictors trained on historical traces from many language models beat self-reported confidence. It also constrains our design, because calibration degraded from an Expected Calibration Error (ECE) of 0.02 on MMLU to 0.10 on TriviaQA. We therefore use the auditor only for ranking. The number of pseudo-labels admitted comes from a ranking quantile, not from trusting the auditor's probabilities.
On the pseudo-labeling side, three methods build their filters from the model's own outputs. FixMatch uses a fixed 0.95 cutoff. FreeMatch uses cutoffs that adapt per class through running averages. SPF fits a two-component Beta mixture to the top confidence score. FreeMatch itself acknowledges that such filters reinforce confident errors. AdaConG weights pseudo-labels by conformal prediction set size, which requires a labeled calibration set, and it scores samples with a fixed nonconformity score rather than a learned one. UMPL feeds student performance back to the teacher through meta-gradients and filters with dropout-based uncertainty, but it has no dedicated error model and is tested only on physics surrogates.
Research Program
The first component is a feature set that means the same thing on any label space. For each unlabeled sample at training step $t$, we compute:
- Output shape: the top-5 sorted logits, logit energy, top-two margin, and entropy on the weakly augmented view.
- View agreement: whether the weak and strong views agree, and the strong view's probability for the pseudo-label.
- History: a per-sample memory of a few scalars, namely how many epochs the current argmax has been stable, the number of class flips so far, and the epoch at which it first exceeded 0.95.
- Class pressure: the ratio between the predicted class's share of current pseudo-labels and a uniform prior.
- Training progress: the fraction of training completed.
We would also test an optional low-dimensional projection of penultimate-layer activations, following the evidence from Chen et al. that hidden layers reveal errors which the final softmax hides. Because raw activations are tied to a particular architecture, they appear in an ablation rather than in the main method.
The second component is a trace corpus. We run FixMatch-style training on the source datasets at several label budgets and seeds, and label every logged pseudo-label as correct or incorrect using the withheld ground truth. The auditor is a small gradient-boosted tree ensemble, in the spirit of Risk Advisor, and a small MLP. Training examples are weighted toward high-confidence samples, because those are the errors that matter.
The third component is deployment. On the target run, the auditor scores each unlabeled sample, and the loss weight is the sample's rank-based trust. We admit the top fraction $\rho_t$ of the batch by trust, and $\rho_t$ follows the same schedule that SPF's posterior implies on the source runs. The target run therefore uses no hyperparameter tuned on target labels.
Baselines
The naive reference point is FixMatch with $\tau=0.95$. The SPF paper reports a FixMatch error of 13.81% on CIFAR-10 with 4 labels per class and 58.40% on Oxford Flowers 102. The FixMatch paper itself reports 88.61% accuracy with 40 labels.
The strongest existing filter in the file on our setting is SPF-RA, which reports a 7.46% error on CIFAR-10 at 4 labels per class and 28.32% on Flowers-102. It is the strongest competitor because it is the self-referential filter at its best: adaptive and threshold-free. FreeMatch is a second strong baseline. For it, the file gives only relative reductions: 5.78% over FlexMatch at 1 label per class on CIFAR-10, and 13.59% on STL-10 at 4 labels per class. AdaConG is also included, with its conformal calibration set carved out of the same labeled budget.
All methods share one backbone, one augmentation pipeline, one iteration count, and three to five seeds, all run in the same codebase. Each baseline receives the same tuning budget on source datasets only. The one-time cost of building the trace corpus is reported separately and amortized. Beating SPF-RA means lower mean error on held-out target datasets, outside seed variance, with no target-label tuning.
Three ablations isolate the key ingredient:
- Unsupervised version: a two-component Gaussian mixture fitted online to the same multivariate features. This separates the effect of the features from the effect of the cross-dataset supervision.
- In-target supervised version: an auditor trained on a held-out split of the target's own labels, as in the quality-filter paper. This tests whether transfer is needed at all.
- Oracle: an auditor trained on the target's own trace ground truth, which sets the ceiling.
Why not just…?
Why not just use SPF or FreeMatch, which already adapt? Both read only the model's top probability. When a subpopulation is confidently mislabeled, it falls into the "clean" mode of the mixture or clears the adaptive cutoff. FreeMatch concedes this failure. The decisive measurement is the fraction of pseudo-labels with confidence above 0.95 that are wrong and still get admitted, tracked over training for each method. If that fraction does not fall with our auditor, the proposal has no mechanism.
Why not just fit an unsupervised mixture to logit margin and energy instead of learning anything? This is the most serious threat. The Suitability Filter results suggest that much of the gain may come from logit features alone. The multivariate Gaussian-mixture ablation tests exactly this. If it matches the transferred auditor, we will report that the contribution is the features, not the supervision, and narrow the claim to that.
Why not just train the error detector on the target's own labeled data? At 4 labels per class, a held-out split leaves too few examples, and the model's errors on memorized data are nearly absent. We expect this variant to collapse at 40 labels and recover at 250. That crossover is itself a finding about when transfer is needed. If the in-target version wins even at 40 labels, the transfer component is unnecessary.
Why not just ensemble or use dropout uncertainty? The quality-filter paper needed 18 models, and UMPL notes the cost of repeated dropout passes. Members that share an architecture and training data also agree on the same systematic errors (Token-Level Ensemble Distillation). We include a 3-member ensemble-disagreement baseline under matched compute rather than dismissing it.
Proposed Experiments
The main protocol holds out one dataset at a time across CIFAR-10, CIFAR-100, STL-10, and Flowers-102, training the auditor on traces from the remaining three.
Detection quality is measured first, independently of accuracy, using three metrics on the target's logged pseudo-labels:
- AUROC: how well the auditor ranks wrong pseudo-labels above correct ones.
- Confident-error recall: the recall of wrong pseudo-labels among those with confidence above 0.95.
- Admitted precision: the precision of the admitted set at each epoch.
The first decisive check is whether the auditor's AUROC on the held-out dataset exceeds that of the top-probability score and of the top-two margin. It must do so especially in the first 10% of training, where the known loss bump originates. If transfer fails there, we stop.
The second decisive check is final top-1 error against SPF-RA at 4 labels per class on CIFAR-10 and on Flowers-102. Flowers-102 is the hardest transfer target because its classes are fine-grained.
The remaining experiments probe the claim's limits:
- Architecture transfer: train on WideResNet traces and deploy on a ResNet.
- Distribution shift: evaluate on STL-10, whose unlabeled pool contains classes outside the label set.
- Scale: evaluate on an ImageNet subset to see whether class-pressure features survive 1,000 classes. The file gives no reported FreeMatch accuracy at 100 labels per class there, only FixMatch's 28.54% error with 10% of labels.
We will also report an error-type breakdown: which confusions the auditor catches that the confidence score misses.
References
- Generalized Correctness Models: Learning Calibrated and Cross-Model Correctness Predictors from Historical Patterns
- Confidence Estimation for Information Extraction
- Confidence Estimation for Machine Translation.
- Confidence Analysis for Multi-Class Object Recognition using the Intermediate Values from Machine Learning Algorithms
- Suitability Filter: A Statistical Framework for Classifier Evaluation in Real-World Deployment Settings
- Auto-Annotation Quality Prediction for Semi-Supervised Learning with Ensembles
- FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence
- FreeMatch: Self-adaptive Thresholding for Semi-supervised Learning
- Towards Self-Adaptive Pseudo-Label Filtering for Semi-Supervised Learning
- Adaptive Conformal Guidance for Learning under Uncertainty
- Uncertainty-Informed Meta Pseudo Labeling for Surrogate Modeling with Limited Labeled Data
- Confidence Scoring Using Whitebox Meta-models with Linear Classifier Probes
- Responsible model deployment via model-agnostic uncertainty learning
- Token-Level Ensemble Distillation for Grapheme-to-Phoneme Conversion