A cheap labeler, such as a vision-language model that labels images zero-shot, helps a student model only while the student is worse than the labeler. Once the student, trained on a growing set of expert ("gold") labels, catches up, the cheap labels start to hurt. On three medical-imaging benchmarks this crossover is sharp, and training past it costs up to 0.10 AUC. Noise-robust losses soften the damage but do not move the point where it begins (From Theory to Decision Rule). That crossover is an average over the whole dataset. Even after the student beats the teacher overall, the teacher may still be the better labeler on particular regions of the data: certain classes, certain confidence ranges, certain kinds of image.
We propose a curriculum that runs in reverse of the usual direction. As the student improves, it retires weak labels region by region, keeping only those where the teacher is still measurably more accurate than the current student. Curriculum learning normally asks which samples to add as the model gets stronger. Here the question is which borrowed supervision to drop. The bet is that this moves the crossover point itself, which the file says loss-level fixes cannot do. A second payoff is that the post-crossover unlabeled pool turns from a liability into a modest, measurable asset.
Core Claim
Post-crossover harm from weak labels is not spread evenly. It comes from the subset of weak labels in regions where the student now out-predicts the teacher. A selection rule that compares teacher accuracy with current-student accuracy, region by region, and re-estimates that comparison as gold data and training progress, will do two things. First, it will keep the student's AUC at or above the gold-only curve at every gold budget, so there is no degradation phase. Second, it will stay strictly above that curve for a range of gold budgets beyond the classical crossover.
The ingredient that matters is the comparison against the current student. A fixed property of the teacher, such as its confidence, is not enough. A skeptical reviewer's natural alternative is "just keep the teacher's confident labels." We treat that as the primary competitor and design an ablation to settle it.
Where This Sits
The strongest motivation is the crossover paper itself (From Theory to Decision Rule). It finds that the crossover sits at roughly 100 gold labels on PCAM, 20–50 on ISIC 2019, and 250–500 on NIH-CXR. It also finds that the crossover does not depend on student capacity. Its proposed decision rule is all-or-nothing: compare gold-trained validation AUC to the teacher's, then use the weak labels or don't. That rule predicts whether weak labels help with 76% accuracy across 129 configurations. It tests only static weak-label sets and never asks whether a partial, adaptive subset could beat both options. Our proposal makes exactly that change.
The saturation literature tells us what to expect. The analysis of the semi-supervised Gibbs algorithm (How Does Pseudo-Labeling Affect the Generalization Error) shows that error flattens as the unlabeled-to-labeled ratio grows, at a floor set by the information shared between the labeled data and the pseudo-labels. Industrial NLU work found gains peaking at 500k unlabeled samples and then declining (Industry Scale Semi-Supervised Learning for NLU). Read together, these suggest a simple picture: once the shared information has been extracted, extra weak data adds mostly noise. If that information is concentrated in identifiable regions, selecting by region should recover it without importing the noise. The Gibbs analysis assumes an idealized learner, so we treat its picture as a hypothesis to check, not a guarantee.
Several curriculum-style methods for semi-supervised learning are close, but each paces something different. FlexMatch (FlexMatch) lowers per-class confidence thresholds for classes the model is learning slowly. Curriculum Labeling (Curriculum Labeling) admits a growing top percentile of confident pseudo-labels and re-initializes the model each round. The appearance-based curriculum (Appearance-Based Curriculum) adds unlabeled images in order of distance from the labeled set whenever validation plateaus. All three only ever add data, and all three judge a pseudo-label by the learner's own confidence or by geometry. None compares two labelers or retires supervision.
SemiReward (SemiReward) learns a per-sample quality score. It is co-trained on its own accepted pseudo-labels, so it can drift with the student. Its scores are also not anchored to a held-out estimate of whether the teacher beats the student.
The theory of easy-to-hard self-improvement (A Task-centric Theory for Iterative Self-Improvement) supplies the framing we reuse. Its main result is that a curriculum helps only inside a feasible-initialization region, and that the effective training set expands as the model improves. Our setting is the mirror image: as the model improves, the set of trustworthy borrowed labels shrinks. Self-paced learning (Self-Paced Learning for Latent Variable Models) supplies the mechanics: binary per-sample selection variables, updated alternately with the model parameters.
Research Program
We partition the weakly labeled pool into cells. A cell is defined by the teacher's predicted class, a coarse bin of teacher confidence, and a coarse bin of student confidence. A clustering of features from the pretrained backbone is an optional additional axis.
At round $t$, weak sample $i$ in cell $c$ is admitted when
$$ \mathrm{LCB}_{\delta}\big[\hat a_T(c) - \hat a_S^{t}(c)\big] > 0 . $$
Here $\hat a_T(c)$ is the teacher's accuracy on gold examples falling in cell $c$, and $\hat a_S^{t}(c)$ is the round-$t$ student's accuracy on the same examples. The LCB is a lower confidence bound at level $\delta$ on their paired difference. Cells with too few gold points fall back to their parent cell (same class, all confidence bins) through shrinkage. That fallback matters on ISIC, where the crossover sits at 20–50 gold labels.
Each round has three steps. Retrain the student on gold labels plus the admitted weak labels. Recompute the cells and accuracies using the new student. Re-select which weak labels are admitted. Training continues from current weights; there is no re-initialization. This avoids the cost that Curriculum Labeling pays to escape confirmation bias, and confirmation bias is controlled instead by anchoring every decision to gold.
The student's own gold training points cannot fairly score it, so student accuracies are cross-fitted. Gold labels are split into $k$ folds, and each fold is scored by a student trained without it. This costs roughly $k$ times the training, which we report. We also test a cheaper variant that holds out a fixed slice of the gold labels as a scoring set.
The analysis side asks one question: is post-crossover harm actually concentrated? We decompose the AUC loss from admitting all weak labels into per-cell contributions, and measure how much of the loss comes from cells where the student beats the teacher. If harm turns out to be uniform across cells, the core claim is false. That would be worth reporting, because it would mean the binary decision rule is the best one can do.
Baselines
The do-nothing reference is gold-only training with the same backbone. This is effectively what the all-or-nothing decision rule from From Theory to Decision Rule recommends past the crossover. The naive reference is training on gold plus all weak labels mixed uniformly, the setting that produces the up-to-0.10 AUC drop. The oracle decision rule chooses, at each gold budget, the better of these two curves. Beating that oracle is our primary success criterion: our curve must sit at or above it everywhere and strictly above it for some interval past the crossover.
The file reports crossover locations and the size of the drop, but no absolute AUC values for these curves. No reported result found on Lacuna for gold-only or weak-label AUC on PCAM, ISIC 2019, or NIH-CXR. We regenerate all curves ourselves with the paper's protocol: weak-label counts up to 50,000, the BiomedCLIP teacher, and DenseNet, ResNet, and ConvNeXt students.
The strongest loss-level competitors are generalized cross-entropy and co-teaching trained on all weak labels. These are reported not to move the crossover, so they serve as the test that our gain comes from selection rather than loss shape.
The strongest selection competitor is teacher-confidence thresholding: keep weak labels whose teacher confidence exceeds a threshold tuned on gold validation. We also run a FlexMatch-style per-class adaptive version of it. For reference, the file reports FlexMatch at 39.94% error against FixMatch's 46.42% on CIFAR-100 with 400 labels (FlexMatch). We additionally run SemiReward's learned scorer as the filter. The file reports that SemiReward adds 1.9–3.7% accuracy when plugged into FlexMatch or FreeMatch (SemiReward), via its codebase Westlake-AI/SemiReward. We also run Curriculum Labeling's percentile pacing applied to teacher labels.
Every method uses the same gold labels, weak pool, backbones, and epoch budget. Each gets the same number of hyperparameter trials, tuned on the same gold validation split. We report wall-clock cost alongside AUC so that the cost of cross-fitting is visible.
Three ablations isolate the key ingredient. The first drops the student term, ranking cells by teacher gold accuracy alone. The second freezes the cell selection after the first round, so the curriculum never updates. The third admits random cells matched in count to our selection. Our method beats the first ablation only if the student-relative comparison carries information beyond teacher reliability. It beats the second only if the retiring schedule matters. It beats the third only if which cells are kept matters.
Why not just…?
Why not just apply the decision rule and stop using weak labels past the crossover? This is the honest default, and it is never worse than its better branch when the rule predicts correctly. It errs about one time in four (From Theory to Decision Rule), and by construction it throws away any teacher advantage that survives in specific regions. Our primary metric compares against the oracle version of this rule. If we cannot beat the oracle past the crossover, the regional-advantage hypothesis fails and the simple rule wins.
Why not just keep the teacher's high-confidence labels? This is the most serious threat. If teacher confidence is well calibrated, confident labels are exactly the useful ones, and comparing against the student adds nothing. Our argument against it is specific. A confidence threshold is fixed as the student grows, so it keeps admitting confident teacher labels in regions the student has already mastered. Those labels are redundant at best and harmful wherever the teacher is confidently wrong. The no-student-term ablation and the tuned confidence baseline settle this directly. If either matches our method, we will report that teacher confidence suffices and that the curriculum adds nothing.
Why not just switch to the student's own pseudo-labels on the unlabeled pool once it beats the teacher? Self-training with FlexMatch-style thresholds is a reasonable post-crossover move. However, self-training is known to drift as errors reinforce across rounds (Learning Dual Retrieval Module), and it discards the teacher's remaining regional advantage. We include it as a baseline, and we also include a hybrid that uses student labels in retired cells. If the hybrid dominates, the deliverable becomes a rule for deciding, per region, who labels.
Proposed Experiments
The main experiment reproduces the gold-budget sweeps on PCAM, ISIC 2019, and NIH-CXR. Gold counts span from well below to well above each crossover, weak pools scale up to 50,000, and results are averaged over five seeds and three student families.
We report three quantities. The first is AUC as a function of gold count. The second is the shift in crossover: the gold count at which each method first falls below gold-only. The third is the area between each method's curve and the oracle curve past the classical crossover.
The decisive checks are four. Our method must show no degradation phase at any gold budget. It must keep a positive area over the oracle that shrinks toward zero as gold grows; we expect that shrinkage because cells where the teacher still wins should run out. The per-cell harm decomposition must show that harm concentrates in student-dominated cells. And the student-relative ablation must beat the confidence-only ablation.
Because the crossover was reported to be independent of student capacity, we also test a prediction that follows from our rule. Larger students should retire cells earlier in training at a fixed gold budget, even though the global crossover does not move.
Two stress tests probe the known weak points. NIH-CXR's multi-label class imbalance is where sparse gold cells make the confidence bound loose. A deliberate shift, in which gold labels are drawn from a subpopulation of the weak pool, probes the regime where the original decision rule is known to break.
A secondary experiment on one USB text task and one USB vision task (USB) uses a fixed pretrained model as the weak teacher. Its purpose is to check that the effect is not specific to medical imaging.
References
- From Theory to Decision Rule: Calibrating the Noisy-Label Crossover for VLM Weak Supervision Across Three Medical-Imaging Benchmarks
- How Does Pseudo-Labeling Affect the Generalization Error of the Semi-Supervised Gibbs Algorithm?
- Industry Scale Semi-Supervised Learning for Natural Language Understanding
- FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labeling
- Curriculum Labeling: Revisiting Pseudo-Labeling for Semi-Supervised Learning
- Appearance-Based Curriculum for Semi-Supervised Learning with Multi-Angle Unlabeled Data
- SemiReward: A General Reward Model for Semi-supervised Learning
- A Task-centric Theory for Iterative Self-Improvement with Easy-to-Hard Curricula
- Self-Paced Learning for Latent Variable Models
- Learning Dual Retrieval Module for Semi-supervised Relation Extraction
- USB: A Unified Semi-supervised Learning Benchmark for Classification