Self-training methods teach a model on unlabeled data by using its own confident predictions as labels, called pseudo-labels. When a wrong pseudo-label gets through the confidence filter, the model is pushed toward the wrong class. It then becomes more confident in that mistake on similar images, which lets more wrong labels through. This loop is called confirmation bias. Current defenses act on which labels are accepted (confidence thresholds) or on what the target looks like (soft labels, mixup). None of them limits how far a single optimizer step driven by pseudo-labels can move the weights. We propose to limit exactly that. The gradient from the pseudo-labeled loss is computed separately from the gradient of the labeled loss. For each unit of the network (one row of a weight matrix, for example the classifier row of one class), the pseudo-label gradient is capped at a fixed fraction of that unit's weight norm. The labeled gradient is left alone.
In a colleague's terms, the proposal borrows adaptive gradient clipping from normalizer-free networks and applies it only to the pseudo-label signal, unit by unit. The goal is to slow down the self-reinforcing feedback loop without throttling ordinary learning. The claim that makes this more than "use a smaller unlabeled loss weight" is empirical and can be falsified. We expect confirmation-bias episodes to appear as short bursts in which the pseudo-label gradient on a few units grows large relative to those units' weights. If that holds, a cap that stays inactive most of the time should beat any constant down-weighting. If the bursts do not exist, the method has no reason to work, and the first experiment is designed to find that out.
What exists and where the gap is
Two papers connect gradient clipping and label noise, and they supply the method's framing. Can gradient clipping mitigate label noise? shows that ordinary norm clipping of a convex loss is equivalent to a Huberised loss, which is still vulnerable to label noise. It proposes clipping only the derivative of the base loss, which gives a saturating "partially Huberised" (PHuber) loss. On CIFAR-100 with 60% symmetric noise, PHuber-CE reaches 18.5% accuracy against 11.4% for cross-entropy and 9.0% for cross-entropy with standard clipping. Optimized Gradient Clipping for Noisy Label Learning adapts the clipping threshold over time by fitting a two-component mixture to the minibatch losses.
Both papers study fixed, externally corrupted labels in supervised learning. Both rely, explicitly or implicitly, on noisy samples having large losses. That assumption fails in self-training. A wrong pseudo-label accepted at confidence 0.95 has, by construction, a small loss on the view that produced it. These papers therefore constrain our design: sample-level loss shaping and small-loss separation are the wrong tools for the pseudo-label noise that matters most. We do not claim per-sample robustness. Menon et al.'s equivalence holds only at batch size one in any case. Our claim concerns how much each step can amplify an existing error.
The mechanism comes from High-Performance Large-Scale Image Recognition Without Normalization. That paper introduced unit-wise adaptive gradient clipping (AGC), which caps each unit's gradient relative to its weight norm, and used it to train normalizer-free networks on clean supervised ImageNet. We reuse the clipping rule. We change what it is applied to: only the unlabeled stream, and never the combined gradient.
The semi-supervised methods motivate the gap. FixMatch (a fixed 0.95 threshold with weak-to-strong augmentation consistency) and FlexMatch (per-class curriculum thresholds) treat an accepted pseudo-label as ground truth during backpropagation. FlexMatch also admits noise on imbalanced SVHN because thresholds for rare classes stay low. Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learning fights the problem in data and loss space with soft labels, mixup, and a minimum number of labeled samples per batch. It also documents that Wide ResNet-28-2 collapses at 500 labels, but it offers no optimizer-level remedy. Semi-Supervised Learning with Meta-Gradient updates the pseudo-label vectors with meta-gradients computed on the labeled set, not the parameter step. Mitigating Dataset Bias by Using Per-Sample Gradient warns that per-sample gradient methods cost 1.5× to 2× in runtime. It also warns that upweighting high-gradient samples would amplify label errors. Our design avoids both problems by working on aggregate, per-unit gradients.
Core Claim
Write $G_u^{(i)}$ for the minibatch gradient of the unlabeled (pseudo-label) loss with respect to unit $i$, and $W_i$ for that unit's weights. We replace $G_u^{(i)}$ with
$$\tilde G_u^{(i)} = G_u^{(i)} \cdot \min\left(1,\; \frac{\lambda_u \max(\|W_i\|, \epsilon)}{\|G_u^{(i)}\|}\right),$$
where $\lambda_u$ is the allowed relative step and $\epsilon$ is a small floor for units near zero. The labeled gradient $G_\ell$ passes through unchanged, so the update direction is built from $G_\ell + w_u \tilde G_u$, with $w_u$ the usual unlabeled-loss weight.
The intuition is simple. One step can then change a unit's weights by at most a fraction of order $\lambda_u$ times the learning rate from pseudo-labels alone. A wrong-class classifier row therefore cannot be dragged far in a single burst, and the labeled signal and the augmentation consistency get time to correct it before the error compounds.
We make three claims:
- Bursts exist. Collapse episodes are preceded by spikes in the unit-wise ratio $\|G_u^{(i)}\|/\|W_i\|$, concentrated on a few units.
- The cap beats constant down-weighting. Because it binds only during those spikes, it outperforms any constant $w_u$ at an equal tuning budget.
- Stream separation and unit resolution are necessary. Clipping the combined gradient, or clipping globally, does not give the same benefit.
Research Program
The work has three stages.
Stage 1: instrumentation. We run FixMatch and FlexMatch with the true labels of the "unlabeled" data hidden from training but logged. At every step we record the unit-wise pseudo-label gradient ratios and the precision of the accepted pseudo-labels for each class. We also record a persistence rate: the fraction of wrong accepted labels that are still wrong and accepted $k$ epochs later. This stage tests whether the spikes lead drops in precision and rises in persistence. It is useful as a published diagnostic even if the method fails.
Stage 2: the method. We implement the split-stream cap. Labeled and unlabeled losses are backpropagated separately. FixMatch's unlabeled batch is seven times larger than the labeled one, so the extra labeled backward pass is cheap. We expect well under the 1.5× to 2× overhead of per-sample methods and will report measured wall-clock time.
Stage 3: a minimal analysis. In a linear softmax self-training model with a fixed confidence threshold, we derive how the relative weight change allowed per step bounds the growth of the wrongly accepted mass per step. This yields a prediction that can be checked against Stage 1 logs: the cap should reduce persistence roughly in proportion to how often it binds.
Baselines
Tuned FixMatch. The naive reference point is FixMatch at $\tau = 0.95$, reimplemented in our codebase with the same backbone (WRN-28-2 for CIFAR-10 and SVHN), schedule, augmentations and seeds as every other arm. It anchors the reported numbers: 94.93% accuracy at 250 labels and 88.61% at 40 labels on CIFAR-10, with 64.28% median at one label per class. Our reimplementation must land within about one point of these values before any comparison counts.
FlexMatch. The strongest thresholding competitor is FlexMatch, because it already attacks which pseudo-labels get through, class by class. The file reports only a relative figure: a 13.96% error reduction versus FixMatch on CIFAR-100 at four labels per class, and 18.96% on STL-10. No absolute FlexMatch accuracy was found on Lacuna for CIFAR-10, CIFAR-100 or SVHN, so we will run it ourselves under identical conditions. Beating it means a higher mean accuracy over five seeds by more than the pooled seed standard deviation. We will test this on CIFAR-100 at four labels per class and on SVHN. We also add our cap on top of FlexMatch to test whether the two are complementary.
Arazo et al.'s regularized pseudo-labeling. The strongest confirmation-bias-specific competitor is the M* configuration of Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learning (soft labels, mixup, minimum labeled samples per batch). It reports 13.68% error on CIFAR-10 with 500 labels, down from 52.44% for naive pseudo-labeling. We use this exact setting with WRN-28-2, the architecture they found most fragile.
SVHN reference points. At 1,000 labels we cite meta-gradient SSL at 3.15% error (Semi-Supervised Learning with Meta-Gradient) and Tri-net at 3.45% (Tri-net for Semi-Supervised Deep Learning) as external numbers, not rerun.
Ablations isolating the key ingredient. Every ablation receives the same 12-configuration tuning budget on a held-out split drawn from the labeled data:
- Constant down-weighting: a sweep over $w_u$, plus a linear warm-up of $w_u$.
- Combined-stream AGC: AGC applied to the combined gradient, as in Brock et al.
- Global cap: the unlabeled stream clipped by a single global norm.
- Layer-wise cap: the unlabeled stream clipped layer by layer.
- Proposed method: the unit-wise, unlabeled-only cap.
Why not just…?
Why not just lower the unlabeled loss weight or warm it up? This is the most serious threat. A constant weight also bounds the average pseudo-label step. Our argument works only if the harmful steps come in rare, concentrated bursts, so that a constant weight small enough to suppress them also slows useful learning the rest of the time. Stage 1 tests the premise directly, and the equal-budget $w_u$ sweep tests the conclusion. If the best constant weight matches the cap within seed noise, we will report that clipping adds nothing beyond reweighting.
Why not just use PHuber loss or OGC on the unlabeled term? Both bound the influence of samples with large loss (Can gradient clipping mitigate label noise?, Optimized Gradient Clipping for Noisy Label Learning). Wrong pseudo-labels accepted at high confidence can have small loss on the weak view, though their loss on the strong view may be larger. We will include both methods, applied to the strong-view unlabeled loss, as arms. We will also log whether the mixture model used by OGC separates correct from incorrect accepted pseudo-labels. If it does, OGC is a real competitor and the comparison decides between the two.
Why not just clip the whole gradient with AGC, or rely on Adam? Clipping the combined gradient mixes the labeled anchor with the pseudo-label signal. It therefore throttles ground-truth learning exactly when the pseudo-label gradient spikes. Adam's per-coordinate normalization rescales the combined gradient in the same way. Combined-stream AGC and an Adam variant are explicit arms. The split-stream claim fails if either one matches the proposed method.
Why not just use soft labels and mixup? Arazo et al.'s interventions change the targets, not the step size, so they are compatible in principle. We run the cap on top of M* in the 500-label setting. A gain there shows that the two mechanisms are not redundant.
Proposed Experiments
The benchmarks are CIFAR-10 with 40, 250 and 500 labels, plus one label per class; CIFAR-100 with four labels per class; and SVHN with 1,000 labels. SVHN is the imbalanced case where FlexMatch admits noise. Every setting uses five seeds.
Beyond accuracy we measure precision of accepted pseudo-labels over training, the persistence rate, the variance of final accuracy across seeds, and wall-clock overhead.
The decisive checks, in order, are:
- Premise: Stage 1 shows ratio spikes that lead drops in pseudo-label precision. If not, the method is abandoned and the diagnostic is published.
- Persistence: the cap lowers persistence more than the best constant $w_u$ at equal final pseudo-label coverage.
- Accuracy: it beats tuned FlexMatch by more than seed noise on CIFAR-100 at four labels per class and on SVHN.
- Ablations: the unit-wise, unlabeled-only variant beats the combined-stream and global variants.
- Fragile regimes: it narrows the spread across seeds in the one-label-per-class and WRN-28-2 at 500 labels settings, where confirmation bias is most documented.
References
- Can gradient clipping mitigate label noise?
- Optimized Gradient Clipping for Noisy Label Learning
- High-Performance Large-Scale Image Recognition Without Normalization
- FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence
- FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labeling
- Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learning
- Semi-Supervised Learning with Meta-Gradient
- Mitigating Dataset Bias by Using Per-Sample Gradient
- Tri-net for Semi-Supervised Deep Learning