Sampled Pseudo-Labels as Saddle-Escape Noise in Self-Training

Self-training is the simplest way to use unlabeled data. A model labels unlabeled examples with its own best guess, called a pseudo-label, and then trains on those guesses. It fails in a familiar way. Wrong guesses get reinforced, the model grows more confident in them, and accuracy plateaus well below what the data allows. This failure is known as confirmation bias. Nearly every fix works on the data side: better confidence thresholds, augmentation, or auxiliary networks that score label quality. This note takes the optimizer's side.

The optimization literature has a precise account of why stochastic gradient descent (SGD) escapes saddle points, which are flat spots where the gradient vanishes but some direction still slopes downhill. SGD escapes them because its mini-batch noise has variance along those downhill directions. We argue that hard pseudo-labels destroy exactly this noise. Labels drawn at random from the model's own predicted distribution restore it, without adding any systematic bias. We propose to measure this effect directly, build a cheap escape step around it, and test whether it breaks confirmation-bias plateaus on standard low-label benchmarks.

Core Claim

Consider the gradient of cross-entropy loss with respect to a model's output scores (its logits) on one unlabeled example. It equals $p - y$, where $p$ is the predicted class distribution and $y$ is the one-hot target. This gives three regimes for the unlabeled term:

  • Hard argmax pseudo-labels. These produce a gradient that is biased toward the current guess. On examples above the confidence threshold the gradient is also nearly deterministic, and examples below the threshold contribute nothing. The noise the optimizer sees is therefore almost absent on the uncertain examples, which is where the errors live.
  • Soft self-targets ($y = p$). These remove the bias, but they also remove all signal and all noise apart from augmentation mismatch.
  • Sampled labels ($y \sim p$). These keep the mean at zero but leave noise behind:

$$ \mathbb{E}_{y\sim p}[p-y]=0,\qquad \mathrm{Cov}_{y\sim p}[p-y]=\mathrm{diag}(p)-pp^\top . $$

Pushed back through the network's Jacobian $J$ (the derivative of logits with respect to weights), this covariance becomes $J^\top(\mathrm{diag}(p)-pp^\top)J$. That is the per-example Fisher information. It is large exactly along weight directions that would change predictions on uncertain examples, which are the directions that could flip a wrong pseudo-label.

The claim has three parts. First, at confirmation-bias plateaus, hard pseudo-labeling violates the Correlated Negative Curvature (CNC) condition of Escaping Saddles with Stochastic Gradients. That condition requires gradient noise to have variance along the downhill directions. Here, the noise has little projection onto the downhill directions of the clean-label loss. Second, switching uncertain examples to sampled labels during detected plateaus restores that projection. Third, this beats isotropic weight-space noise of equal magnitude, because isotropic noise thins out as $O(1/d)$ along any single direction, as that paper shows empirically.

Relation to Prior Work

The research file reports no prior work that treats pseudo-label confirmation bias as a saddle-escape problem in parameter space. Three lines of work come close.

Gradual Domain Adaptation via Gradient Flow uses Langevin noise, but it moves data points between domains. It does not touch the model's weights or their curvature.

The saddle-to-saddle analyses in deep networks, Saddle-to-Saddle Dynamics Explains A Simplicity Bias Across Neural Network Architectures and Saddle-To-Saddle Dynamics in Deep ReLU Networks, explain long plateaus under clean supervision only. They motivate our view that plateaus are geometric objects, but they say nothing about targets the model generates for itself.

FixMatch observes that SGD with Nesterov momentum is critical and Adam fails in self-training, but offers no mechanism. Our measurements are designed partly to explain that observation.

Two further papers constrain the design.

Confirmation Bias in Gaussian Mixture Models proves that both hard (K-means) and soft (EM) self-assignment stay correlated with arbitrary initial hypotheses indefinitely. This is a negative result for soft pseudo-labels. It tells us that soft targets alone will not escape, and it predicts that only label-space randomness, not label-space averaging, can. Drawing random assignments instead of computing hard or posterior-weighted ones is an old idea in mixture-model estimation. What we add is the claim that, in deep networks, this randomness acts specifically as curvature-aligned escape noise, together with tests that verify or falsify that claim.

Accelerated Gradient Descent Escapes Saddle Points Faster than Gradient Descent supplies the momentum-reset component. Its curvature-exploitation step zeroes the momentum buffer when instability is detected. We borrow this because accumulated momentum from biased pseudo-gradients would otherwise carry the model straight back into the basin it is trying to leave.

Research Program

Diagnosis

The first part of the program is diagnostic, and it is possible only because benchmark "unlabeled" data actually has ground-truth labels we can withhold. We run FixMatch from the USB codebase bundled with Westlake-AI/SemiReward on CIFAR-10 with 40 labels and CIFAR-100 with 400 labels, and save checkpoints along the run. At each checkpoint we compute the lowest Hessian eigenvectors of two losses on a fixed subset, using Hessian-vector-product Lanczos:

  • the self-training loss, built from the pseudo-labels;
  • the clean loss, built from the true labels.

We then compute the variance of mini-batch gradient noise projected onto the clean loss's most negative eigenvector. We do this under hard, soft and sampled pseudo-labels at the same weights.

This answers questions the file lists as open. Are plateaus local minima of the self-training loss that are still saddles, or even non-stationary points, of the clean loss? And does the hard-label variance along clean negative curvature fall well below the sampled-label variance?

The escape step

The second part is the method, which we call sampled-label escape. A plateau detector watches two quantities: the per-epoch fraction of pseudo-labels that flip, and the slope of the unlabeled loss. It uses no labeled validation data. When both fall below set levels, the method does three things for a short window of $K$ steps:

  1. It resets the momentum buffer.
  2. For examples below the confidence threshold, it replaces their hard pseudo-labels with labels sampled from the temperature-scaled weak-augmentation prediction.
  3. Optionally, it raises the step size, mirroring the periodic amplification in CNC-SGD.

Outside these windows, training is unchanged FixMatch or FlexMatch. The extra cost is one categorical sample per example.

Theory

The third part is a small theoretical result in the setting of the Gaussian-mixture analysis. The model is a two-component linear classifier trained by gradient self-training. The goal is to show that the wrong self-consistent fixed point is linearly stable under hard labels and unstable under sampled labels. A sketch of why: the sampled-label noise covariance has a nonzero component along the direction that separates the two components, and that is the flip direction. We will claim only what we can prove there.

Baselines

Unmodified FixMatch. This is the naive reference point: hard labels with a fixed 0.95 threshold, the standard Wide ResNet-28-2 recipe, and the same iteration budget. On CIFAR-10 it reports 88.61% accuracy with 40 labels and 94.93% with 250 labels (FixMatch). Any gain must show up as a mean over at least three seeds that exceeds the seed spread. Gains should be largest on the seeds where FixMatch plateaus low, because the method claims to fix plateaus, not to add accuracy everywhere.

FlexMatch. FlexMatch is the strongest heuristic that needs no extra parameters. It lowers thresholds per class according to how well each class is learning. It reports a 13.96% error reduction over FixMatch on CIFAR-100 with 4 labels per class, and 18.96% on STL-10. It is the fairest competitor because it also adds negligible compute and no validation data. We run it from the same USB code with the same budget. We also run FlexMatch plus our escape step, because the two act on different things and should compose. Beating it means higher accuracy on CIFAR-100 and STL-10, and specifically avoiding the SVHN degradation FlexMatch attributes to class imbalance.

PabLO. PabLO is the strongest reported number. It fits a learned confidence function under an explicit error constraint, using a labeled calibration set. On CIFAR-100 with 400 labels it reports 67.44% top-1 accuracy, against 30.38% for FixMatch. On CIFAR-10 with 40 labels it reports roughly 89%, up from roughly 57%. We run it from harit7/PabLO-SSL. Our method uses no calibration set, so we compare in two ways:

  • Our method alone, with the calibration labels added back to the training set. Here beating PabLO means matching it within seed noise.
  • PabLO plus our escape step. Here beating PabLO means exceeding 67.44%.

We do not expect to beat PabLO outright on CIFAR-100, and we say so.

Ablations isolating the key ingredient. Three controls, all applied with the same trigger and window:

  • Isotropic weight perturbation. This is Perturbed Gradient Descent as in How to Escape Saddle Points Efficiently. The noise norm is matched to the measured norm of the sampled-label noise.
  • Soft self-targets. Same window, soft targets instead of sampled ones.
  • Momentum reset only. Same window, no change to labels.

The structure claim survives only if sampled labels beat all three.

Why not just…?

Why not just add isotropic noise to the weights, or raise the learning rate? This is the most direct import from the saddle literature, and it is our main ablation. The prediction against it comes from Escaping Saddles with Stochastic Gradients. There, isotropic noise variance along the most negative eigenvector decays as $O(1/d)$, while structured gradient noise does not. With millions of weights, a kick of norm comparable to one gradient step barely moves the model along the flip direction. If norm-matched isotropic noise or step-size spikes recover as much accuracy as sampled labels, our structure claim is falsified. The contribution would then shrink to the diagnosis plus a generic noise schedule.

Why not just use soft pseudo-labels with Mixup? Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learning does exactly this. It shows that it helps but is not enough on its own. On CIFAR-10 with 500 labels, Mixup alone leaves 32.10% error versus 13.68% with an added constraint on labeled samples per batch, and Wide ResNet-28-2 still collapses. The Gaussian-mixture result shows that soft self-assignment stays locked to initial hypotheses. Our account explains why: soft targets cancel the bias but also cancel the noise. The soft-target ablation, and the measured noise projection under soft targets, test this directly.

Why not just use a better threshold? FlexMatch and PabLO decide which labels enter the loss. This is a real threat on CIFAR-100, where PabLO's 67.44% is far ahead of FixMatch. But thresholds act on the admitted set, while our step acts on the uncertain set that thresholds throw away. They should therefore stack, and the composition runs will show it. If adding our step to PabLO gives no gain, the honest conclusion is that good selection already prevents the plateaus our step targets.

Why not just switch off the unlabeled loss during a plateau? With 40 labels, the labeled loss is memorized almost immediately. Its gradient and gradient noise are then near zero, so a labeled-only phase should stall. We include it as a cheap control within the same window.

Proposed Experiments and Decisive Checks

Datasets and setting. We use CIFAR-10 (40 and 250 labels), CIFAR-100 (400 labels and 4 per class), STL-10 (4 per class) and SVHN, with Wide ResNet-28-2. To probe the architecture fragility reported by Arazo et al., we also use their 13-layer CNN on CIFAR-10 with 500 labels, where naive pseudo-labeling reaches 52.44% error. All methods share the USB training loop, the iteration budget and the hyperparameter search budget. Our method gets the same search budget as FlexMatch, over trigger levels, $K$ and temperature.

Metrics. We report:

  • final top-1 accuracy;
  • the fraction of seeds that end in a low plateau;
  • pseudo-label error on the unlabeled pool over time;
  • the measured noise projection onto the clean-loss negative curvature.

What would make us drop the idea. Three findings would refute the mechanism outright:

  • The diagnosis shows that hard-label noise already projects onto clean negative curvature about as much as sampled-label noise does.
  • Plateau checkpoints turn out to be minima of the clean loss as well. If so, there is nothing to escape, and confirmation bias is a different problem from the one we describe.
  • Norm-matched isotropic perturbation equals sampled-label escape.

What would confirm it. We would want to see three things together:

  • The noise projection collapses at plateaus under hard labels.
  • The projection is restored under sampled labels.
  • Accuracy gains concentrate on the seeds and datasets with the lowest FixMatch plateaus.

A secondary prediction is that the momentum reset matters: escape windows without the reset should recover less. That would give a concrete reason for FixMatch's observed dependence on the optimizer.