Self-training teaches a model on unlabeled data by letting it label that data itself. A natural way to catch its wrong guesses is to train several models and distrust any example on which they disagree. This note argues that the safeguard quietly destroys itself. When every ensemble member is trained toward the same consensus pseudo-labels, the shared targets pull the members together. Their disagreement shrinks over training, and the disagreement filter loses its power in the late, low-label regime where it is most needed. The same failure is well understood in particle-based Bayesian inference. There, Stein Variational Gradient Descent (SVGD) moves a set of "particles" (candidate parameter vectors) under two forces: a pull toward high posterior probability and a push between nearby particles. In high dimensions the push vanishes and the particles clump into one point.
We propose to measure this "pseudo-label variance collapse" directly and to fix it with two changes taken from the SVGD literature. First, each member is trained on pseudo-labels produced by the other members, never by itself. This removes the self-reinforcing term, as damped SVGD does. Second, members repel each other in the space of predictions on unlabeled inputs, not in weight space, where distances stop being informative in high dimensions. The goal is a small ensemble whose disagreement stays informative across training and so keeps detecting "confidently wrong" pseudo-labels.
Core Claim
Understanding the Variance Collapse of SVGD in High Dimensions traces SVGD's underestimated variance, which scales as $\mathcal{O}(n/d)$ for $n$ particles in dimension $d$, to a specific cause. SVGD computes its driving force from its own current, already-biased particles, which closes a feedback loop. MMD-descent draws on external target samples and keeps the variance near 1.0. Pseudo-labeling has the same structure: the targets are computed from the model's own current predictions. Our claim has two parts.
The first part is a measurable diagnosis. For an ensemble trained on shared consensus pseudo-labels, two quantities decay over training: the pairwise prediction disagreement on the unlabeled pool, and that disagreement's ability to rank wrong pseudo-labels above correct ones. Ensembles of plain SVGD particles in weight space decay faster still, because their repulsion is already near zero.
The second part is a remedy. Leave-one-out targets combined with function-space repulsion keep the disagreement informative throughout training. Selecting pseudo-labels by this kept-alive disagreement should beat input-perturbation stability filters at matched compute.
The function-space repulsion borrows the move of Approximate Bayesian Inference with Stein Functional Variational Gradient Descent. That paper runs Stein descent over function evaluations rather than weights, but only for regression and bandits. We apply it to classification outputs on unlabeled data.
Concretely, let $p_k(x)$ be member $k$'s class-probability vector on unlabeled input $x$, for $K$ members. Let $\hat y_{-k}(x)$ be the argmax of the average of the other members' predictions on a weakly augmented view. Member $k$ minimizes
$$ \mathcal{L}_k = \sum_{x} m(x)\,\mathrm{CE}\big(p_k(\tilde x), \hat y_{-k}(x)\big) \;-\; \beta \sum_{j \neq k} \sum_{x} \kappa\big(p_k(x), p_j(x)\big), $$
where $\tilde x$ is a strongly augmented view and $\mathrm{CE}$ is cross-entropy. The mask $m(x)\in\{0,1\}$ keeps $x$ only if the members' disagreement on $x$ falls below a running class-wise quantile. The kernel $\kappa$ is an RBF kernel on per-example probability vectors, a space with only as many dimensions as there are classes, so the distance concentration that defeats weight-space kernels in Sliced Kernelized Stein Discrepancy does not arise. The weight $\beta$ sets the repulsion strength. Subtracting the kernel term pushes members apart. A labeled cross-entropy term on the labeled set, applied to each member, is omitted from the equation for brevity.
Research Program
The work has three parts.
The first part is analysis in a model simple enough to solve. We study linear-Gaussian self-training, where $K$ linear classifiers are refit on pseudo-labels. We will derive how the spread of the ensemble contracts per round as a function of three quantities: the self-weight $1/K$ in the consensus label, the ratio of unlabeled to labeled data, and the parameter-to-sample ratio. That last ratio is the regime where Approximate Bayes Optimal Pseudo-Label Selection found Bayesian selection matters most. Our working conjecture is that consensus labels contract the spread geometrically, and that leave-one-out targets remove the leading contraction term in the same way damped SVGD's shrinkage of the self-correlation term does. This is a hypothesis to prove or refute, not a result we already hold.
The second part is diagnosis in deep self-training. Benchmark labels are known, so at every checkpoint we can measure two things on the unlabeled pool: mean pairwise disagreement, and the area under the ROC curve (AUROC) of disagreement as a detector of wrong pseudo-labels. We track both for independent-seed ensembles trained on consensus labels, for weight-space SVGD particles, for Tri-net-style cross-labeling, and for our method. The open question is whether detection AUROC actually decays, and when.
The third part is building the method. For efficiency, members share a convolutional trunk and own separate final blocks and classifier heads. Function-space repulsion matters most in this shared-trunk setting, because nothing else forces the heads apart. We will also run fully separate members as an upper reference, to show what sharing costs.
Baselines
The naive reference point is FixMatch from FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence. It trains a single model with a fixed 0.95 confidence threshold and weak-to-strong augmentation consistency, and it reports 88.61% accuracy on CIFAR-10 with 40 labels. It defines the floor and shares our augmentation pipeline.
The strongest thresholding baseline is FreeMatch from FreeMatch: Self-adaptive Thresholding for Semi-supervised Learning. It replaces the fixed threshold with one that adapts per class during training and adds a fairness term against collapse onto easy classes. It reports gains only relative to FlexMatch, for example a 13.59% error reduction on STL-10 with 4 labels per class. The file contains no absolute STL-10 figure, so we will rerun it ourselves.
The strongest existing selector is Local Variance Match from Leveraging Local Variance for Pseudo-Label Selection in Semi-supervised Learning. It accepts a pseudo-label only if the prediction is both confident and stable across two or three input perturbations. It reports 5.41% error on CIFAR-10 with 10 labels and 35.87% on CIFAR-100 with 400 labels. It is the most direct competitor because it targets the same "confidently wrong" examples with a cheaper stability signal.
To make comparisons fair, every method uses the same WideResNet backbone, the same labeled splits and seeds, and the same augmentation pipeline. Every method gets a matched budget of total training FLOPs: a single-model baseline is given as many extra iterations as our multi-head ensemble consumes. Every method also gets the same number of hyperparameter trials, tuned on a held-out labeled slice.
Beating LVM means lower mean error than its reported 5.41% on CIFAR-10 with 10 labels, and lower than our rerun of it under matched FLOPs, across at least five label splits.
Three ensemble baselines and ablations isolate our ingredients. An independent-seed ensemble trained on shared consensus labels and filtered by disagreement tests whether plain ensembling suffices. Tri-net cross-labeling from Tri-net for Semi-Supervised Deep Learning already trains each network on the others' labels, so it tests leave-one-out alone. Our method with $\beta=0$, compared against our method using consensus labels, separates the two ingredients.
Why not just…?
Why not just use LVM's input perturbations? Perturbing the input measures how smoothly one fixed model responds near an example. It does not measure what a differently fitted model would predict. Approximate Bayes Optimal Pseudo-Label Selection argues that parameter uncertainty is what guards against confirmation bias in high-dimensional regimes, though it shows this only on small tabular models. This alternative is a real threat, because LVM's reported numbers are strong. Our head-to-head comparison at matched FLOPs settles it, together with a check of whether combining both signals beats either one alone. If LVM matches us, our contribution reduces to the diagnosis.
Why not just train an independent deep ensemble and filter by consensus? Auto-Annotation Quality Prediction for Semi-Supervised Learning with Ensembles shows that consensus filtering works, but with 18 models and a learned quality filter. Our hypothesis is that shared targets erase the members' independence during training. The consensus-ensemble baseline, with its disagreement-AUROC curve over training, tests this directly. If its AUROC does not decay, the central claim is wrong and we will report that.
Why not just use Tri-net-style cross-labeling, which already avoids self-labels? It does avoid self-labels. The ablation with $\beta=0$ is that method under matched conditions. Our prediction is that cross-labeling slows the collapse but does not stop it, because members still converge through the labeled loss and through each other's labels. If $\beta>0$ adds nothing over cross-labeling, repulsion is unnecessary and we will drop it.
Why not just use weight-space SVGD or Stein mixtures? Understanding the Variance Collapse of SVGD in High Dimensions predicts negligible particle spread when there are few particles and $d \sim 10^6$. ELBOing Stein: Variational Bayes with Stein Mixture Inference avoids that collapse by evolving Gaussian guide distributions instead of points, but at a high cost per step and only in supervised settings. We include weight-space SVGD in the diagnosis to confirm its collapse. We do not include Stein mixtures because of their cost, which we state as a limitation.
Proposed Experiments
The decisive check is the diagnosis curve: detection AUROC of wrong pseudo-labels plotted against training step on CIFAR-10 with 10 and 40 labels. The prediction is decay for consensus ensembles and weight-space SVGD, partial decay for cross-labeling, and a flat curve for our method. If consensus ensembles do not decay, the proposal fails at its first step.
We then report accuracy on four settings: CIFAR-10 with 10 and 40 labels, CIFAR-100 with 400 labels, and STL-10 with 4 labels per class, each against all baselines under matched FLOPs. We also report pseudo-label precision and recall per class, which shows whether repulsion protects the hard classes that FreeMatch's fairness term targets.
A sweep over $\beta$ and $K$ from 2 to 8 shows whether the gain grows with ensemble size. A sweep over the ratio of labeled examples to parameters tests whether the gain concentrates where it does in Approximate Bayes Optimal Pseudo-Label Selection. Finally, we validate the linear-Gaussian theory on synthetic data, comparing predicted against measured spread contraction per round.
References
- Understanding the Variance Collapse of SVGD in High Dimensions
- Approximate Bayesian Inference with Stein Functional Variational Gradient Descent
- Sliced Kernelized Stein Discrepancy
- Approximately Bayes-Optimal Pseudo Label Selection
- FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence
- FreeMatch: Self-adaptive Thresholding for Semi-supervised Learning
- Leveraging Local Variance for Pseudo-Label Selection in Semi-supervised Learning
- Tri-net for Semi-Supervised Deep Learning
- Auto-Annotation Quality Prediction for Semi-Supervised Learning with Ensembles
- ELBOing Stein: Variational Bayes with Stein Mixture Inference