In long-tailed semi-supervised learning, a model gets a small labeled set in which some classes are common and others appear only a handful of times. It also gets a large unlabeled pool, which it labels for itself and then retrains on. For a rare class with three to five labeled examples, those few "seed" images are the model's entire definition of the class. Bootstrapping research in NLP has long shown that iterative self-labeling amplifies whatever quirks its seeds carry. Yet the image-classification methods built for this setting report results on one or a few fixed labeled splits, and they never ask how much a rare class's final accuracy depends on which particular examples were drawn.
We propose to measure this dependence directly and then reduce it. Our hypothesis is that much of the run-to-run variation in tail-class accuracy is a lottery over seed draws. We expect a cheap statistic, computed before training, to predict which tail classes will fail: how typical a class's seeds are relative to the unlabeled pool's own density. Weighting each seed by that typicality when the model forms its class anchors should shrink the losing end of the lottery without extra labels. The test is whether this narrows the gap between a lucky and an unlucky draw more than switching to a stronger long-tail method does.
Core Claim
Our claim has three parts. First, in standard long-tailed semi-supervised benchmarks, the spread of per-class tail accuracy across random draws of the labeled tail seeds is large compared with the gaps between published methods. Second, a seed-reliability score predicts that spread before any training happens. The score is the average density of a class's seeds within a frozen self-supervised embedding of the unlabeled pool, together with how dispersed the seeds are. Third, a simple correction removes much of the worst-draw damage. The correction down-weights atypical seeds when computing class prototypes, meaning the per-class mean feature vectors that several methods use as anchors, and in the labeled loss.
The new contribution is treating the seed draw itself as the variable under study, and using geometry measured on the unlabeled pool to correct it. Prior work treats the labeled set as fixed and redesigns the loss or the pseudo-label filter.
Grounding in Prior Work
The motivation comes from two literatures that have not been joined. Bootstrapping studies show that seeds lacking coverage or containing noise send the loop into drift that it never recovers from. Shutova et al. found metaphor systems fail entirely on domains their seeds miss. De Belder et al. found relation extractors break when seed pairs are noisy or redundant. Understanding seed selection in bootstrapping showed that random seeds give high variance on poorly separated, imbalanced text data, and proposed Expected Model Rotation, which chooses seeds by how much they would move a linear decision boundary. That method assumes the practitioner chooses the seeds, and it works on shallow models. We instead take the common case where the seeds are given, and we work with deep features.
On the long-tail side, DARP showed that self-labeling amplifies imbalance: a true imbalance ratio of 150 becomes an effective ratio above 1,000 in the pseudo-labels. That is exactly the amplifier the bootstrapping work describes. A line of methods then repairs the loop at different points:
- InPL filters unlabeled samples by logit energy rather than softmax confidence.
- UDTS filters by dropout-based uncertainty with class-wise moving thresholds.
- CCL uses a dual classification head and smoothed pseudo-labels.
- CoLA calibrates logit adjustment on a proxy validation set resampled from the labeled seeds.
- GBC regularizes paths in feature space between unlabeled features and a memory bank of class anchors called a Prototype Atlas.
All five take the labeled seeds as given. Two of them openly depend on seed quality. GBC's authors note that the Atlas is fragile early in training when labeled data is scarce. CoLA's proxy validation set inherits whatever narrowness the tail seeds have.
These papers motivate us, because they expose the dependency. They also constrain us: any correction must slot under their loops rather than replace them. Towards Inter-class and Intra-class Imbalance adds a warning. Its finding that naive hard-example mining collapses beyond 20% label noise implies we must not treat "atypical" as "informative," which is the trap that MAK's loss-based measure of tailness risks.
Research Program
The first component is a measurement protocol. For a benchmark split, we hold the head-class labeled examples fixed and redraw the labeled examples of the medium and tail classes many times (on the order of 20 draws). The unlabeled pool and the training seed stay fixed. Separately, we vary only the training seed, which isolates data-draw variance from optimization noise. We report three quantities per class: mean tail accuracy, its standard deviation across draws, and worst-decile accuracy. We run this for FixMatch and for each long-tail method above, which shows whether newer methods reduce sensitivity or only raise the mean.
The second component is the reliability score. We embed the unlabeled pool once with a frozen self-supervised encoder. For each labeled seed embedding $z_i$ we compute a k-nearest-neighbour density estimate $\hat p(z_i)$ over unlabeled embeddings. A class's reliability combines two things: the mean log-density of its seeds, and the dispersion of those seeds relative to the local neighbourhood scale. We then test whether this score, computed before training, predicts final per-class recall across draws.
The third component is the correction, Seed-Typicality Anchoring. Wherever a method forms a class anchor from labeled data, we replace the plain mean with a typicality-weighted one:
$$\mu_c = \frac{\sum_{i \in S_c} w_i z_i}{\sum_{i \in S_c} w_i}, \qquad w_i = \hat p(z_i)^{\alpha}.$$
Here $S_c$ is the seed set of class $c$ and $\alpha \ge 0$ sets how strongly atypical seeds are discounted. Setting $\alpha = 0$ recovers the ordinary mean. The anchors this applies to are GBC's Atlas initialization and the class centers in the labeled supervised term. The same weights scale each seed's labeled loss, and they reweight the draw of CoLA's proxy validation set. The correction is deliberately small: one density pass at the start, plus a reweighting.
Two properties keep this from being a threshold hack. It never discards a seed. And because density is measured on the unlabeled pool, it uses information the labeled set alone cannot provide. That second property is the point at which the bootstrapping concern ("the loop cannot see what its seeds miss") gets an answer.
Baselines
The naive reference point is FixMatch trained on the drawn split, together with a supervised model trained only on the labeled seeds. DARP showed that the latter can beat FixMatch on minority classes, so both are needed to establish the floor.
The strongest existing competitors are GBC and CoLA. GBC reports 68.6% top-1 on ImageNet-127, which surpasses Meta-Expert by 1.2%. It also reports 27.5% top-1 on ImageNet-1K at 64×64, a 2.5% gain over SimPro (GBC). For CIFAR-10-LT, the file gives only relative numbers: InPL reports +3% over FixMatch (InPL), UDTS reports +5.26% over FixMatch (UDTS), and GBC reports nearly 1% over SimPro in the reversed setting. CoLA reports more than 1% over the runner-up across distribution shifts on CIFAR-100-LT (CoLA). We found no absolute CIFAR-LT accuracies for GBC or CoLA on Lacuna, so we will reproduce them.
We run every method with its published hyperparameters, the same backbone, the same epochs, and the same set of seed draws. Our variant adds only the weighting. The tuning budget for $\alpha$ is matched to one extra hyperparameter search on a held-out draw that is never used for evaluation.
Beating a baseline means two things. Worst-decile tail accuracy must rise and the across-draw standard deviation must fall, both by more than the training-seed noise floor, while mean overall accuracy stays within noise.
The key ablations are $\alpha = 0$, which tests whether any gain comes from the weighting itself, and weights randomly permuted within each class, which keeps the reweighting magnitude but destroys the typicality information. We also include an inverted variant that up-weights atypical seeds, MAK-style, to test the noise-versus-tailness concern directly.
Why not just…?
Why not just report means over more splits? This fixes the reporting problem, and our protocol does exactly that. It does nothing for a practitioner who has one labeled set and cannot redraw it. Our measurement half adopts this suggestion. The correction half addresses what it leaves open.
Why not just start from a strong self-supervised encoder, which may make seed choice irrelevant? This is the most serious threat. If pretrained features already place every seed near its class mode, typicality weights will be nearly uniform, and the effect will vanish. We test this head-on by running the full protocol both from scratch and from the same frozen self-supervised initialization used to compute densities. If the variance collapses under pretraining alone, we will report that as the main finding. It would be a useful negative result for the field, and it would narrow our correction to the from-scratch regime.
Why not just use a robust method such as GBC or CoLA? These may already reduce sensitivity, and the measurement protocol checks that directly. Two facts suggest they do not fully do so. Both methods build their anchors or proxy validation sets from the seeds, and GBC's authors acknowledge early-epoch fragility of the Atlas. Our correction is applied inside these methods rather than competing with them.
Why not choose better seeds actively, as in Expected Model Rotation? That requires control over which examples get labeled. We include typicality-based seed selection as an upper-bound arm. It shows how much of the gap reweighting recovers when the seeds cannot be chosen.
Why not just label one more example per tail class? That costs annotation. We report this arm to give a price in labels for the gap our correction closes.
Proposed Experiments
We start with CIFAR-10-LT and CIFAR-100-LT at imbalance ratios 100 and 150, under consistent, uniform, and reversed unlabeled distributions, following CCL and CoLA. We then scale to STL-10-LT and ImageNet-127. On ImageNet-127, fewer draws (around 5) keep compute feasible.
We also stress seed quality directly. In a controlled-corruption study, we replace one tail seed per class with either a mislabeled image or a correct but atypical one (identified by low density). This lets us test whether the weighting separates the two cases, which is the distinction DuBE warns about.
A small text check on the 20 Newsgroups splits sl-8-1 and ol-8-1 from Understanding seed selection in bootstrapping tests whether the reliability score predicts the variance that study observed under random seeds.
We treat three outcomes as decisive. The first is whether the pre-training reliability score correlates with per-class recall across draws, within each class, which controls for class identity. The second is whether typicality weighting raises worst-decile tail accuracy over the permuted-weight control inside GBC and CoLA. The third is whether either effect survives self-supervised initialization. If the first fails, the proposal reduces to a variance audit. If the second fails while the first holds, seed sensitivity is real but not repairable by reweighting, and the case for active seed acquisition strengthens.
References
- Metaphor Identification Using Verb and Noun Clustering
- Does Google own Youtube? Entity Relationship Extraction with Minimal Supervision
- Understanding seed selection in bootstrapping
- Distribution Aligning Refinery of Pseudo-label for Imbalanced Semi-supervised Learning
- InPL: Pseudo-labeling the Inliers First for Imbalanced Semi-supervised Learning
- Uncertainty-aware Sampling for Long-tailed Semi-supervised Learning
- Continuous Contrastive Learning for Long-Tailed Semi-Supervised Recognition
- CoLA: Co-Calibrated Logit Adjustment for Long-Tailed Semi-Supervised Learning
- Geometric Regularization for Long-Tailed Semi-Supervised Learning via Gaussian Feature Bridges
- Towards Inter-class and Intra-class Imbalance in Class-imbalanced Learning
- Improving Contrastive Learning on Imbalanced Seed Data via Open-World Sampling