Trusting Specialized Critics Only Where They Are Experts: Reliability-Weighted Pseudo-Labeling

Semi-supervised learning trains a classifier from a few labeled examples plus a large pool of unlabeled ones, usually by letting the model label the unlabeled pool itself. These guesses are called pseudo-labels, and the model keeps only those it is confident about. The known weakness is that this admission rule feeds on itself. Confident regions of the data get more pseudo-labels and become more confident, while hard regions stay starved, and confidently wrong guesses get locked in. We argue this is a form of mode collapse, the failure in adversarial training where a generator covers only a few regions of the data. Here it is the selection process that collapses onto easy regions of the unlabeled pool.

The proposal borrows the cure from multi-discriminator generative models. We train several lightweight critic heads that deliberately specialize on different regions of the data, so that someone is responsible for every region. Specialization creates a new problem, though. A specialist is reliable inside its region and close to useless outside it, so averaging their votes uniformly is exactly the wrong move. We therefore combine the heads with a reliability model from crowdsourcing that estimates, separately for each region, how much to trust each head. Those estimates come from the heads' agreement patterns on unlabeled data and are anchored on the few labeled examples. In two sentences: specialized critics give coverage of hard regions, and region-dependent reliability weighting keeps specialists from voting where they know nothing. Neither ingredient works alone; together they should stop pseudo-label selection from collapsing.

Core Claim

The claim is that a reliability-weighted consensus over deliberately specialized critic heads produces pseudo-labels that are both more accurate and more evenly spread across the unlabeled pool than three alternatives: single-model confidence thresholds, a single learned quality scorer, and uniform ensembles. The coverage benefit should be largest in the lowest-label regimes.

The contribution is the pairing. MCL-GAN shows that splitting discrimination across specialists trained by Multiple Choice Learning (only the best-scoring few heads get credit for each sample) forces coverage of all modes. It lowers CIFAR-10 FID (Fréchet Inception Distance) from 37.7 for DCGAN to 26.87. Its heads are symmetric, though, and it never asks how much to trust each one. Classical aggregation work answers the trust question but assumes static raters:

  • Platanios et al. infer error rates of frozen classifiers from agreement alone.
  • Lien et al. show that 5–10% supervised items can anchor rater weights.
  • Yan et al.'s LGP makes annotator expertise depend on the input and smooths it with a graph built over unlabeled points. This is the closest conceptual precursor, but it is shallow, binary, and aimed at human annotators.

We apply input-dependent reliability to learned critics that change during training and that are made to be input-dependent experts on purpose.

Research Program

The model is a standard semi-supervised backbone with $K$ small classifier heads on shared features. On labeled data, heads are trained by Multiple Choice Learning, following MCL-GAN's recipe:

  • the $k$ lowest-loss heads for each example receive the cross-entropy gradient;
  • the remaining heads are pushed toward a uniform prediction;
  • a balance term prevents one head from winning everything.

The shared features of labeled and unlabeled data are clustered into $C$ regions, refreshed every few epochs. In this step the unlabeled marginal does work beyond supplying pseudo-labels: it defines the regions over which reliability is estimated, in the spirit of LGP's manifold prior.

For an unlabeled example $x$ in region $c(x)$, each head casts a vote $v_k$. The vote is a hard class under weak augmentation. The consensus posterior is

$$p(y \mid x) \propto \pi_y \prod_{k=1}^{K} \theta_{k,c(x)}[y, v_k],$$

where $\pi_y$ is the class prior and $\theta_{k,c}$ is head $k$'s confusion matrix in region $c$. This is a Dawid–Skene model whose confusion matrices vary by region. The matrices are fit by EM on the current unlabeled batch history. Labeled examples enter as observed-truth "salt" items, in the sense of Bansal et al. and Lien et al.

The pseudo-label is the argmax of $p(y \mid x)$. Following RGPO's finding that scaling the loss beats hard filtering, the training weight is a smooth function of the posterior maximum rather than a hard cut. The student, which is the backbone with an ordinary head, trains on strong augmentations as in FixMatch. The critic heads train on the same consensus targets, restricted again by Multiple Choice Learning.

One constraint shapes the design. Dependence-aware aggregation via Ising models warns that judges sharing training data produce correlated errors, which fool agreement-based estimators. Our heads share a backbone, so this applies directly. We take two measures:

  1. Anchor the confusion matrices on labeled items rather than relying on agreement alone.
  2. Measure pairwise error correlation between heads on held-out labeled data. If it is high, add a stop-gradient on the backbone for head updates.

If correlation remains high even after these measures, the independence model is wrong for this setting. That would be a reportable negative result.

We will implement everything inside LAMDA-SSL, which already provides FixMatch, FlexMatch, Mean Teacher, and Co-Training. All methods then share data loaders, augmentations, and backbones.

Baselines

The naive reference is supervised training on the labeled subset alone. Above it, vanilla FixMatch with a fixed confidence threshold of 0.95 represents the self-reinforcing selection we claim collapses. Towards Understanding Why FixMatch Generalizes Better reports that FixMatch beats supervised learning on STL-10 by over 28%. Its SA-FixMatch variant, which masks the features the model already relies on, adds 1.5% on Imagewoof and 1.38% on ImageNet. We include SA-FixMatch because its gains also come from forcing the model to look elsewhere, so it competes directly with our coverage argument.

FlexMatch is the cheapest direct answer to class-level selection imbalance, since it adjusts thresholds per class.

The strongest existing competitor is SemiReward. It replaces confidence with a learned scorer of pseudo-label quality and reports accuracy gains of 1.9%–3.7% when added to FlexMatch/FreeMatch, along with 1.7x–3.5x faster convergence. The file reports these gains only relative to the host methods, so no absolute SemiReward accuracy on CIFAR-100, STL-10, or Yahoo! Answers was found on Lacuna. We will rerun it ourselves rather than quote numbers.

Fairness rules:

  • All methods use the same backbone, label splits, and augmentations.
  • Iterations are matched, and wall-clock time is reported.
  • Each method gets the same hyperparameter search budget (equal number of configurations).
  • Our $K$ heads add only head-sized parameters. We also report a SemiReward variant whose rewarder is enlarged to match our extra parameter count.

Beating SemiReward means higher test accuracy averaged over three seeds in the lowest-label split of at least two of the three image benchmarks, plus higher coverage of hard regions as defined in the experiments section.

Three ablations isolate the key ingredient:

  • Specialized heads with uniform averaging. We predict this is worse than a non-specialized ensemble, because specialists vote outside their niche.
  • Non-specialized heads with the same region-dependent reliability model. This tests whether specialization contributes anything.
  • Winner-take-all routing. Each region trusts only the head that won Multiple Choice Learning there, with no EM.

Why not just…?

Why not just average an ensemble of heads? Averaging independent heads reduces variance. It gives no incentive to cover hard regions, since all heads train on the same confident examples and inherit the same blind spots. The uniform-ensemble and specialized-uniform ablations test this directly. If a plain non-specialized ensemble with reliability weighting matches the full method, then specialization is unnecessary and we will drop it.

Why not just use per-class thresholds like FlexMatch? This is a real threat when collapse happens at the level of whole classes. Collapse inside a class is invisible to it. Examples are rare poses or backgrounds, which the feature-masking analysis of FixMatch suggests are exactly the secondary features supervised learning ignores. Our coverage metric is computed over feature clusters as well as classes. If FlexMatch matches us at both levels, the reliability machinery is not earning its cost.

Why not just use SemiReward's learned scorer? A single scorer has one notion of quality everywhere and can itself overfit to well-covered regions. SemiReward also notes its filtering is coarse, at the level of whole samples. Our method differs by keeping several scorers and estimating where each is trustworthy. The head-to-head comparison and the per-cluster pseudo-label accuracy curves settle this.

Why not just trust the Multiple Choice Learning winner in each region? This is the most serious alternative, because the winner assignment is already a crude reliability signal. It fails when regions drift as the backbone changes, and when two heads are partially competent in the same region. Both situations are common early in training. If winner-take-all routing matches full EM within seed noise, we will report that and simplify the method.

Proposed Experiments and Decisive Checks

Benchmarks:

  • CIFAR-100 and STL-10 (the standard semi-supervised folds of STL-10, with 100k unlabeled images);
  • Imagewoof, as a fine-grained stress test where intra-class modes matter;
  • Yahoo! Answers, to show the method is not specific to vision.

Each uses a lowest and a moderate label budget, three seeds each. Metrics are test accuracy, pseudo-label accuracy, and convergence iterations.

Two collapse diagnostics carry the argument. Both use feature clusters defined by an independent pretrained encoder, so that they do not reward our own clustering.

  1. Coverage over time. We track the entropy of admitted pseudo-label mass across clusters during training.
  2. Hard-cluster recall. We take the clusters with lowest initial confidence and measure the fraction admitted with correct labels by mid-training.

The core prediction is that FixMatch and SemiReward show falling or flat coverage in hard clusters early on, while our method raises it without lowering pseudo-label accuracy.

A calibration check tests the reliability model itself. We compare the estimated per-region confusion matrices against matrices computed from held-out labeled examples. If the estimates are poorly calibrated, any accuracy gain must come from something other than the reliability model, and we would say so.

Finally, we sweep the Multiple Choice Learning balance weight and $K$. MCL-GAN reports delicate tuning of these, so we will show whether reliability weighting makes results less sensitive to them.

The idea fails if it does not beat SemiReward and FlexMatch in the lowest-label regime at matched compute. It also fails if its gains vanish under the winner-take-all ablation.