Moving the Weak-Label Crossover with Gold-Anchored Error Entropy

When labels are scarce, practitioners add cheap labels: pseudo-labels from the model itself, or zero-shot labels from a foundation model. This works at first and then backfires. Once a model trained on the small expert-labeled ("gold") set becomes as accurate as the cheap labeler, more cheap labels make it worse, because the model starts copying the labeler's mistakes. On medical imaging this loss reaches 0.10 test AUC, and robust losses soften the drop without delaying it. We propose a training objective that lets a model keep benefiting from cheap labels past that point. It works by learning where the cheap labeler is wrong, using the few gold labels as anchors and spreading that information to nearby unlabeled examples.

The tool is the minimum error entropy (MEE) criterion. Instead of pushing every prediction error toward zero, MEE makes the distribution of errors as concentrated as possible, which pulls errors into a few sharp peaks. We apply it conditionally: the errors of examples that look alike (they sit close together in an embedding space) are pushed toward the same value, and gold examples fix what that value is. If gold labels in a neighborhood show the labeler calls that kind of image "positive" when it is negative, the objective pulls the model toward disagreeing with the labeler across the whole neighborhood. Per-sample robust losses cannot express this, because they look at each example's error in isolation. The core question is whether this pairwise coupling in error space moves the crossover point, the gold-set size beyond which cheap labels stop helping. The proposal is built so that a negative answer is just as informative.

Why this intersection, and what is already known

Three bodies of evidence motivate the idea. First, saturation is real and has a structure. How Does Pseudo-Labeling Affect the Generalization Error of the Semi-Supervised Gibbs Algorithm? proves that generalization error falls as the unlabeled-to-labeled ratio grows but then hits a floor set by the information shared between labeled and pseudo-labeled data. Realistic Evaluation of Deep Semi-Supervised Learning Algorithms finds that gains from extra unlabeled SVHN data level off around 80,000 examples. From Theory to Decision Rule: Calibrating the Noisy-Label Crossover for VLM Weak Supervision Across Three Medical-Imaging Benchmarks measures the crossover with vision-language model labels. It falls at about 100 gold labels on PCAM, 20–50 on ISIC 2019 and 250–500 on NIH-CXR. That paper also shows that neither student capacity nor noise-robust losses (Generalized Cross Entropy, Co-teaching) move the crossover. This last result is the constraint that shapes the proposal. Another per-sample robust loss is unlikely to succeed where those two failed, so simply swapping cross-entropy for MEE on pseudo-labels is not worth doing.

Second, the MEE literature supplies the pieces that make a different mechanism possible. Restricted Minimum Error Entropy Criterion for Robust Classification shows that plain MEE collapses in classification. It also shows that the ideal error distribution under label noise has three peaks, at 0 for correct labels and at −1 and +1 for false negatives and false positives. That paper enforces this shape by restricting a quantized MEE (from Quantized Minimum Error Entropy Criterion) to the codebook {−1, 0, +1}, and it tolerates heavy label flipping. Its evaluation is supervised-only, binary, and uses a hand-set outlier proportion. We reuse its codebook and its half-quadratic optimizer, add unlabeled data, and replace the hand-set proportion with gold anchoring.

Third, MEE theory explains why a global error-entropy objective will fail on cheap labels, and what fixes it. Consistency Analysis of an Empirical Minimum Error Entropy Algorithm shows that when noise depends on the input, minimizing error entropy need not recover the true function unless the kernel bandwidth diverges. A Note on the W-S Lower Bound of the MEE Estimation shows that MEE reaches its optimum exactly when the error is independent of the observation. Cheap-label noise is the clearest case of input-dependent noise: foundation-model labelers fail in specific semantic regions. Conditioning the entropy on the input neighborhood is therefore the natural fix, because it makes noise roughly input-independent within each neighborhood. Distributed Minimum Error Entropy Algorithms is the only prior work joining MEE with unlabeled data. It proves that unlabeled data improves MEE rates for kernel regression, but it has no experiments, no classification, and no pseudo-label noise. We test an empirical analogue of its claim in exactly the regime it leaves out.

Core Claim

Training on cheap labels with a gold-anchored, neighborhood-conditional error-entropy term will move the crossover to substantially larger gold-set sizes. Below the crossover, accuracy will match standard training. The reason is that the objective lets the student disagree with the labeler in a structured way that transfers from gold examples to their unlabeled neighbors, instead of forcing all disagreement toward zero.

Concretely, let $w_i$ be the cheap label of example $i$, $p_\theta(x_i)$ the student's predicted probability, and $e_i = w_i - p_\theta(x_i)$ its error against the cheap label. Classes are handled one-vs-rest, so each coordinate is binary. For gold examples we know the true label $y_j$, so we fix their anchor error at the labeler's actual error $e_j^\ast = w_j - y_j \in \{-1, 0, +1\}$. We maximize a conditional information potential over minibatch pairs:

$$V_{\mathrm{cond}}(\theta) = \frac{\sum_{i,j} k(z_i, z_j)\, G_h\big(Q(e_i) - Q(e_j)\big)}{\sum_{i,j} k(z_i, z_j)}$$

Here $z$ is a frozen embedding of the input and $k$ is a similarity kernel on it. $G_h$ is a Gaussian kernel of bandwidth $h$ on errors, and $Q$ quantizes errors to the three-peak codebook. Pairs range over weak–weak and weak–gold combinations, with gold errors held fixed at $e^\ast$. The training loss is cross-entropy on gold labels minus $\alpha \log V_{\mathrm{cond}}$. Maximizing $V_{\mathrm{cond}}$ concentrates the error distribution within each neighborhood, which is the same as minimizing a conditional Rényi entropy. The weak–gold pairs decide which peak a neighborhood's errors concentrate on. The weak–weak pairs carry that decision further, to neighborhoods with no gold examples nearby. Because only nearby pairs contribute and the codebook has three entries, the usual quadratic cost of MEE drops to roughly linear in batch size times neighborhood size.

Research Program

The first stage builds the objective and checks its mechanism on a controlled problem. We use binary CIFAR-10 subtasks with synthetic labelers whose errors are either random or confined to chosen embedding regions. Here we can verify the predicted behavior directly. Global restricted MEE and per-sample robust losses should copy region-confined errors once the student is accurate. The conditional objective should learn to disagree in exactly those regions, with the disagreement mass at ±1 tracking the true per-region error rate. We also map how sensitive the method is to bandwidth $h$ and to the embedding kernel width. Within-neighborhood noise is close to input-independent, so we expect a fixed small $h$ to suffice and the bandwidth paradox to matter less. This is a hypothesis to test, not a guarantee: the cited fast rates are for unimodal noise, and ours is three-peaked.

The second stage is the main test, on the three medical benchmarks of the crossover paper. We use the same VLM labelers (BiomedCLIP, QuiltNet, PubMedCLIP), the same gold-set grid from 10 labels upward, and weak pools of up to 50,000 examples. NIH-CXR's multi-label structure fits the per-coordinate codebook directly. ISIC 2019's eight classes test the one-vs-rest extension that the restricted-MEE paper left open. The embedding $z$ comes from the labeler's own image encoder, so no extra pretrained resource is introduced.

The third stage moves from fixed weak labels to iterative self-training. We plug the objective into FixMatch's pseudo-label branch on CIFAR-10 with 40 labels and CIFAR-100 with 400 labels, using the harit7/PabLO-SSL codebase. We scale the unlabeled pool on SVHN past the roughly 80,000-example plateau. Here the "labeler" is the model itself, so the gold anchors are the only outside signal. This tests whether coupling to gold error structure helps against confirmation bias, or whether it feeds the shared-information trap described in the Gibbs analysis. That analysis predicts harm when pseudo-labels depend too heavily on the labeled set. Our coupling ties neighborhoods to the labeler's errors on gold examples rather than to the student's own predictions. Even so, with tiny gold sets it could overfit those few anchors, so we will report this explicitly.

Baselines

The naive reference points are gold-only training with the same pretrained backbone, and plain binary cross-entropy on gold plus all weak labels. Together they define the crossover itself, the gold-set size at which the second stops beating the first. Their reported behavior gives the target: crossovers near 100, 20–50 and 250–500 gold labels on PCAM, ISIC 2019 and NIH-CXR, and post-crossover drops of up to 0.10 AUC (From Theory to Decision Rule). We count the crossover as moved only if weak labels still improve AUC over gold-only at least twice the reported crossover gold count on at least two of the three datasets.

The established robust losses are Generalized Cross Entropy and Co-teaching, run exactly as in the crossover study. We also run a stronger Co-teaching variant whose forget rate is set to the labeler error rate estimated from gold. That gives it the same gold information our method uses. To count as beating these, our method must move the crossover where they do not; reducing the post-crossover drop alone is not enough.

The strongest simple competitor is gold-guided neighborhood noise correction. For each weak example, we estimate from the nearest gold examples in the same embedding how likely the labeler is to be wrong there. We then reweight or flip the weak label accordingly, either by kernel-weighted voting or by forward correction with a per-neighborhood confusion matrix, and train with cross-entropy. This baseline uses the same gold labels, embedding and kernel, so it isolates whether the entropy objective adds anything beyond "estimate a local noise map from gold." We will give it the same tuning budget as our method, selected by the same small-gold cross-validation protocol.

The ablations are global restricted MEE (no conditioning) and our objective with the weak–weak term removed. The first tests whether conditioning matters. The second reduces the method to soft nearest-gold matching in error space and tests whether propagation beyond gold neighborhoods matters.

In the semi-supervised track, the baselines are FixMatch and PabLO. FixMatch reports 88.61% accuracy on CIFAR-10 with 40 labels (FixMatch). PabLO reports 67.44% top-1 on CIFAR-100 with 400 labels, against 30.38% for its FixMatch run (Rethinking Confidence Scores and Thresholds in Pseudolabeling-based SSL). PabLO also reports FixMatch at about 57% on CIFAR-10 with 40 labels, which conflicts with FixMatch's own 88.61%. We will rerun both under one codebase and report our own numbers. PabLO is the bar on CIFAR-100. Because it holds out labeled data for calibration, we run it both with its held-out split and with that split folded back into training, so that every method sees the same total labels. For SVHN, no reported FixMatch or PabLO result was found on Lacuna, so our reruns define the reference.

Why not just…?

Why not just fine-tune a pretrained model on the gold labels and skip cheap labels entirely? On natural images this is a serious contender. ImageNet transfer reached 12.09% CIFAR-10 error versus 13.13% for the best semi-supervised method in Realistic Evaluation of Deep Semi-Supervised Learning Algorithms. For that reason our gold-only baseline is always a fine-tuned pretrained encoder, not a model trained from scratch. Past the crossover, gold-only fine-tuning is by definition the better option. Our claim is only meaningful if we beat it there, and the crossover-shift criterion is exactly that comparison.

Why not just stop adding weak labels at the crossover, using the published decision rule? The rule in From Theory to Decision Rule is 76% accurate across 129 configurations, and even a perfect version only reaches the maximum of gold-only and naive weak training. We report that maximum, chosen with hindsight, as an oracle switch. Beating the oracle switch past the crossover is the cleanest evidence that weak labels carry signal that standard training cannot extract.

Why not just use a robust loss like Generalized Cross Entropy or Co-teaching? These are already shown not to move the crossover. Our strengthened Co-teaching, with a gold-estimated forget rate, tests whether that failure was only a tuning issue. If tuned Co-teaching moves the crossover as far as our method, the entropy formulation adds nothing and we will say so.

Why not just estimate where the labeler is wrong from nearby gold examples and correct those labels? This is the real threat, and it may work as well. The case for our objective is that with 10–100 gold labels most weak examples have no informative gold neighbor. The weak–weak entropy term can carry corrections through chains of similar unlabeled examples, while nearest-gold voting cannot. The deciding experiment compares our method with the weak–weak term removed, the noise-correction baseline, and the full method, as the gold count shrinks. If the full method does not pull ahead specifically at the smallest gold sizes and in regions far from any gold example, the contribution reduces to local noise correction, and we will report it that way.

Decisive Checks

Three results settle the proposal. The first is the crossover curve: AUC against gold-set size for every method on PCAM, ISIC 2019 and NIH-CXR, with crossovers estimated by bootstrap over five seeds. The second is a stratified error analysis. We split test examples by distance to the nearest gold example and by whether the labeler is wrong on them, which shows whether gains come from corrections that propagated past gold neighborhoods, as the mechanism predicts. The third is the unlabeled-scaling curve on SVHN and CIFAR-100 at fixed labels, which shows whether the plateau moves later or merely flattens. A moved crossover, together with gains concentrated in labeler-error regions far from gold, would establish conditional error entropy as a way to extract signal from cheap labels after standard training saturates. A tie with gold-guided noise correction would show that the MEE machinery is unnecessary for this problem, which the field currently does not know.