Auditing Noisy Teachers with Latent Label Inference

When a large "teacher" model labels unlabeled data for a small "student" to learn from, the student inherits the teacher's mistakes. The most damaging mistakes are confident ones. Existing fixes estimate how noisy each teacher label is from the teacher's own confidence, and this cannot catch a teacher that is confidently wrong. We propose a different check. Treat the true label of each unlabeled example as a hidden variable, and ask whether that label can explain a second, independent description of the example: a compact embedding from a self-supervised vision encoder. A hypothesized label is supported when a small class-conditional generative model, conditioned on that label, reconstructs the embedding well. The teacher's prediction is modeled as a noisy observation of the hidden label. The student classifier is the model's label-inference network, and it is the only part kept at deployment.

This takes the core move of deep generative semi-supervised learning and applies it where it can finally scale. That move, from Semi-supervised Learning with Deep Generative Models, is to infer a missing label by summing over candidate labels and scoring how well each explains the data. Two changes make it tractable. First, the generative model reconstructs a low-dimensional semantic embedding rather than pixels, so it is cheap and focuses on class content instead of stroke width or lighting. Second, the sum over labels is restricted to the teacher's top few candidates. This restriction still yields a valid lower bound on the data likelihood.

Core Claim

Semi-supervised distillation methods model teacher noise as a function of the teacher's confidence. SLaM does this by mapping the gap between the teacher's top two scores to a probability of being correct, using isotonic regression fit on a clean validation split. Because that mapping is monotone in confidence, a high-margin error is by construction treated as reliable.

Our claim is that a class-conditional generative model over an independent embedding supplies an instance-level signal that confidence cannot. Suppose the teacher says "wolf" with high margin, but the embedding is far better reconstructed under "husky". The posterior over the hidden label then moves toward husky even though the teacher was confident. We expect the gain over confidence-based noise modeling to be concentrated on exactly these high-margin teacher errors, and we will measure it there directly.

The mechanism depends on one condition that we treat as a hypothesis to test rather than an assumption. The embedding's errors must be only partly correlated with the teacher's errors. If the embedding comes from the teacher's own penultimate layer, the check should become circular and the gain should vanish. That prediction is a built-in falsification test.

The Model

For an image $x$, let $e$ be its compressed embedding from a frozen self-supervised encoder, $t$ the teacher's soft prediction, $y$ the hidden true label, and $z$ a continuous variable absorbing within-class variation. The generative story is $p(y)\,p(z)\,p_\theta(e\mid y,z)\,p_\psi(t\mid y,x)$. The last factor is a noise model: a confusion distribution over teacher outputs, conditioned on the true label and on a low-dimensional descriptor of the input, so that noise can be instance-dependent.

The student $q_\phi(y\mid x)$ reads the raw image. A small encoder $q(z\mid e,y)$ handles the continuous variable. For an unlabeled image with candidate set $S(x)$ (the teacher's top-$k$ classes), the training objective is

$$\log p(e,t) \;\ge\; \sum_{y\in S(x)} q_\phi(y\mid x)\Big[\mathbb{E}_{q(z\mid e,y)}\log p_\theta(e\mid y,z) + \log p_\psi(t\mid y,x) + \log p(y) - \mathrm{KL}\big(q(z\mid e,y)\,\|\,p(z)\big)\Big] + \mathcal{H}\big(q_\phi(y\mid x)\big),$$

where $q_\phi$ is supported only on $S(x)$ and $\mathcal{H}$ is its entropy.

Restricting the support of $q_\phi$ is simply a smaller variational family. The bound therefore stays a valid lower bound on $\log p(e,t)$, and its looseness equals the posterior mass outside $S(x)$. This addresses the linear-in-class-count cost the file identifies for the original M2 model while introducing no gradient bias. The cost per unlabeled image is $k$ passes through a decoder on a vector of a few hundred dimensions, which is negligible next to the student's forward pass.

Labeled images contribute the same terms with $y$ observed, plus a cross-entropy term on $q_\phi$. The labeled images are what pin down the class-to-embedding association, which prevents the noise model from explaining everything away. We additionally regularize $p_\psi$ toward the identity confusion.

For the embedding, we borrow the angle-preserving compression head of Faithful Label-free Knowledge Distillation. That work showed that cosine geometry, not raw feature magnitude, carries the semantics of foundation-model features. We therefore use a von Mises–Fisher-style decoder on the unit sphere rather than a Gaussian with squared-error reconstruction.

Relation to Prior Work

Two papers supply the method. The generative semi-supervised model of Kingma et al. supplies label marginalization. It reconstructs pixels and sums over all classes, which the file flags as prohibitive at 1,000 classes; we change both. CosPress supplies the embedding geometry. It uses that geometry only for label-free feature matching and never infers discrete labels; we use it as the observation a latent label must explain.

Three papers motivate the proposal by showing that teacher noise is structured. SLaM models noise through margin plus a golden set. MIND models noise from foundation-model annotators as a mixture of transition matrices tied to orthogonal feature subspaces. Both are discriminative and neither checks a label against a generative account of the input. DIVA shows that separating class and nuisance latents lets unlabeled data help: ROC AUC on a held-out malaria patient rose from 71.9% to 79.4%. This supports our use of $z$ to keep within-class variation out of $y$.

Two papers set the open-set comparison and constrain what a variational approach should look like. Knowledge Distillation Meets Open-Set Semi-supervised Learning uses the teacher's classifier head as a critic of student features. Variational Information Distillation applies a variational bound to continuous activations with learned variance. Neither treats the label as latent. Their success does show that variational distillation objectives are trainable at CIFAR and ImageNet scale.

Research Program

The work has three parts.

First, we build the model above. We vary the decoder from a single Gaussian or vMF component per class, which is essentially a mixture model in embedding space, up to a conditional network with continuous $z$. We vary the noise model from global confusion to instance-conditioned confusion.

Second, we analyze where the gain comes from. We partition unlabeled examples by teacher margin and by teacher correctness, and measure how often each method recovers the true label on high-margin teacher errors. We also measure the looseness of the bound as a function of $k$ by comparing against full marginalization on CIFAR-100, where full marginalization is affordable.

Third, we test the independence hypothesis. We swap the embedding source among an independent self-supervised encoder, the teacher's own features, and a weaker self-supervised encoder, and check whether the gain tracks how much the embedding's errors overlap with the teacher's.

Baselines

The naive reference point is a student trained on the labeled subset alone, together with vanilla soft-label distillation from the teacher on all unlabeled data, as in Distilling the Knowledge in a Neural Network. On ImageNet with 5% labels, SLaM reports vanilla distillation at 41.67% top-1.

The strongest existing competitor is SLaM, since it targets exactly this setting of a noisy teacher labeling unlabeled data. It reports 48.1% top-1 on ImageNet with 5% labels. We run SLaM with the same teacher, student architecture, augmentation, epochs and hyperparameter search budget as our method. Because our method uses the labeled set, SLaM receives that same labeled set, held out from teacher training, as its golden calibration split. This way neither method gets more clean data. Beating it means a higher student top-1 averaged over at least three seeds with non-overlapping confidence intervals. It also means a clearly higher recovery rate on the high-margin teacher-error subset, which is the specific prediction. The file gives no reported SLaM number on CIFAR-100, so there we compare against our own reproduction only.

For the open-set experiment, the strongest baseline is SRD, reported at 76.24% for a ResNet8x4 student with CIFAR-100 labeled data and Tiny-ImageNet unlabeled data, against 72.50% for the plain student. We rerun it under our budget.

Four ablations isolate the key ingredient. The first removes the generative term, leaving a latent-label model whose only evidence is the teacher output through $p_\psi$; this is a discriminative noisy-channel model close in spirit to SLaM but without the golden set. The second replaces the independent embedding with the teacher's own features. The third replaces embedding reconstruction with pixel reconstruction at small $k$, which is the direct M2 descendant. The fourth sets $k$ to the full class count on CIFAR-100.

Why not just…?

Why not just use SLaM, which already models teacher noise? SLaM is the real competitor, and on examples where teacher confidence tracks correctness we expect it to be as good as our method. Its limitation is structural: a monotone map from margin to accuracy assigns high reliability to every high-margin label. MIND's evidence that foundation-model annotators make systematic, instance-dependent errors suggests such confident errors are common. The margin-stratified analysis settles this directly. If our recovery rate on high-margin errors is no better than SLaM's, the proposal's central claim fails.

Why not just run label propagation or a class-conditional mixture on the foundation embeddings, with no deep generative model? This is a genuine threat, and we have built it into the design instead of arguing around it. The simplest decoder in our program, one vMF component per class plus a global confusion matrix, is exactly this alternative written as a probabilistic model. If it matches the deep conditional decoder, the paper's conclusion becomes that the useful ingredient is latent label inference over an independent embedding with a teacher-noise channel, and that deep decoders are unnecessary. That would be a clean and useful result, not a failure.

Why not just do M2-style generative semi-supervised learning on pixels with a teacher-restricted label sum? That is the obvious bridge, but pixel likelihood is dominated by nuisance detail. Kingma et al.'s results are on MNIST, SVHN and NORB, and the file reports no ImageNet-scale success. Each pixel-decoder pass is also expensive. The pixel ablation at CIFAR-100 scale measures the accuracy and compute cost of this choice directly.

Why not just distill the teacher's features with angle-preserving matching and then train a classifier? CosPress-style transfer improves the student's representation. However, it never decides which label an unlabeled example has, so on the classification loss the student still copies the teacher's labels. We include CosPress feature matching followed by vanilla soft-label distillation as a baseline. If it matches our method, the benefit came from the embedding and not from label inference.

Proposed Experiments

The main experiment follows SLaM's protocol. The teacher is trained on a small labeled fraction of CIFAR-100 and of ImageNet-1K (5% for ImageNet, to match the reported 48.1% and 41.67%), then labels the remainder. The student is a smaller network. Embeddings come from a frozen self-supervised encoder of the kind CosPress distills from, compressed to a few hundred dimensions with the angle-preserving head.

We report student top-1 accuracy, recovery rate on high-margin teacher errors, calibration of the inferred noise model against true confusion on held-out labeled data, and wall-clock cost as a function of $k$.

The second experiment uses the open-set split of CIFAR-100 labeled data with Tiny-ImageNet unlabeled data. Here the generative term offers a natural extra behavior: an example whose embedding is poorly explained by every candidate class receives low likelihood. We test whether down-weighting such examples by their bound value beats SRD's 76.24%. We also report how well that bound value flags out-of-distribution unlabeled images, measured as AUROC.

The decisive checks are three. The gain over SLaM must concentrate on high-margin teacher errors. It must shrink toward zero when the embedding is taken from the teacher itself. At CIFAR-100, top-$k$ restriction with small $k$ must be within noise of full marginalization while costing a fraction of the compute. If the first two hold, the proposal has shown something that confidence-based distillation cannot do. If only the third holds, the contribution reduces to an efficiency trick for generative semi-supervised learning, and we would report it as such.