Class-conditional image generators are trained on images paired with class labels, but most image collections are unlabeled. The obvious fix is to label the extra images with a classifier and train on those guesses, called pseudo-labels. Some of those guesses are wrong, and a generator told "this is a goldfinch" while looking at a sparrow learns to draw the wrong thing. This proposal starts from a structural fact about next-scale generators, models that first draw a tiny version of the image and then repeatedly draw a sharper version conditioned on the blurrier ones. In these models the class label probably matters a great deal at the coarse steps, where layout and object identity are decided, and very little at the fine steps, where the model mostly adds texture that follows from the coarser image. If that holds, a wrong label does most of its damage at a few coarse scales, and the fine scales can learn from every unlabeled image without trusting any label.
The concrete proposal is to measure, for each scale of the generator, how much information the class label carries about that scale's tokens. That measurement then sets how confident a pseudo-label must be before it is allowed to condition that scale. Low-confidence images still train the fine scales, with the label replaced by a "no label" token, so no data is thrown away. The approach is new in that it routes a noisy label to parts of the generative hierarchy instead of deciding once, per image, whether to trust it. It also comes with a falsifiable prediction: the gains should track the measured per-scale label information, and they should shrink on fine-grained data where class identity lives in texture.
Background and the gap
Next-scale autoregression was introduced by Visual Autoregressive Modeling (VAR). VAR encodes an image with a multi-scale vector-quantized autoencoder, which turns an image into grids of discrete codes at resolutions from 1×1 up to 16×16. A transformer then predicts each whole grid from all coarser ones and reaches ImageNet 256×256 FID 1.73. M-VAR separates attention within a scale from conditioning across scales and reports FID 1.78 with better speed. Both papers assume clean, complete class labels. Both also inherit the open problem that an error committed at a coarse scale traps every finer scale into refining a wrong structure. We take the architecture from these papers and ask what happens when the conditioning signal itself is unreliable.
On the noise side, FreeMatch sets confidence thresholds from moving averages of the model's global and per-class confidence. Combating Noise via region uncertainty softens uncertain detection targets. Auto-annotation quality prediction uses an 18-model ensemble to mask unreliable pixels. All three make one trust decision per sample, region, or pixel. None asks which part of the learner should receive the label.
The closest conceptual precedent is Scale Decoupled Distillation. It pools a teacher's predictions at 1×1, 2×2, and 4×4 spatial grids and reweights the regions where local and global predictions disagree, which shows that label information is not uniform across scale. That work is discriminative and spatial, whereas we work across the generative frequency hierarchy.
A confirmation bias from hierarchical approximate inference motivates the core hypothesis. It shows formally that in sequential hierarchical inference, an ambiguous top-level belief biases how lower-level evidence is integrated. By analogy, a wrong class token at the coarse scales should corrupt everything below it. Fine scales conditioned on a correct coarse image should not need the label at all.
The two papers that already connect generation and pseudo-labels leave this question open. DiffRect uses latent diffusion to clean noisy segmentation pseudo-labels for a discriminative segmenter. It improves the labels and does not study how a generator consumes them. MED-GAN is the most direct prior on semi-supervised class-conditional synthesis. It fixes a discriminator that relies on background shortcuts, reporting CUB-200 FID 16.90 and CIFAR-10 Inception Score 8.47. It has no scale hierarchy to exploit. We use it as a cross-family reference point, not as the method we modify.
Core Claim
In a next-scale generator trained with pseudo-labels, the harm from label noise is concentrated at the scales where the label is informative. Gating label use per scale, by pseudo-label confidence weighted by measured label informativeness, should beat any per-sample trust rule. The comparison is at matched image quality and matched compute, and the advantage should be largest on class consistency.
Research Program
The program has three parts.
Part 1: measure where the label lives. On the labeled subset, we train a small VAR with class-conditioning dropout, the standard trick that lets one model act as both conditional and unconditional. For each scale $k$ we then estimate
$$\Delta_k = \mathbb{E}\left[-\log p(r_k \mid r_{<k}) + \log p(r_k \mid r_{<k}, c)\right].$$
Here $r_k$ is the token grid at scale $k$, $r_{<k}$ are all coarser grids, $c$ is the true class, and the expectation runs over held-out labeled images. $\Delta_k$ is a plug-in estimate of the conditional mutual information between the class and scale $k$, measured in nats per image.
Part 2: a controlled noise study. We inject symmetric and classifier-confusion-structured label noise at known rates into fully labeled data. For each scale we record how much $\Delta_k$, per-scale token likelihood, and final sample quality degrade. This directly tests the primacy-bias hypothesis.
Part 3: the gated training rule. An unlabeled image receives pseudo-label $\hat{y}$ with confidence $q$ from a FreeMatch classifier. At scale $k$, the model sees $\hat{y}$ with probability $q^{\alpha_k}$ and the null token otherwise, where $\alpha_k = \Delta_k / \bar{\Delta}$ and $\bar{\Delta}$ is the mean of $\Delta_k$ over scales. Scales where the label carries much information therefore demand high confidence. Scales where it carries little accept nearly any label, and the model learns to ignore labels there anyway.
This requires the class embedding to enter per scale rather than only through a single start token. VAR already conditions through class-dependent layer normalization, so we make that normalization depend on a per-scale class-or-null token. The change adds negligible parameters. At sampling time, the same per-scale structure permits per-scale classifier-free guidance, which we also evaluate.
Optional extension. Because $p(r_{\le k} \mid c)$ for small $k$ is a cheap generative classifier, we can compare its class ranking with the discriminative pseudo-label. The generator then acts as a second opinion that is not confident in the same ways as the classifier. This targets the known failure in which thresholds miss confidently wrong labels. It is reported separately so the main claim does not depend on it.
Baselines
The naive reference is a conditional VAR trained on the labeled subset only. Its unconditional variant is trained on all images, so it measures what ignoring unlabeled data costs.
The strongest existing recipe is per-sample confidence filtering with FreeMatch pseudo-labels. Images above the self-adaptive threshold condition every scale, and the rest are dropped. This is the direct transfer of the best thresholding method in the file, and it is what a practitioner would do today. A stronger variant keeps the rejected images with the null token at every scale.
The ablation that isolates our ingredient is uniform confidence dropout. Every scale sees $\hat{y}$ with probability $q$, which is the same total label budget and differs only in not distinguishing scales. We also run a random-profile control: the same per-scale probabilities with the $\alpha_k$ values shuffled across scales. This tests whether the measured profile matters rather than merely having some profile.
The upper reference is the same small VAR trained with all true labels. VAR's reported full-scale ImageNet FID of 1.73 (VAR) is only a ceiling for context, since we train smaller models. No reported result was found on Lacuna for semi-supervised next-scale generation, so every comparison is rerun by us.
Fairness conditions. All runs share the same tokenizer, the same FreeMatch classifier and pseudo-labels, the same depth-16 model, the same number of training steps, and the same guidance-scale sweep.
What counts as beating a baseline. Gating must beat uniform dropout and threshold-filtering on class-consistency accuracy at equal or better FID. Class-consistency accuracy is measured by a held-out classifier trained on real labeled data, applied to samples generated for requested classes.
On CUB-200 and CIFAR-10 we additionally report MED-GAN's published FID 16.90 and IS 8.47 (MED-GAN). These are cross-family references only, since the model families differ.
Why not just…?
Why not just threshold pseudo-labels globally and train VAR normally? This is the default. FreeMatch already shows fixed thresholds drop hard classes, with reported error reductions of 13.59% on STL-10 and 5.78% on CIFAR-10 relative to FlexMatch. Any per-sample threshold still discards fine-scale training signal from rejected images. It also cannot stop a confidently wrong label from reaching the coarse scales. The threshold baseline with null-token reuse is designed to close as much of this gap as possible. If it matches gating, our claim fails.
Why not just use confidence-weighted label dropout everywhere? This is the real threat. It is simple, it keeps all data, and it may already capture most of the benefit. Our gate differs from it only in the per-scale profile, which is why uniform dropout and the shuffled-profile control are our key ablations. If gating's advantage over uniform dropout does not grow as injected noise increases in Part 2, we will conclude the hierarchy adds nothing and say so.
Why not just clean the labels first, for example with DiffRect-style generative rectification? DiffRect rectifies dense segmentation masks. For image-level class labels, cleaning reduces to a better classifier, and residual errors remain whatever it does. Gating is complementary: it limits the damage of whatever errors survive. We test this by running gating on top of the best available pseudo-labels, including the extension's generative second opinion.
Proposed Experiments and Decisive Checks
We run on ImageNet 256×256 with 10% and 100-per-class labels, using the 100-per-class setting that FreeMatch evaluated. We also run on CUB-200 with 10% labels and on CIFAR-10 at 4,000 labels. Metrics are FID, Inception Score, precision and recall, class-consistency accuracy, and per-scale token negative log-likelihood.
The first decisive check comes from the injected-noise study. It should show that corrupting labels at coarse scales alone reproduces most of the class-consistency loss from corrupting all scales. Corrupting fine scales alone should cost almost nothing.
The second check is that gating beats uniform dropout, with a margin that widens as noise increases.
The third check is a built-in falsification test on CUB-200, where species differ in plumage texture. There, $\Delta_k$ should be flatter across scales and gating's advantage should shrink. If the gate helps on CUB as much as on ImageNet despite a flat $\Delta_k$ profile, the mechanism is not what we claim. If the $\Delta_k$ profile predicts where gains appear across all three datasets, it becomes a diagnostic tool for any hierarchical generator trained on imperfect conditioning.
References
- Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction
- M-VAR: Decoupled Scale-wise Autoregressive Modeling for High-Quality Image Generation
- FreeMatch: Self-adaptive Thresholding for Semi-supervised Learning
- Combating Noise: Semi-supervised Learning by Region Uncertainty Quantification
- Auto-Annotation Quality Prediction for Semi-Supervised Learning with Ensembles
- Scale Decoupled Distillation
- A confirmation bias in perceptual decision-making due to hierarchical approximate inference
- DiffRect: Latent Diffusion Label Rectification for Semi-supervised Medical Image Segmentation
- Mask-Embedded Discriminator With Region-Based Semantic Regularization for Semi-Supervised Class-Conditional Image Synthesis