Abort Early: Multi-Scale Decoders for Cheap Rejection in Scale-Wise Synthetic Data Generation

Pipelines that train a small model on data produced by a large one pay most of their bill for samples that get thrown away. An image generator produces a candidate at full resolution, a teacher classifier inspects it, and a large share of candidates is discarded after every generation FLOP has already been spent. Scale-wise generators already build each image coarse to fine. A visual autoregressive (VAR) model, for example, predicts a 1×1 grid of discrete image tokens, then 2×2, and so on up to the final grid. Nearly all of the compute sits in the last few scales, because token counts grow quadratically with resolution. If we could judge a candidate reliably from its coarse scales, we could stop generating the ones that will fail before paying for their expensive fine scales.

The obstacle is that the coarse stages of these generators are not built to be looked at. We propose to fine-tune the generator's image decoder with a multi-scale reconstruction loss: a penalty applied to images decoded at every intermediate scale against correspondingly downsampled real images. The goal is that a partially generated sample decodes into a faithful low-resolution picture. A cheap teacher can then accept or reject it mid-generation. The central claim is that multi-scale reconstruction supervision is the specific ingredient that makes early rejection trustworthy, and that it cuts the GPU-hours per accepted synthetic sample substantially without lowering the accuracy of the student model trained on the data.

Core Claim and Positioning

The closest existing work is ProtoVAR, which distills ImageNet into small synthetic training sets. It replaces diffusion with VAR, steers coarse-scale token logits toward a bank of prototype features, and then screens finished candidates with a fast teacher. It reports roughly 70× faster generation than diffusion transformers. ProtoVAR is the motivating result: it shows that scale-wise generation is the right substrate for cheap synthesis. It still filters only after complete synthesis, and it uses the coarse scales only for steering, never for deciding. Our proposal keeps its generator and its filter but moves the filter's decision to intermediate scales.

The method comes from HieraTok. HieraTok trains a tokenizer with RGB reconstruction losses at every scale from 16×16 to 256×256, which improves reconstruction FID (a distance between real and reconstructed image distributions) from 1.47 to 1.07. It also speeds up downstream generator training by 1.38×. HieraTok never uses its coarse scales at sampling time and never evaluates synthetic data for training classifiers.

Several other papers explain why multi-scale losses should make coarse stages meaningful. simple diffusion uses a weighted sum of downsampled losses to push a pixel-space diffusion model toward large-scale structure. Im2Vec and PICASSO show that pyramid $L_2$ losses give usable coarse gradients when fine-scale error carries no signal. In all of these works the multi-scale loss is a training aid for the generator's own quality. We use it for a different purpose: to make intermediate states legible to an external judge.

The overhead side of the problem is well documented. DISCO routes every generated counterfactual through a teacher filter. UDAPDR needs six generation and distillation stages. GC-DDPM spends 62 hours on 8 GPUs to synthesize one handwriting training set. None of these reduces the cost of candidates that are rejected. Two other lines attack overhead differently. ManifoldGD builds multi-scale prototype coresets but still runs full diffusion for every sample. Simple and Fast Distillation makes each sample cheaper to generate, which is complementary to making rejected samples cheaper.

The expected savings can be written down directly. Let $c_k$ be the fraction of full generation cost incurred up to scale $k$, let $r$ be the fraction of candidates the filter ultimately rejects, and suppose early rejection at scale $k$ catches a fraction $q$ of those. The cost per accepted sample then falls by a factor of roughly

$$ \frac{1}{1 - r\,q\,(1 - c_k)}. $$

This follows from counting the FLOPs saved on caught rejects. In VAR's ten-scale schedule, the last two scales hold about 62% of the tokens and a larger share of the attention cost, so $1 - c_k \gtrsim 0.6$ when we exit before them. With the high rejection rates typical of large candidate pools, such as ProtoVAR's 1,000 candidates per class for IPC 10, the savings are large if $q$ is high. The whole project therefore hinges on measuring $q$. It must be measured together with its mirror image: the rate at which samples that the full-resolution filter would accept get falsely aborted.

Research Program

We start from a pretrained class-conditional VAR on ImageNet at 256×256 and the ProtoVAR pipeline. The first step checks the premise. We decode partial residual token sums at each intermediate scale with the stock decoder. We then measure two things: how faithful these decodes are to downsampled versions of the final image, and how often the teacher's verdict on them agrees with its verdict on the final image.

The second step fine-tunes only the decoder, leaving the generator frozen. The decoder is trained so that decoding the first $k$ scales reconstructs the image downsampled to scale $k$, with the loss summed over all $k$ in the manner of HieraTok. We add the constraint that final-scale reconstruction must not degrade, verified by reconstruction FID. Because the generator and its token distribution are unchanged, samples that survive are identical to the baseline's. Any change in student accuracy therefore comes from which samples are kept, not from what the samples look like.

The third step builds the early judge. This is the same fast teacher ProtoVAR uses, applied to the upsampled coarse decode and briefly adapted on pairs of coarse decodes and full-resolution verdicts. Its threshold is calibrated so the false-abort rate stays below a chosen budget.

As an extension, we replace the accept/reject criterion with a difficulty band. Samples whose coarse-scale teacher confidence falls in a middle range are kept, following the finding in Difficulty Controlled Diffusion that samples near class centroids carry little training signal. That paper conditions the generator on difficulty. Here difficulty is instead selected cheaply, at coarse scales, from an unmodified generator. This tests whether coarse scales predict difficulty, which is a stronger claim than predicting validity.

Baselines

The naive reference point is the ProtoVAR pipeline exactly as published: full-resolution generation of the whole candidate pool followed by target-guided teacher filtering, run on the same GPU model with the same candidate pool size. The file gives ProtoVAR's gains only relative to prior methods (+0.7% on ImageNet-1K IPC 10 with ResNet-18, +2.3% on ImageWoof). No absolute ProtoVAR accuracy was found on Lacuna for any of these settings, so we will reproduce ProtoVAR's numbers ourselves and report them alongside its paper's relative claims. Beating this baseline means matching its student accuracy within seed noise while using measurably fewer total GPU-hours, including the one-time decoder fine-tuning. A second, stronger form of beating it is higher accuracy at an equal GPU-hour budget, achieved by spending the savings on a larger candidate pool.

The strongest existing distillation competitor in the file is ManifoldGD. It reports 64.1% on ImageNette at IPC 10, 2.2 points above MGD, and +3.2% on ImageWoof at IPC 20. We run it with its released settings on the same student architectures and report both accuracy and wall-clock generation cost. Its generation cost is full multi-step diffusion, so our comparison is primarily on the accuracy-versus-GPU-hours curve.

The key ablation isolates the multi-scale reconstruction loss. It is the identical early-exit pipeline, with the same teacher, the same adaptation data and the same threshold calibration, but using the stock decoder. If this variant achieves the same catch rate $q$ at the same false-abort rate, then the multi-scale loss is not the operative ingredient, and the paper's central claim fails.

A second ablation uses a single-scale decoder fine-tune with equal compute. It is trained only on the final scale for the same number of steps, which separates "more decoder training" from "multi-scale decoder training."

Why not just…?

Why not just run the teacher on the stock decoder's coarse output? This is the ablation above, and it is a real threat. The stock decoder may already produce recognizable blurry images from partial sums. The file offers indirect evidence against it: HieraTok found that single-scale tokenizers yield poorly structured intermediate representations, and scale consistency had to be enforced by explicit multi-scale supervision. This is not proof for VAR's decoder specifically. The step-one measurement settles the question directly. If stock coarse decodes already give high teacher agreement, we drop the decoder fine-tune and report early exit as the contribution, with the multi-scale loss as a negative result.

Why not just train a probe on the coarse token maps to predict the final filter verdict, with no decoding at all? This is cheaper still and may work. We will train such a probe with the same labeled pairs and compare the curves of false-abort rate against catch rate. Our expectation is that a decoded image lets us reuse a strong pretrained teacher rather than learning a verdict predictor from limited pairs, and that it matters most for the difficulty-band criterion, where the teacher's calibrated confidence is the signal. If the probe matches the decoder route, the simpler method wins and we will say so.

Why not just make every sample cheaper with few-step distillation, or steer the generator so fewer samples are rejected? Few-step distillation in the style of Simple and Fast Distillation reduces the cost of every sample but does nothing about wasted samples, and the two approaches compose. Steering the generator, as ProtoVAR's prototype guidance and Difficulty Controlled Diffusion both do, lowers the rejection rate $r$, which shrinks our savings. We therefore report results on top of ProtoVAR's guidance rather than instead of it. If guided rejection rates turn out to be small, early exit buys little, and the savings formula makes that outcome visible before any student is trained.

Proposed Experiments

The first decisive check needs no student training. On a held-out set of ImageNet-1K class-conditional candidates, we plot the agreement between early and final teacher verdicts at each exit scale, for the stock decoder, the multi-scale decoder and the token probe. We report the catch rate $q$ at false-abort rates of 1% and 5%, plus the realized cost factor $c_k$ measured in wall-clock time on one RTX 4090, the hardware ProtoVAR used. The decoder must also keep its final-scale reconstruction FID within a small tolerance of the original.

The second check is downstream. We build distilled sets for ImageNet-1K at IPC 10, and for ImageNette and ImageWoof at IPC 10 and 20. We train ResNet-18 and the other architectures ProtoVAR reports, and plot student top-1 accuracy against total GPU-hours for full-filter ProtoVAR, early-exit ProtoVAR and ManifoldGD, with three seeds each. The proposal succeeds if early exit with the multi-scale decoder dominates full filtering on this curve and the stock-decoder ablation does not.

A final check tests whether the difficulty-band variant beats accept/reject filtering at matched cost on ImageWoof, where fine-grained boundary samples should matter most. This check determines whether coarse scales predict only whether a sample is valid, or also how useful it is for training.