Training a small image classifier by imitating a large one (knowledge distillation) no longer requires real data. A pretrained image generator can synthesize the training set, and the large "teacher" classifier labels each image with its full probability distribution over classes. The cost has moved, though. A diffusion model is a generator that turns random noise into an image over dozens of denoising steps, and running one to produce more than a million images is the dominant expense of the pipeline. Much of that output also teaches the student little. On some images the student already agrees with the teacher. On others the teacher's label is a sharp, overconfident spike that carries no information about how classes relate. Selective distillation methods have learned to discard such examples. They discard them only after the full generation cost has been paid.
This note proposes moving the selection decision inside the generation process. Partway through denoising, we decode the generator's current estimate of the final image, ask the teacher and the current student how they would label it, and stop sampling any image whose predicted training value is low. The compute saved goes to new candidates. The claim is narrow and testable. Distillation value measured on an early, blurry estimate of an image predicts its value once finished, well enough that aborting low-value trajectories gives a better student per GPU-hour than generating everything, generating everything more cheaply, or filtering after generation.
Core Claim
The closest existing work falls into three groups, and each leaves the same opening.
The first group generates blindly at scale. Is Synthetic Data From Diffusion Models Ready for Knowledge Distillation? (DM-KD) synthesizes images from class labels with Stable Diffusion or DiT and distills a black-box teacher through a divergence between probability distributions. Its results motivate this proposal in two ways. First, accuracy stops improving at roughly 2 million samples, so generating more images is not the answer. Second, low-fidelity images made with low guidance produce softer, more informative teacher distributions than polished ones, so image quality is not what we should be buying. DM-KD never asks which individual images are worth generating.
The second group filters after generation. DISCO over-generates counterfactual text with GPT-3 and keeps only candidates that a DeBERTa-v2 teacher flips confidently. The filter works, since 75,000 filtered examples beat pools of more than 1,000,000 noisy ones, but all compute spent on rejected candidates is lost. ADAM selects training pairs by teacher confidence, and AdaSPEC keeps only the tokens a small student can learn. Both are training-time gates applied to data that already exists.
The third group steers the prompt toward the student's weaknesses. Towards Active Synthetic Data Generation for Finetuning Language Models picks the seed problems the student fails on and asks GPT-4o for variants, reaching 1.3x to 2x data efficiency over random generation. MT-PATCHER generates translation examples targeted at specific student errors. Both make their decision before generation begins, at the level of the prompt, and then commit to whatever comes out.
None of these makes a per-instance, student-aware decision during generation. That is the specific gap identified in the file: no method rejects partial generations early using bounds from a teacher-student gate. Diffusion sampling suits this kind of mid-generation decision better than text decoding because cost is spread evenly over the denoising steps. Each step also produces a full-size estimate of the final image that a classifier can score.
The selection signal we use is a learnability gap. For a candidate image $x$, the per-image utility is
$$ u(x) = \mathrm{KL}\big(p_T(x)\,\|\,p_S(x)\big)\cdot \mathbb{1}\big[H(p_T(x)) \ge \tau\big], $$
where $p_T$ and $p_S$ are the teacher's and the current student's class distributions, $\mathrm{KL}$ is the Kullback-Leibler divergence, $H$ is entropy, and $\tau$ is a minimum teacher entropy. The divergence term follows the student-centric logic of active generation: spend effort where the student still differs from the teacher. The entropy floor follows DM-KD's and ADAM's observation that near-one-hot teacher outputs carry little transferable structure.
A trajectory is checked at a fraction $k/T$ of its $T$ denoising steps, where $\hat{x}_0^{(k)}$ denotes the generator's estimate of the final image at step $k$. It continues only if $u(\hat{x}_0^{(k)})$ exceeds a running quantile $q$ of recent scores. Every aborted trajectory saves roughly a fraction $1-k/T$ of its sampling cost.
The cost of the gate is one latent decode plus one forward pass each from the teacher and student. We will measure this cost and charge it to the method rather than assume it is negligible.
Research Program
The work has three stages, and each can stop the project.
The first stage is a measurement study that comes before any training. On a fixed pool of DiT or Stable Diffusion trajectories, we record $u$ for each image at every checkpoint $k/T \in \{0.1, 0.2, \ldots, 1.0\}$, at several student training stages. We then compute the rank correlation between early and final utility, as a function of $k$ and of the guidance scale. Low guidance is the regime DM-KD recommends, and it may also make early estimates more or less informative, which is worth knowing in its own right.
The second stage is the closed-loop method. Generation proceeds in rounds. The student is refreshed between rounds, so utility follows its changing errors. Within each round, trajectories are checked at the chosen $k$ and aborted below quantile $q$. Surviving images are fully denoised and enter the distillation set with a standard distillation loss. We use no gate at training time, so that any gain can be attributed to what was generated rather than to how it was weighted.
The third stage asks when the method helps. We vary the teacher between ResNet18 and ResNet50 to test DM-KD's finding that the weaker teacher distills better. Our hypothesis is that the entropy and divergence gate removes much of the stronger teacher's overconfidence on generation artifacts and narrows that gap. We also run an optional label-consistency term, which aborts trajectories where the teacher's top class contradicts the class the generator was conditioned on. This term acts as a weak, free substitute for the ground-truth verifiers that OPDVR and oracle distillation require. It is reported only as an ablation, because DM-KD's results suggest that "ugly," ambiguous images are often the useful ones.
Development runs on 100-class ImageNet subsets at reduced resolution. Headline results are on full ImageNet-1K with DM-KD's setup: a ResNet18 student, a class-conditional diffusion generator, and guidance scale $s=1$.
Baselines
The naive reference point is DM-KD itself. It uniformly generates class-conditional images at $s=1$ with full denoising and distills with KL divergence. The file reports 68.25% top-1 for a ResNet18 student on ImageNet-1K with 1.28M synthetic images (DM-KD). We will reproduce this number in our own code before comparing anything against it. For every method we will plot accuracy against total GPU-hours, counting generation, gate evaluation, and training.
The strongest cheap competitor is uniform generation with fewer denoising steps, matched to our method's total compute. DM-KD's own finding that low-fidelity images distill well makes this a serious threat. A shorter sampler also saves compute on every image, not only on rejected ones. We will tune its step count over the same grid we tune $k$ on. To beat it, our method must reach higher accuracy at equal GPU-hours on ImageNet-1K. The success threshold is a margin larger than the spread across three seeds.
The post-hoc filter baseline generates every candidate fully, scores it with the same $u$, and keeps the same fraction of survivors, in the style of DISCO adapted to images. It isolates whether early scoring loses selection quality relative to final scoring. We expect slightly better selection from it and a much higher cost per kept image. Our method wins if it matches this baseline's accuracy at a fraction of the compute.
The strongest prompt-level competitor adapts active synthetic generation to class-conditional diffusion. Between rounds it reallocates the generation budget across classes in proportion to the mean student-teacher KL divergence on each class, then generates every image in full. This tests whether choosing which classes to generate captures most of the benefit without any per-instance decision. It gets the same rounds, the same student refresh schedule, and the same tuning budget.
Two ablations isolate the key ingredient. The first, random early abort, stops trajectories at the same step and the same rate as our method but chooses them at random. This separates the value of the gate from simple reallocation of compute to more samples. The second, divergence-only gating, removes the entropy floor and tests whether the DM-KD and ADAM argument about sharp teacher distributions matters for selection.
Why not just…?
Why not just sample every image with fewer steps? This is the most serious alternative, and it may win. DM-KD shows that the student does not need polished images, so a short sampler may already capture most of what full denoising adds. Our method differs because it spends full denoising on images that pass the gate and almost nothing on the rest. Short sampling spends a reduced amount on every image without regard to its value. The compute-matched comparison in the Baselines section settles the question directly. If short sampling matches us, we will report that the overhead problem in diffusion distillation is solved by cheaper samplers rather than by selection, and stop.
Why not just steer generation toward the classes the student gets wrong? The class-level reallocation baseline answers this. Active generation works at the prompt level, and in text its gains came from choosing seeds (Towards Active Synthetic Data Generation). For class-conditional images, much of the variation in utility may lie within a class: the pose, clutter, or artifact that makes the teacher soft or overconfident. We will measure the within-class and between-class variance of $u$ in stage one. If most of the variance lies between classes, the per-instance gate adds little, and the class-level baseline should be preferred. The two can also be combined, and we will report that combination.
Why not just filter after generating, or gate the loss at training time? Post-hoc filtering (DISCO) and training-time gating (ADAM, AdaSPEC) improve the student but do nothing about generation cost, which is the quantity this proposal targets. They are the right comparison for selection quality, and the post-hoc filter baseline measures exactly how much selection quality early scoring gives up.
Why not use an adversarial data-free generator instead of a diffusion model? Data-free generators trained to maximize teacher-student disagreement exploit the teacher's errors on unrealistic inputs. That weakness is documented by CSD, which reports 60.88% versus 56.46% for CMI on CIFAR-100, and by HiDFD. A pretrained diffusion model keeps images close to natural ones. Our gate seeks disagreement only among those images, and the entropy floor rules out the teacher's overconfident corners. No reported result was found on Lacuna for adversarial data-free distillation at ImageNet-1K scale under our protocol, so we will not compare against it head to head.
Proposed Experiments
The decisive first check is the stage-one correlation study. If the rank correlation between utility at $k/T = 0.2$ and final utility is weak at every guidance scale and every student stage, early abort cannot work, and the project ends with a short negative result on the limits of predicting sample value mid-generation.
If the correlation holds, the main experiment is on ImageNet-1K. For DM-KD-uniform, short-sampling, post-hoc filter, class-level active, random abort, and our method, we report ResNet18 top-1 accuracy against total GPU-hours, with three seeds each. We also report the GPU-hours each method needs to reach the reproduced 68.25%. A second experiment varies the teacher between ResNet18 and ResNet50 and tests whether the gate shrinks the weak-teacher advantage. A third reports the scaling curve of each method out to and beyond the roughly 2M-sample plateau DM-KD observed. It tests whether selection moves that plateau or only reaches it more cheaply.
The project succeeds if our method beats compute-matched short sampling and class-level reallocation by more than seed variance. It also needs to recover at least most of the post-hoc filter's accuracy at substantially lower cost. Any other outcome will be reported as evidence about which of the three cheaper strategies is enough.
References
- Is Synthetic Data From Diffusion Models Ready for Knowledge Distillation?
- DISCO: Distilling Counterfactuals with Large Language Models
- ADAM: Dense Retrieval Distillation with Adaptive Dark Examples
- AdaSPEC: Selective Knowledge Distillation for Efficient Speculative Decoders
- Towards Active Synthetic Data Generation for Finetuning Language Models
- MT-PATCHER: Selective and Extendable Knowledge Distillation from Large Language Models for Machine Translation
- On-policy Distillation with Verifiable Reward
- Towards Oracle Knowledge Distillation with Neural Architecture Search
- Customizing Synthetic Data for Data-Free Student Learning
- Hybrid Data-Free Knowledge Distillation