Training a small image classifier by imitating a large one, known as knowledge distillation, increasingly runs on synthetic images from diffusion models (generators that turn random noise into images by repeated denoising). Generation is the expensive part. Every image costs dozens of denoising steps, and the student then sees each image many times. This proposal asks how to get more teaching signal out of each generated image, so that fewer images need to be made. The idea combines two existing tools. A self-supervised Vision Transformer, a model trained without labels that nonetheless learns to attend to whole objects, shows where the informative regions of each synthetic image are. A small teacher then labels several such regions. For each region we store only the few most probable classes plus a single number for "everything else." Those compact labels are cached once and reused across all training epochs.
The surprising fact that makes this worth testing comes from Is Synthetic Data From Diffusion Models Ready for Knowledge Distillation? (DM-KD). That paper found that low-quality synthetic images, generated with weak guidance toward the class prompt, distill better than polished ones, because the teacher's predictions on them are less peaked and carry more information about how classes relate. Standard top-K truncation, which keeps only the K highest-probability classes and rescales them to sum to one, deletes exactly that spread. So the field's cheapest way to cut storage and bandwidth may be destroying the field's best-known source of signal on synthetic data. We propose a way to keep both, along with a test that says which is right.
Core Claim
We make three claims. Each comes with a check that could refute it.
First, the "ugly image advantage" in DM-KD lives mostly in how much probability the teacher puts outside its top few classes, not in the identity of the tail classes. If so, a truncated label that keeps the top-K classes plus one bucket holding all remaining mass will preserve the advantage. Ordinary renormalized top-K will erase it.
Second, a single generated image yields several useful training views if those views are chosen by an attention map and labeled separately. This raises distillation signal per generated image, so a student can match dense, full-image DM-KD while generating substantially fewer images. Because generation dominates cost, that is where the savings come from.
Third, the attention model and the label teacher should be different networks. This resolves a tension between two findings. DM-KD reports that stronger teachers overfit diffusion artifacts and distill worse (its "weak teacher paradox"). The salience literature, by contrast, finds that clean object-level attention appears only in strong self-supervised backbones such as DINOv2, as shown by Distilling Self-Supervised Vision Transformers for Weakly-Supervised Few-Shot Classification & Segmentation. A large DINOv2 can decide where to look without ever producing class labels, while a weak teacher supplies the labels.
The truncated label has a clean form. Let $p$ be the teacher's class distribution on a region and $S$ its top-$K$ classes. Let $\tau = 1 - \sum_{k \in S} p_k$ be the leftover mass. Let $q$ be the student's distribution, with $q_{\text{tail}} = 1 - \sum_{k \in S} q_k$. The loss is
$$\mathcal{L} = \sum_{k \in S} p_k \log \frac{p_k}{q_k} + \tau \log \frac{\tau}{q_{\text{tail}}}.$$
This is the exact KL divergence between the two distributions after all non-top-K classes are merged into one bin. By the data-processing inequality it is a lower bound on the full KL. It never inflates the boundary classes the way renormalization does, which Pre-training Distillation for Large Language Models and the truncation literature flag as a calibration hazard. We set the support size $K$ per region by the nucleus rule of that paper: keep classes until cumulative mass reaches $p$, capped at $K_{\max}$. Diffuse regions therefore get more stored classes and confident ones get fewer.
Research Program
The pipeline has four stages, and each reuses an existing method.
Generation follows DM-KD. A class-conditional diffusion model synthesizes ImageNet-1K images at low guidance. We also sweep guidance and step count, because the interaction between image quality and label format is the central scientific question.
Salience extraction follows FocusDC, which passes downsampled images through a ViT to get attention maps cheaply and reports a 4.2× FLOP reduction from downsampling. We read the attention of DINOv2's [CLS] token over image patches. That token summarizes the whole image, and its attention row is a known zero-cost foreground map, as in EffoVPR. FocusDC uses attention to select and stitch real patches into a fixed canvas of three foregrounds and one background. We do not stitch, and we do not fix the count. Each high-attention connected region, up to $R_{\max}$, becomes a crop, and one global view is always kept. Busy scenes therefore yield more views than single-object images, which addresses the fixed-budget limitation FocusDC itself acknowledges.
Labeling uses a deliberately weak teacher, ResNet18 following DM-KD. The teacher labels each crop. We store the crop box, a 4-byte augmentation seed in the style of TinyViT, and the adaptive top-K-plus-tail label.
Student training replays the cached views under the loss above. No teacher runs during training.
Overhead is accounted end to end in GPU-hours, split into generation, salience, labeling, and student training, plus cache bytes. The headline curve plots student accuracy against the number of generated images.
Baselines
The naive reference point is DM-KD itself. It runs dense full-image KL from a ResNet18 teacher on 1.28M low-guidance synthetic images, with the teacher queried online. The paper reports 68.25% ImageNet-1K top-1 for a ResNet18 student. We reproduce it on our own generated images with the same generator, guidance, and student schedule, so the comparison is not polluted by generator differences. Matching 68.25% with materially fewer generated images and lower total GPU-hours counts as beating it. Exceeding it at equal image count is a secondary win.
The strongest competitor is a cached random-crop relabeling scheme, which is the strongest because it already squeezes many views out of each image. It samples random resized crops, labels each with the same teacher, and caches top-K labels and seeds. This combines TinyViT's sparse offline cache with the region lookup table of FDBPL, which caches top-K teacher labels per random crop and reports 2.2× faster training. It is run with the same number of views per image, the same teacher, the same byte budget, and the same student epochs. The only difference is where the crops come from. To beat it, salience-chosen crops must reach a given accuracy with fewer generated images, beyond seed variance over three runs. I found no reported result on Lacuna for random-crop cached relabeling on diffusion-synthesized ImageNet-1K, so we establish this number ourselves.
The third baseline is a global truncated cache: one view per image with TinyViT-style top-K. TinyViT reports a 481 GB cache for ImageNet-21k at K=100.
Three ablations isolate the key ingredients. The first replaces the tail-bucket loss with renormalized top-K at identical K, which isolates the tail term. The second replaces adaptive K with a fixed K of equal average size. The third replaces DINOv2 attention with the ResNet18 teacher's own class activation map, which tests whether a separate strong salience model matters. Each ablation uses the same budget and tuning, with temperature and learning rate tuned per arm on a held-out slice of synthetic data.
Why not just…?
Why not just use random crops with cached labels? This is the real threat. Low-guidance class-conditional diffusion images are often roughly object-centered, so random crops may already land on the object most of the time. The evidence that attention-chosen regions beat unguided ones comes from settings unlike ours: FocusDC on real-image condensation, and the weakly supervised segmentation gains in the DINO distillation work. We therefore treat this as an open empirical question. The random-crop baseline at matched views and bytes settles it. If the two curves coincide, we will report that salience adds nothing for classification on synthetic data. We will then retain only the tail-bucket and multi-view findings, and test salience on denser multi-object generations, where its advantage should be largest.
Why not just store dense labels? On ImageNet-1K, dense fp16 labels cost about 2 KB per view. Eight views over a million images is a few tens of gigabytes, which is not prohibitive. We say so plainly. Storage savings matter mainly with large label spaces, such as TinyViT's 21,841 classes, and in a secondary experiment we have an ImageNet-21k-pretrained teacher label the same synthetic images. On ImageNet-1K, the dense arm serves as an accuracy ceiling. The more important question there is scientific: does truncation erase the ugly-image advantage? The guidance-by-label-format sweep answers it.
Why not just make generation cheaper directly, with fewer denoising steps or token pruning? Methods like CAT Pruning, which drops unchanging image tokens mid-denoising for up to 1.90× speedup on SD3, cut the cost per image. Our method cuts the number of images. The two compound rather than compete. We include a low-step generation arm to check that the gains from extra views persist on cheaper, uglier images.
Why not just use the teacher's own attention, or a strong teacher for everything? DM-KD's weak teacher paradox argues against a strong label teacher on synthetic data. Weak supervised CNNs' activation maps collapse onto discriminative parts, as the DINO distillation paper documents. The third ablation measures whether this matters in practice.
Proposed Experiments
The first decisive check is the label-format sweep. We cross guidance scale (DM-KD's low setting and two higher ones) with dense, renormalized top-K, and top-K-plus-tail labels, on full-image single views. If the tail-bucket arm tracks dense labels at every guidance level, while renormalized top-K loses its edge specifically at low guidance, the first claim holds. If renormalized top-K matches dense labels at low guidance, the advantage is not about tail mass and the claim fails.
The second decisive check is the images-versus-accuracy curve. We train at 160K, 320K, 640K, and 1.28M generated images, at a fixed student compute budget, for DM-KD, the global cache, random-crop relabeling, and our method. Success means our curve reaches 68.25% at a smaller image count than DM-KD and lies above the random-crop curve. We report total GPU-hours at the crossing point.
The third check is the teacher-separation ablation, together with a ResNet50 label teacher. This tests whether moving the strong model into a label-free salience role avoids the paradox. Secondary evaluations repeat the best configuration with a MobileNet-v2 student, and with the 21k-way teacher to measure cache bytes where truncation genuinely matters.
References
- Is Synthetic Data From Diffusion Models Ready for Knowledge Distillation?
- Distilling Self-Supervised Vision Transformers for Weakly-Supervised Few-Shot Classification & Segmentation
- Pre-training Distillation for Large Language Models: A Design Space Exploration
- FocusDC: Real-World Scene Infusion for Robust Dataset Condensation
- EffoVPR: Effective Foundation Model Utilization for Visual Place Recognition
- TinyViT: Fast Pretraining Distillation for Small Vision Transformers
- FDBPL: Faster distillation-based prompt learning for region-aware vision-language models adaptation
- CAT Pruning: Cluster-Aware Token Pruning For Text-to-Image Diffusion Models