Dataset distillation compresses a large training set into a handful of synthetic images per class, so that a network trained on those few images performs nearly as well as one trained on everything. The method works well at very small budgets and then stops working. When the budget grows past about 100 to 200 images per class, distilled sets lose their edge and can do worse than picking real images at random. The DC-BENCH benchmark calls this the "sprint specialist" phenomenon. The likely cause is a familiar failure from global optimization called premature convergence: a population of search points collapses onto the same basin and stops exploring. In distillation the population is the set of synthetic images. Each image is optimized against the same objective, so each one drifts toward the class prototype, and the hundredth image mostly repeats the first.
We propose to synthesize images in blocks and to "deflate" each new block. Deflation means penalizing a new block for occupying feature directions that earlier blocks already cover, measured against how much the real data occupies those directions. The idea is borrowed from neural similarity deflation in recurrent networks, where each new network is pushed out of the subspace explained by its predecessors. The resulting cost grows linearly with the image budget. It needs no Batch Normalization statistics, so it runs on Vision Transformers. It also gives an adaptive signal for when to open a new block and when to stop. The central bet is that this turns added image budget into added information, and so lets distilled sets keep scaling where they currently flatten out.
Core Claim
The diminishing returns at high budget come from a specific mechanism. Synthetic images are optimized symmetrically and independently, so the marginal image is drawn toward directions that are already represented. A sequential, residual-targeted objective should restore near-additive gains, and it should do so better than symmetric pairwise repulsion or hand-set staggering. Concretely, we predict that the accuracy-versus-budget curve of a deflated distilled set stays above both random selection and diversity-selected real images (k-center) at 200 images per class, a regime where DC-BENCH reports that current distillation loses its advantage.
Relation to Prior Work
Several recent papers already treat the collapse of synthetic images as a diversity problem, and the proposal has to be different from all of them.
Two papers attack it symmetrically. DiRe adds a pairwise cosine-diversity loss to decoupled pipelines such as SRe2L. It raises CIFAR-10 accuracy at 100 images per class (IPC) from 57.5% to 71.2%, but the pairwise cost is quadratic in IPC, and its authors themselves point to subspace or determinantal alternatives. DEDA maximizes variance and decorrelates last-layer features, and it degrades at IPC=100 when the diversity and discriminability terms fight each other. Both push images apart from each other. Neither asks whether a new image adds directions the real data actually contains. Our penalty is asymmetric and sequential: earlier blocks are frozen, and later blocks are scored only against the real-data residual.
Two papers attack it through the optimization process. DWA perturbs the teacher's weights and separates the mean and variance terms of the Batch Normalization matching loss. It reaches 37.9% on ImageNet-1K at IPC=10, but it is tied to Batch Normalization and cannot run natively on LayerNorm transformers. DELT staggers when subsets of images begin optimization, with a manually set schedule. Our method also synthesizes in stages, but what each stage is pushed toward and how long it runs come from measured residual variance rather than a fixed calendar.
E²D and DELT both report that prolonged optimization erodes diversity. That finding motivates the proposal, and it also constrains it: deflation must not simply be a slower route to early stopping, and an ablation below tests exactly that. SATM and LCMat flatten the student's loss landscape but contain no term acting between synthetic images. We see them as complementary rather than competing.
The mechanism itself comes from outside distillation. Iterative Neural Similarity Deflation escapes low-dimensional attractors by penalizing what previously trained networks already predict, and we apply the same idea to data instead of networks. Gradient Starvation explains why redundancy hurts: easy features absorb the gradient and leave the subtle ones unlearned. DEDA hypothesizes that low-diversity distilled sets cause exactly this starvation in students, and our residual targeting attacks it directly. The repulsion schemes from swarm optimization, such as DPSO, constrain our design. Their documented weakness is a persistent drag on easy landscapes, which is why our penalty is scaled relative to the real data's occupancy of each direction rather than applied unconditionally.
Research Program
Work proceeds class by class. Let $\phi_l(x)$ be the teacher's features for image $x$ at layer $l$. For a ViT these are pooled tokens; for a CNN they are pooled channels. Let $\mu_{c,l}$ be the real class mean at that layer. After $k$ blocks have been synthesized, we run a randomized SVD on the centered features of those blocks to get an orthonormal basis $U_{k,l}$ of their top $r$ directions. We compute the same projection for the real class data, which gives the real share of variance in that span, $\rho_{k,l}$. Each image $x$ in block $k+1$ then minimizes
$$\mathcal{L}_{k+1}(x) = \mathcal{L}_{\text{base}}(x) + \lambda \sum_{l} \left[ \frac{\lVert U_{k,l}^{\top}(\phi_l(x)-\mu_{c,l})\rVert^2}{\lVert \phi_l(x)-\mu_{c,l}\rVert^2} - \rho_{k,l} \right]_+$$
In this objective, $\mathcal{L}_{\text{base}}$ is the host pipeline's synthesis loss: teacher cross-entropy plus statistic matching for SRe2L-style methods, or feature-mean matching for ViTs. The bracket $[\cdot]_+$ keeps only positive values, so a new image is penalized only when it puts more of its variation into already-covered directions than real images do. A deflated block therefore cannot be pushed into directions the data does not support, and that is the safeguard against the drag that repulsion shows on easy landscapes.
We apply the penalty at intermediate layers as well as the last one. DEDA reports that final-layer features collapse into tight clusters, which would make last-layer deflation nearly empty.
The cost per block is one projection per image, plus one randomized SVD over an $r$-dimensional basis. The total cost is therefore linear in IPC, compared with quadratic for DiRe.
The schedule is driven by feedback, replacing DELT's fixed partition. A new block opens when the residual real variance $1-\rho_{k,l}$ explained by the current block stops falling. Blocks keep being added until the budget is spent.
We will build the method as a plug-in for the SRe2L, DWA and DELT codebases, using the DWA repository. We will measure Coverage and Vendi Score as DiRe does. We will also report a cumulative explained-variance curve of synthetic versus real features as the primary diagnostic of collapse.
Baselines
The naive reference points are uniform random selection of real images and k-center selection of real images, evaluated under DC-BENCH's evaluation protocol. DC-BENCH reports random selection at 25.67% on CIFAR-10 at IPC=10. We found no reported result on Lacuna for random or k-center selection at IPC=200 on any benchmark here, and these are exactly the numbers the scaling claim depends on, so we will run them ourselves. Beating them at IPC=200 is the core test.
The strongest existing method varies by regime. On CIFAR-10 at IPC=100, SRe2L+DiRe reports 71.2% against 57.5% for plain SRe2L (DiRe). On ImageNet-1K, DWA reports 37.9% at IPC=10 and 59.2% at IPC=100 (DWA). DELT reports 66.1% at IPC=50 with a ResNet-101 student (DELT). We will re-run each of these with the same teacher, the same student training recipe, the same soft-label budget, and GPU-hours matched to within 10%. Each method receives the same hyperparameter search budget. That means 12 trials for $\lambda$ and $r$ in our method, and 12 trials for DiRe's loss weights and for DWA's step count and perturbation size.
The criteria for beating these baselines are as follows. We must be at least 1.5 points above the best of them at IPC=100 and 200, with non-overlapping three-seed intervals, while staying within one point at IPC=10. For ViT-based synthesis, where DWA cannot run natively and we found no reported result on Lacuna for any diversity-regularized method, the comparison is against SRe2L-style feature matching with and without DiRe.
Three ablations isolate the key ingredient. The first keeps our block schedule with $\lambda=0$; this is pure staggering, essentially DELT with adaptive timing. The second replaces the deflation term with DiRe's pairwise cosine loss inside the same block structure. The third removes the real-occupancy offset $\rho$ to give unconditional deflation. If the first ablation matches the full method, the gain comes from the schedule rather than from deflation, and the core claim fails.
Why not just…?
Why not just select diverse real images with k-center or random sampling at high budgets? This is the most serious threat. DC-BENCH shows random selection catching up with distillation at 200+ IPC, so at large budgets a well-chosen real subset may be all one needs. The evidence that synthesis still matters comes from low budgets: AdaCore reaches only 5.43% against 7.65% for LCMat-S on CIFAR-100 at a 0.5% fraction (LCMat). Nothing in the file settles the question at IPC=200. Our scaling experiment settles it. If k-center matches deflated synthesis at IPC=200, we will report that distillation's value is confined to low budgets, and we will stop claiming otherwise.
Why not just stop optimization early, as E²D and DELT suggest? Early stopping keeps diversity by keeping images close to their real initializations. It does not direct later images toward uncovered directions. The $\lambda=0$ ablation, which uses identical staging and iteration counts, tests this directly. A deflation benefit that disappears under matched iteration budgets would mean early stopping is enough.
Why not just add DiRe's pairwise repulsion? DiRe works, as its 71.2% shows. But it is symmetric: it spreads images apart regardless of where real data actually has variance, and its cost is quadratic in IPC. Our second ablation swaps it into the same pipeline. We expect the two to tie at IPC=50 and deflation to win at IPC≥100 on both accuracy and synthesis time. If they tie everywhere, deflation offers only a speed advantage, and we will say so.
Why not just augment the distilled set more heavily at training time? DC-BENCH reports that evaluation-time augmentation adds more than 10 points but does not close the cross-architecture gap. DiRe's 57.5% baseline already includes standard augmentation. All runs here use identical augmentation, so any remaining gap reflects what the images themselves contain.
Decisive Experiments
The first experiment is the scaling curve. It covers IPC of 10, 50, 100 and 200 on CIFAR-100 and ImageNet-1K, plus IPC=50 on Tiny ImageNet, with a ResNet-18 teacher and ResNet-18 and ResNet-101 students. Every method, including random and k-center selection, appears on the same plot. Success means the deflated curve keeps a positive, non-shrinking gap over k-center selection through IPC=200.
The second experiment tests portability. We distill with a ViT teacher on ImageNet-1K at IPC=50 and evaluate on both ViT and ResNet students. This probes the Batch Normalization dependence of the leading diversity method. It also probes the cross-architecture weakness that SATM improves only slightly, from 46.4% to 47.7% when transferring from ConvNet to ResNet18.
The third experiment is mechanistic. We plot the cumulative explained-variance curves of synthetic and real features, and we measure student gradient starvation: the per-sample gradient norm after the easy feature has been fit, following Gradient Starvation. This checks whether accuracy gains coincide with closing the gap between the synthetic and real variance spectra. If accuracy improves while the spectra stay mismatched, the gain does not come from deflation, and we would be misreading our own method.
References
- DC-BENCH: Dataset Condensation Benchmark
- DiRe: Diversity-promoting Regularization for Dataset Condensation
- Diversity-Enhanced Distribution Alignment for Dataset Distillation
- Diversity-Driven Synthesis: Enhancing Dataset Distillation through Directed Weight Adjustment
- DELT: A Simple Diversity-driven EarlyLate Training for Dataset Distillation
- Accelerating Large-Scale Dataset Distillation via Exploration-Exploitation Optimization
- Enhancing Generalization via Sharpness-Aware Trajectory Matching for Dataset Condensation
- Loss-Curvature Matching for Dataset Selection and Condensation
- Discovering alternative solutions beyond the simplicity bias in recurrent neural networks
- Gradient Starvation: A Learning Proclivity in Neural Networks
- Divergence-Guided Particle Swarm Optimization