Dataset distillation (DD) aims to condense massive datasets into a handful of synthetic "proxy" images. When a model trains on these few images, it ideally achieves performance comparable to training on the full original dataset. However, traditional distillation methods suffer from a significant practical bottleneck: rigidity.
As noted by Sajedi et al., most frameworks optimize raw pixels for a specific "images-per-class" (IPC) budget. If a practitioner decides to change that budget—perhaps to fit a smaller mobile cache or to utilize extra GPU memory—they must typically restart the entire, computationally expensive distillation process from scratch. Recent research has moved toward making distilled data flexible, allowing it to scale or shrink on-the-fly without redundant re-optimization.
The Problem of Static Rigidity
In standard dataset condensation, the synthetic images are optimized as a collective set. This creates two primary issues:
- Re-distillation Overhead: Khaki et al. and Wang 0036 et al. highlight that any change to the distillation ratio requires a complete re-computation. This lack of incremental scaling makes it difficult to experiment with different data budgets.
- Subset Degradation: You cannot simply take a subset of a distilled dataset. As Xiao et al. demonstrate, if you condense a dataset to 50 IPC and then try to use only 10 of those images, the performance drops significantly compared to a set specifically distilled for 10 IPC. The information is spread across the specific fixed-size set, making individual samples less representative in isolation.
From Static Pixels to Generative Models
One solution to the rigidity problem is to stop distilling into static images and start distilling into generative models. Instead of saving 10 discrete images, the "essence" of the dataset is stored within the weights of a generator, such as a GAN.
Li et al. propose optimizing a generator to balance global semantic structures with fine-grained local details. This approach allows for arbitrary sampling; if you need more data, you simply sample more from the latent space. Similarly, Wang 0036 et al. introduce DiM (Distilling into Generative Model), which overcomes the fixed-ratio constraint by using a generative model to synthesize informative training data on demand.
To make this even faster, Liu and Wang developed MGDD (Meta Generator for Dataset Distillation). By meta-learning a generator on large-scale data like ImageNet, they enable the system to rapidly adapt to new datasets, providing a 22× speedup and the ability to produce synthetic sets of any size without re-optimization.
Nested Condensation and Pruning
Another direction focuses on making the synthetic sets themselves more robust to resizing. This is often achieved through "slimmable" or "nested" structures.
Multisize Dataset Condensation (MDC)
To solve the subset degradation problem, He et al. and Xiao et al. introduce an adaptive subset loss. This ensures that every prefix of the synthetic dataset (e.g., the first 5 images, the first 10, etc.) is itself a highly representative proxy for the original data. This "nested" property allows a single synthetic dataset to be truncated to any size to fit fluctuating hardware resources on edge devices.
Post-Condensation Pruning
If a large synthetic dataset already exists, can we resize it without the original data? Ye et al. propose Slimmable Dataset Condensation, which enables "on-the-fly" resizing of distilled sets to meet storage or bandwidth constraints by using significance-aware parameterization.
Taking this further, He et al. introduce YOCO (You Only Condense Once). The authors argue for a decoupled workflow: condense a high-density dataset once, then use a brief training phase to assign importance scores to samples. This allows practitioners to prune the condensed set into smaller, optimal subsets for various memory constraints without ever returning to the expensive full-scale optimization loop.
Efficiency through Attention
Beyond size flexibility, the underlying mechanisms of distillation are becoming more efficient. Sajedi et al. show that matching the spatial attention maps of real and synthetic images (DataDAM) is significantly faster than traditional gradient matching. Khaki et al. extend this with ATOM (Attention Mixer), which captures both spatial and channel-wise context. These attention-based methods improve the quality of each synthetic sample, ensuring that even when the dataset is scaled down, the remaining images retain the most critical features for model training.
Key Foundational Work
You Only Condense Once: Two Rules for Pruning Condensed Datasets (2023) — contributed 3 concept elements to this direction.
Slimmable Dataset Condensation (2023) — contributed 3 concept elements to this direction.
ATOM: Attention Mixer for Efficient Dataset Distillation (2024) — contributed 2 concept elements to this direction.