Dataset distillation aims to condense a massive dataset into a tiny pool of synthetic samples. When a downstream neural network trains on this synthetic fraction, it should achieve performance close to a model trained on the full dataset. As noted by Ilia Sucholutsky et al. in Soft-Label Dataset Distillation and Text Dataset Distillation, shrinking data down to a handful of highly informative, soft-labeled samples can drastically accelerate downstream training.
However, the process of creating these synthetic samples introduces a severe computational paradox. To distill a dataset, standard methods solve a bi-level optimization problem. The inner loop trains a model on the synthetic data, while the outer loop updates the synthetic data's pixels to minimize the model's loss on the real dataset.
This formulation requires backpropagating through the entire training trajectory of the inner model. If the inner model trains for $T$ steps, computing the gradient of the outer objective requires storing the intermediate states of all $T$ steps. This unrolled gradient tracking creates an immense memory bottleneck. As a result, early dataset distillation methods were strictly confined to low-resolution toy datasets like MNIST or CIFAR-10.
The High-Resolution Scaling Wall
When moving from $32 \times 32$ images to industry-standard resolutions like ImageNet ($224 \times 224$ or $512 \times 512$), the bi-level optimization framework completely breaks down. As Ruonan Yu et al. point out in Dataset Distillation: A Comprehensive Review, tracking parameter trajectories across high-dimensional spaces introduces prohibitive memory demands that quickly exceed standard hardware capacities.
The scaling bottleneck stems from two compounding factors:
- Dimensionality Expansion: Optimizing every raw pixel directly means the parameter search space expands linearly with image resolution and the number of images per class (IPC).
- Optimization Complexity: High-resolution images contain intricate structural variations and fine-grained details. Forcing a model to learn these details through unrolled trajectories requires longer inner-loop training sequences ($T$), exponentially increasing the unrolled gradient memory.
To overcome this wall, researchers have shifted away from direct, pixel-level unrolled optimization toward three core architectural paradigms: latent space distillation, distribution matching, and decoupled multi-stage pipelines.
Paradigm 1: Distilling in Latent Spaces
Direct pixel optimization often yields synthetic images that lack visual coherence or training utility because the optimization gets lost in high-dimensional noise. Why optimize raw pixels when we can optimize a compact, highly semantic representation instead?
To resolve this, Jianfu Zhang et al. proposed Dataset Distillation in Latent Space. Instead of treating raw pixels as learnable parameters, their framework, LatentDD, performs the distillation within the compact latent space of a pretrained autoencoder. This slashes the spatial dimensionality of the optimization space while allowing the synthetic data to capture denser, more complex features.
[Real High-Res Images] ---> [Pretrained Encoder] ---> [Compact Latent Space]
|
(Distillation occurs here)
|
[Synthetic High-Res Images] <-- [Pretrained Decoder] <-------+
Similarly, Jianyang Gu et al. introduced Efficient Dataset Distillation via Minimax Diffusion, which shifts the distillation target from individual images to the parameter space of a generative model. By optimizing a latent diffusion model to synthesize training data via a minimax criterion, the authors bypass sample-wise pixel optimization entirely, accelerating distillation by over 20x.
Paradigm 2: Bypassing the Inner Loop via Distribution Matching
Another way to eliminate the unrolled gradient bottleneck is to replace the bi-level training loop with a single-level proxy objective. Instead of tracking how an inner model's weights change over time, we can force the synthetic data to share the same statistical properties as the real data.
Bo Zhao et al. pioneered this approach with Dataset Condensation with Distribution Matching. Their method embeds both real and synthetic images using randomly initialized neural networks and minimizes the distance between their feature distributions. Because it requires no inner-loop training or trajectory tracking, it collapses a complex temporal optimization into a static geometric alignment problem, scaling dataset condensation to ImageNet-1K for the first time.
To capture more than just the mean of the distribution, subsequent works refined this geometry alignment. Ganlong Zhao et al. introduced Improved Distribution Matching for Dataset Condensation, which incorporates a memory-efficient model queue to stabilize the embedding space. Furthermore, Wei Wei et al. developed Dataset condensation with latent quantile matching to align fine-grained distribution shapes rather than just average statistics, ensuring that synthetic datasets retain critical outlier and sub-class information.
Paradigm 3: Decoupling and Constant-Memory Pipelines
For methods that still rely on matching training trajectories to capture deep sequential knowledge, researchers have found clever ways to restructure the optimization math to avoid the $O(T)$ memory explosion.
Justin Cui et al. developed TESLA (Scaling Up Dataset Distillation to ImageNet-1K with Constant Memory), a framework that mathematically reduces the memory complexity of trajectory matching from linear to constant. This allowed them to run high-fidelity trajectory matching on a single GPU for large-scale datasets.
Taking decoupling to its logical extreme, Zeyuan Yin et al. proposed Squeeze, Recover and Relabel: Dataset Condensation at ImageNet Scale From A New Perspective. Their framework, SRe$^2$L, completely breaks the bi-level optimization into three independent, single-level stages:
- Squeeze: Pretrain a standard model on the massive real dataset to isolate its knowledge.
- Recover: Synthesize high-resolution images by optimizing them to match the batch normalization statistics of the pretrained model.
- Relabel: Use the pretrained model to generate soft labels for the recovered images.
By completely isolating image synthesis from model training, the authors achieved a 52x speedup and set new accuracy benchmarks on full-scale ImageNet, proving that scaling high-resolution distillation does not inherently require expensive, joint optimization loops.