Introduction
Training modern deep learning models requires vast amounts of data, leading to significant computational and storage costs. Dataset Distillation (DD) has emerged as a promising solution, aiming to condense massive datasets into a small set of synthetic images that retain the essential information needed for training. However, traditional DD methods are often bottlenecked by slow optimization processes, poor visual realism, and limited generalization across different model architectures.
This paper introduces RDED (Realistic, Diverse, and Efficient Dataset Distillation), a new paradigm that addresses these challenges by replacing complex optimization with an efficient patch-selection strategy.
The RDED Paradigm
Most existing distillation methods rely on "bi-level optimization," an iterative process that is both memory-intensive and slow. RDED shifts this focus toward selection and reconstruction. Instead of synthesizing pixels from scratch, it identifies and extracts high-information patches from the original dataset and combines them to form distilled images.
As shown in the figure above, the proposed paradigm avoids the expensive inner and outer loops of traditional methods, leading to a dramatic reduction in synthesis time while simultaneously improving accuracy.
Methodology: Selection over Synthesis
RDED operates in two primary stages grounded in $\mathcal{V}$-information theory:
- Patch Extraction: A pre-trained "observer" model evaluates patches from the original dataset. Patches that the model can easily and accurately classify are assigned high "realism scores" and selected for distillation.
- Information Reconstruction: To maintain diversity, RDED selects multiple high-scoring patches (typically four) from different images within the same class. These patches are stitched into a single grid-like distilled image.
- Soft Labeling: Rather than using hard labels, RDED employs the observer model to generate probability distributions (soft labels) for the distilled images. This captures the complex, multi-patch information contained within each synthetic frame.
Key Insights: Realism and Generalization
A central argument of the paper is that realism is the key to cross-architecture generalization. Traditional synthetic images often consist of abstract noise patterns that are "over-optimized" for a specific model architecture.
While these patterns (as seen above) might trigger a specific response in the training model, they often fail to transfer to different architectures. By using real image patches, RDED ensures that the distilled data contains natural features that any robust vision model can learn from.
Experimental Results
The researchers evaluated RDED on benchmarks ranging from CIFAR-10 to the full ImageNet-1K dataset. The results demonstrate a significant leap in performance:
- Efficiency: RDED distilled ImageNet-1K (10 images per class) in just 7 minutes on a single GPU—over 50x faster than the previous state-of-the-art (SRe$^2$L), which required 6 hours.
- Accuracy: On ImageNet-1K (IPC 10), RDED achieved 42.3% top-1 accuracy, nearly doubling the performance of previous methods (21.7%).
- Generalization: Data distilled using a ResNet-18 backbone remained highly effective when used to train MobileNet-V2 and EfficientNet-B0, proving the robustness of the realistic patch-based approach.
Limitations and Future Work
Despite its strengths, RDED has limitations. At extremely low data budgets (e.g., 1 image per class), optimization-based methods can sometimes pack more information into a single set of pixels than a simple selection of patches. Additionally, the grid-like structure of the distilled images lacks natural visual cohesion, which may affect tasks beyond standard classification. Future research may focus on more seamless patch integration or the use of generative priors to enhance visual quality further.
Conclusion
RDED represents a significant shift in dataset distillation, proving that intelligent selection of real data can outperform complex synthesis. By prioritizing realism and diversity through an optimization-free approach, RDED makes large-scale dataset distillation accessible and effective for a wider range of applications and architectures.
Summary updated