On the Diversity and Realism of Distilled Dataset: An Efficient Dataset Distillation Paradigm

Peng Sun, Bei Shi, Daiwei Yu, Tao Lin

Introduction

Training modern deep learning models requires vast amounts of data, leading to significant computational and storage costs. Dataset Distillation (DD) has emerged as a promising solution, aiming to condense massive datasets into a small set of synthetic images that retain the essential information needed for training. However, traditional DD methods are often bottlenecked by slow optimization processes, poor visual realism, and limited generalization across different model architectures.

This paper introduces RDED (Realistic, Diverse, and Efficient Dataset Distillation), a new paradigm that addresses these challenges by replacing complex optimization with an efficient patch-selection strategy.

The RDED Paradigm

Most existing distillation methods rely on "bi-level optimization," an iterative process that is both memory-intensive and slow. RDED shifts this focus toward selection and reconstruction. Instead of synthesizing pixels from scratch, it identifies and extracts high-information patches from the original dataset and combines them to form distilled images.

Comparison of the traditional optimization-based paradigm versus the proposed RDED paradigm and its performance gains.

As shown in the figure above, the proposed paradigm avoids the expensive inner and outer loops of traditional methods, leading to a dramatic reduction in synthesis time while simultaneously improving accuracy.

Methodology: Selection over Synthesis

RDED operates in two primary stages grounded in $\mathcal{V}$-information theory:

  1. Patch Extraction: A pre-trained "observer" model evaluates patches from the original dataset. Patches that the model can easily and accurately classify are assigned high "realism scores" and selected for distillation.
  2. Information Reconstruction: To maintain diversity, RDED selects multiple high-scoring patches (typically four) from different images within the same class. These patches are stitched into a single grid-like distilled image.
  3. Soft Labeling: Rather than using hard labels, RDED employs the observer model to generate probability distributions (soft labels) for the distilled images. This captures the complex, multi-patch information contained within each synthetic frame.

Key Insights: Realism and Generalization

A central argument of the paper is that realism is the key to cross-architecture generalization. Traditional synthetic images often consist of abstract noise patterns that are "over-optimized" for a specific model architecture.

Synthetic images generated by optimization-based methods often appear as abstract noise patterns.

While these patterns (as seen above) might trigger a specific response in the training model, they often fail to transfer to different architectures. By using real image patches, RDED ensures that the distilled data contains natural features that any robust vision model can learn from.

Experimental Results

The researchers evaluated RDED on benchmarks ranging from CIFAR-10 to the full ImageNet-1K dataset. The results demonstrate a significant leap in performance:

  • Efficiency: RDED distilled ImageNet-1K (10 images per class) in just 7 minutes on a single GPU—over 50x faster than the previous state-of-the-art (SRe$^2$L), which required 6 hours.
  • Accuracy: On ImageNet-1K (IPC 10), RDED achieved 42.3% top-1 accuracy, nearly doubling the performance of previous methods (21.7%).
  • Generalization: Data distilled using a ResNet-18 backbone remained highly effective when used to train MobileNet-V2 and EfficientNet-B0, proving the robustness of the realistic patch-based approach.

Limitations and Future Work

Despite its strengths, RDED has limitations. At extremely low data budgets (e.g., 1 image per class), optimization-based methods can sometimes pack more information into a single set of pixels than a simple selection of patches. Additionally, the grid-like structure of the distilled images lacks natural visual cohesion, which may affect tasks beyond standard classification. Future research may focus on more seamless patch integration or the use of generative priors to enhance visual quality further.

Conclusion

RDED represents a significant shift in dataset distillation, proving that intelligent selection of real data can outperform complex synthesis. By prioritizing realism and diversity through an optimization-free approach, RDED makes large-scale dataset distillation accessible and effective for a wider range of applications and architectures.

Summary updated

Abstract

Contemporary machine learning requires training large neural networks on massive datasets and thus faces the challenges of high computational demands. Dataset distillation, as a recent emerging strategy, aims to compress real-world datasets for efficient training. However, this line of research currently struggle with large-scale and high-resolution datasets, hindering its practicality and feasibility. To this end, we re-examine the existing dataset distillation methods and identify three properties required for large-scale real-world applications, namely, realism, diversity, and efficiency. As a remedy, we propose RDED, a novel computationally-efficient yet effective data distillation paradigm, to enable both diversity and realism of the distilled data. Extensive empirical results over various neural architectures and datasets demonstrate the advancement of RDED: we can distill the full ImageNet-1K to a small dataset comprising 10 images per class within 7 minutes, achieving a notable 42% top-1 accuracy with ResNet-18 on a single RTX-4090 GPU (while the SOTA only achieves 21% but requires 6 hours).
BibTeX
@inproceedings{sun2024diversity,
  author = {Peng Sun and Bei Shi and Daiwei Yu and Tao Lin},
  title = {{On the Diversity and Realism of Distilled Dataset: An Efficient Dataset Distillation Paradigm}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.00897},
}
Figure