Abstract
Training modern deep learning models is computationally intensive, often requiring massive datasets and significant time. Dataset condensation aims to mitigate this by creating small, synthetic versions of large datasets. However, traditional methods suffer from slow bi-level optimization and poor scalability. This paper introduces Distribution Matching (DM), a method that synthesizes data by matching feature distributions in randomly sampled embedding spaces. DM achieves a 45x speedup over previous state-of-the-art methods and is the first to successfully scale dataset condensation to the full ImageNet-1K benchmark.
Introduction
As deep learning models grow in complexity, the datasets required to train them have expanded to millions of images. Dataset condensation (or distillation) seeks to compress these massive datasets into a handful of highly informative synthetic images. Ideally, a model trained on just 10 or 50 synthetic images per class should achieve performance comparable to one trained on the full dataset.
Existing condensation techniques typically rely on matching the gradients of a training model. This creates a "bi-level optimization" problem where the synthetic data must be updated within an outer loop that simulates a full training trajectory. This process is computationally expensive, memory-intensive, and requires second-order derivatives, limiting its application to small-scale datasets like MNIST or CIFAR-10.
Methodology: Distribution Matching
To overcome the computational bottleneck, the authors propose Distribution Matching (DM). Instead of tracking a specific training trajectory, DM focuses on the underlying data distribution.
The DM Framework
- Embedding Spaces: The method uses a family of deep neural networks as "embedding functions." Crucially, these networks are randomly initialized and never trained. At each iteration, a new random initialization is sampled, providing a diverse set of perspectives on the data.
- MMD Loss: The core objective is to minimize the Maximum Mean Discrepancy (MMD) between the real training images and the synthetic images. By ensuring the mean feature vector of the synthetic set matches that of the real set across many different random embedding spaces, the synthetic data captures the essential characteristics of the original distribution.
- Single-Level Optimization: Because the network weights are fixed during each step, the complex bi-level optimization is replaced with a straightforward, single-level gradient descent on the pixels of the synthetic images. This decoupling makes the synthesis process significantly faster and more robust.
Key Results and Scalability
The authors evaluated DM across standard benchmarks, including MNIST, CIFAR-10/100, and TinyImageNet. Most notably, they successfully applied DM to ImageNet-1K (resized to 64x64), a feat previously unattainable for condensation methods.
Performance and Speed
DM demonstrates a massive efficiency gain. On CIFAR-10, synthesizing 500 images takes less than 20 minutes with DM, whereas the previous state-of-the-art method, DSA, requires 15 hours. This represents a 45x speedup.
Despite the speed increase, DM often yields better accuracy. For example, on CIFAR-100 with 50 images per class, DM achieved 43.6% accuracy, surpassing DSA's 42.8%. On the full ImageNet-1K, a model trained on a condensed set representing only 3.8% of the original data achieved 11.4% accuracy, proving the method's viability at scale.
Generalization and Applications
Because DM matches distributions across random embeddings rather than a specific gradient path, the resulting synthetic data generalizes better to different architectures. Data condensed using a simple ConvNet remains effective when training a ResNet. Furthermore, these condensed sets prove useful in practical scenarios like Neural Architecture Search (NAS) and Continual Learning, where they help prevent "catastrophic forgetting" by providing a compact, high-quality memory of previous tasks.
Conclusion
Distribution Matching represents a paradigm shift in dataset condensation. By replacing complex gradient matching with a simpler distribution-based objective, the authors have unlocked the ability to condense large-scale datasets efficiently. While a performance gap remains between condensed sets and full datasets, DM provides a scalable foundation for future research in data-efficient deep learning.
Summary updated