Dataset Condensation with Distribution Matching

Bo Zhao, Hakan Bilen

Abstract

Training modern deep learning models is computationally intensive, often requiring massive datasets and significant time. Dataset condensation aims to mitigate this by creating small, synthetic versions of large datasets. However, traditional methods suffer from slow bi-level optimization and poor scalability. This paper introduces Distribution Matching (DM), a method that synthesizes data by matching feature distributions in randomly sampled embedding spaces. DM achieves a 45x speedup over previous state-of-the-art methods and is the first to successfully scale dataset condensation to the full ImageNet-1K benchmark.

Introduction

As deep learning models grow in complexity, the datasets required to train them have expanded to millions of images. Dataset condensation (or distillation) seeks to compress these massive datasets into a handful of highly informative synthetic images. Ideally, a model trained on just 10 or 50 synthetic images per class should achieve performance comparable to one trained on the full dataset.

Existing condensation techniques typically rely on matching the gradients of a training model. This creates a "bi-level optimization" problem where the synthetic data must be updated within an outer loop that simulates a full training trajectory. This process is computationally expensive, memory-intensive, and requires second-order derivatives, limiting its application to small-scale datasets like MNIST or CIFAR-10.

Methodology: Distribution Matching

To overcome the computational bottleneck, the authors propose Distribution Matching (DM). Instead of tracking a specific training trajectory, DM focuses on the underlying data distribution.

Conceptual overview of Distribution Matching: Real data is mapped to multiple embedding spaces where the mean feature vectors of real and synthetic data are matched.

The DM Framework

  1. Embedding Spaces: The method uses a family of deep neural networks as "embedding functions." Crucially, these networks are randomly initialized and never trained. At each iteration, a new random initialization is sampled, providing a diverse set of perspectives on the data.
  2. MMD Loss: The core objective is to minimize the Maximum Mean Discrepancy (MMD) between the real training images and the synthetic images. By ensuring the mean feature vector of the synthetic set matches that of the real set across many different random embedding spaces, the synthetic data captures the essential characteristics of the original distribution.
  3. Single-Level Optimization: Because the network weights are fixed during each step, the complex bi-level optimization is replaced with a straightforward, single-level gradient descent on the pixels of the synthetic images. This decoupling makes the synthesis process significantly faster and more robust.

Key Results and Scalability

The authors evaluated DM across standard benchmarks, including MNIST, CIFAR-10/100, and TinyImageNet. Most notably, they successfully applied DM to ImageNet-1K (resized to 64x64), a feat previously unattainable for condensation methods.

Performance and Speed

DM demonstrates a massive efficiency gain. On CIFAR-10, synthesizing 500 images takes less than 20 minutes with DM, whereas the previous state-of-the-art method, DSA, requires 15 hours. This represents a 45x speedup.

Comparison of training time and accuracy: DM (red) reaches higher testing accuracy significantly faster than the previous state-of-the-art, DSA (blue).

Despite the speed increase, DM often yields better accuracy. For example, on CIFAR-100 with 50 images per class, DM achieved 43.6% accuracy, surpassing DSA's 42.8%. On the full ImageNet-1K, a model trained on a condensed set representing only 3.8% of the original data achieved 11.4% accuracy, proving the method's viability at scale.

Generalization and Applications

Because DM matches distributions across random embeddings rather than a specific gradient path, the resulting synthetic data generalizes better to different architectures. Data condensed using a simple ConvNet remains effective when training a ResNet. Furthermore, these condensed sets prove useful in practical scenarios like Neural Architecture Search (NAS) and Continual Learning, where they help prevent "catastrophic forgetting" by providing a compact, high-quality memory of previous tasks.

Conclusion

Distribution Matching represents a paradigm shift in dataset condensation. By replacing complex gradient matching with a simpler distribution-based objective, the authors have unlocked the ability to condense large-scale datasets efficiently. While a performance gap remains between condensed sets and full datasets, DM provides a scalable foundation for future research in data-efficient deep learning.

Summary updated

Abstract

Computational cost of training state-of-the-art deep models in many learning problems is rapidly increasing due to more sophisticated models and larger datasets. A recent promising direction for reducing training cost is dataset condensation that aims to replace the original large training set with a significantly smaller learned synthetic set while preserving the original information. While training deep models on the small set of condensed images can be extremely fast, their synthesis remains computationally expensive due to the complex bi-level optimization and secondorder derivative computation. In this work, we propose a simple yet effective method that synthesizes condensed images by matching feature distributions of the synthetic and original training images in many sampled embedding spaces. Our method significantly reduces the synthesis cost while achieving comparable or better performance. Thanks to its efficiency, we apply our method to more realistic and larger datasets with sophisticated neural architectures and obtain a significant performance boost <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> . We also show promising practical benefits of our method in continual learning and neural architecture search.
BibTeX
@inproceedings{zhao2023dataset,
  author = {Bo Zhao and Hakan Bilen},
  title = {{Dataset Condensation with Distribution Matching}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  doi = {10.1109/wacv56688.2023.00645},
}
Figure