Abstract
The computational cost of training state-of-the-art deep models is rapidly increasing due to more sophisticated models and larger datasets. A recent promising direction for reducing training cost is dataset condensation, which aims to replace the original large training set with a significantly smaller learned synthetic set while preserving the original information. While training deep models on the small set of condensed images can be extremely fast, their synthesis remains computationally expensive due to complex bi-level optimization and second-order derivative computation. In this work, we propose a simple yet effective method that synthesizes condensed images by matching feature distributions of the synthetic and original training images in many sampled embedding spaces. Our method significantly reduces the synthesis cost while achieving comparable or better performance. Thanks to its efficiency, we apply our method to more realistic and larger datasets with sophisticated neural architectures and obtain a significant performance boost. We also show promising practical benefits of our method in continual learning and neural architecture search.
Introduction
Modern deep learning requires massive datasets and weeks of computation, creating a significant barrier to entry and a large environmental footprint. Dataset condensation offers a solution by synthesizing a tiny version of a dataset (e.g., 10 or 50 images per class) that retains the performance of the full original data. However, existing methods often rely on "bi-level optimization"—a process that requires training a model within the loop of generating synthetic data. This makes the synthesis process itself prohibitively slow and prevents scaling to high-resolution datasets like ImageNet.
Methodology: Distribution Matching (DM)
We introduce Distribution Matching (DM), a direct approach that bypasses the expensive inner-loop optimization of previous methods. Instead of simulating the learning process, DM focuses on the data's statistical properties.
The core process involves three steps:
- Feature Embedding: Utilizing a family of deep neural networks with different random initializations as embedding functions.
- Maximum Mean Discrepancy (MMD): Calculating the distance between the distribution of real images and synthetic images within these embedding spaces.
- Optimization: Updating the pixels of the synthetic images to minimize this distance.
Crucially, DM treats randomly initialized networks as fixed "distance-preserving" lenses. This insight allows the method to match distributions without updating network weights during synthesis, leading to massive efficiency gains.
Experimental Results
We evaluated DM across several benchmarks, including MNIST, CIFAR-10/100, TinyImageNet, and the full ImageNet-1K.
Efficiency and Speed
DM achieves a 45x speedup over the previous state-of-the-art (DSA) on CIFAR-10. While previous methods required 15 hours to generate a condensed set, DM completes the task in under 20 minutes on a single GPU. This efficiency is visualized in the training accuracy vs. time comparison below.
Scaling to ImageNet
DM is the first condensation technique to successfully scale to the full ImageNet-1K dataset. Using only 50 images per class (a tiny fraction of the 1.3M original images), DM achieved 11.4% accuracy, significantly outperforming random selection and setting a new milestone for the field.
Key Insights
- Random Networks as Feature Extractors: A key discovery is that pre-trained networks are not required. Randomly initialized networks preserve distances in embedding spaces—keeping same-class samples close and different-class samples apart—providing a robust basis for distribution matching.
- Mathematical Robustness: We demonstrate that matching gradients of the last layer (a common prior technique) is essentially a weighted version of matching feature distributions. By weighing all features equally, DM becomes more robust and significantly faster.
- Downstream Utility: DM-generated sets are highly effective for Continual Learning, where they serve as a compact memory of old tasks, and Neural Architecture Search (NAS), where they act as high-correlation proxies for the full dataset at a fraction of the cost.
Conclusion
Distribution Matching represents a significant step forward in making dataset condensation practical and scalable. While currently most effective for classification, future work will explore extending this framework to complex tasks like object detection and semantic segmentation, as well as narrowing the performance gap between condensed and full datasets.
Summary updated