Abstract
Dataset distillation aims to create a small synthetic dataset that mimics the training performance of a large-scale original dataset on neural networks. Existing approaches typically rely on heavy iterative optimization across thousands of neural network initializations, leading to significant computational overhead. This paper introduces a "few-shot" dataset distillation framework that utilizes a learned distillation space. By optimizing synthetic data within this specific space, the method achieves performance comparable to state-of-the-art techniques while requiring only a single or a few network evaluations. We propose a Translative Pre-Training approach where a translator network, pre-trained on large-scale data like ImageNet, maps images from an arbitrary neural space to the target distillation space. Extensive experiments demonstrate that this method achieves a ~15× acceleration with satisfactory generalization across various datasets and architectures.
The Challenge of Efficient Dataset Distillation
Dataset Distillation (DD) is a powerful technique for condensing massive datasets into a handful of representative synthetic images. Ideally, a model trained on these few images should achieve performance close to one trained on the full dataset. However, the current state-of-the-art (SOTA) methods are computationally expensive. They often require simulating training across thousands of random network initializations to update synthetic pixels, a process that can take days or weeks for a single dataset.
This paper addresses the need for few-shot dataset distillation, where high-quality synthetic data is produced with minimal network evaluations. The goal is to move away from grueling iterative loops toward a more efficient, transferable distillation process.
Methodology: Learning a Distillation Space
The core innovation is the concept of a Distillation Space. The authors hypothesize that synthetic data optimized for a dedicated, learned network space can approximate the performance of data optimized across a vast ensemble of random networks.
To implement this, the authors utilize Translative Pre-Training:
- The Translator ($G$): An auto-encoder network is trained to map images from a fixed "random" neural space into the optimized "distillation space."
- Pre-training and Adaptation: The translator is first pre-trained on a large dataset (ImageNet) to learn general features of effective distilled images. When applied to a new target dataset (e.g., CIFAR-10), it only requires a few adaptation steps to produce high-quality synthetic data.
- Decoupling Optimization: By using a translator, the method decouples the image distillation from the network space optimization. This eliminates the need to back-propagate through thousands of unrolled training steps, drastically reducing memory and time costs.
Key Results and Efficiency
The proposed method achieves a massive acceleration over existing benchmarks. While some SOTA methods like FRePo require extensive TFLOPs and time, the translative approach reaches comparable accuracy with significantly lower overhead.
- Performance: On CIFAR-10 (1 image per class), the method achieves 44.0% accuracy. When combined with down-sampling parameterization (Ours-FS-DS), accuracy jumps to 51.4%, outperforming many "heavy" distillation methods.
- Speed: The method is approximately 15× faster than FRePo. Distillation tasks that previously took days can now be completed in minutes or hours after the initial pre-training phase.
- Generalization: Synthetic data produced via this method generalizes well across different architectures, such as AlexNet, VGG, and ResNet, even when distilled using a standard ConvNet.
Conclusion
By introducing the distillation space and translative pre-training, this work significantly lowers the barrier to entry for dataset distillation. The ability to transfer distillation "knowledge" from ImageNet to smaller datasets like CIFAR demonstrates that the characteristics of effective synthetic data are not entirely dataset-specific. Future work will focus on further reducing memory overhead during the adaptation phase and exploring more diverse architectural designs for the translator network.
Summary updated