Few-Shot Dataset Distillation via Translative Pre-Training

Songhua Liu, Xinchao WangICCV 2023 · OpenReview

Abstract

Dataset distillation aims to create a small synthetic dataset that mimics the training performance of a large-scale original dataset on neural networks. Existing approaches typically rely on heavy iterative optimization across thousands of neural network initializations, leading to significant computational overhead. This paper introduces a "few-shot" dataset distillation framework that utilizes a learned distillation space. By optimizing synthetic data within this specific space, the method achieves performance comparable to state-of-the-art techniques while requiring only a single or a few network evaluations. We propose a Translative Pre-Training approach where a translator network, pre-trained on large-scale data like ImageNet, maps images from an arbitrary neural space to the target distillation space. Extensive experiments demonstrate that this method achieves a ~15× acceleration with satisfactory generalization across various datasets and architectures.

The Challenge of Efficient Dataset Distillation

Dataset Distillation (DD) is a powerful technique for condensing massive datasets into a handful of representative synthetic images. Ideally, a model trained on these few images should achieve performance close to one trained on the full dataset. However, the current state-of-the-art (SOTA) methods are computationally expensive. They often require simulating training across thousands of random network initializations to update synthetic pixels, a process that can take days or weeks for a single dataset.

This paper addresses the need for few-shot dataset distillation, where high-quality synthetic data is produced with minimal network evaluations. The goal is to move away from grueling iterative loops toward a more efficient, transferable distillation process.

Methodology: Learning a Distillation Space

The core innovation is the concept of a Distillation Space. The authors hypothesize that synthetic data optimized for a dedicated, learned network space can approximate the performance of data optimized across a vast ensemble of random networks.

The proposed distillation space approach (c) optimizes synthetic data in a dedicated space to match target performance efficiently.

To implement this, the authors utilize Translative Pre-Training:

  1. The Translator ($G$): An auto-encoder network is trained to map images from a fixed "random" neural space into the optimized "distillation space."
  2. Pre-training and Adaptation: The translator is first pre-trained on a large dataset (ImageNet) to learn general features of effective distilled images. When applied to a new target dataset (e.g., CIFAR-10), it only requires a few adaptation steps to produce high-quality synthetic data.
  3. Decoupling Optimization: By using a translator, the method decouples the image distillation from the network space optimization. This eliminates the need to back-propagate through thousands of unrolled training steps, drastically reducing memory and time costs.

Key Results and Efficiency

The proposed method achieves a massive acceleration over existing benchmarks. While some SOTA methods like FRePo require extensive TFLOPs and time, the translative approach reaches comparable accuracy with significantly lower overhead.

Accuracy vs. Log-Scale Running Time (d) highlighting the 15× speedup and superior accuracy of the proposed method (Ours) compared to baselines.
  • Performance: On CIFAR-10 (1 image per class), the method achieves 44.0% accuracy. When combined with down-sampling parameterization (Ours-FS-DS), accuracy jumps to 51.4%, outperforming many "heavy" distillation methods.
  • Speed: The method is approximately 15× faster than FRePo. Distillation tasks that previously took days can now be completed in minutes or hours after the initial pre-training phase.
  • Generalization: Synthetic data produced via this method generalizes well across different architectures, such as AlexNet, VGG, and ResNet, even when distilled using a standard ConvNet.

Conclusion

By introducing the distillation space and translative pre-training, this work significantly lowers the barrier to entry for dataset distillation. The ability to transfer distillation "knowledge" from ImageNet to smaller datasets like CIFAR demonstrates that the characteristics of effective synthetic data are not entirely dataset-specific. Future work will focus on further reducing memory overhead during the adaptation phase and exploring more diverse architectural designs for the translator network.

Summary updated

Abstract

Dataset distillation aims at a small synthetic dataset to mimic the training performance on neural networks of a given large dataset. Existing approaches heavily rely on an iterative optimization to update synthetic data and multiple forward-backward passes over thousands of neural network spaces, which introduce significant overhead for computation and are inconvenient in scenarios requiring high efficiency. In this paper, we focus on few-shot dataset distillation, where a distilled dataset is synthesized with only a few or even a single network. To this end, we introduce the notion of distillation space, such that synthetic data optimized only in this specific space can achieve the effect of those optimized through numerous neural networks, with dramatically accelerated training and reduced computational cost. To learn such a distillation space, we first formulate the problem as a quad-level optimization framework and propose a bi-level algorithm. Nevertheless, the algorithm in its original form has a large memory footprint in practice due to the back-propagation through an unrolled computational graph. We then convert the problem of learning the distillation space to a first-order one based on image translation. Specifically, the synthetic images are optimized in an arbitrary but fixed neural space and then translated to those in the targeted distillation space. We pre-train the translator on some large datasets like ImageNet so that it requires only a limited number of adaptation steps on the target dataset. Extensive experiments demonstrate that the translator after pre-training and a limited number of adaptation steps achieves comparable distillation performance with state of the arts, with ∼ 15× acceleration. It also exerts satisfactory generalization performance across different datasets, storage budgets, and numbers of classes.
BibTeX
@inproceedings{liu2023few,
  author = {Songhua Liu and Xinchao Wang},
  title = {{Few-Shot Dataset Distillation via Translative Pre-Training}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  url = {https://openreview.net/forum?id=SFiJuldPxq},
}
Figure