Dataset Condensation via Expert Subspace Projection

Zhiheng Ma, Dezheng Gao, Shaolei Yang, Xing Wei, Yihong Gong

TL;DR

This paper introduces Expert Subspace Projection (ESP), a novel method for dataset condensation that significantly reduces memory and computation costs while preserving long-range training dynamics. ESP achieves this by projecting gradients onto a learned subspace of the original dataset's training trajectory, outperforming previous state-of-the-art methods in synthetic data generation for downstream tasks.

What Problem Does This Paper Address?

The ever-increasing size of datasets in deep learning poses significant challenges for storage, computation, and training time. While dataset condensation aims to create smaller, synthetic datasets that mimic the performance of the original, large datasets, existing methods like trajectory matching, though effective, are computationally expensive due to the need to unroll gradients across many training steps. This unrolling leads to high memory usage, often forcing the use of sub-sampled synthetic data, which can introduce bias and degrade performance. The core problem is to distill the essential knowledge from large datasets into compact synthetic ones efficiently, without sacrificing performance or introducing biases.

Methods

The proposed method, Expert Subspace Projection (ESP), tackles the memory and computation bottleneck of trajectory matching. Instead of directly mimicking the entire training trajectory of a model on the real dataset, ESP leverages the subspace spanned by this trajectory.

Here's a breakdown of the key components:

  • Expert Trajectory: The paper first trains a model on the original dataset (Dreal) to generate an "expert trajectory," which is a sequence of weight snapshots. This trajectory captures the long-range dynamics of the training process.
  • Expert Subspace: An "expert subspace" is constructed from this expert trajectory. This subspace is a lower-dimensional representation that effectively captures the essential information of the training dynamics.
  • Projected Gradient Descent (Inner Optimization): When training with the synthetic dataset (Dsyn), ESP uses projected gradient descent. Instead of updating the model's parameters directly with the computed gradient, the gradient is first projected onto the expert subspace. This ensures that the synthetic model's training stays within the learned subspace, effectively mimicking the long-range dynamics without the need for extensive gradient unrolling.
  • Projection Loss (Outer Optimization): A "projection loss" is introduced to penalize any deviation of the gradient from the expert subspace. This loss, combined with a distribution matching loss (to align feature distributions between real and synthetic data), forms the outer optimization objective for updating the synthetic dataset.

This approach significantly reduces memory requirements because the inner optimization only involves a single gradient step, unlike trajectory matching which requires unrolling gradients over multiple steps.

Datasets

The paper evaluates its method on several standard image classification benchmarks:

  • CIFAR10/100: Widely used datasets with 32x32 pixel images in 10 and 100 classes, respectively.
  • TinyImageNet: A downsized version of ImageNet, often used for benchmarking deep learning models, with 64x64 pixel images across 200 classes.
  • SVHN (Street View House Number): A dataset of house numbers from Google Street View, useful for digit recognition tasks.

The authors pre-compute expert trajectories on ConvNet-3 for each dataset to expedite the condensation process.

Key Insights and Ideas

  • Subspace is Sufficient: The core insight is that the subspace spanned by the expert trajectory, rather than the entire trajectory itself, is sufficient to capture the crucial long-range training dynamics. This allows for significant computational and memory savings.
  • Gradient Projection as a Proxy: Projecting the gradient onto this expert subspace acts as an effective proxy for mimicking the complex, multi-step gradient unrolling required by trajectory matching.
  • Memory Efficiency Enables Full Dataset Training: By drastically reducing memory overhead, ESP allows for training on the complete synthetic dataset without resorting to batching or slicing. This avoids potential biases introduced by sub-sampling and leads to better performance.
  • Complementarity of Losses: The combination of the projection loss (for long-range dynamics) and distribution matching loss (for static feature alignment) is crucial for achieving state-of-the-art results. The projection loss constrains higher-order information, while distribution matching captures lower-level texture and visual details.

Key Results

  • State-of-the-Art Performance: ESP achieves state-of-the-art results on SVHN, CIFAR10, CIFAR100, and TinyImageNet datasets.
  • Outperforms Trajectory Matching: On CIFAR10 with 1 Image/Class, ESP outperforms trajectory matching by 16.7%.
  • Significant Improvement over SOTA: ESP surpasses the previous state-of-the-art method by 3.2% on CIFAR10.
  • Memory Efficiency: ESP demonstrates significantly lower memory consumption compared to trajectory matching, with memory usage remaining largely unaffected by the student trajectory length or the number of images per class.
  • Cross-Architecture Generalization: The synthetic datasets generated by ESP show robust performance across different network architectures (ResNet, DenseNet, EfficientNet), indicating good generalization capabilities.
  • Batch Size Advantage: Due to its memory efficiency, ESP can train on the full synthetic dataset, leading to better performance than methods that require batching.

Limitations and Future Work

  • Bi-level Optimization: ESP still operates within a bi-level optimization framework. The authors acknowledge that extending this to very large datasets with high-resolution images remains a challenge, similar to other bi-level methods.
  • Future Directions: Promising avenues for future work include disentangling the outer and inner optimization steps and approximating the inner optimization using a convex proxy model to further reduce computational requirements and memory usage.

Summary updated

Abstract

The rapid growth in dataset sizes in modern deep learning has significantly increased data storage costs. Furthermore, the training and time costs for deep neural networks are generally proportional to the dataset size. Therefore, reducing the dataset size while maintaining model performance is an urgent research problem that needs to be addressed. Dataset condensation is a technique that aims to distill the original dataset into a much smaller synthetic dataset while maintaining downstream training performance on any agnostic neural network. Previous work has demonstrated that matching the training trajectory between the synthetic dataset and the original dataset is more effective than matching the instantaneous gradient, as it incorporates long-range information. Despite the effectiveness of trajectory matching, it suffers from complex gradient unrolling across iterations, which leads to significant memory and computation overhead. To address this issue, this paper proposes a novel approach called Expert Subspace Projection (ESP), which leverages long-range information while avoiding gradient unrolling. Instead of strictly enforcing the synthetic dataset’s training trajectory to mimic that of the real dataset, ESP only constrains it to lie within the subspace spanned by the training trajectory of the real dataset. The memory-saving advantage offered by our method facilitates unbiased training on the complete set of synthetic images and seamless integration with other dataset condensation techniques. Through extensive experiments, we have demonstrated the effectiveness of our approach. Our method outperforms the trajectory matching method on CIFAR10 by 16.7% in the setting of 1 Image/Class, surpassing the previous state-of-the-art method by 3.2%.
BibTeX
@inproceedings{ma2023dataset,
  author = {Zhiheng Ma and Dezheng Gao and Shaolei Yang and Xing Wei and Yihong Gong},
  title = {{Dataset Condensation via Expert Subspace Projection}},
  booktitle = {Sensors},
  year = {2023},
  url = {https://openreview.net/forum?id=TivABokQrU},
}
Figure