An Efficient Dataset Condensation Plugin and Its Application to Continual Learning

Enneng Yang, Li Shen, Zhenyi Wang, Tongliang Liu, Guibing Guo

Introduction

Training modern deep neural networks requires massive datasets, leading to high computational costs and storage burdens. Dataset Condensation (DC) aims to solve this by "distilling" a large dataset into a tiny synthetic one. If done well, a model trained on this tiny proxy should perform nearly as well as one trained on the original data.

However, most existing DC methods treat images as a collection of independent pixels, optimizing every single pixel value in a high-dimensional space. The authors of this paper argue this is inefficient because natural images are highly structured and "locally connected"—they actually exist on a much lower-dimensional manifold. By ignoring this structure, traditional DC methods waste "parameter budget" on redundant information, limiting how much knowledge can be packed into a small memory buffer. To address this, the authors introduce a "low-rank" plugin for dataset condensation that exploits this low intrinsic dimensionality.

The Low-Rank Data Condensation Plugin

The core innovation is the Low-rank Data Condensation Plugin. Instead of learning a synthetic image $S$ of size $H \times W$, the plugin decomposes the image into two low-rank matrices, $A$ (height $\times$ rank) and $B$ (rank $\times$ width), such that $S \approx A \times B$.

Key aspects of this approach include:

  • Plug-and-Play Architecture: This isn't a standalone algorithm but a structural modification that can be "plugged into" existing state-of-the-art DC methods like Gradient Matching (DC/DSA) or Distribution Matching (DM). The authors demonstrate this by creating Low-rank Dataset Condensation (LoDC) and Low-rank Distribution Matching (LoDM).
  • Efficiency and Information Density: Because the rank $r$ can be very small (e.g., $r=2$), the number of parameters needed to represent an image drops drastically. Under a strict "Same Memory" constraint, the low-rank method can store significantly more images. For example, on CIFAR-10, the low-rank approach can store roughly 8 times more images than traditional pixel-based methods using the same amount of storage.
  • Orthogonality: The method is orthogonal to the underlying matching strategy. Whether matching gradients, trajectories, or distributions, the low-rank representation consistently provides a performance boost.

Key Results and Applications

The authors evaluated their approach on standard computer vision benchmarks, including MNIST, CIFAR-10, CIFAR-100, and TinyImageNet.

Accuracy and Memory Boost: On CIFAR-10 with a budget of 10 images per class, the low-rank version (LoDM) achieved 59.8% accuracy, a massive jump over the standard DM's 48.9%. For the same number of images, the low-rank method reduced storage consumption by 7x to 8x across the MNIST and CIFAR datasets. Interestingly, the images generated by the low-rank plugin often look "sharper" and more recognizable to humans than the blurry, noisy images typically produced by standard DC methods.

Cross-Architecture Transfer: A critical test for synthetic data is whether features learned on one architecture transfer to another. Synthetic data generated using one network (e.g., ConvNet) transferred exceptionally well to others (e.g., ResNet, VGG), proving the learned features are robust and not just overfitted to a specific model.

Cross-architecture transfer performance comparing DSA, DM, and the proposed LoDM.

Continual Learning (CL): In CL, models learn tasks sequentially and often "forget" old ones. The authors use their low-rank synthetic images to fill a small "replay buffer," preserving more diverse information about past tasks than raw pixel storage would allow. When used as a buffer for the GDumb baseline, LoDM achieved a final accuracy of 37.92% on a 5-task CIFAR-100 challenge, outperforming standard DM (33.81%) and random coreset selection (27.9%).

Limitations and Future Work

While highly effective, the approach has a few limitations:

  • Manual Rank Selection: Currently, the rank $r$ is a hyperparameter that must be manually tuned for different datasets. The authors suggest that adaptive rank assignment—where the system automatically determines the optimal rank for each class or dataset—is a promising future direction.
  • Computational Overhead: While memory-efficient, the reconstruction of images from matrices during training adds a small computational step, though the authors note this is negligible compared to the overall training time.
  • Task Expansion: While this paper focuses on image classification and continual learning, future work will explore applying the low-rank plugin to other areas like Neural Architecture Search (NAS) and Federated Learning.

Summary updated

Abstract

Dataset condensation (DC) distills a large real-world dataset into a small synthetic dataset, with the goal of training a network from scratch on the latter that performs similarly to the former. State-of-the-art (SOTA) DC methods have achieved satisfactory results through techniques such as accuracy, gradient, training trajectory, or distribution matching. However, these works all perform matching in the high-dimension pixel spaces, ignoring that natural images are usually locally connected and have lower intrinsic dimensions, resulting in low condensation efficiency. In this work, we propose a simple-yet-efficient dataset condensation plugin that matches the raw and synthetic datasets in a low-dimensional manifold. Specifically, our plugin condenses raw images into two low-rank matrices instead of parameterized image matrices. Our plugin can be easily incorporated into existing DC methods, thereby containing richer raw dataset information at limited storage costs to improve the downstream applications' performance. We verify on multiple public datasets that when the proposed plugin is combined with SOTA DC methods, the performance of the network trained on synthetic data is significantly improved compared to traditional DC methods. Moreover, when applying the DC methods as a plugin to continual learning tasks, we observed that our approach effectively mitigates catastrophic forgetting of old tasks under limited memory buffer constraints and avoids the problem of raw data privacy leakage.
BibTeX
@inproceedings{yang2023efficient,
  author = {Enneng Yang and Li Shen and Zhenyi Wang and Tongliang Liu and Guibing Guo},
  title = {{An Efficient Dataset Condensation Plugin and Its Application to Continual Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=Murj6wcjRw},
}
Figure