Introduction
Training modern deep neural networks requires massive datasets, leading to high computational costs and storage burdens. Dataset Condensation (DC) aims to solve this by "distilling" a large dataset into a tiny synthetic one. If done well, a model trained on this tiny proxy should perform nearly as well as one trained on the original data.
However, most existing DC methods treat images as a collection of independent pixels, optimizing every single pixel value in a high-dimensional space. The authors of this paper argue this is inefficient because natural images are highly structured and "locally connected"—they actually exist on a much lower-dimensional manifold. By ignoring this structure, traditional DC methods waste "parameter budget" on redundant information, limiting how much knowledge can be packed into a small memory buffer. To address this, the authors introduce a "low-rank" plugin for dataset condensation that exploits this low intrinsic dimensionality.
The Low-Rank Data Condensation Plugin
The core innovation is the Low-rank Data Condensation Plugin. Instead of learning a synthetic image $S$ of size $H \times W$, the plugin decomposes the image into two low-rank matrices, $A$ (height $\times$ rank) and $B$ (rank $\times$ width), such that $S \approx A \times B$.
Key aspects of this approach include:
- Plug-and-Play Architecture: This isn't a standalone algorithm but a structural modification that can be "plugged into" existing state-of-the-art DC methods like Gradient Matching (DC/DSA) or Distribution Matching (DM). The authors demonstrate this by creating Low-rank Dataset Condensation (LoDC) and Low-rank Distribution Matching (LoDM).
- Efficiency and Information Density: Because the rank $r$ can be very small (e.g., $r=2$), the number of parameters needed to represent an image drops drastically. Under a strict "Same Memory" constraint, the low-rank method can store significantly more images. For example, on CIFAR-10, the low-rank approach can store roughly 8 times more images than traditional pixel-based methods using the same amount of storage.
- Orthogonality: The method is orthogonal to the underlying matching strategy. Whether matching gradients, trajectories, or distributions, the low-rank representation consistently provides a performance boost.
Key Results and Applications
The authors evaluated their approach on standard computer vision benchmarks, including MNIST, CIFAR-10, CIFAR-100, and TinyImageNet.
Accuracy and Memory Boost: On CIFAR-10 with a budget of 10 images per class, the low-rank version (LoDM) achieved 59.8% accuracy, a massive jump over the standard DM's 48.9%. For the same number of images, the low-rank method reduced storage consumption by 7x to 8x across the MNIST and CIFAR datasets. Interestingly, the images generated by the low-rank plugin often look "sharper" and more recognizable to humans than the blurry, noisy images typically produced by standard DC methods.
Cross-Architecture Transfer: A critical test for synthetic data is whether features learned on one architecture transfer to another. Synthetic data generated using one network (e.g., ConvNet) transferred exceptionally well to others (e.g., ResNet, VGG), proving the learned features are robust and not just overfitted to a specific model.
Continual Learning (CL): In CL, models learn tasks sequentially and often "forget" old ones. The authors use their low-rank synthetic images to fill a small "replay buffer," preserving more diverse information about past tasks than raw pixel storage would allow. When used as a buffer for the GDumb baseline, LoDM achieved a final accuracy of 37.92% on a 5-task CIFAR-100 challenge, outperforming standard DM (33.81%) and random coreset selection (27.9%).
Limitations and Future Work
While highly effective, the approach has a few limitations:
- Manual Rank Selection: Currently, the rank $r$ is a hyperparameter that must be manually tuned for different datasets. The authors suggest that adaptive rank assignment—where the system automatically determines the optimal rank for each class or dataset—is a promising future direction.
- Computational Overhead: While memory-efficient, the reconstruction of images from matrices during training adds a small computational step, though the authors note this is negligible compared to the overall training time.
- Task Expansion: While this paper focuses on image classification and continual learning, future work will explore applying the low-rank plugin to other areas like Neural Architecture Search (NAS) and Federated Learning.
Summary updated