Abstract
As deep learning models and datasets rapidly scale up, network training is extremely time-consuming and resource-costly. Instead of training on the entire dataset, learning with a small synthetic dataset becomes an efficient solution. Extensive research has been explored in the direction of dataset condensation, among which gradient matching achieves state-of-the-art performance. The gradient matching method directly targets the training dynamics by matching the gradient when training on the original and synthetic datasets. However, there are limited deep investigations into the principle and effectiveness of this method. In this work, we delve into the gradient matching method from a comprehensive perspective and answer the critical questions of what, how, and where to match. We propose to match the multi-level gradients to involve both intra-class and inter-class gradient information. We demonstrate that the distance function should focus on the angle, considering the magnitude simultaneously to delay the overfitting. An overfitting-aware adaptive learning step strategy is also proposed to trim unnecessary optimization steps for algorithmic efficiency improvement. Ablation and comparison experiments demonstrate that our proposed methodology shows superior accuracy, efficiency, and generalization compared to prior work.
Introduction
Training deep learning models on massive datasets is incredibly resource-heavy. Whether performing Neural Architecture Search (NAS), hyperparameter optimization, or training on edge devices, there is a critical need for "full-scale" performance from "pocket-sized" datasets. Dataset Condensation (DC) addresses this by synthesizing a tiny set of images (e.g., 1 or 10 per class) that mimics the training dynamics of the original data.
A leading approach in this field is Gradient Matching, which attempts to align the gradients of synthetic data with those of the original dataset. However, existing methods often suffer from overfitting to the tiny synthetic set or ignore global relationships between classes, leading to sub-optimal performance. This work investigates the principles of gradient matching to improve its effectiveness and efficiency.
Methodology
The authors propose three key enhancements to the standard gradient matching flow, collectively referred to as Ours-MDA (Multi-level, Distance-aware, Adaptive):
1. Multi-level Matching
Traditional methods typically match gradients within the same class (intra-class). This paper introduces a term to match the average gradient across all classes (inter-class). This ensures the synthetic data captures not just class-specific features, but also global context—how a "dog" differs from a "cat" across the entire distribution.
2. Angle and Magnitude Distance
Previous research focused primarily on the angle between gradients (cosine similarity). The authors argue that ignoring magnitude is a mistake. As a model overfits the synthetic set, gradient magnitude drops toward zero. By incorporating a magnitude-aware distance (like Euclidean distance), the method effectively "delays" this overfitting, keeping the learning process active.
3. Adaptive Learning Steps
In the algorithm's inner loop, model parameters $\theta$ are updated to imitate the training process. Rather than using a fixed number of steps, the authors propose an adaptive schedule that reduces the number of updates as training progresses. This focuses optimization on early stages where synthetic data diverges most from real data, improving efficiency.
Results and Discussion
The researchers evaluated their method on standard benchmarks including MNIST, FashionMNIST, SVHN, CIFAR-10, and CIFAR-100.
Key Findings: * Accuracy: On CIFAR-10 with 10 images per class (IPC), the method achieved 50.2% accuracy, a significant improvement over the 44.9% baseline. * Efficiency: The adaptive learning step strategy reduced computation time by 25% to 30% in 10 IPC experiments. * Generalization: Synthetic data generated with a simple ConvNet generalized effectively when used to train more complex architectures like ResNet-18. * Overfitting Mitigation: Incorporating gradient magnitude into the loss function forced the optimizer to find synthetic samples that maintain a "healthy" learning signal, preventing early memorization.
Conclusion
While Ours-MDA significantly advances dataset condensation, challenges remain regarding scalability to massive datasets like ImageNet-1k and the persistent performance gap between synthetic and full datasets. Future work may explore semi-supervised learning and more generalized distance functions to further bridge this gap. Nonetheless, the proposed multi-level matching and adaptive strategies provide a robust framework for efficient deep learning on condensed data.
Summary updated