Delving into Effective Gradient Matching for Dataset Condensation

Zixuan Jiang, Jiaqi Gu, Mingjie Liu, David Z. Pan

Abstract

As deep learning models and datasets rapidly scale up, network training is extremely time-consuming and resource-costly. Instead of training on the entire dataset, learning with a small synthetic dataset becomes an efficient solution. Extensive research has been explored in the direction of dataset condensation, among which gradient matching achieves state-of-the-art performance. The gradient matching method directly targets the training dynamics by matching the gradient when training on the original and synthetic datasets. However, there are limited deep investigations into the principle and effectiveness of this method. In this work, we delve into the gradient matching method from a comprehensive perspective and answer the critical questions of what, how, and where to match. We propose to match the multi-level gradients to involve both intra-class and inter-class gradient information. We demonstrate that the distance function should focus on the angle, considering the magnitude simultaneously to delay the overfitting. An overfitting-aware adaptive learning step strategy is also proposed to trim unnecessary optimization steps for algorithmic efficiency improvement. Ablation and comparison experiments demonstrate that our proposed methodology shows superior accuracy, efficiency, and generalization compared to prior work.

Introduction

Training deep learning models on massive datasets is incredibly resource-heavy. Whether performing Neural Architecture Search (NAS), hyperparameter optimization, or training on edge devices, there is a critical need for "full-scale" performance from "pocket-sized" datasets. Dataset Condensation (DC) addresses this by synthesizing a tiny set of images (e.g., 1 or 10 per class) that mimics the training dynamics of the original data.

A leading approach in this field is Gradient Matching, which attempts to align the gradients of synthetic data with those of the original dataset. However, existing methods often suffer from overfitting to the tiny synthetic set or ignore global relationships between classes, leading to sub-optimal performance. This work investigates the principles of gradient matching to improve its effectiveness and efficiency.

Methodology

The authors propose three key enhancements to the standard gradient matching flow, collectively referred to as Ours-MDA (Multi-level, Distance-aware, Adaptive):

1. Multi-level Matching

Traditional methods typically match gradients within the same class (intra-class). This paper introduces a term to match the average gradient across all classes (inter-class). This ensures the synthetic data captures not just class-specific features, but also global context—how a "dog" differs from a "cat" across the entire distribution.

2. Angle and Magnitude Distance

Previous research focused primarily on the angle between gradients (cosine similarity). The authors argue that ignoring magnitude is a mistake. As a model overfits the synthetic set, gradient magnitude drops toward zero. By incorporating a magnitude-aware distance (like Euclidean distance), the method effectively "delays" this overfitting, keeping the learning process active.

The process of updating the synthetic dataset $\mathcal{S}$ by matching its gradient to the target dataset $\mathcal{T}$.

3. Adaptive Learning Steps

In the algorithm's inner loop, model parameters $\theta$ are updated to imitate the training process. Rather than using a fixed number of steps, the authors propose an adaptive schedule that reduces the number of updates as training progresses. This focuses optimization on early stages where synthetic data diverges most from real data, improving efficiency.

The inner loop update of model parameters $\theta$ using gradient descent on the synthetic data.

Results and Discussion

The researchers evaluated their method on standard benchmarks including MNIST, FashionMNIST, SVHN, CIFAR-10, and CIFAR-100.

Key Findings: * Accuracy: On CIFAR-10 with 10 images per class (IPC), the method achieved 50.2% accuracy, a significant improvement over the 44.9% baseline. * Efficiency: The adaptive learning step strategy reduced computation time by 25% to 30% in 10 IPC experiments. * Generalization: Synthetic data generated with a simple ConvNet generalized effectively when used to train more complex architectures like ResNet-18. * Overfitting Mitigation: Incorporating gradient magnitude into the loss function forced the optimizer to find synthetic samples that maintain a "healthy" learning signal, preventing early memorization.

Conclusion

While Ours-MDA significantly advances dataset condensation, challenges remain regarding scalability to massive datasets like ImageNet-1k and the persistent performance gap between synthetic and full datasets. Future work may explore semi-supervised learning and more generalized distance functions to further bridge this gap. Nonetheless, the proposed multi-level matching and adaptive strategies provide a robust framework for efficient deep learning on condensed data.

Summary updated

Abstract

As deep learning models and datasets rapidly scale up, network training is extremely time-consuming and resource-costly. Instead of training on the entire dataset, learning with a small synthetic dataset becomes an efficient solution. Extensive research has been explored in the direction of dataset condensation, among which gradient matching achieves state-of-the-art performance. The gradient matching method directly targets the training dynamics by matching the gradient when training on the original and synthetic datasets. However, there are limited deep investigations into the principle and effectiveness of this method. In this work, we delve into the gradient matching method from a comprehensive perspective and answer the critical questions of what, how, and where to match. We propose to match the multi-level gradients to involve both intra-class and inter-class gradient information. We demonstrate that the distance function should focus on the angle, considering the magnitude simultaneously to delay the overfitting. An overfitting-aware adaptive learning step strategy is also proposed to trim unnecessary optimization steps for algorithmic efficiency improvement. Ablation and comparison experiments demonstrate that our proposed methodology shows superior accuracy, efficiency, and generalization compared to prior work.
BibTeX
@inproceedings{jiang2023delving,
  author = {Zixuan Jiang and Jiaqi Gu and Mingjie Liu and David Z. Pan},
  title = {{Delving into Effective Gradient Matching for Dataset Condensation}},
  booktitle = {COINS},
  year = {2023},
  doi = {10.1109/coins57856.2023.10189244},
}
Figure