Delving into Effective Gradient Matching for Dataset Condensation

Zixuan Jiang, Jiaqi Gu, Mingjie Liu, David Z. Pan

TL;DR

Dataset condensation aims to shrink massive datasets into tiny, high-impact synthetic versions. This paper enhances the state-of-the-art "gradient matching" approach by incorporating global (inter-class) information, a more robust distance metric that considers gradient magnitude, and an adaptive training strategy that cuts computation time by up to 30%.

What Problem Does This Paper Address?

Training deep learning models on massive datasets is incredibly resource-heavy. Whether you are performing Neural Architecture Search (NAS), hyperparameter optimization, or training on edge devices, you often need a way to get "full-scale" performance from a "pocket-sized" dataset.

Dataset Condensation (DC) solves this by synthesizing a tiny set of images (e.g., just 1 or 10 per class) that mimics the training dynamics of the original data. A popular method for this is Gradient Matching, which tries to make the gradients (the "direction" the model learns in) of the synthetic data match those of the real data. However, existing methods often overfit the tiny synthetic set or ignore the relationships between different classes, leading to sub-optimal performance and wasted compute.

Methods

The authors propose three key enhancements to the standard gradient matching flow:

  1. Multi-level Matching: Traditional methods only match gradients within the same class (intra-class). This paper adds a term to match the average gradient across all classes (inter-class). This ensures the synthetic data captures not just what makes a "dog" a dog, but also how a "dog" differs from a "cat" globally.
  2. Angle and Magnitude Distance: Previous work focused almost entirely on the angle between gradients (using cosine similarity). The authors argue that ignoring magnitude is a mistake. As a model starts to overfit the tiny synthetic set, the gradient magnitude drops to near zero. By including a magnitude-aware distance (like Euclidean distance), they effectively "delay" this overfitting.
  3. Adaptive Learning Steps: In the inner loop of the algorithm, the model parameters are updated to "imitate" the training process. Instead of using a fixed number of steps, the authors propose a schedule ($\zeta_\theta(t)$) that reduces the number of updates as training progresses. This focuses the optimization on the early stages where the synthetic data diverges most from the real data.

Datasets

The researchers evaluated their method on five standard image classification benchmarks:

  • MNIST and FashionMNIST (grayscale)
  • SVHN (house numbers)
  • CIFAR-10 and CIFAR-100 (natural images)
  • Tiny ImageNet (used for supplementary experiments to test scalability)

All datasets were tested in low-data regimes, specifically synthesizing 1, 10, or 50 images per class (IPC).

Key Insights and Ideas

  • Global Context is Key: Matching gradients class-by-class is like learning to recognize individual letters without understanding how they form words. By adding inter-class matching, the synthetic images become better at representing the overall data distribution.
  • Magnitude as an Overfitting Signal: When gradient magnitude vanishes, the model has "memorized" the synthetic samples and stopped learning useful features. Incorporating magnitude into the loss function forces the optimizer to find synthetic samples that keep the learning process active and "healthy."
  • Efficiency via Pruning: You don't need to train the model to completion in every iteration of the condensation process. By using an adaptive schedule to trim unnecessary optimization steps, the authors significantly improved algorithmic efficiency without sacrificing accuracy.

Key Results

The proposed method, dubbed Ours-MDA (Multi-level, Distance-aware, Adaptive), achieved state-of-the-art results:

  • Accuracy Boost: On CIFAR-10 with 10 images per class, the method achieved 50.2% accuracy, a significant jump over the baseline's 44.9%.
  • Efficiency Gains: The adaptive learning step strategy reduced computation time by 25% to 30% in the 10 IPC experiments.
  • Generalization: Synthetic data generated using a simple ConvNet architecture generalized well when used to train more complex models like ResNet-18, proving the learned data isn't just "hacking" a specific architecture.

Limitations and Future Work

While the results are impressive, the authors acknowledge a few hurdles:

  • Scalability: The exploration process is still computationally intensive, making it difficult to scale to massive datasets like the full ImageNet-1k without significant hardware.
  • The "Performance Gap": Even with these improvements, training on 10 images per class still yields lower accuracy than training on the full dataset (e.g., ~50% vs ~85% on CIFAR-10).
  • Future Directions: The authors suggest exploring semi-supervised learning (using unlabeled data to help the condensation) and developing even more generalized distance functions.

Summary updated

Abstract

As deep learning models and datasets rapidly scale up, model training is extremely time-consuming and resource-costly. Instead of training on the entire dataset, learning with a small synthetic dataset becomes an efficient solution. Extensive research has been explored in the direction of dataset condensation, among which gradient matching achieves state-of-the-art performance. The gradient matching method directly targets the training dynamics by matching the gradient when training on the original and synthetic datasets. However, there are limited deep investigations into the principle and effectiveness of this method. In this work, we delve into the gradient matching method from a comprehensive perspective and answer the critical questions of what, how, and where to match. We propose to match the multi-level gradients to involve both intra-class and inter-class gradient information. We demonstrate that the distance function should focus on the angle, considering the magnitude simultaneously to delay the overfitting. An overfitting-aware adaptive learning step strategy is also proposed to trim unnecessary optimization steps for algorithmic efficiency improvement. Ablation and comparison experiments demonstrate that our proposed methodology shows superior accuracy, efficiency, and generalization compared to prior work.
BibTeX
@inproceedings{jiang2023delving,
  author = {Zixuan Jiang and Jiaqi Gu and Mingjie Liu and David Z. Pan},
  title = {{Delving into Effective Gradient Matching for Dataset Condensation}},
  booktitle = {COINS},
  year = {2023},
  doi = {10.1109/coins57856.2023.10189244},
}

Loading related content...

Figure