Dataset Distillation with Convexified Implicit Gradients

Noel Loo, Ramin Hasani, Mathias Lechner, Daniela Rus

Introduction

Dataset distillation is the process of condensing a massive dataset into a small, synthetic version that retains the essential information of the original. The goal is to create a handful of synthetic images that, when used for training, yield a model with performance comparable to one trained on the full dataset. This task is fundamentally a bi-level optimization problem: the inner loop involves training a model on the synthetic set, while the outer loop updates the synthetic images to minimize the loss of that model on the original validation set.

Historically, backpropagating through an entire training process has been computationally prohibitive. Existing methods often rely on surrogates, such as matching gradients or training trajectories, which are faster but often sacrifice accuracy. This paper introduces RCIG (Reparameterized Convexified Implicit Gradients), an algorithm that leverages implicit differentiation and model linearization to achieve a significant leap in distillation performance.

Methodology: The RCIG Framework

The authors propose a three-step evolution to overcome the instability and bias typically associated with implicit gradients in deep learning.

1. Implicit Gradients (IG)

Rather than unrolling the entire training loop—a process that is memory-intensive and computationally expensive—RCIG utilizes the Implicit Function Theorem. This allows the calculation of how synthetic data affects final model parameters directly at the "equilibrium" point of training, bypassing the need to store intermediate states.

2. Convexification via Linearization

Vanilla implicit gradients often fail in deep learning because the loss landscapes of deep networks are non-convex and complex. RCIG addresses this by "convexifying" the inner problem. By linearizing the network using a finite-width Neural Tangent Kernel (NTK) approximation, the authors transform the inner optimization into a well-behaved problem where gradients are more reliable and mathematically stable.

3. Reparameterization and Analytic Solutions

To further reduce optimization bias, the authors split the network into "body" parameters and "final layer" parameters. They demonstrate that for a linearized model, the optimal final layer can be solved analytically in closed form. By learning only the body parameters and solving for the head, RCIG significantly accelerates convergence and improves final accuracy. Additionally, a bias-free subsampling technique is introduced to manage large distilled sets (e.g., 50 images per class) within GPU memory constraints.

Experimental Results

RCIG was evaluated across a diverse range of benchmarks, including standard datasets (MNIST, CIFAR-10/100), fine-grained benchmarks (CUB-200), and large-scale challenges (resized ImageNet-1K).

Performance Gains

RCIG outperformed previous state-of-the-art (SOTA) methods on 17 out of 22 reported benchmarks. Notable achievements include:

  • ImageNet (1 image/class): RCIG achieved 15.6% accuracy, representing a 108% improvement over the previous best of 7.5%.
  • Tiny-ImageNet (1 image/class): Achieved 25.6%, a 66% gain over the previous SOTA.
  • CIFAR-100 (1 image/class): Achieved 39.3%, a 37% gain over previous methods.

Cross-Architecture Robustness

Synthetic data generated by RCIG demonstrated impressive transferability. Data created using a simple ConvNet architecture transferred effectively to more complex architectures like ResNet or AlexNet, particularly when Batch Normalization was employed during the evaluation phase.

Additional Insights: Privacy and Security

Beyond performance, the paper highlights a significant side effect of the RCIG distillation process: enhanced privacy. Models trained on RCIG-distilled data show high resistance to Membership Inference Attacks (MIA). Because the synthetic images are concentrated, abstract representations rather than raw data points, they do not leak sensitive information about specific individuals in the original training set.

Limitations and Future Directions

Despite its success, RCIG faces certain challenges:

  • Overfitting: On small datasets with high class counts (e.g., CUB-200), the algorithm can overfit, reaching 100% training accuracy while showing lower generalization on test sets.
  • Memory Bottlenecks: While subsampling helps, the method still requires a full forward pass on the distilled set, which may limit scalability for extremely large-scale tasks.
  • Initialization: Performance on high-resolution images is sensitive to initialization; starting with real images currently yields better results than random noise, suggesting that improved initialization strategies are a fertile ground for future research.

Conclusion

RCIG represents a major advancement in dataset distillation. By combining the mathematical elegance of implicit gradients with the stability of model linearization, it provides a robust framework for creating highly efficient synthetic datasets, setting new benchmarks for the field while offering inherent privacy benefits.

Summary updated

Abstract

We propose a new dataset distillation algorithm using reparameterization and convexification of implicit gradients (RCIG), that substantially improves the state-of-the-art. To this end, we first formulate dataset distillation as a bi-level optimization problem. Then, we show how implicit gradients can be effectively used to compute meta-gradient updates. We further equip the algorithm with a convexified approximation that corresponds to learning on top of a frozen finite-width neural tangent kernel. Finally, we improve bias in implicit gradients by parameterizing the neural network to enable analytical computation of final-layer parameters given the body parameters. RCIG establishes the new state-of-the-art on a diverse series of dataset distillation tasks. Notably, with one image per class, on resized ImageNet, RCIG sees on average a 108% improvement over the previous state-of-the-art distillation algorithm. Similarly, we observed a 66% gain over SOTA on Tiny-ImageNet and 37% on CIFAR-100.
BibTeX
@inproceedings{loo2023dataset,
  author = {Noel Loo and Ramin Hasani and Mathias Lechner and Daniela Rus},
  title = {{Dataset Distillation with Convexified Implicit Gradients}},
  booktitle = {International Conference on Machine Learning},
  year = {2023},
  url = {https://openreview.net/forum?id=D0fHlOIyhZ},
}
Figure