Introduction
Dataset distillation (DD) is a technique designed to condense massive datasets into a small number of highly informative synthetic images. This reduction significantly lowers the computational cost of training deep learning models. While Distribution Matching (DM) has emerged as an efficient framework for DD, it faces two primary challenges: feature dispersion, where synthetic images within a class are too scattered in the feature space, and oversimplified matching, which typically only aligns the mean of feature distributions, ignoring complex inter-feature correlations.
Proposed Methodology
To address these limitations, this paper introduces two "plug-and-play" constraints: Class Centralization and Local Covariance Matching.
Class Centralization Constraint ($L_{CC}$)
This constraint encourages synthetic samples of the same class to cluster tightly around their class center in the embedding space. By penalizing samples that deviate from the center, $L_{CC}$ ensures high class discrimination, making it easier for models to learn clear decision boundaries.
Local Covariance Matching Constraint ($L_{CM}$)
Standard DM methods often fail to capture the relationships between different features. The authors propose matching the covariance matrices of real and synthetic data. To overcome the "small sample problem" inherent in tiny synthetic sets, they utilize local covariance. By treating each spatial location in a feature map as an individual sample, the method generates sufficient data points to estimate a robust covariance matrix even from a single image.
Experimental Results
The proposed method was evaluated on standard benchmarks including SVHN, CIFAR-10, CIFAR-100, and TinyImageNet.
Performance and Efficiency
The addition of these constraints leads to significant performance gains. For instance, on CIFAR-10 with 10 images per class (IPC), the method achieved 55.1% accuracy, a 6.6% improvement over the baseline. Furthermore, the method demonstrates superior convergence speed, reaching peak accuracy in fewer than 3,000 iterations, compared to the 20,000+ iterations required by previous approaches.
Feature Visualization
t-SNE visualizations confirm that the centralization constraint effectively transforms scattered feature "clouds" into dense, distinct clusters. This increased class separation is a key driver of the observed accuracy improvements.
Conclusion
By exploiting both inter-sample (centralization) and inter-feature (covariance) relationships, this approach significantly enhances the quality of distilled datasets. The constraints are architecture-agnostic and provide a robust solution for high-compression dataset distillation. Future work may focus on improving the visual interpretability of synthetic images and developing scale-specific strategies for high-resolution data.
Summary updated