Exploiting Inter-sample and Inter-feature Relations in Dataset Distillation

Wenxiao Deng, Wenbin Li, Tianyu Ding, Lei Wang, Hongguang Zhang, Kuihua Huang, Jing Huo, Yang Gao

Introduction

Dataset distillation (DD) is a technique designed to condense massive datasets into a small number of highly informative synthetic images. This reduction significantly lowers the computational cost of training deep learning models. While Distribution Matching (DM) has emerged as an efficient framework for DD, it faces two primary challenges: feature dispersion, where synthetic images within a class are too scattered in the feature space, and oversimplified matching, which typically only aligns the mean of feature distributions, ignoring complex inter-feature correlations.

Proposed Methodology

To address these limitations, this paper introduces two "plug-and-play" constraints: Class Centralization and Local Covariance Matching.

Class Centralization Constraint ($L_{CC}$)

This constraint encourages synthetic samples of the same class to cluster tightly around their class center in the embedding space. By penalizing samples that deviate from the center, $L_{CC}$ ensures high class discrimination, making it easier for models to learn clear decision boundaries.

Local Covariance Matching Constraint ($L_{CM}$)

Standard DM methods often fail to capture the relationships between different features. The authors propose matching the covariance matrices of real and synthetic data. To overcome the "small sample problem" inherent in tiny synthetic sets, they utilize local covariance. By treating each spatial location in a feature map as an individual sample, the method generates sufficient data points to estimate a robust covariance matrix even from a single image.

The proposed covariance matching constraint reshapes feature maps to calculate stable local covariance matrices even with very few images.

Experimental Results

The proposed method was evaluated on standard benchmarks including SVHN, CIFAR-10, CIFAR-100, and TinyImageNet.

Performance and Efficiency

The addition of these constraints leads to significant performance gains. For instance, on CIFAR-10 with 10 images per class (IPC), the method achieved 55.1% accuracy, a 6.6% improvement over the baseline. Furthermore, the method demonstrates superior convergence speed, reaching peak accuracy in fewer than 3,000 iterations, compared to the 20,000+ iterations required by previous approaches.

Feature Visualization

t-SNE visualizations confirm that the centralization constraint effectively transforms scattered feature "clouds" into dense, distinct clusters. This increased class separation is a key driver of the observed accuracy improvements.

t-SNE visualization showing how the proposed method (right) creates much tighter, more discriminative class clusters compared to standard DM (left).

Conclusion

By exploiting both inter-sample (centralization) and inter-feature (covariance) relationships, this approach significantly enhances the quality of distilled datasets. The constraints are architecture-agnostic and provide a robust solution for high-compression dataset distillation. Future work may focus on improving the visual interpretability of synthetic images and developing scale-specific strategies for high-resolution data.

Summary updated

Abstract

Dataset distillation has emerged as a promising approach in deep learning, enabling efficient training with small synthetic datasets derived from larger real ones. Particularly, distribution matching-based distillation methods attract attention thanks to its effectiveness and low computational cost. However, these methods face two primary limitations: the dispersed feature distribution within the same class in synthetic datasets, reducing class discrimination, and an exclusive focus on mean feature consistency, lacking precision and comprehensiveness. To address these challenges, we introduce two novel constraints: a class centralization constraint and a covariance matching constraint. The class centralization constraint aims to enhance class discrimination by more closely clustering samples within classes. The covariance matching constraint seeks to achieve more accurate feature distribution matching between real and synthetic datasets through local feature covariance matrices, particularly beneficial when sample sizes are much smaller than the number of features. Experiments demonstrate notable improvements with these constraints, yielding performance boosts of up to 6.6% on CIFAR10, 2.9% on SVHN, 2.5% on CIFAR100, and 2.5% on TinyImageNet, compared to the state-of-the-art relevant methods. In addition, our method maintains robust performance in cross-architecture settings, with a maximum performance drop of 1.7% on four architectures. Code is available at https://github.com/VincenDen/IID.
BibTeX
@inproceedings{deng2024exploiting,
  author = {Wenxiao Deng and Wenbin Li and Tianyu Ding and Lei Wang and Hongguang Zhang and Kuihua Huang and Jing Huo and Yang Gao},
  title = {{Exploiting Inter-sample and Inter-feature Relations in Dataset Distillation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
}
Figure