Condensed Sample-Guided Model Inversion for Knowledge Distillation

Kuluhan Binici, Shivam Aggarwal, Cihan Acar, Nam Trung Pham, Karianto Leman, Gim Hee Lee, Tulika MitraarXiv 2024 · DOI · arXiv

Introduction

Knowledge Distillation (KD) is a cornerstone technique for transferring knowledge from a large "teacher" model to a smaller "student" model. However, its effectiveness is often hampered by the lack of original training data due to privacy or storage constraints. While Dataset Condensation (DC) and Model Inversion (MI) offer potential workarounds by creating synthetic data, both face significant hurdles. Condensed datasets are often too small to provide sufficient variety, while standard model inversion produces noisy samples that may not align with the target distribution. This paper introduces a novel framework that bridges these two approaches, using condensed samples as prototypes to guide the model inversion process.

The Challenge of Data Scarcity and Distribution Gaps

Dataset Condensation aims to compress large datasets into a handful of representative samples. While efficient, these tiny sets often fail to capture the full diversity of the original feature space. As shown in Figure 6, condensed samples can be redundant or leave large regions of the data distribution uncovered, leading to poor student performance.

Visualization of feature space coverage. Condensed samples (colored clusters) often leave significant portions of the original data distribution (shaded area) uncovered or contain redundant information.

Conversely, Model Inversion (MI) generates data by exploiting the teacher model's internal knowledge. Without guidance, MI often produces "hallucinated" images that are semantically disconnected from the original data. Simply combining condensed data with MI-generated data fails because of the inherent domain gap between the two distributions.

Methodology: Feature Alignment via Prototypes

The core innovation of this work is a Feature Alignment Module designed to synchronize the distributions of synthetic and condensed data. Instead of generating data in a vacuum, the system uses condensed samples as "anchors" or prototypes in the teacher's feature space.

The proposed framework. (a) The Feature Alignment Module uses a discriminator to align synthetic features with condensed prototypes. (b) Conceptual view of model inversion guided by condensed templates in feature space.

The process involves a generator, a teacher model, and a discriminator. The generator produces synthetic images, which are then processed by the teacher alongside the condensed prototypes. A discriminator acts as a judge, attempting to distinguish between the high-level features (specifically at the penultimate layer) of the synthetic images and those of the condensed samples. By training the generator to "trick" the discriminator, the synthetic data is forced to align with the feature distribution of the condensed prototypes. This alignment is performed on a class-specific basis, ensuring that synthetic "Class A" images match the characteristics of "Class A" prototypes.

Experimental Results and Insights

The framework was evaluated on CIFAR-10, CIFAR-100, and ImageNet-200 benchmarks across various teacher-student architectures, including ResNet-34 and MobileNet-v2. The results demonstrate significant improvements over existing data-free KD methods:

  • Accuracy Gains: On CIFAR-100, the method improved student accuracy by up to 11.44% compared to standard baselines.
  • Visual Fidelity: Qualitatively, the guided model inversion (termed CMI*) produces images that are significantly more recognizable—resembling actual objects like cars or birds—than the abstract, noisy patterns typical of standard MI.
  • Robustness to Scarcity: The method remains highly effective even with extreme data limitations, such as a single condensed sample per class.

The research highlights that condensed samples are most valuable not as a final training set, but as a source of inductive bias for generative models. This "plug-and-play" module can be integrated into various existing MI algorithms, enhancing their performance across heterogeneous model pairs.

Conclusion

By treating condensed samples as prototypes for model inversion, this work effectively addresses the limitations of both dataset condensation and data-free knowledge distillation. The proposed feature alignment mechanism bridges the domain gap, enabling the creation of diverse, high-quality synthetic data that significantly improves student model performance. This approach paves the way for more efficient and privacy-preserving model training in data-constrained environments.

Summary updated

BibTeX
@misc{binici2024condensed,
  author = {Kuluhan Binici and Shivam Aggarwal and Cihan Acar and Nam Trung Pham and Karianto Leman and Gim Hee Lee and Tulika Mitra},
  title = {{Condensed Sample-Guided Model Inversion for Knowledge Distillation}},
  year = {2024},
  eprint = {2408.13850},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2408.13850},
}
Figure