Abstract
Knowledge distillation (KD) has enabled remarkable progress in model compression and knowledge transfer. However, KD requires a large volume of original data or their representation statistics that are not usually available in practice. Data-free KD has recently been proposed to resolve this problem, wherein teacher and student models are fed by a synthetic sample generator trained from the teacher. Nonetheless, existing data-free KD methods rely on fine-tuning of weights to balance multiple losses, and ignore the diversity of generated samples, resulting in limited accuracy and robustness. To overcome this challenge, we propose robustness and diversity seeking data-free KD (RDSKD). The generator loss function is crafted to produce samples with high authenticity, class diversity, and inter-sample diversity. Without real data, the objectives of seeking high sample authenticity and class diversity often conflict with each other, causing frequent loss fluctuations. We mitigate this by exponentially penalizing loss increments. With MNIST, CIFAR-10, and SVHN datasets, our experiments show that RDSKD achieves higher accuracy with more robustness over different hyperparameter settings, compared to other data-free KD methods such as DAFL, MSKD, ZSKD, and DeepInversion.
Introduction
Knowledge Distillation (KD) is a popular technique for model compression, where a large, complex "teacher" model transfers its "knowledge" to a smaller, efficient "student" model. Traditionally, this requires the original training dataset. However, in many real-world scenarios—such as medical imaging or proprietary industrial data—the original data is unavailable due to privacy concerns or storage limitations.
"Data-free" KD attempts to solve this by using a generative model to synthesize training samples based on the teacher's internal logic. The challenge is that these generators often suffer from mode collapse (producing very similar images for the same class) and unstable training. Existing methods like DAFL (Data-Free Learning) require tedious hyperparameter tuning to balance conflicting goals: making samples look "authentic" to the teacher while ensuring they are diverse enough for the student to learn effectively.
Methodology
The RDSKD framework improves the synthetic data generation process through two primary innovations designed to enhance both the quality and variety of the training signal.
Diversity Seeking Regularization ($L_{DS}$)
To prevent the generator from producing nearly identical images for a single class, RDSKD introduces a term inspired by mode-seeking GANs. This term encourages the generator to produce images that are visually distinct even if the teacher classifies them similarly. Crucially, the method uses the distance between the teacher's softmax outputs as a metric for similarity, leveraging the teacher's "dark knowledge" to guide diversity.
Robustness Seeking Loss Function
Usually, researchers balance different loss terms (like authenticity vs. diversity) using fixed weights (hyperparameters). RDSKD replaces this with an exponentially penalized loss function. If a specific loss term starts to increase compared to the previous training epoch, the penalty grows exponentially. This automatically balances conflicting objectives without requiring manual fine-tuning of weights, making the training process much more robust to different learning rates.
Experimental Results
The researchers evaluated RDSKD on three standard image classification benchmarks: MNIST, SVHN, and CIFAR-10. They compared their results against several baselines, including DAFL, MSKD, ZSKD, and DeepInversion.
- Accuracy: RDSKD consistently outperformed other data-free methods. On CIFAR-10, it achieved 90.8% accuracy, outperforming DAFL (90.4%) and significantly beating DeepInversion (35.1%) and ZSKD (10.5%) in this specific setting.
- Stability: In experiments varying the learning rate, RDSKD maintained high accuracy (e.g., ~97% on MNIST) across a wide range of settings where other methods saw performance drops of 10% or more.
- Efficiency: The method is computationally efficient, requiring only 20 epochs to train a high-performing generator, whereas some competing methods require significantly more iterations.
Discussion and Insights
A key observation of this work is the conflict between "authenticity" (making the teacher very certain about a sample) and "class diversity" (ensuring all classes are represented). This conflict causes loss fluctuations in older methods, which RDSKD mitigates through its exponential regulation strategy.
Furthermore, the paper notes that the highest quality images (measured by metrics like FID) don't always lead to the best student accuracy. What matters most is the transferability—how well the synthetic samples represent the teacher's decision boundaries. RDSKD’s $L_{DS}$ term specifically targets this "intra-sample" diversity to ensure the student learns a comprehensive representation.
Conclusion
RDSKD provides a robust and effective solution for data-free knowledge distillation by addressing the core issues of mode collapse and hyperparameter sensitivity. By automating the balance between authenticity and diversity, it achieves state-of-the-art performance with significantly less manual tuning. Future work could explore extending this framework to high-resolution datasets and non-image modalities.
Summary updated