Robustness and Diversity Seeking Data-Free Knowledge Distillation

Pengchao Han, Jihong Park, Shiqiang Wang, Yejun Liu

TL;DR

RDSKD is a novel "data-free" knowledge distillation framework that trains a small student model using only a pre-trained teacher, without needing the original training data. By introducing a specialized diversity-seeking regularizer and an exponentially regulated loss function, the authors achieve higher accuracy and significantly more stable training than previous state-of-the-art methods.

What Problem Does This Paper Address?

Knowledge Distillation (KD) is a popular technique for model compression, where a large, complex "teacher" model transfers its "knowledge" to a smaller, efficient "student" model. Traditionally, this requires the original training dataset. However, in many real-world scenarios—such as medical imaging or proprietary industrial data—the original data is unavailable due to privacy concerns or storage limitations.

"Data-free" KD attempts to solve this by using a generative model to synthesize training samples based on the teacher's internal logic. The challenge is that these generators often suffer from mode collapse (producing very similar images for the same class) and unstable training. Existing methods like DAFL (Data-Free Learning) require tedious hyperparameter tuning to balance conflicting goals: making samples look "authentic" to the teacher while ensuring they are diverse enough for the student to learn effectively.

Methods

The authors propose RDSKD, which improves the synthetic data generation process through two primary innovations:

  1. Diversity Seeking Regularization ($L_{DS}$): To prevent the generator from producing nearly identical images for a single class, the authors introduce a term inspired by mode-seeking GANs. This term encourages the generator to produce images that are visually distinct even if the teacher classifies them similarly. Crucially, they use the distance between the teacher's softmax outputs as a metric for similarity, leveraging the teacher's "dark knowledge" to guide diversity.
  2. Robustness Seeking Loss Function: Usually, researchers balance different loss terms (like authenticity vs. diversity) using fixed weights (hyperparameters). RDSKD replaces this with an exponentially penalized loss function. If a specific loss term starts to increase compared to the previous training epoch, the penalty grows exponentially. This automatically balances conflicting objectives without requiring manual fine-tuning of weights, making the training process much more robust to different learning rates.

The student model is then trained using standard KD techniques (Kullback-Leibler divergence) on the synthetic samples produced by this optimized generator.

Datasets

The researchers evaluated RDSKD on three standard image classification benchmarks:

  • MNIST: Handwritten digits (10 classes).
  • SVHN: Street View House Numbers (10 classes).
  • CIFAR-10: General object classification (10 classes).

They compared their results against several baselines, including DAFL, MSKD, ZSKD, and DeepInversion.

Key Insights and Ideas

  • The Authenticity-Diversity Conflict: The authors observed that seeking "authenticity" (making the teacher very certain about a sample) often conflicts with "class diversity" (ensuring all classes are represented). This conflict causes the loss to fluctuate wildly in older methods.
  • Intra-class Diversity Matters: It isn't enough to just generate samples for every class; you need variety within each class. RDSKD’s $L_{DS}$ term specifically targets this "intra-sample" diversity.
  • Transferability vs. Quality: Interestingly, the paper notes that the highest quality images (measured by metrics like FID or Inception Score) don't always lead to the best student accuracy. What matters most is the transferability—how well the synthetic samples represent the teacher's decision boundaries.
  • Hyperparameter Independence: By using the exponential regulation strategy, the authors created a method that is far less sensitive to the generator's learning rate ($\eta_G$) and the distillation temperature ($\tau$) than previous approaches.

Key Results

RDSKD consistently outperformed other data-free methods across all tested datasets:

  • Accuracy: On CIFAR-10, RDSKD achieved 90.8% accuracy, outperforming DAFL (90.4%) and significantly beating DeepInversion (35.1%) and ZSKD (10.5%) in this specific setting.
  • Stability: In experiments varying the learning rate, RDSKD maintained high accuracy (e.g., ~97% on MNIST) across a wide range of settings where other methods (like DAFL) saw performance drops of 10% or more.
  • Efficiency: The method is computationally efficient, requiring only 20 epochs to train a high-performing generator, whereas some competing methods require significantly more iterations or complex optimization.

Limitations and Future Work

While the paper demonstrates strong results on standard benchmarks, the authors do not extensively test the method on high-resolution or large-scale datasets like ImageNet, where data-free generation is significantly more difficult.

Future work could explore:

  • Applying RDSKD to more complex architectures beyond ResNets and LeNet.
  • Investigating the "transferability" of samples further to understand exactly which features of synthetic data are most beneficial for the student.
  • Extending the framework to non-image data (e.g., audio or natural language).

Summary updated

Abstract

Knowledge distillation (KD) has enabled remarkable progress in model compression and knowledge transfer. However, KD requires a large volume of original data or their representation statistics that are not usually available in practice. Data-free KD has recently been proposed to resolve this problem, wherein teacher and student models are fed by a synthetic sample generator trained from the teacher. Nonetheless, existing data-free KD methods rely on fine-tuning of weights to balance multiple losses, and ignore the diversity of generated samples, resulting in limited accuracy and robustness. To overcome this challenge, we propose robustness and diversity seeking data-free KD (RDSKD) in this paper. The generator loss function is crafted to produce samples with high authenticity, class diversity, and inter-sample diversity. Without real data, the objectives of seeking high sample authenticity and class diversity often conflict with each other, causing frequent loss fluctuations. We mitigate this by exponentially penalizing loss increments. With MNIST, CIFAR-10, and SVHN datasets, our experiments show that RDSKD achieves higher accuracy with more robustness over different hyperparameter settings, compared to other data-free KD methods such as DAFL, MSKD, ZSKD, and DeepInversion.
BibTeX
@inproceedings{han2021robustness,
  author = {Pengchao Han and Jihong Park and Shiqiang Wang and Yejun Liu},
  title = {{Robustness and Diversity Seeking Data-Free Knowledge Distillation}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=tPMGH36Nui},
}

Loading related content...

Figure