Abstract
Knowledge distillation (KD) is a powerful technique for model compression, typically requiring the original training dataset. However, privacy and security concerns often restrict access to this data. This paper introduces a "data-free" KD method that synthesizes a transfer set by modeling the teacher model's intermediate feature space as a multivariate normal distribution. By utilizing weight-based correlations and soft target sampling, the proposed method generates high-quality pseudo-samples that effectively capture the original data distribution. Evaluations on MNIST and CIFAR-10 show that our method achieves performance comparable to standard KD using real data, significantly outperforming prior data-free approaches.
Introduction
The deployment of large-scale AI models on edge devices necessitates efficient compression techniques like Knowledge Distillation. Standard KD involves a "teacher-student" paradigm where the student mimics the teacher's output on the original training set. In practice, this dataset is frequently unavailable due to privacy regulations (e.g., medical data), massive storage requirements (e.g., ImageNet), or proprietary protections. The central challenge addressed here is transferring knowledge without any access to the original training images.
Methodology: Soft Targeted Transfer Set Synthesis
The proposed framework follows a three-step pipeline to synthesize a "transfer set" of pseudo-samples:
- Feature Space Modeling: Instead of focusing solely on the final softmax layer, the method models the distribution of an intermediate layer (typically the penultimate fully connected layer). This space is assumed to follow a multivariate normal distribution.
- Weight-Based Correlation: A correlation matrix is derived directly from the teacher's pre-trained weights. This insight leverages the fact that trained weights implicitly encode the relationships between features in the original data.
- Pseudo-Sample Synthesis: Soft targets are sampled from the modeled distribution. Starting from random noise, backpropagation is used to "evolve" the input into an image that triggers the desired intermediate features in the teacher. An activation loss term is included to ensure the synthesized images produce strong, "vivid" responses.
The synthesis process is sensitive to the variance parameter $\sigma$, which influences the magnitude of the sampled targets. As shown in Figure 3, the maximum value of the targets increases linearly with $\sigma$, affecting the "strength" of the synthesized signals.
Experimental Results
The approach was evaluated using LeNet-5, AlexNet, and ResNet-34 architectures on the MNIST and CIFAR-10 benchmarks.
- MNIST: The student model achieved 99.08% accuracy, nearly matching the teacher's 99.32% and surpassing the previous state-of-the-art Zero-Shot KD (ZSKD) by 0.31%.
- CIFAR-10: Using AlexNet, the method reached 73.91% accuracy, a significant improvement over ZSKD (69.56%) and DAFL (70.23%). On ResNet, the student reached 93.31%, remarkably close to the 94.40% achieved with real data.
Figure 1 illustrates the impact of the hyperparameter $\sigma$ on final accuracy. The method demonstrates a broad tolerance range, with optimal performance typically found between $\sigma=1.5$ and $\sigma=2.0$.
Discussion and Conclusion
A key insight of this work is that modeling shallower layers provides more generalized features, reducing the "label mismatch" problem where synthesized images do not visually correspond to their assigned labels. Furthermore, the results confirm that the teacher's weights contain sufficient structural information to reconstruct a representative transfer set.
While highly effective, the method involves computational overhead due to the iterative nature of image synthesis. Future work will focus on scaling this approach to high-dimensional datasets like ImageNet-1k and further optimizing the hyperparameter selection process.
Summary updated