Learning Student Networks in the Wild

Hanting Chen, Tianyu Guo, Chang Xu, Wenshuo Li, Chunjing Xu, Chao Xu, Yunhe WangCVPR 2021 · OpenReview

Abstract

Data-free learning for student networks is a new paradigm for solving users' anxiety caused by the privacy problem of using original training data. Since the architectures of modern convolutional neural networks (CNNs) are compact and sophisticated, the alternative images or meta-data generated from the teacher network are often broken. Thus, the student network cannot achieve the comparable performance to that of the pre-trained teacher network especially on the large-scale image dataset. Different to previous works, we present to maximally utilize the massive available unlabeled data in the wild. Specifically, we first thoroughly analyze the output differences between teacher and student network on the original data and develop a data collection method. Then, a noisy knowledge distillation algorithm is proposed for achieving the performance of the student network. In practice, an adaptation matrix is learned with the student network for correcting the label noise produced by the teacher network on the collected unlabeled images. The effectiveness of our DFND (Data-Free Noisy Distillation) method is then verified on several benchmarks to demonstrate its superiority over state-of-the-art data-free distillation methods. Experiments on various datasets demonstrate that the student networks learned by the proposed method can achieve comparable performance with those using the original dataset.

Introduction

Knowledge Distillation (KD) has become a standard technique for compressing heavy teacher models into lightweight student models. However, traditional KD relies on access to the original training data, which is often restricted due to privacy concerns, proprietary secrets, or logistical constraints. Existing "data-free" distillation methods typically employ Generative Adversarial Networks (GANs) to synthesize training samples. While innovative, these synthetic samples often lack the complexity and diversity of real-world data distributions, leading to significant performance degradation, particularly on large-scale tasks like ImageNet.

This paper introduces Data-Free Noisy Distillation (DFND), a framework that shifts the paradigm from data generation to data harvesting. Instead of creating synthetic images, DFND leverages massive, unlabeled datasets available "in the wild" (e.g., Flickr or public repositories) to train the student network.

Methodology

Overview of the DFND framework, illustrating data selection from the wild and the use of a Noisy Adaptation Matrix to correct teacher predictions.

Data Collection in the Wild

The core challenge in using wild data is identifying which samples are relevant to the teacher's original task. The authors derive a surrogate principle based on teacher confidence: samples where the teacher network exhibits low entropy (high confidence) are statistically more likely to align with the original data distribution. By filtering a massive pool of unlabeled data for these high-confidence samples, a high-quality proxy dataset is constructed.

Noisy Distillation with Adaptation

Even high-quality wild data will not perfectly match the original classes, and the teacher's predictions on this data will inevitably contain noise. To address this, DFND introduces a Noisy Adaptation Matrix ($Q$). This matrix models the probability transition between the latent "true" label and the noisy label predicted by the teacher. During training, the student network learns the task while the $Q$ matrix is simultaneously optimized to "clean" the teacher's supervision, effectively adapting the teacher's knowledge to the wild data distribution.

Visual comparison of (a) original training data, (b) data selected from the wild using the proposed entropy heuristic, and (c) unselected data.

Experimental Results

The researchers evaluated DFND across several benchmarks, demonstrating its ability to bridge the gap between data-free and data-driven distillation.

  • CIFAR-10: DFND achieved 94.02% accuracy, nearly matching standard KD (94.34%) and outperforming synthetic data-free methods like DAFL (92.22%).
  • ImageNet Scaling: Using unlabeled images from Flickr1M, DFND achieved 61.75% Top-1 accuracy on ImageNet. This is a significant milestone, as most synthetic data-free methods fail to scale to ImageNet-level complexity. For comparison, the teacher-student pair using actual ImageNet data achieved 68.67%.
  • Semantic Segmentation: On the NYUv2 dataset, DFND achieved a Mean IoU of 0.378, surpassing previous adversarial data-free methods (0.364) and approaching the upper bound of original-data distillation (0.380).

Discussion and Conclusion

DFND demonstrates that real-world "wild" data, even if imperfectly labeled, is a far more potent source of information for student networks than GAN-generated synthetic images. The use of a learnable noise adaptation matrix allows the student to benefit from the natural features and textures of real images while mitigating the impact of teacher errors.

While highly effective, the method's performance is tied to the availability of wild data that shares some domain overlap with the original task. Future work may explore more sophisticated initialization strategies for the noise adaptation matrix to handle even higher levels of label noise in extremely diverse wild datasets. Overall, DFND provides a robust, scalable solution for privacy-preserving model compression in real-world scenarios.

Summary updated

Abstract

Data-free learning for student networks is a new paradigm for solving users' anxiety caused by the privacy problem of using original training data. Since the architectures of modern convolutional neural networks (CNNs) are compact and sophisticated, the alternative images or meta-data generated from the teacher network are often broken. Thus, the student network cannot achieve the comparable performance to that of the pre-trained teacher network especially on the large-scale image dataset. Different to previous works, we present to maximally utilize the massive available unlabeled data in the wild. Specifically, we first thoroughly analyze the output differences between teacher and student network on the original data and develop a data collection method. Then, a noisy knowledge distillation algorithm is proposed for achieving the performance of the student network. In practice, an adaptation matrix is learned with the student network for correcting the label noise produced by the teacher network on the collected unlabeled images. The effectiveness of our DFND (Data-Free Noisy Distillation) method is then verified on several benchmarks to demonstrate its superiority over state-of-the-art data-free distillation methods. Experiments on various datasets demonstrate that the student networks learned by the proposed method can achieve comparable performance with those using the original dataset. Code is available at https://github.com/huawei-noah/Data-Efficient-Model-Compression
BibTeX
@inproceedings{chen2021learning,
  author = {Hanting Chen and Tianyu Guo and Chang Xu and Wenshuo Li and Chunjing Xu and Chao Xu and Yunhe Wang},
  title = {{Learning Student Networks in the Wild}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=35dcQJibLI},
}
Figure