: Dynastic Data-Free Knowledge Distillation
Lists
Only you see this
Abstract
Data-free knowledge distillation further broadens the applications of the distillation model. Nevertheless, the problem of providing diverse data with rich expression patterns needs to be further explored. In this paper, a novel dynastic data-free knowledge distillation ($D^{3}K$) model is proposed to alleviate this problem. In this model, a dynastic supernet generator (D-SG) with a flexible network structure is proposed to generate diverse data. The D-SG can adaptively alter architectural configurations and activate different subnet generators in different sequential iteration spaces. The variable network structure increases the complexity and capacity of the generator, and strengthens its ability to generate diversified data. In addition, a novel additive constraint based on the differentiable dhash (D-Dhash) is designed to guide the structure parameter selection of the D-SG. This constraint forces the D-SG to constantly jump out of the fixed generation mode and generate diverse data in semantics and instance. The effectiveness of the proposed model is verified on the experimental benchmark datasets (MNIST, CIFAR-10, CIFAR-100, and SVHN).
BibTeX
@article{li2023dynastic,
author = {Xiufang Li and Qigong Sun and Licheng Jiao and Fang Liu and Xu Liu and Lingling Li and Puhua Chen and Yi Zuo},
title = {{: Dynastic Data-Free Knowledge Distillation}},
journal = {IEEE Trans. Multimedia},
year = {2023},
url = {https://openreview.net/forum?id=YG4TJDdviw},
}Loading related content...