Functionally Similar Multi-Label Knowledge Distillation

Binghan Chen, Jianlong Hu, Xiawu Zheng, Wei Lin, Fei Chao, Rongrong Ji

Abstract

Existing multi-label knowledge distillation methods simply use regression or single-label classification methods without fully exploiting the essence of multi-label classification, resulting in student models’ inadequate performance and poor functional similarity to teacher models. In this paper, we reinterpret multi-label classification as multiple intra-class ranking tasks, with each class corresponding to a ranking task. Furthermore, we define the knowledge of multi-label classification models as the ranking of intra-class samples. On the one hand, we propose to evaluate the functional similarity between multi-label classification models with Kendall’s tau and rank-biased overlap, which are common metrics for evaluating ranking similarity. On the other hand, we propose a new functionally similar multi-label knowledge distillation method called FSD, which enables student models to learn the ranking of intra-class samples from teacher models. Finally, experimental results validate that FSD outperforms existing methods, especially for functional similarity. Specifically, we achieve a mAP of 73.38% and a mKDT of 0.686 on COCO, which are 2.22% and 0.19 better than existing methods, respectively.
BibTeX
@inproceedings{chen2024functionally,
  author = {Binghan Chen and Jianlong Hu and Xiawu Zheng and Wei Lin and Fei Chao and Rongrong Ji},
  title = {{Functionally Similar Multi-Label Knowledge Distillation}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
  doi = {10.1109/icassp48485.2024.10447660},
}

Loading related content...

Figure