UniKD: Universal Knowledge Distillation for Mimicking Homogeneous or Heterogeneous Object Detectors

Shanshan Lao, Guanglu Song, Boxiao Liu, Yu Liu, Yujiu Yang

Introduction

Knowledge Distillation (KD) is a cornerstone technique for compressing large, high-performing "teacher" models into smaller, efficient "student" models. In the domain of object detection, traditional KD methods typically rely on pixel-by-pixel feature map mimicking. While effective for homogeneous model pairs (e.g., a small CNN learning from a larger CNN), these methods struggle with heterogeneous pairs, such as a CNN-based student learning from a Transformer-based teacher. This "semantic gap" arises because different architectures process information in fundamentally distinct ways, making their intermediate feature maps incomparable.

Furthermore, "offline KD"—where teacher features are pre-computed to save training time—often becomes impractical due to the massive storage requirements (terabytes) of high-resolution feature maps. To address these challenges, we present UniKD, a universal framework that shifts the distillation paradigm from feature mimicking to query-based knowledge extraction.

Methodology: The Adaptive Knowledge Extractor

The core of UniKD is the Adaptive Knowledge Extractor (AKE), a module designed to act as a universal "translator" between models. Instead of forcing the student to replicate the teacher's exact feature values, the AKE uses a transformer decoder architecture with deformable cross-attention to extract high-level semantic information using a fixed set of "probes" or queries.

The two-stage UniKD pipeline: (a) absorbing knowledge from the teacher and (b) injecting it into the student.

The training process follows a two-stage pipeline:

  1. Absorbing Knowledge (Teacher Stage): The AKE is attached to a frozen teacher model. It is trained to generate "knowledge embeddings" (content and positional) that can accurately predict object classes and bounding boxes. This stage teaches the AKE to identify and extract the most relevant information from the teacher's features.
  2. Injecting Knowledge (Student Stage): The pre-trained AKE is frozen and attached to the student model. The student is then trained to produce features that, when processed by the AKE, yield embeddings matching those of the teacher.

This approach ensures the student learns what the teacher knows, rather than how the teacher represents it internally.

Experimental Setup

UniKD was evaluated on the MS-COCO 2017 dataset, comprising 118k training and 5k validation images. To demonstrate universality, the authors tested a diverse array of detectors (RetinaNet, Faster R-CNN, FCOS, ATSS, RepPoints, Deformable DETR) and backbones (ResNet, ResNeXt, ConvNeXt, Swin Transformer, Uniformer).

Key Results and Insights

UniKD demonstrates significant advantages in both performance and efficiency:

  • Bridging the Semantic Gap: UniKD successfully transfers knowledge from a Transformer-based Deformable DETR to a CNN-based RetinaNet, achieving a 2.0 mAP boost. In contrast, traditional methods often fail or even degrade performance in such heterogeneous scenarios.
  • State-of-the-Art Performance: In homogeneous settings, UniKD outperforms specialized methods like FGD and FKD. For instance, it improves a RetinaNet-Res18 student from 31.7 to 35.3 mAP.
  • Storage Efficiency: By storing compact knowledge embeddings instead of full feature maps, UniKD reduces disk usage by over 23x compared to standard feature distillation, making offline KD feasible for large-scale datasets.
Comparison of storage efficiency and performance between UniKD and traditional feature-based distillation (FitNet) in offline mode.

Conclusion

UniKD provides a robust, "plug-and-play" solution for universal knowledge distillation in object detection. By focusing on query-based extraction, it overcomes the architectural constraints of traditional feature mimicking while offering superior storage efficiency. Future work may explore extending this paradigm to other vision tasks such as instance segmentation or video tracking.

Summary updated

Abstract

Knowledge distillation (KD) has become a standard method to boost the performance of lightweight object detectors. Most previous works are feature-based, where students mimic the features of homogeneous teacher detectors. However, distilling the knowledge from the heterogeneous teacher fails in this manner due to the serious semantic gap, which greatly limits the flexibility of KD in practical applications. Bridging this semantic gap now requires case-by-case algorithm design which is time-consuming and heavily relies on experienced adjustment. To alleviate this problem, we propose Universal Knowledge Distillation (UniKD), introducing additional decoder heads with deformable cross-attention called Adaptive Knowledge Extractor (AKE). In UniKD, AKEs are first pretrained on the teacher’s output to infuse the teacher’s content and positional knowledge into a fixed-number set of knowledge embeddings. The fixed AKEs are then attached to the student’s backbone to encourage the student to absorb the teacher’s knowledge in these knowledge embeddings. In this query-based distillation paradigm, detection-relevant information can be dynamically aggregated into a knowledge embedding set and transferred between different detectors. When the teacher model is too large for online inference, its output can be stored on disk in advance to save the computation overhead, which is more storage efficient than feature-based methods. Extensive experiments demonstrate that our UniKD can plug and play in any homogeneous or heterogeneous teacher-student pairs and significantly outperforms conventional feature-based KD.
BibTeX
@inproceedings{lao2023unikd,
  author = {Shanshan Lao and Guanglu Song and Boxiao Liu and Yu Liu and Yujiu Yang},
  title = {{UniKD: Universal Knowledge Distillation for Mimicking Homogeneous or Heterogeneous Object Detectors}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.00585},
}
Figure