Introduction
Knowledge Distillation (KD) is a cornerstone technique for compressing large, high-performing "teacher" models into smaller, efficient "student" models. In the domain of object detection, traditional KD methods typically rely on pixel-by-pixel feature map mimicking. While effective for homogeneous model pairs (e.g., a small CNN learning from a larger CNN), these methods struggle with heterogeneous pairs, such as a CNN-based student learning from a Transformer-based teacher. This "semantic gap" arises because different architectures process information in fundamentally distinct ways, making their intermediate feature maps incomparable.
Furthermore, "offline KD"—where teacher features are pre-computed to save training time—often becomes impractical due to the massive storage requirements (terabytes) of high-resolution feature maps. To address these challenges, we present UniKD, a universal framework that shifts the distillation paradigm from feature mimicking to query-based knowledge extraction.
Methodology: The Adaptive Knowledge Extractor
The core of UniKD is the Adaptive Knowledge Extractor (AKE), a module designed to act as a universal "translator" between models. Instead of forcing the student to replicate the teacher's exact feature values, the AKE uses a transformer decoder architecture with deformable cross-attention to extract high-level semantic information using a fixed set of "probes" or queries.
The training process follows a two-stage pipeline:
- Absorbing Knowledge (Teacher Stage): The AKE is attached to a frozen teacher model. It is trained to generate "knowledge embeddings" (content and positional) that can accurately predict object classes and bounding boxes. This stage teaches the AKE to identify and extract the most relevant information from the teacher's features.
- Injecting Knowledge (Student Stage): The pre-trained AKE is frozen and attached to the student model. The student is then trained to produce features that, when processed by the AKE, yield embeddings matching those of the teacher.
This approach ensures the student learns what the teacher knows, rather than how the teacher represents it internally.
Experimental Setup
UniKD was evaluated on the MS-COCO 2017 dataset, comprising 118k training and 5k validation images. To demonstrate universality, the authors tested a diverse array of detectors (RetinaNet, Faster R-CNN, FCOS, ATSS, RepPoints, Deformable DETR) and backbones (ResNet, ResNeXt, ConvNeXt, Swin Transformer, Uniformer).
Key Results and Insights
UniKD demonstrates significant advantages in both performance and efficiency:
- Bridging the Semantic Gap: UniKD successfully transfers knowledge from a Transformer-based Deformable DETR to a CNN-based RetinaNet, achieving a 2.0 mAP boost. In contrast, traditional methods often fail or even degrade performance in such heterogeneous scenarios.
- State-of-the-Art Performance: In homogeneous settings, UniKD outperforms specialized methods like FGD and FKD. For instance, it improves a RetinaNet-Res18 student from 31.7 to 35.3 mAP.
- Storage Efficiency: By storing compact knowledge embeddings instead of full feature maps, UniKD reduces disk usage by over 23x compared to standard feature distillation, making offline KD feasible for large-scale datasets.
Conclusion
UniKD provides a robust, "plug-and-play" solution for universal knowledge distillation in object detection. By focusing on query-based extraction, it overcomes the architectural constraints of traditional feature mimicking while offering superior storage efficiency. Future work may explore extending this paradigm to other vision tasks such as instance segmentation or video tracking.
Summary updated