Introduction
Knowledge Distillation (KD) is a powerful technique for transferring knowledge from a large, complex "teacher" model to a smaller, more efficient "student" model. While highly effective in image classification, KD faces significant challenges in object detection due to the dual requirements of classification and localization. Traditional methods often rely on rigid one-to-one feature matching between the teacher and student Feature Pyramids (FPN), leading to two primary issues: semantic misalignment and architectural incompatibility. G-DetKD addresses these by introducing a flexible, general framework capable of distilling knowledge across diverse detector families.
Methodology
The G-DetKD framework consists of three primary components designed to overcome the limitations of traditional distillation:
Semantic-Guided Feature Imitation (SGFI)
Standard KD methods often fail when the teacher and student have different pyramid levels or when features at different levels share similar semantic information. SGFI employs a soft-matching mechanism that evaluates the semantic content across all pyramid levels. It automatically identifies the most relevant teacher features to guide specific student regions, enabling effective cross-level imitation.
Contrastive Knowledge Distillation (CKD)
To capture the structural relationships between different object regions, G-DetKD incorporates a contrastive learning objective. Using an InfoNCE loss and a memory queue for negative samples, CKD encourages the student to learn representations that mirror the teacher's for identical objects while remaining distinct for different ones. This helps the student understand the global context and relationships within an image.
Class-aware Localization KD
Beyond raw coordinate imitation, G-DetKD leverages the teacher's "uncertainty" in localization. The regression loss is weighted by the teacher's class probabilities, allowing the student to learn how localization precision varies across different object categories.
Experimental Results
The framework was evaluated on major benchmarks including MS COCO, Pascal VOC, and Berkeley Deep Drive (BDD100k).
Performance on COCO
G-DetKD consistently delivers substantial performance gains across various architectures. For instance, a ResNet-50 Faster R-CNN student improved from 37.4% to 44.0% AP when distilled from a powerful X101-FasterRCNN-Instaboost teacher. Notably, the framework excels in cross-architecture distillation; distilling from a two-stage Faster R-CNN to a one-stage RetinaNet student resulted in an improvement from 32.6% to 43.3% AP.
Generalization and Robustness
Tests on Pascal VOC and BDD100k confirmed the framework's ability to generalize to different data distributions and challenging environments, such as autonomous driving scenarios.
Conclusion and Future Work
G-DetKD provides a versatile solution for object detector distillation, effectively bridging the gap between heterogeneous architectures. While the method shows state-of-the-art results, future research could explore its application to even more diverse model pairs, such as distilling from Vision Transformers to traditional CNN-based detectors. Key considerations for implementation include the sensitivity of the contrastive memory queue and the necessity of careful negative sample filtering to maintain training stability.
Summary updated