G-DetKD: Towards General Distillation Framework for Object Detectors via Contrastive and Semantic-guided Feature Imitation

Lewei Yao, Renjie Pi, Hang Xu, Wei Zhang, Zhenguo Li, Tong Zhang

Introduction

Knowledge Distillation (KD) is a powerful technique for transferring knowledge from a large, complex "teacher" model to a smaller, more efficient "student" model. While highly effective in image classification, KD faces significant challenges in object detection due to the dual requirements of classification and localization. Traditional methods often rely on rigid one-to-one feature matching between the teacher and student Feature Pyramids (FPN), leading to two primary issues: semantic misalignment and architectural incompatibility. G-DetKD addresses these by introducing a flexible, general framework capable of distilling knowledge across diverse detector families.

Methodology

The G-DetKD framework consists of three primary components designed to overcome the limitations of traditional distillation:

Semantic-Guided Feature Imitation (SGFI)

Standard KD methods often fail when the teacher and student have different pyramid levels or when features at different levels share similar semantic information. SGFI employs a soft-matching mechanism that evaluates the semantic content across all pyramid levels. It automatically identifies the most relevant teacher features to guide specific student regions, enabling effective cross-level imitation.

Contrastive Knowledge Distillation (CKD)

To capture the structural relationships between different object regions, G-DetKD incorporates a contrastive learning objective. Using an InfoNCE loss and a memory queue for negative samples, CKD encourages the student to learn representations that mirror the teacher's for identical objects while remaining distinct for different ones. This helps the student understand the global context and relationships within an image.

Overview of the G-DetKD framework highlighting the Student-Teacher interaction through SGFI and CKD.

Class-aware Localization KD

Beyond raw coordinate imitation, G-DetKD leverages the teacher's "uncertainty" in localization. The regression loss is weighted by the teacher's class probabilities, allowing the student to learn how localization precision varies across different object categories.

Experimental Results

The framework was evaluated on major benchmarks including MS COCO, Pascal VOC, and Berkeley Deep Drive (BDD100k).

Performance on COCO

G-DetKD consistently delivers substantial performance gains across various architectures. For instance, a ResNet-50 Faster R-CNN student improved from 37.4% to 44.0% AP when distilled from a powerful X101-FasterRCNN-Instaboost teacher. Notably, the framework excels in cross-architecture distillation; distilling from a two-stage Faster R-CNN to a one-stage RetinaNet student resulted in an improvement from 32.6% to 43.3% AP.

Performance comparison of G-DetKD against baseline models across various architectures on the COCO dataset.

Generalization and Robustness

Tests on Pascal VOC and BDD100k confirmed the framework's ability to generalize to different data distributions and challenging environments, such as autonomous driving scenarios.

Conclusion and Future Work

G-DetKD provides a versatile solution for object detector distillation, effectively bridging the gap between heterogeneous architectures. While the method shows state-of-the-art results, future research could explore its application to even more diverse model pairs, such as distilling from Vision Transformers to traditional CNN-based detectors. Key considerations for implementation include the sensitivity of the contrastive memory queue and the necessity of careful negative sample filtering to maintain training stability.

Summary updated

Abstract

In this paper, we investigate the knowledge distillation (KD) strategy for object detection and propose an effective framework applicable to both homogeneous and heterogeneous student-teacher pairs. The conventional feature imitation paradigm introduces imitation masks to focus on informative foreground areas while excluding the background noises. However, we find that those methods fail to fully utilize the semantic information in all feature pyramid levels, which leads to inefficiency for knowledge distillation between FPN-based detectors. To this end, we propose a novel semantic-guided feature imitation technique, which automatically performs soft matching between feature pairs across all pyramid levels to provide the optimal guidance to the student. To push the envelop even further, we introduce contrastive distillation to effectively capture the information encoded in the relationship between different feature regions. Finally, we propose a generalized detection KD pipeline, which is capable of distilling both homogeneous and heterogeneous detector pairs. Our method consistently outperforms the existing detection KD techniques, and works when (1) components in the framework are used separately and in conjunction; (2) for both homogeneous and heterogenous student-teacher pairs and (3) on multiple detection benchmarks. With a powerful X101-FasterRCNN-Instaboost detector as the teacher, R50-FasterRCNN reaches 44.0% AP, R50-RetinaNet reaches 43.3% AP and R50-FCOS reaches 43.1% AP on COCO dataset.
BibTeX
@inproceedings{yao2021g,
  author = {Lewei Yao and Renjie Pi and Hang Xu and Wei Zhang and Zhenguo Li and Tong Zhang},
  title = {{G-DetKD: Towards General Distillation Framework for Object Detectors via Contrastive and Semantic-guided Feature Imitation}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.00357},
}
Figure