Improve Object Detection with Feature-based Knowledge Distillation: Towards Accurate and Efficient Detectors

Linfeng Zhang, Kaisheng Ma

Overview

Researchers from Tsinghua University have developed a novel feature-based knowledge distillation (KD) framework designed to significantly boost the performance of lightweight object detectors. By focusing the "student" model's attention on foreground objects and teaching it the global relationships between pixels, the proposed method achieves up to a 4.1 Average Precision (AP) improvement on the COCO benchmark across various detector architectures.

The Challenge with Knowledge Distillation in Object Detection

Knowledge Distillation is a highly successful technique in image classification where a large, powerful "teacher" model trains a smaller, efficient "student" model to mimic its behavior. However, applying KD to object detection has historically been challenging. The authors identify two primary reasons for this performance gap:

  1. Foreground-Background Imbalance: In a typical image, the background occupies the vast majority of pixels. Standard distillation treats every pixel with equal priority, causing the student to waste its limited capacity learning background noise rather than the crucial features of actual objects.
  2. Missing Relationships: Most distillation methods look at pixels in isolation. Object detection, however, relies heavily on the relationships between different parts of an image (e.g., a "car" is often found on a "road"). Existing KD methods fail to pass this relational "dark knowledge" from teacher to student.

Proposed Methodology

To overcome these limitations, the authors propose two novel distillation components:

Attention-Guided Distillation

Instead of using a simple binary mask or distilling everything equally, the researchers use the teacher’s own internal "attention"—both spatial and channel-wise—to create a fine-grained mask. This forces the student to prioritize learning the features of pixels that the teacher deems most important, which are typically the foreground objects.

Attention-Guided Distillation Framework

Non-Local Distillation

To capture the "big picture," the authors employ non-local modules. These modules calculate how much every pixel in a feature map relates to every other pixel. By distilling these relationship matrices, the student learns the global structure and context that the teacher has mastered.

Crucially, these modules are only utilized during the training phase. The final student model remains lightweight and introduces zero additional computational overhead during inference and deployment.

Key Insights

  • The "High AP Teacher" Advantage: In image classification, a teacher that is too accurate can sometimes hurt a student due to the "capacity gap." This paper finds the opposite is true for object detection: a stronger teacher (higher AP) almost always results in a better student. Because detection is a harder task, students seem to need all the high-quality guidance they can get.
  • Model Agnosticism: Because the distillation happens at the feature map level (the "backbone") rather than the specific detection "heads," the method is highly versatile. It was successfully tested on one-stage (RetinaNet), two-stage (Faster R-CNN), anchor-based, and anchor-free (RepPoints) detectors.
  • Small Object Mastery: Qualitative analysis showed that the distilled students became significantly better at detecting small objects and were less likely to produce duplicate "double-box" errors compared to baseline models.

Results and Performance

The primary evaluation was conducted on MS COCO 2017, the gold standard for object detection containing over 120k images. Supplementary results on the Cityscapes dataset also demonstrated the method's effectiveness in autonomous driving contexts.

The performance gains are consistent and substantial across various architectures:

Performance improvements across various baseline models on MS COCO 2017
  • Faster R-CNN (ResNet101): Performance jumped from 39.8 AP to 43.9 AP (a +4.1 improvement).
  • Average Gains: The method provided an average boost of 2.9 AP for two-stage and one-stage models, and 2.2 AP for anchor-free models.
  • Efficiency: A student model with a ResNet50 backbone trained using this method actually outperformed a much larger ResNet101 baseline model while being significantly faster.
  • Segmentation: The benefits also extended to instance segmentation, with Mask R-CNN seeing a 2.0 AP improvement in mask accuracy.

Limitations and Future Directions

While the results are impressive, the paper notes a few areas for consideration:

  • Training Complexity: The non-local distillation requires calculating large relationship matrices, which increases the memory and time required during the training phase.
  • Hyperparameter Tuning: The method introduces four new hyperparameters to balance the different loss components. While a sensitivity study showed the method is relatively robust, finding the optimal balance for a brand-new architecture might require some tuning effort.

The authors suggest that exploring how these techniques apply to other complex tasks beyond detection—such as video-based tracking or 3D perception—could be a promising next step for future research.

Summary updated

Abstract

Knowledge distillation, in which a student model is trained to mimic a teacher model, has been proved as an effective technique for model compression and model accuracy boosting. However, most knowledge distillation methods, designed for image classification, have failed on more challenging tasks, such as object detection. In this paper, we suggest that the failure of knowledge distillation on object detection is mainly caused by two reasons: (1) the imbalance between pixels of foreground and background and (2) lack of distillation on the relation between different pixels. Observing the above reasons, we propose attention-guided distillation and non-local distillation to address the two problems, respectively. Attention-guided distillation is proposed to find the crucial pixels of foreground objects with attention mechanism and then make the students take more effort to learn their features. Non-local distillation is proposed to enable students to learn not only the feature of an individual pixel but also the relation between different pixels captured by non-local modules. Experiments show that our methods achieve excellent AP improvements on both one-stage and two-stage, both anchor-based and anchor-free detectors. For example, Faster RCNN (ResNet101 backbone) with our distillation achieves 43.9 AP on COCO2017, which is 4.1 higher than the baseline. Codes have been released on Github.
BibTeX
@inproceedings{zhang2021improve,
  author = {Linfeng Zhang and Kaisheng Ma},
  title = {{Improve Object Detection with Feature-based Knowledge Distillation: Towards Accurate and Efficient Detectors}},
  booktitle = {International Conference on Learning Representations},
  year = {2021},
}
Figure