CrossKD: Cross-Head Knowledge Distillation for Dense Object Detection

JiaBao Wang, Yuming Chen, ZhaoHui Zheng, Xiang Li, Ming-Ming Cheng, Qibin Hou

TL;DR

CrossKD is a novel knowledge distillation (KD) framework that significantly boosts the performance of small object detectors by feeding their internal features into a larger "teacher" model's detection head. This "cross-head" approach resolves the long-standing problem of "target conflict," where a student model struggles to simultaneously satisfy the demands of the ground-truth labels and the teacher's predictions.

What Problem Does This Paper Address?

In the world of deep learning, Knowledge Distillation (KD) is the process of training a small, efficient "student" model to mimic a large, high-performing "teacher." In object detection, this usually happens in two ways: feature imitation (matching internal feature maps) or prediction mimicking (matching final outputs).

The authors identify a major bottleneck in prediction mimicking: Target Conflict.

Modern detectors use dynamic label assignment, meaning the "ground truth" for a specific pixel might change depending on the model's architecture. When a student tries to mimic a teacher's predictions, it often receives contradictory signals. For example, the ground-truth label might tell the student a pixel is "background," while the teacher’s prediction (which the student is told to mimic) says it’s "foreground." This tug-of-war confuses the student and limits its performance, often making prediction mimicking less effective than simply matching internal features.

Methods

The authors propose CrossKD (Cross-Head Knowledge Distillation) to bypass this conflict.

Instead of forcing the student’s final output to match the teacher’s final output, CrossKD takes an intermediate feature map from the student’s detection head and "delivers" it to the teacher’s detection head.

  1. The Hand-off: An intermediate feature $f^s_i$ from the student is plugged into the corresponding layer of the teacher's head.
  2. Cross-Head Prediction: The teacher’s head (which is frozen) processes this student feature to produce a "cross-head prediction" ($\hat{p}^s$).
  3. The Loss: The distillation loss is calculated between this cross-head prediction ($\hat{p}^s$) and the teacher’s original prediction ($p^t$).

By doing this, the student's early layers learn to produce features compatible with the teacher's high-level logic, while the student's final layers remain free to focus entirely on the ground-truth labels without being pulled in two directions.

Datasets

The researchers evaluated CrossKD on the MS COCO dataset, the gold standard for object detection. They used the trainval135k set for training and the minival set for validation, measuring success via Average Precision (AP).

Key Insights and Ideas

  • Relieving the "Tug-of-War": The core "aha" moment is that the student's own detection head is the site of the conflict. By moving the distillation task to the teacher's head, the student's head is "relieved" from receiving contradictory supervision.
  • Task-Oriented Gradients: Visualization of the gradients shows that CrossKD focuses the student's attention on semantically meaningful areas (actual objects). In contrast, standard feature imitation methods often produce "noisy" gradients that impact the entire feature map, including irrelevant background.
  • Simplicity is Key: Unlike previous attempts to bridge heterogeneous models (like HEAD), CrossKD doesn't require complex "assistant" heads or complicated mapping functions. It simply reuses the teacher's existing architecture.

Key Results

CrossKD achieves state-of-the-art results across various detector architectures:

  • Massive Gains: On a GFL ResNet-50 model, CrossKD boosted the Average Precision (AP) from 40.2 to 43.7 (+3.5 AP), outperforming all existing KD methods like PKD and LD.
  • Surpassing the Teacher: In some cases, the student distilled with CrossKD actually outperformed its own teacher. For instance, a RetinaNet student reached 39.7 AP, surpassing its ResNet-101 teacher (38.9 AP).
  • Heterogeneous Success: The method works even when the teacher and student have completely different architectures. Distilling a transformer-based Swin-T teacher into a ResNet-50 student yielded a 1.5 AP improvement over the baseline.
  • Orthogonality: CrossKD can be combined with feature-based methods. When paired with PKD, it reached a peak of 43.9 AP on GFL.

Limitations and Future Work

While CrossKD is highly effective for dense (one-stage) detectors, the authors note that its application to more complex architectures like two-stage detectors (Faster R-CNN) or query-based detectors (DETR) requires specific adaptations (e.g., passing encoder features into the teacher's decoder).

The authors suggest that future research could extend this "cross-head" philosophy to other domains, such as 3D object detection, where target conflicts and architectural gaps between models are even more pronounced.

Summary updated

Abstract

Knowledge Distillation (KD) has been validated as an effective model compression technique for learning compact object detectors. Existing state-of-the-art KD methods for object detection are mostly based on feature imitation, which is generally observed to be better than prediction mimicking. In this paper, we show that the inconsistency of the optimization objectives between the ground-truth signals and distillation targets is the key reason for the inefficiency of prediction mimicking. To alleviate this issue, we present a simple yet effective distillation scheme, termed CrossKD, which delivers the intermediate features of the student's detection head to the teacher's detection head. The resulting cross-head predictions are then forced to mimic the teacher's predictions. Such a distillation manner relieves the student's head from receiving contradictory supervision signals from the ground-truth annotations and the teacher's predictions, greatly improving the student's detection performance. On MS COCO, with only prediction mimicking losses applied, our CrossKD boosts the average precision of GFL ResNet-50 with 1x training schedule from 40.2 to 43.7, outperforming all existing KD methods for object detection. Code is available at https://github.com/jbwang1997/CrossKD.
BibTeX
@inproceedings{wang2024crosskd,
  author = {JiaBao Wang and Yuming Chen and ZhaoHui Zheng and Xiang Li and Ming-Ming Cheng and Qibin Hou},
  title = {{CrossKD: Cross-Head Knowledge Distillation for Dense Object Detection}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.01563},
}
Figure