Cosine similarity-guided knowledge distillation for robust object detectors

Sangwoo Park, Dong‐Goo Kang, Joonki PaikScientific 2024 · DOI

Abstract

This paper presents a Cosine Similarity-Based Knowledge Distillation (CSKD) for robust, lightweight object detectors. Knowledge Distillation (KD) has been effective in enhancing the performance of compact models in image classification by leveraging deep CNN models. However, the complex and multifaceted nature of object detection, characterized by its modular design and multitasking requirements, poses significant challenges for traditional KD techniques. These challenges are further compounded by the conventional reliance on the Mean Squared Error (MSE) loss function and the limited application of enhanced feature representations to the training phase. Addressing these limitations, the proposed CSKD method combines cosine similarity guidance with MSE loss to facilitate a more effective knowledge transfer from the teacher model to the student model. This is achieved by distilling both intermediate features and prediction outputs, aided by an assistant prediction branch designed to learn directly from the teacher's predictions. This dual-faceted distillation strategy enables the student model to better mimic the teacher model's behavior, leading to improved performance. The proposed method demonstrates versatility and robustness across various object detector architectures without the need for additional feature enhancement layers during training. Notably, employing ResNet-50 as the teacher model and ResNet-18 as the student model, we achieve new benchmarks in KD for object detection across several architectures, including Faster-RCNN, RetinaNet, FCOS, and GFL, with respective mAP scores of 36.6, 35.2, 35.9, and 38.9. These results highlight the effectiveness of CSKD in advancing the state-of-the-art in KD for object detection, offering a compelling solution to the challenges previously faced by traditional KD methods in this domain.

Introduction

In computer vision, a persistent trade-off exists between accuracy and efficiency. Deep, complex models (teachers) achieve high accuracy but are often too computationally expensive for edge devices. Conversely, compact models (students) offer the necessary speed but frequently suffer from reduced performance. Knowledge Distillation (KD) aims to bridge this gap by training a student model to mimic the behavior of a high-performing teacher.

However, applying KD to object detection is significantly more challenging than image classification. Detection models are modular—comprising a backbone, neck, and head—and must simultaneously perform classification and localization. Traditional KD methods often rely on Mean Squared Error (MSE) loss, which forces the student to match the teacher's feature values exactly. This approach can be overly restrictive when a large "capacity gap" exists between the teacher and student. The authors argue that focusing on feature "direction" rather than raw numerical values provides a more achievable and effective optimization goal for compact models.

Methodology

The CSKD framework introduces a two-pronged approach to enhance knowledge transfer:

  1. Cosine Similarity (CS)-Guided Feature Distillation Loss: Instead of relying solely on MSE, the method incorporates Cosine Similarity to measure the directional alignment between teacher and student feature vectors. This similarity is used to generate a "guidance map" that highlights regions of significant discrepancy, directing the student's attention to critical features.
  2. Structural Distillation with Assistant Branches:
    • Assistant Prediction Branch (APB): An additional prediction layer is added to the student model during training. This branch learns exclusively from the teacher’s outputs, allowing the student's head module to absorb the teacher's logic more directly.
  3. Non-Learnable Layers (NLL): Fixed operations such as ROI (Region of Interest) pooling and binary masking are utilized to help the model distinguish foreground objects from background noise. These operations ensure that distillation focuses on informative regions without adding permanent complexity to the student model.

Importantly, these auxiliary components are used only during the training phase; the final deployed student model remains as lightweight as the baseline architecture.

Experimental Setup and Results

The researchers evaluated CSKD across major benchmarks, including COCO, Pascal VOC, and a specialized COCO-Traffic subset. The method was tested across diverse architectures: two-stage (Faster-RCNN), anchor-based one-stage (RetinaNet, GFL), and anchor-free (FCOS) detectors.

Key Performance Gains

Using a ResNet-50 teacher and a ResNet-18 student, CSKD consistently outperformed existing state-of-the-art distillation methods like FGD, GID, and PKD. Notable improvements in Mean Average Precision (mAP) on the COCO dataset include:

  • RetinaNet: Improved from 31.7 to 35.2 (+3.5 mAP).
  • FCOS: Improved from 32.5 to 35.9 (+3.4 mAP).
  • GFL: Achieved a benchmark score of 38.9 mAP.
Visual comparison of feature maps from the Teacher, Student, and CSKD-trained models, alongside the generated Spatial & Channel Guidance Map.

Discussion and Insights

A primary insight of this work is that for a student model with limited capacity, mimicking the directional vector of the teacher's features is more critical than matching exact magnitudes. This "relaxed" constraint facilitates better convergence. Furthermore, by utilizing binary masking and ROI pooling, CSKD effectively addresses the "background problem" in object detection, ensuring that the distillation process is not diluted by uninformative image regions.

Conclusion and Future Work

While CSKD demonstrates superior performance and robustness, it does involve increased computational costs during the training phase due to the calculation of guidance maps. Future research will focus on optimizing the training efficiency and extending the CSKD framework to 3D object detection, which is vital for autonomous driving and robotics applications.

Summary updated

Abstract

This paper presents a Cosine Similarity-Based Knowledge Distillation (CSKD) for robust, lightweight object detectors. Knowledge Distillation (KD) has been effective in enhancing the performance of compact models in image classification by leveraging deep CNN models. However, the complex and multifaceted nature of object detection, characterized by its modular design and multitasking requirements, poses significant challenges for traditional KD techniques. These challenges are further compounded by the conventional reliance on the Mean Squared Error (MSE) loss function and the limited application of enhanced feature representations to the training phase. Addressing these limitations, the proposed CSKD method combines cosine similarity guidance with MSE loss to facilitate a more effective knowledge transfer from the teacher model to the student model. This is achieved by distilling both intermediate features and prediction outputs, aided by an assistant prediction branch designed to learn directly from the teacher's predictions. This dual-faceted distillation strategy enables the student model to better mimic the teacher model's behavior, leading to improved performance. The proposed method demonstrates versatility and robustness across various object detector architectures without the need for additional feature enhancement layers during training. Notably, employing ResNet-50 as the teacher model and ResNet-18 as the student model, we achieve new benchmarks in KD for object detection across several architectures, including Faster-RCNN, RetinaNet, FCOS, and GFL, with respective mAP scores of 36.6, 35.2, 35.9, and 38.9. These results highlights the effectiveness of CSKD in advancing the state-of-the-art in KD for object detection, offering a compelling solution to the challenges previously faced by traditional KD methods in this domain. The code of the proposed CSKD is available at https://github.com/swkdn16/CSKD .
BibTeX
@inproceedings{park2024cosine,
  author = {Sangwoo Park and Dong‐Goo Kang and Joonki Paik},
  title = {{Cosine similarity-guided knowledge distillation for robust object detectors}},
  booktitle = {Scientific},
  year = {2024},
  doi = {10.1038/s41598-024-69813-6},
}
Figure