ScaleKD: Distilling Scale-Aware Knowledge in Small Object Detector

Yichen Zhu, Qiqi Zhou, Ning Liu, Zhiyuan Xu, Zhicai Ou, Xiaofeng Mou, Jian Tang

Abstract

Detecting small objects remains a significant challenge in computer vision due to feature entanglement and sensitivity to localization errors. We present ScaleKD, a novel knowledge distillation (KD) framework specifically optimized for small object detection (SOD). ScaleKD introduces two key components: Scale-Decoupled Feature (SDF) distillation, which separates features by scale using varying dilation rates, and a Cross-Scale Assistant (CSA), a learnable module that refines teacher predictions via multi-scale cross-attention. Our approach significantly improves performance across multiple architectures—improving $AP_S$ by up to 5.9 points on the COCO dataset—without adding any inference-time overhead.

Introduction

Small object detection (SOD) is often considered one of the "final bosses" of computer vision. Objects with limited pixel footprints are easily obscured or "contaminated" by background noise and larger adjacent objects. While high-resolution inputs and larger models can mitigate these issues, they are often impractical for real-time deployment on edge devices.

Knowledge Distillation (KD) offers a potential solution by training a compact "student" model to mimic a high-resolution "teacher." However, standard KD methods face two major hurdles in SOD:

  1. Feature Entanglement: Standard methods treat all object scales uniformly, causing subtle small-object signals to be overwhelmed by dominant large-object features.
  2. Low Error Tolerance: Small objects are highly sensitive to localization noise. A minor bounding box offset in a teacher's prediction can be catastrophic for a tiny object, leading to "noisy" supervision that degrades student performance.

Methods

ScaleKD addresses these challenges through a specialized training pipeline that is discarded after training, ensuring no impact on inference speed.

Overview of the ScaleKD framework featuring Scale-Decoupled Feature distillation and the Cross-Scale Assistant.

Scale-Decoupled Feature (SDF) Distillation

To prevent feature entanglement, SDF employs a multi-branch convolutional block with different dilation rates (1, 2, and 3). Branches with smaller dilation focus on fine-grained details essential for small objects, while larger dilation captures the broader context needed for large objects. These branches share weights to maintain training efficiency while explicitly separating knowledge by scale.

Cross-Scale Assistant (CSA)

The CSA acts as a learnable "smart filter" between the teacher and student. Utilizing multi-scale cross-attention, it bridges the gap between the teacher’s high-resolution features and the student’s standard-resolution features. By refining the teacher's predictions before they supervise the student, the CSA ensures that the transferred "dark knowledge" is accurate and helpful, mitigating the impact of teacher noise.

Key Results

ScaleKD was evaluated on the MS COCO 2017 and VisDrone datasets. VisDrone serves as a rigorous "stress test" for SOD due to its high-altitude drone perspective.

Performance improvements of ScaleKD across various detector architectures and backbones on the COCO dataset.

The framework demonstrated significant quantitative gains across various architectures:

  • COCO Performance: For a RetinaNet (ResNet50) student, ScaleKD improved the Small Object AP ($AP_S$) from 20.0 to 25.9 (+5.9 points), with an overall AP increase of 5.4 points.
  • VisDrone Performance: ScaleKD outperformed specialized SOD distillation techniques like ZoomInNet. A RetinaNet-MobileNetV2 student achieved 26.08 AP, nearly matching models with backbones eight times heavier.
  • Versatility: Success was observed across two-stage (Faster R-CNN), anchor-based one-stage (RetinaNet), and anchor-free (FCOS, RepPoints) detectors.

Conclusion

ScaleKD highlights the necessity of scale-aware distillation for improving small object detection. By decoupling features and refining teacher supervision, it provides a robust way to boost small model performance without increasing deployment costs. While the current implementation requires a specific training "warm-up" phase and hyper-parameter tuning, its success across diverse architectures and tasks (including instance segmentation) marks a significant step forward. Future work will explore extending these scale-aware principles to 3D detection and video sequences.

Summary updated

Abstract

Despite the prominent success of general object detection, the performance and efficiency of Small Object Detection (SOD) are still unsatisfactory. Unlike existing works that struggle to balance the tradeoff between inference speed and SOD performance, in this paper, we propose a novel Scale-aware Knowledge Distillation (ScaleKD), which transfers knowledge of a complex teacher model to a compact student model. We design two novel modules to boost the quality of knowledge transfer in distillation for SOD: 1) a scale-decoupled feature distillation module that disentangled teacher's feature representation into multi-scale embedding that enables explicit feature mimicking of the student model on small objects. 2) a cross-scale assistant to refine the noisy and uninformative bounding boxes prediction student models, which can mislead the student model and impair the efficacy of knowledge distillation. A multi-scale cross-attention layer is established to capture the multi-scale semantic information to improve the student model. We conduct experiments on COCO and VisDrone datasets with diverse types of models, i.e., two-stage and one-stage detectors, to evaluate our proposed method. Our ScaleKD achieves superior performance on general detection performance and obtains spectacular improvement regarding the SOD performance.
BibTeX
@inproceedings{zhu2023scalekd,
  author = {Yichen Zhu and Qiqi Zhou and Ning Liu and Zhiyuan Xu and Zhicai Ou and Xiaofeng Mou and Jian Tang},
  title = {{ScaleKD: Distilling Scale-Aware Knowledge in Small Object Detector}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.01889},
}
Figure