Abstract
Detecting small objects remains a significant challenge in computer vision due to feature entanglement and sensitivity to localization errors. We present ScaleKD, a novel knowledge distillation (KD) framework specifically optimized for small object detection (SOD). ScaleKD introduces two key components: Scale-Decoupled Feature (SDF) distillation, which separates features by scale using varying dilation rates, and a Cross-Scale Assistant (CSA), a learnable module that refines teacher predictions via multi-scale cross-attention. Our approach significantly improves performance across multiple architectures—improving $AP_S$ by up to 5.9 points on the COCO dataset—without adding any inference-time overhead.
Introduction
Small object detection (SOD) is often considered one of the "final bosses" of computer vision. Objects with limited pixel footprints are easily obscured or "contaminated" by background noise and larger adjacent objects. While high-resolution inputs and larger models can mitigate these issues, they are often impractical for real-time deployment on edge devices.
Knowledge Distillation (KD) offers a potential solution by training a compact "student" model to mimic a high-resolution "teacher." However, standard KD methods face two major hurdles in SOD:
- Feature Entanglement: Standard methods treat all object scales uniformly, causing subtle small-object signals to be overwhelmed by dominant large-object features.
- Low Error Tolerance: Small objects are highly sensitive to localization noise. A minor bounding box offset in a teacher's prediction can be catastrophic for a tiny object, leading to "noisy" supervision that degrades student performance.
Methods
ScaleKD addresses these challenges through a specialized training pipeline that is discarded after training, ensuring no impact on inference speed.
Scale-Decoupled Feature (SDF) Distillation
To prevent feature entanglement, SDF employs a multi-branch convolutional block with different dilation rates (1, 2, and 3). Branches with smaller dilation focus on fine-grained details essential for small objects, while larger dilation captures the broader context needed for large objects. These branches share weights to maintain training efficiency while explicitly separating knowledge by scale.
Cross-Scale Assistant (CSA)
The CSA acts as a learnable "smart filter" between the teacher and student. Utilizing multi-scale cross-attention, it bridges the gap between the teacher’s high-resolution features and the student’s standard-resolution features. By refining the teacher's predictions before they supervise the student, the CSA ensures that the transferred "dark knowledge" is accurate and helpful, mitigating the impact of teacher noise.
Key Results
ScaleKD was evaluated on the MS COCO 2017 and VisDrone datasets. VisDrone serves as a rigorous "stress test" for SOD due to its high-altitude drone perspective.
The framework demonstrated significant quantitative gains across various architectures:
- COCO Performance: For a RetinaNet (ResNet50) student, ScaleKD improved the Small Object AP ($AP_S$) from 20.0 to 25.9 (+5.9 points), with an overall AP increase of 5.4 points.
- VisDrone Performance: ScaleKD outperformed specialized SOD distillation techniques like ZoomInNet. A RetinaNet-MobileNetV2 student achieved 26.08 AP, nearly matching models with backbones eight times heavier.
- Versatility: Success was observed across two-stage (Faster R-CNN), anchor-based one-stage (RetinaNet), and anchor-free (FCOS, RepPoints) detectors.
Conclusion
ScaleKD highlights the necessity of scale-aware distillation for improving small object detection. By decoupling features and refining teacher supervision, it provides a robust way to boost small model performance without increasing deployment costs. While the current implementation requires a specific training "warm-up" phase and hyper-parameter tuning, its success across diverse architectures and tasks (including instance segmentation) marks a significant step forward. Future work will explore extending these scale-aware principles to 3D detection and video sequences.
Summary updated