Confidence-driven Bounding Box Localization for Small Object Detection

Huixin Sun, Baochang Zhang, Yanjing Li, Xianbin Cao

Abstract

Small object detection (SOD) remains a significant challenge in computer vision, typically attributed to low resolution and background clutter. This research identifies a more fundamental mathematical hurdle: gradient instability caused by the sharp curvature of standard regression losses. We present Uncertainty-Aware Gradient Stabilization (UGS), a framework that replaces traditional bounding box regression with a stabilized classification-based approach and an uncertainty-guided refinement module. UGS significantly reduces gradient variance and achieves state-of-the-art performance across multiple benchmarks.

The Curvature Culprit in Small Object Detection

While low pixel counts are a known issue, this paper uncovers a deeper optimization problem. Through Hessian matrix analysis, the authors demonstrate that standard localization losses (such as $L_2$ or IoU loss) exhibit much sharper curvature for small objects than for large ones. In optimization terms, the "loss landscape" for small objects is characterized by steep cliffs and narrow valleys.

This leads to "salient" gradients—unstable, high-magnitude signals that cause the model to oscillate or diverge during training. In contrast, larger objects provide smoother learning signals, allowing for stable convergence. Quantitative analysis shows that small objects suffer from significantly higher gradient variance compared to their larger counterparts.

Gradient variance across object scales. Small objects exhibit high variance ("salient" gradients) compared to larger objects which reach "convergence" smoothly.

Uncertainty-Aware Gradient Stabilization (UGS)

The proposed UGS framework tackles this instability through three primary components designed to provide a "confidence-driven" learning signal.

1. Classification-based Localization with IN Labels

Instead of asking the model to regress a continuous value (which can produce unbounded gradients), UGS quantizes the localization space into discrete intervals. They employ Interval Non-uniform (IN) Labels, using a grid that is denser near zero (exponential spacing). This allows for finer precision where it matters most for small objects. By treating localization as a classification task, the resulting gradients are naturally bounded and more stable.

The UGS localization branch transforms regression into a classification task using non-uniform intervals and two-hot soft targets to stabilize training.

2. Uncertainty Minimization (UM) Loss

To further stabilize training, an entropy-based loss is introduced. This explicitly penalizes high-variance, "flat" probability distributions, forcing the model to produce more certain and "sharp" spatial predictions.

3. Uncertainty-guided Refinement (UR) Module

Inspired by adversarial training, the UR module identifies regions of high uncertainty and introduces small perturbations into the feature maps. This forces the network to learn robust, discriminative features that can withstand noise, which is critical in the low-signal-to-noise environments typical of SOD.

Key Results and Performance

The UGS framework consistently improved every baseline detector tested:

  • VisDrone: Boosted the state-of-the-art DINO-5scale by 2.6 AP and improved Faster R-CNN by 2.9 AP.
  • SODA-A: Improved Rotated RetinaNet by 4.5 AP, demonstrating effectiveness for oriented bounding boxes.
  • COCO 2017: Improved Faster R-CNN (ResNet-50) by 1.5 AP overall and 1.4 AP specifically for small objects ($AP_s$).
  • Stability: Quantitative analysis confirmed that UGS reduces gradient variance by 2.5x to 2.9x compared to standard Smooth-$L_1$ or $L_2$ losses for small objects.

Conclusion

By shifting the perspective from a data-centric problem to an optimization-centric one, UGS provides a robust solution for small object detection. The transition from continuous regression to bounded, classification-based localization effectively tames the unstable gradients inherent in SOD, leading to significant gains in both training stability and final detection accuracy.

Summary updated

BibTeX
@misc{sun2023confidence,
  author = {Huixin Sun and Baochang Zhang and Yanjing Li and Xianbin Cao},
  title = {{Confidence-driven Bounding Box Localization for Small Object Detection}},
  year = {2023},
  eprint = {2303.01803},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2303.01803},
}
Figure