Abstract
Deep learning-based dense object detectors have achieved great success in the past few years and have been applied to numerous multimedia applications such as video understanding. However, the current training pipeline for dense detectors is compromised to lots of conjunctions that may not hold. In this paper, we investigate three such important conjunctions: 1) only samples assigned as positive in classification head are used to train the regression head; 2) classification and regression share the same input feature and computational fields defined by the parallel head architecture; and 3) samples distributed in different feature pyramid layers are treated equally when computing the loss. We first carry out a series of pilot experiments to show disentangling such conjunctions can lead to persistent performance improvement. Then, based on these findings, we propose Disentangled Dense Object Detector (DDOD), in which simple and effective disentanglement mechanisms are designed and integrated into the current state-of-the-art dense object detectors. Extensive experiments on MS COCO benchmark show that our approach can lead to 2.0 mAP, 2.4 mAP and 2.2 mAP absolute improvements on RetinaNet, FCOS, and ATSS baselines with negligible extra overhead. Notably, our best model reaches 55.0 mAP on the COCO test-dev set and 93.5 AP on the hard subset of WIDER FACE, achieving new state-of-the-art performance on these two competitive benchmarks.
Introduction
Single-stage (dense) object detectors like RetinaNet or FCOS are popular because they are fast and simple. However, their training pipelines often rely on "conjunctions"—assumptions that classification and bounding-box regression should be treated identically. The authors argue that this "one-size-fits-all" approach is a bottleneck, identifying three areas where entanglement hurts performance: label assignment, spatial features, and scale-wise supervision.
Methods
The authors propose DDOD, a training framework that introduces three specific disentanglement mechanisms:
- Label Assignment Disentanglement: Instead of a shared set of positive samples, DDOD uses a cost-based assigner that independently selects the best candidates for classification and regression. It uses a hyperparameter to tune the balance between classification scores and localization accuracy for each branch.
- Spatial Feature Disentanglement: Standard convolutions in detection heads are replaced with Deformable Convolutions (DCN). This allows the model to adaptively "shift" its gaze: the classification head learns to focus on discriminative semantic parts, while the regression head focuses on geometric boundaries.
- Pyramid Supervision Disentanglement: To fix scale imbalance in Feature Pyramid Networks (FPN), a FPN Hierarchical Loss re-weights the loss for each layer based on the number of positive samples it contains, ensuring sparse signals from large objects aren't drowned out by small objects.
Key Insights and Ideas
A core motivation for DDOD is the observation that classification and regression tasks have diverging needs. Pilot experiments showed that the optimal IoU threshold for classification is higher (0.5) than for regression (0.4), proving that the regression branch benefits from seeing "looser" samples that the classification branch would consider background.
Furthermore, using GradCAM, the authors visualized that the classification branch naturally wants to look at the center of objects, while the regression branch wants to look at the edges. Forcing them to share the exact same pixels via standard convolutions creates a "tug-of-war" that limits both.
Results and Performance
DDOD consistently improved every baseline it was tested against on the MS COCO and WIDER FACE benchmarks:
- RetinaNet: +2.0 mAP improvement.
- FCOS: +2.4 mAP improvement.
- ATSS: +2.2 mAP improvement.
- State-of-the-Art: The best model reached 55.0 mAP on the COCO test-dev set and 93.5 AP on the WIDER FACE "Hard" subset.
Crucially, because these changes primarily affect the training phase, the performance boost comes with negligible extra overhead during inference, making it highly practical for real-world deployment.
Conclusion
DDOD demonstrates that breaking traditional conjunctions in dense detector training leads to significant performance boosts. While currently focused on single-stage detectors, the authors note that the problem of task entanglement likely exists in "sparse" (two-stage) detectors like Faster R-CNN as well, suggesting a promising direction for future research into multi-stage detection pipelines.
Summary updated