Abstract
Object detection aims to localize and classify objects, but these two tasks are sensitive to different object regions. Consequently, some locations predict high-quality bounding boxes but low classification scores, and vice versa. This spatial entanglement leads to a misalignment problem. We propose a plug-in Spatial-disentangled and Task-aligned operator (SALT) to solve this. SALT predicts two task-aware point sets located in each task's sensitive regions, reassigning features to align them with the corresponding anchor point. Additionally, we introduce a Self-distillation regression (SDR) loss to transfer knowledge from refined to coarse regression stages. Our SALT-Net consistently boosts state-of-the-art dense detectors by ~2 AP on MS-COCO, achieving 53.8 AP with a Res2Net-101-DCN backbone.
Introduction: The Misalignment Problem
In dense object detection, a single feature map location typically handles both classification and regression. However, these tasks have conflicting spatial preferences. Classification focuses on salient parts (e.g., a person's head), while regression requires a global view of boundaries to be accurate. This leads to a spatial misalignment where high classification scores do not necessarily correspond to the most accurate bounding boxes. This confuses the Non-Maximum Suppression (NMS) algorithm, which relies on classification scores to rank detection quality.
Proposed Methods: SALT and SDR
To address this, we introduce SALT-Net, featuring two key components designed to disentangle these tasks spatially and align them functionally.
1. SALT (Spatial-disentangled and Task-aligned operator)
Instead of forcing one feature point to perform both tasks, SALT predicts two sets of "task-aware" points: regression-aware points located near object boundaries and classification-aware points on salient regions. It uses bilinear interpolation to reassign features from these specific regions back to the original anchor point. This ensures the classifier receives "classification features" and the regressor receives "regression features," effectively disentangling them.
2. SDR (Self-Distillation Regression) Loss
The network operates in two stages: a coarse initial prediction and a refined second prediction. The SDR loss treats the refined stage as a "teacher," transferring knowledge back to the coarse stage only when the refined prediction is demonstrably better (higher IoU). This creates a positive feedback loop that improves initial prediction quality without adding inference cost.
Key Insights
- Invariance vs. Equivariance: Classification benefits from translation invariance, whereas regression requires translation equivariance (the box must move exactly as the object moves).
- Skip Connections: While beneficial for classification, long-range skip connections can degrade regression accuracy by introducing features from different depths that interfere with precise coordinate decoding.
- Log-Space Stability: Predicting transforms in log-space for task-aware points proved more stable and accurate than direct distance predictions during training.
Experimental Results
Evaluated on the MS-COCO benchmark, SALT-Net demonstrates significant performance gains across various backbones and base detectors:
- Universal Improvement: As a plug-and-play module, SALT boosted popular detectors like FCOS and RepPoints by 1.6 to 2.3 AP.
- State-of-the-Art Performance: SALT-Net with a Res2Net-101-DCN backbone achieved 53.8 AP on the COCO test-dev set.
- Efficiency: A ResNet-50 version of SALT-Net achieves 46.1 AP, outperforming many models with significantly heavier backbones while maintaining a competitive speed-accuracy trade-off.
Conclusion
SALT-Net effectively resolves the spatial "tug-of-war" between classification and regression in dense detectors. By disentangling features and employing self-distillation, it provides a robust, plug-in improvement for modern detection frameworks. Future research may further explore optimized feature fusion strategies to mitigate the negative impact of skip connections on regression tasks.
Summary updated