Rethinking the Misalignment Problem in Dense Object Detection

Yang Yang, Min Li, Bo Meng, Junxing Ren, Degang Sun, Zihao Huang

Abstract

Object detection aims to localize and classify objects, but these two tasks are sensitive to different object regions. Consequently, some locations predict high-quality bounding boxes but low classification scores, and vice versa. This spatial entanglement leads to a misalignment problem. We propose a plug-in Spatial-disentangled and Task-aligned operator (SALT) to solve this. SALT predicts two task-aware point sets located in each task's sensitive regions, reassigning features to align them with the corresponding anchor point. Additionally, we introduce a Self-distillation regression (SDR) loss to transfer knowledge from refined to coarse regression stages. Our SALT-Net consistently boosts state-of-the-art dense detectors by ~2 AP on MS-COCO, achieving 53.8 AP with a Res2Net-101-DCN backbone.

Introduction: The Misalignment Problem

In dense object detection, a single feature map location typically handles both classification and regression. However, these tasks have conflicting spatial preferences. Classification focuses on salient parts (e.g., a person's head), while regression requires a global view of boundaries to be accurate. This leads to a spatial misalignment where high classification scores do not necessarily correspond to the most accurate bounding boxes. This confuses the Non-Maximum Suppression (NMS) algorithm, which relies on classification scores to rank detection quality.

Visualizing the misalignment problem: (a) shows detection results where high classification (Cls) doesn't match high IoU. (b) and (c) show the spatial distribution of these scores.

Proposed Methods: SALT and SDR

To address this, we introduce SALT-Net, featuring two key components designed to disentangle these tasks spatially and align them functionally.

1. SALT (Spatial-disentangled and Task-aligned operator)

Instead of forcing one feature point to perform both tasks, SALT predicts two sets of "task-aware" points: regression-aware points located near object boundaries and classification-aware points on salient regions. It uses bilinear interpolation to reassign features from these specific regions back to the original anchor point. This ensures the classifier receives "classification features" and the regressor receives "regression features," effectively disentangling them.

Comparison between (a) Coupled heads, where tasks share features, and (b) Decoupled heads, which SALT facilitates by spatially disentangling task-specific features.

2. SDR (Self-Distillation Regression) Loss

The network operates in two stages: a coarse initial prediction and a refined second prediction. The SDR loss treats the refined stage as a "teacher," transferring knowledge back to the coarse stage only when the refined prediction is demonstrably better (higher IoU). This creates a positive feedback loop that improves initial prediction quality without adding inference cost.

Key Insights

  • Invariance vs. Equivariance: Classification benefits from translation invariance, whereas regression requires translation equivariance (the box must move exactly as the object moves).
  • Skip Connections: While beneficial for classification, long-range skip connections can degrade regression accuracy by introducing features from different depths that interfere with precise coordinate decoding.
  • Log-Space Stability: Predicting transforms in log-space for task-aware points proved more stable and accurate than direct distance predictions during training.

Experimental Results

Evaluated on the MS-COCO benchmark, SALT-Net demonstrates significant performance gains across various backbones and base detectors:

  • Universal Improvement: As a plug-and-play module, SALT boosted popular detectors like FCOS and RepPoints by 1.6 to 2.3 AP.
  • State-of-the-Art Performance: SALT-Net with a Res2Net-101-DCN backbone achieved 53.8 AP on the COCO test-dev set.
  • Efficiency: A ResNet-50 version of SALT-Net achieves 46.1 AP, outperforming many models with significantly heavier backbones while maintaining a competitive speed-accuracy trade-off.

Conclusion

SALT-Net effectively resolves the spatial "tug-of-war" between classification and regression in dense detectors. By disentangling features and employing self-distillation, it provides a robust, plug-in improvement for modern detection frameworks. Future research may further explore optimized feature fusion strategies to mitigate the negative impact of skip connections on regression tasks.

Summary updated

Abstract

Object detection aims to localize and classify the objects in a given image, and these two tasks are sensitive to different object regions. Therefore, some locations predict high-quality bounding boxes but low classification scores, and some locations are quite the opposite. A misalignment exists between the two tasks, and their features are spatially entangled. In order to solve the misalignment problem, we propose a plug-in Spatial-disentangled and Task-aligned operator (SALT). By predicting two task-aware point sets that are located in each task's sensitive regions, SALT can reassign features from those regions and align them to the corresponding anchor point. Therefore, features for the two tasks are spatially aligned and disentangled. To minimize the difference between the two regression stages, we propose a Self-distillation regression (SDR) loss that can transfer knowledge from the refined regression results to the coarse regression results. On the basis of SALT and SDR loss, we propose SALT-Net, which explicitly exploits task-aligned point-set features for accurate detection results. Extensive experiments on the MS-COCO dataset show that our proposed methods can consistently boost different state-of-the-art dense detectors by $\sim$2 AP. Notably, SALT-Net with Res2Net-101-DCN backbone achieves 53.8 AP on the MS-COCO test-dev.
BibTeX
@misc{yang2021rethinking,
  author = {Yang Yang and Min Li and Bo Meng and Junxing Ren and Degang Sun and Zihao Huang},
  title = {{Rethinking the Misalignment Problem in Dense Object Detection}},
  year = {2021},
  eprint = {2108.12176},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2108.12176},
}
Figure