Imagine an object detector processing an image of a crowded street. It spots a pedestrian. The classification network is 99% certain of the category because it clearly detects distinct textures like a jacket and a backpack. However, the bounding box regression network draws a sloppy, oversized box that extends far into the empty background. Conversely, another candidate box perfectly hugs the pedestrian's contours, but because it only captures a portion of the object's texture, its classification score is a lukewarm 65%.
During the final Non-Maximum Suppression (NMS) step, which box does the system keep? Standard pipelines rely entirely on the classification score to rank candidates. As a result, the highly confident but poorly localized box suppresses the perfectly aligned one.
This phenomenon is known as classification-localization misalignment. As identified across various domains—from 2D object detection by Lars Petersson et al. in Improving Object Localization with Fitness NMS and Bounded IoU Loss to multi-instance pose estimation by Matteo Ruggero Ronchi et al. in Benchmarking and Error Diagnosis in Multi-instance Pose Estimation—a model's confidence in what an object is frequently fails to correlate with its precision regarding where it is.
Why the Tasks Diverge
Why do deep networks struggle to synchronize these two tasks? The root cause lies in how modern detectors are trained and structured.
Independent Optimization
Classification and localization are typically treated as distinct sub-problems handled by separate network branches or heads. As Guanglu Song et al. observe in Decoupled DETR: Spatially Disentangling Localization and Classification for Improved End-to-End Object Detection, the training process rarely explicitly synchronizes the quality of the predicted category with the precision of the bounding box. Classification heads look for global context and discriminative features, while regression heads focus on exact spatial boundaries.
Static Label Assignment
During training, anchors or proposals are assigned positive or negative labels based on static spatial overlap (IoU) thresholds before the network learns to regress them. Xu Tang et al. demonstrated a striking flaw in this approach in HAMBox: Delving Into Mining High-Quality Anchors on Face Detection: a vast majority of successful detections actually originate from anchors initially labeled as background. When high-performing regressed anchors are forced into background categories due to rigid initial placement, the model loses vital training signals that could bridge the gap between classification and regression.
Solution 1: Predicting Localization Quality
To fix this, researchers introduced mechanisms that force the network to estimate its own localization quality. Instead of relying solely on classification confidence during NMS, the pipeline incorporates an explicit measure of spatial fit.
Standard NMS: [Class Confidence] ──> Rank & Suppress
IoU-Aware NMS: [Class Confidence] ┐
├──> [Combined Score] ──> Rank & Suppress
[Predicted IoU] ┘
The authors of Improving Object Localization with Fitness NMS and Bounded IoU Loss proposed Fitness NMS, which factors estimated localization quality directly into the suppression stage. Similarly, Shengkai Wu et al. designed an IoU-aware head in IoU-aware Single-stage Object Detector for Accurate Localization. By multiplying the classification score by this predicted IoU, the final ranking metric reflects both class certainty and geometric accuracy.
In crowded or densely packed environments, Eran Goldman demonstrated in Precise Detection in Densely Packed Scenes. that adding a Soft-IoU layer to predict detection quality prevents high-confidence background noise from overriding precise object boundaries.
Solution 2: Joint Optimization and Ranking
Rather than patching the network post-hoc with an extra prediction head, another line of work focuses on unifying the optimization objectives during training. The goal is to enforce a monotonic relationship: higher classification scores must correspond to better spatial alignment.
Shuai Li et al. developed a dual weighting scheme in A Dual Weighting Label Assignment Scheme for Object Detection that dynamically calculates loss weights, prioritizing anchors that excel at both classification and localization simultaneously.
In the context of visual tracking, Feng Tang et al. introduced Ranking-Based Optimization in Ranking-Based Siamese Visual Tracking. By explicitly penalizing the model when a poorly localized proposal receives a higher classification score than a well-aligned one, they successfully aligned the two tasks at zero additional computational cost during inference.
The Impact on Semi-Supervised Learning
The consequences of classification-localization misalignment are particularly severe in semi-supervised object detection (SSOD), where a "Teacher" model generates pseudo-labels to train a "Student" model.
If the Teacher relies strictly on classification confidence to filter pseudo-labels, it will inevitably pass along highly confident but spatially inaccurate bounding boxes. As Yen‐Cheng Liu et al. point out in Unbiased Teacher v2: Semi-supervised Object Detection for Anchor-free and Anchor-based Detectors, this introduces misleading gradients into the Student's regression branch, severely degrading spatial accuracy.
To combat this, modern SSOD frameworks decouple the filtering process:
- Box Jittering: Mengde Xu et al. introduced a "Soft Teacher" mechanism in End-to-End Semi-Supervised Object Detection with Soft Teacher that uses box jittering to assess localization reliability independently of class confidence. If a box's predicted coordinates remain stable under random shifts, it is deemed reliable.
- Regression Uncertainty: Similarly, Honggyu Choi et al. utilized regression uncertainty in Semi-Supervised Object Detection with Object-wise Contrastive Learning and Regression Uncertainty to guide pseudo-labeling, ensuring that only geometrically sound boxes are used to train the spatial boundaries of the detector.