In modern object detection pipelines, a model typically processes an image through two distinct, parallel pathways: a classification branch that determines what an object is, and a localization branch that determines where it is. During inference, thousands of candidate bounding boxes are generated, and a post-processing step called Non-Maximum Suppression (NMS) filters them down to the final detections.
Crucially, standard NMS relies almost entirely on the classification confidence score to rank and filter these candidates. This architectural separation introduces a fundamental flaw known as prediction inconsistency. As noted by Keyang Wang et al. in Reconcile Prediction Consistency for Balanced Object Detection, a bounding box can receive an exceptionally high classification confidence score despite having poor spatial alignment with the actual object. Conversely, a perfectly bounded box might be discarded because its classification score is slightly lower. This decoupling limits the reliability of object detectors, particularly in precision-sensitive or crowded environments where accurate spatial positioning is just as critical as correct categorization (Kemal Oksuz et al. in On Calibration of Object Detectors: Pitfalls, Evaluation and Baselines).
Predicting Spatial Quality Explicitly
To bridge this gap, a prominent strategy involves teaching the network to evaluate its own spatial accuracy. Instead of assuming that classification confidence serves as a reliable proxy for localization quality, researchers introduce dedicated regression branches to estimate a geometric metric—most commonly the Intersection over Union (IoU) between the predicted box and the ground truth.
For example, in Mask Scoring R-CNN, Zhaojin Huang et al. added a specialized MaskIoU head to explicitly predict the pixel-level overlap of instance masks. Similarly, Eran Goldman et al. introduced a Soft-IoU layer in Precise Detection in Densely Packed Scenes. to assess bounding box alignment independently of class confidence.
Once the network can predict its own localization quality, the final ranking score is computed as a composite metric—typically the product of the classification probability and the predicted spatial quality score:
$$\text{Score} = P(\text{Class}) \times \text{IoU}_{\text{predicted}}$$
This multiplicative fusion ensures that a detection can only achieve a high rank if the model is highly certain about both its category assignment and its geometric fidelity.
Unifying Tasks via Joint Supervision
While adding an extra prediction head helps at inference time, it does not fundamentally change the fact that the classification head is trained on discrete binary targets (e.g., object vs. background). To achieve deeper alignment, frameworks modify the training loss functions to supervise the classification branch using continuous localization metrics.
In Generalized Focal Loss: Learning Qualified and Distributed Bounding Boxes for Dense Object Detection, Xiang Li et al. replaced the traditional binary ground-truth targets for the positive class with continuous IoU scores. By forcing the classification head to predict the IoU directly during training, the model unifies classification and localization quality into a single representation. This prevents background regions or poorly aligned proposals from receiving high scores, directly optimizing the network for the ranking metrics used during inference.
Other approaches use mutual supervision or feedback loops. For instance, the framework proposed by Keyang Wang et al. forces the classification and localization branches to supervise each other dynamically, ensuring that the optimization paths of both tasks remain synchronized.
Localization-Aware Post-Processing and Calibration
Aligning confidence and localization also requires rethinking how models filter predictions during post-processing. Standard NMS can inadvertently suppress highly accurate bounding boxes if a nearby, poorly aligned box happens to have a higher classification score.
To resolve this, Lars Petersson et al. developed Fitness NMS in Improving Object Localization with Fitness NMS and Bounded IoU Loss, which incorporates the estimated localization quality directly into the suppression ranking process. This modification ensures that the detection pipeline prioritizes candidates that demonstrate both high semantic certainty and precise spatial fit.
Furthermore, this alignment is essential for the calibration of object detectors. As argued by Aleksei Tiulpin et al. in Beyond Classification: Definition and Density-based Estimation of Calibration in Object Detection, a well-calibrated detector must output confidence scores that reflect the true probability of a detection being spatially correct. By mapping model confidence directly to predicted IoU values (as explored by Kemal Oksuz et al. in MoCaE: Mixture of Calibrated Experts Significantly Improves Object Detection), the model's confidence becomes a reliable metric for real-world deployment, preventing the overconfident hallucinations common in standard architectures.