In modern object detection, a model is typically asked to do two things at once: identify what an object is (classification) and determine where it is (localization). While these tasks are performed by the same network, they are often handled by separate "branches" or "heads." This architectural split creates a fundamental problem: the model’s confidence in its classification often has very little to do with the actual precision of its bounding box.
This discrepancy, often called the confidence-localization gap, is a primary bottleneck in detection performance. When a detector assigns a high confidence score to a poorly aligned box, it doesn't just produce a sloppy result—it actively sabotages the rest of the pipeline.
The Failure of Post-Processing
The most immediate victim of this misalignment is Non-Maximum Suppression (NMS). NMS is the standard heuristic used to clean up redundant detections by keeping the candidate with the highest score and discarding others that overlap with it.
As Lars Petersson and Lachlan Tychsen-Smith point out in Improving Object Localization with Fitness NMS and Bounded IoU Loss, NMS assumes that classification confidence is a reliable proxy for localization quality. If a "confident" but shifted box overlaps with a "less confident" but perfectly centered box, NMS will discard the accurate one. This leads to a sharp drop in Average Precision (AP) at high overlap thresholds, where spatial precision is strictly enforced.
To fix this, researchers have moved toward "IoU-aware" ranking. Instead of relying solely on a softmax score, methods like Fitness NMS and the work of Shengkai Wu in IoU-aware Single-stage Object Detector introduce a predicted localization quality metric. By multiplying the classification score by the predicted Intersection-over-Union (IoU), the model can re-rank candidates so that the most spatially accurate boxes rise to the top.
Bridging the Training-Inference Gap
The root of the misalignment often lies in how models are trained. Standard detection losses treat all "positive" samples—any anchor box that overlaps sufficiently with a ground truth—as equal. This ignores the fact that some anchors are much better positioned than others.
Nan Xue and colleagues argue in Real-Time 3D Object Detection with Inference-Aligned Learning that we must align label assignment with geometric reliability. If the training process doesn't reward the model more for a 90% IoU box than a 50% IoU box, the model has no incentive to produce confidence scores that reflect that difference. Shengkai Wu addressed this by introducing IoU-balanced Loss Functions, which re-weight the classification loss based on the spatial overlap, forcing the model to be more "confident" only when it is also more "accurate."
The Challenge of Unlabeled Data
The confidence-localization gap becomes even more dangerous in semi-supervised learning and domain adaptation. In these settings, a "Teacher" model generates pseudo-labels for a "Student" model to learn from.
If the Teacher is overconfident about a poorly localized box, the Student will "learn" that incorrect boundary as ground truth. Yen‐Cheng Liu and the authors of Unbiased Teacher v2 found that relying on classification scores for pseudo-labeling introduces misleading gradients into the regression branch. Similarly, Samuel Schulter demonstrated in Improving Pseudo Labels for Open-Vocabulary Object Detection that vision-language models often produce highly confident category names paired with noisy, inaccurate boxes.
To combat this, modern frameworks like Consistent-Teacher (Xinjiang Wang et al., /paper/art_cfeb8899509c4425936a913e384986c8) and ConfMix (Giulio Mattolin et al., /paper/art_9965771831aa44bb842ba935341d6e3a) use multi-metric filtering. They only accept a pseudo-label if the model is confident in both the category and the spatial stability of the box (often measured via box jittering or variance estimation).
Architectural Synchronization
Finally, researchers are rethinking the "two-head" architecture itself. In standard detectors, the classification and localization branches often operate in total isolation. Hantao Zhou introduced UniHead to enable cross-task interaction, ensuring the two branches communicate during the prediction process.
Even in Transformer-based architectures, which are naturally more integrated, this gap persists. Guanglu Song and the authors of Decoupled DETR found that while classification and localization need different spatial features (e.g., looking at the whole object vs. its edges), they must be explicitly synchronized to ensure that the "what" and "where" refer to the same high-quality prediction.
By treating confidence not as a simple probability of existence, but as a joint measure of identity and precision, these researchers are moving toward detectors that are not just more certain, but more reliable.