In a standard object detector, the model must answer two distinct questions: What is this? and Where is it? Traditionally, these questions are handled by two separate "heads"—a classification head and a regression head. However, a growing body of research has identified a dangerous "confidence gap" between these two outputs. A model might be 99% certain an object is a "car" (classification) while providing a bounding box that only partially overlaps with the actual vehicle (localization).
This misalignment is not just a theoretical nuance; it is a primary cause of failure in safety-critical systems. If a high-confidence but poorly localized box survives post-processing, it can lead to catastrophic errors in downstream tasks like autonomous braking or robotic grasping.
The Misalignment Problem
The root of the issue lies in how detectors are trained and evaluated. Most detectors use Non-Maximum Suppression (NMS) to filter out redundant boxes. NMS relies entirely on the classification score to decide which box to keep. If a "bad" box has a higher classification score than a "perfect" box, the perfect box is discarded.
As Muhammad Akhtar Munir and colleagues point out in Towards Improving Calibration in Object Detection Under Domain Shift, standard calibration metrics often ignore spatial precision. A model is considered "calibrated" if its 0.9 confidence predictions are correct 90% of the time, but in detection, "correctness" is a gradient of spatial overlap (Intersection over Union, or IoU). David Hall argues in Probabilistic Object Detection: Definition and Evaluation that treating bounding boxes as absolute certainties rather than probability distributions leads to extreme spatial overconfidence.
Unifying Confidence and IoU
To bridge this gap, researchers have moved toward "IoU-aware" confidence. The intuition is simple: the final confidence score should reflect both the categorical certainty and the predicted spatial quality.
One common approach is to add a dedicated branch to the network that predicts the IoU of each proposal. In CIA-SSD, Wu Zheng and others introduce a mechanism to "rectify" classification scores by multiplying them with these predicted IoU values. This ensures that the boxes that survive NMS are those that are both correctly labeled and precisely positioned. Similarly, Jianfeng Wang and the SFace team use an IoU-based re-scoring mechanism to unify disparate detection branches, ensuring the highest-ranked detections have the best spatial alignment.
Training for Alignment
Rather than fixing the scores post-hoc, some researchers argue that the training objective itself must change. In Prime Sample Attention in Object Detection, Kai Chen and colleagues suggest that "prime samples"—those with high IoU—should be prioritized during training. By weighting the classification loss based on spatial overlap, the model learns to naturally assign higher scores to better-localized boxes.
This concept is taken further by Bimsara Pathiraja in Multiclass Confidence and Localization Calibration for Object Detection. Their MCCL framework jointly calibrates classification and localization during training. By minimizing the difference between a box's predicted overlap and the model's internal certainty, the detector is forced to lower its confidence when its spatial predictions are shaky.
The Role of Spatial Context
Calibration is not uniform across an image. Fabian Küppers and colleagues demonstrated in Multivariate Confidence Calibration for Object Detection that detectors often exhibit spatial biases—for example, being more overconfident at the edges of a frame or for objects of a certain scale.
In 3D environments, this problem is even more pronounced. Lingdong Kong and the Calib3D team found that LiDAR-based models are dangerously overconfident about distant, sparse points. Their solution, DeptS, uses depth-aware scaling to force more conservative predictions where data is sparse, aligning the model's "honesty" with the physical reality of the sensor.
Measuring "Honest" Uncertainty
If the goal is to make models "know what they don't know," we need metrics that reward this honesty. Traditional Mean Average Precision (mAP) uses a binary IoU threshold (e.g., 0.5), which treats a "barely passing" box the same as a "perfect" one.
The Probability-based Detection Quality (PDQ) metric, proposed by David Hall, offers a more nuanced alternative. It calculates the geometric mean of spatial quality and label probability. To get a high PDQ score, a detector cannot just be accurate; it must be probabilistically accurate. It must provide a spatial distribution that actually contains the object and a label probability that matches its success rate.
By shifting the focus from pure accuracy to the alignment of confidence and localization, this research direction ensures that when a model says it sees something, we can trust not just what it sees, but exactly where it says it is.