TL;DR
This paper identifies critical flaws in how we currently train and evaluate uncertainty in deep object detectors. By replacing the standard Negative Log Likelihood (NLL) loss with the Energy Score, the authors produce detectors that are better calibrated and more "confident" when they are right. They also provide a more rigorous framework for evaluating how well these models know what they don't know.
What Problem Does This Paper Address?
Deep object detectors are increasingly used in safety-critical systems like self-driving cars. In these settings, it’s not enough for a model to just draw a box around a pedestrian; it needs to say how certain it is about that box's position.
Most current "probabilistic" detectors try to do this by predicting a mean (the box) and a variance (the uncertainty). However, they typically use Negative Log Likelihood (NLL) as their training objective. The authors point out a major pitfall: NLL is a "local" scoring rule. If a model makes a small spatial error but is very "certain" (low entropy), NLL penalizes it infinitely.
To avoid this infinite penalty, models trained with NLL often learn to be "lazy" and output high uncertainty everywhere just to stay safe. Furthermore, the field lacks a standard, theoretically sound way to evaluate these uncertainty estimates. Common metrics like mean Average Precision (mAP) only care about the "mean" prediction, ignoring the quality of the predicted variance entirely.
Methods
The authors propose two major shifts to address these issues:
- Training with the Energy Score (ES): Unlike NLL, the Energy Score is a non-local proper scoring rule. It is distance-sensitive, meaning it doesn't "blow up" if the predicted mean is slightly off from the ground truth. This allows the model to learn lower-entropy (more precise) distributions that are better calibrated.
- A New Evaluation Framework: Instead of relying on a single number like mAP, the authors partition model detections into four categories: True Positives, Duplicates, Localization Errors, and False Positives. They then apply "proper scoring rules" (NLL, Brier Score, and Energy Score) to each category separately. This reveals why a model might be failing—for instance, is it overconfident on its mistakes or underconfident on its successes?
- Variance Networks: They extend three state-of-the-art architectures—DETR, RetinaNet, and Faster R-CNN—with "variance heads" to predict the parameters of a multivariate Gaussian distribution for bounding box coordinates.
Datasets
To rigorously test their approach, the authors utilized several datasets to measure both in-distribution and out-of-distribution performance:
- COCO: Used for standard training and "in-distribution" testing.
- ImageNet-C Corruptions: The authors applied 18 different image corruptions (like blur, snow, or noise) to the COCO validation set to test how models handle "shifted" data.
- OpenImages: Used to create two specific test sets: a Shifted set containing the same 80 categories as COCO but with different lighting and styles, and an Out-of-Distribution (OOD) set containing none of the COCO categories to see if the model correctly flags its own ignorance.
Key Insights and Results
- The "Lazy" NLL Trap: Because object detection involves "near-misses" during training (due to how targets are assigned), NLL forces models to predict high uncertainty even for easy objects. The Energy Score fixes this by rewarding "near-misses" more gracefully.
- Target Assignment Matters: The authors discovered that how a model chooses its training targets fundamentally limits its ability to learn uncertainty. DETR, which uses optimal bipartite matching and sees a wider variety of "errors" during training, produces much more meaningful uncertainty estimates than models relying on strict IOU thresholds like Faster R-CNN.
- Better Calibration and Precision: Models trained with the Energy Score showed significantly lower regression calibration error (CE Reg) compared to NLL. For example, on DETR, the CE Reg dropped from 0.0512 (NLL) to 0.0068 (ES). In many cases, training with the Energy Score actually improved the standard mAP. On the COCO dataset, DETR-ES achieved 39.96% mAP, compared to 37.10% for DETR-NLL.
- Entropy as a Red Flag: The experiments show that while models might still try to detect objects in OOD data, their predictive entropy (uncertainty) spikes significantly. This confirms that variance networks can effectively signal when a model is operating outside its comfort zone.
- Qualitative Difference: Visualizations show that NLL-trained models produce "fuzzy" confidence ellipses that are often much larger than necessary, whereas ES-trained models produce tight, accurate ellipses for clear objects and larger ones only when the object is truly ambiguous.
Limitations and Future Work
- Architecture Constraints: The authors noted that they "bolted on" variance heads to existing deterministic architectures. They suggest that future work should focus on designing new backends specifically optimized for probabilistic output from the ground up.
- Computational Overhead: While the Energy Score is effective, the Monte-Carlo approximation used (sampling 1000 times) adds some computational cost during training, though it remains manageable.
- Open Question: The paper leaves the challenge of how to best integrate these uncertainty estimates into downstream tasks (like path planning for a robot) as an important next step for the community.
Summary updated