TL;DR
This paper demonstrates that object detectors' confidence scores are biased by object size, leading to suboptimal performance. The authors propose a conditional calibration method that accounts for this bias, significantly improving detection accuracy across various architectures.
What Problem Does This Paper Address?
Modern object detectors are crucial for many applications, from autonomous driving to medical imaging. A key aspect of these detectors is their confidence score, which indicates how certain the model is about a detection. However, it's a known issue that these confidence scores are often "miscalibrated," meaning they don't accurately reflect the true probability of a correct detection. Recent work has also suggested that this miscalibration isn't uniform; it can be biased based on factors like object size and position within an image. While these issues have been studied in isolation, this paper argues that they are deeply interconnected and that the conditional confidence bias (specifically, the bias related to bounding box size) actively harms an object detector's overall performance. The paper aims to formally prove this, empirically validate it, and propose a method to mitigate this bias and improve performance.
Methods
The core of the paper's methodology revolves around conditional confidence calibration. Instead of calibrating confidence scores uniformly, they propose calibrating them conditioned on the bounding box size.
- Formal Proof: The authors first formally prove that a conditional bias in object detectors leads to a non-optimal expected Average Precision (AP). This involves analyzing the expected AP and deriving conditions for its maximization.
- Modified Histogram Binning: To empirically validate their findings, they adapt the histogram binning calibration technique. Standard histogram binning groups detections by confidence intervals and estimates the true positive rate for each bin. The authors modify this by first splitting detections into sub-groups based on bounding box size and then applying histogram binning within each sub-group. This allows the calibration to be sensitive to both confidence and object size.
- Test Time Augmentation (TTA) Analysis: They investigate how TTA, a common technique to improve robustness by averaging predictions over augmented images, interacts with this bias. They hypothesize that TTA can exacerbate the problem and show that individually calibrating predictions for each augmentation scale can be more effective.
- Optimization Metrics: They explore various loss functions (Brier Loss, Log Loss, Mean Squared Error) for optimizing the calibration process, finding that Mean Squared Error (MSE) performs best for their conditional calibration approach.
Datasets
- COCO [24]: The authors extensively use the COCO dataset for training and evaluating their methods. They specifically mention using the COCO validation set for calibration and the test-dev benchmark for final evaluation.
Key Insights and Ideas
- Conditional Bias is Detrimental: The most significant insight is the formal proof and empirical validation that the confidence bias related to object size isn't just an artifact, but actively reduces the performance of object detectors.
- Calibration as a Solution: The paper shows that by explicitly accounting for this size bias during calibration, performance can be significantly improved. This suggests that confidence calibration is not just about making scores "honest" but can be a direct pathway to better detection accuracy.
- TTA Amplifies the Problem: Test Time Augmentation, while beneficial, can actually amplify the negative effects of the conditional bias. This is because scaling images down forces detectors to predict smaller objects, where the bias is more pronounced.
- Bias Exists Even in Training Data: The confidence bias is present not only in validation/test data but also in the training data itself. This is crucial because it means the bias can be addressed without relying solely on train-test distribution shifts.
- Model Size Matters: Larger, more performant object detectors seem to inherently learn to reduce this conditional bias to some extent, showing smaller performance gains from the proposed calibration.
Key Results
- Performance Gains: The proposed conditional calibration method achieved significant performance improvements across various object detection architectures. Gains ranged from a slight decrease to a substantial increase of up to 0.6 mAP and 0.8 mAP50 on the test-dev benchmark.
- Modified Histogram Binning Effectiveness: The modified histogram binning, conditioned on bounding box size, was shown to be effective. For example, on CenterNet, it improved mAP from 35.7 (with standard conditional binning) to 40.1 (with their method).
- TTA Improvement: Individually calibrating predictions for each scale augmentation in TTA was about three times more effective than calibrating the combined TTA predictions.
- Optimization Metric Performance: Optimizing the calibration using the estimated Mean Squared Error (LMSE) on the validation split led to better performance gains than optimizing directly on AP or using simpler loss functions like Brier or Log Loss.
Limitations and Future Work
- Parameter Sensitivity: The performance gains are sensitive to the choice of parameters for splitting bounding box sizes (B) and confidence bins (C). Finding optimal parameters for each class is an open challenge.
- Two-Stage Detectors: The proposed method showed little to no performance improvement on two-stage detectors (like Faster R-CNN). The authors hypothesize this is because the confidence prediction is more separated from the bounding box regression in these architectures.
- Generalizability of Calibration Methods: While they found their modified histogram binning effective for demonstrating the bias, they acknowledge it might not be the superior calibration method overall. Future work could explore applying their conditional calibration principle to other, more advanced calibration techniques.
- Broader Impact: The paper acknowledges that while their work can improve beneficial applications, it could also be used to enhance harmful systems if intentionally designed. They believe the net impact will be positive by reducing unintended harms from biased systems.
Summary updated