TL;DR
This paper introduces a "calibrated trade-off dial" for autonomous driving object detectors, allowing engineers to explicitly set the "price" of safety—specifically, how many false alarms they are willing to accept to avoid a single missed object. By applying Conformal Risk Control (CRC) to a frozen YOLOv8x detector, the authors provide a way to maintain strict safety guarantees across varying weather conditions like fog, rain, and snow without the need for expensive model retraining.
What Problem Does This Paper Address?
Camera-based object detectors are notoriously sensitive to adverse weather. In fog or heavy rain, a detector’s confidence scores often drop, leading to missed detections (lower recall). While you can recover these misses by lowering the confidence threshold, doing so typically results in a flood of false alarms—ghost cars or pedestrians that cause dangerous "phantom braking."
Most research focuses on retraining models to be more robust, but this doesn't solve the deployment problem: at what specific threshold should the detector operate? Furthermore, a single global threshold that works in clear weather often fails catastrophically in fog. The authors address the need for a decision rule that adapts to weather conditions while providing a statistical guarantee on the maximum number of objects the system is allowed to miss.
Methods
The authors propose a system that wraps a frozen, pre-trained detector (YOLOv8x) with a secondary calibration layer.
- Feature Extraction: For every candidate box the detector produces, the authors tap into internal signals beyond just the final score. This includes class probabilities, "distribution-focal-loss" (which indicates how certain the model is about the box edges), and "neck" features from the neural network.
- Calibrated Probability Estimation: These features are fed into a Gradient-Boosted Tree (GBT) model. This model doesn't predict a class; instead, it estimates a highly calibrated probability $p(d)$ that the detection is a real object.
- The Trade-off Dial ($\lambda$): The user provides a single parameter, $\lambda$, representing the number of false alarms they will accept to avoid one miss. The system then applies a cost-optimal decision rule: keep a detection if $p(d) \geq 1/(1+\lambda)$.
- Conformal Risk Control (CRC): To ensure safety, the authors use CRC to calculate a per-condition threshold. This provides a finite-sample guarantee that the expected miss rate will stay below a user-defined budget (e.g., 20%), even as weather changes.
Datasets
- COCO: Used for the initial reproduction of baseline conformal methods.
- BDD100K: The primary dataset for fitting the trade-off dial. It contains over 21,000 labeled training frames across various times of day and weather conditions.
- DAWN: A specialized dataset for adverse weather (fog, rain, snow, sand) used exclusively for evaluation to test how well the calibration transfers to real-world shifts.
Key Insights and Ideas
- Safety has a variable price: The authors found that maintaining a 20% miss budget is relatively "cheap" in clear weather but incredibly "expensive" in adverse conditions. In DAWN’s snow bin, it costs 1.6 false alarms per image, but in BDD100K’s night scenes, that price jumps to 117 false alarms per image.
- Frozen detectors are underutilized: You don't always need to retrain. The internal features of a "frozen" model (like box-edge certainty) contain rich information that can be used to recalibrate its outputs for new environments.
- The Batching Bug: During their reproduction of previous work (SeqCRC), the authors discovered that YOLOv8's official wrapper rescales boxes differently depending on batch size. This subtle engineering detail significantly impacted localization accuracy, highlighting the difficulty of reproducing "state-of-the-art" conformal detection results.
Key Results
- Baseline Performance: Using a per-weather CRC threshold (Phase 1) successfully held the miss fraction within a 20% budget across all DAWN weather bins, whereas a single global threshold failed significantly in fog.
- Cost-Optimal Decisions: The proposed per-detection rule (Option B) was shown to be more efficient than a simple score threshold.
- Reproduction Accuracy: The authors successfully reproduced the confidence and classification steps of SeqCRC on COCO, with test risks sitting at or near their 0.1 budgets (e.g., 0.085 to 0.0997).
Limitations and Future Work
- The "Miss Floor": The authors acknowledge a hard limit: if the frozen detector never sees an object at all (zero confidence), no amount of threshold tuning can recover it. In some conditions, 11–18% of objects were simply never detected, making certain low-miss budgets impossible to certify.
- Exchangeability: The CRC guarantees rely on the assumption that images within a weather "bin" (e.g., "rain at night") are exchangeable. If the rain suddenly gets much heavier than what was seen in the calibration set, the guarantee may weaken.
- Future Work: The authors plan to fully integrate the CRC guarantee directly onto the $\lambda$ dial (Phase 4), allowing the system to automatically "slow down" or stop the vehicle if the required safety budget cannot be met at a reasonable false-alarm price.
Summary updated