Weather Adaptive Autonomous Driving

Nicholas Taylor, Alexander Picon, Sudip DhakalarXiv 2026 · arXiv

TL;DR

This paper introduces a "calibrated trade-off dial" for autonomous driving object detectors, allowing engineers to explicitly set the "price" of safety—specifically, how many false alarms they are willing to accept to avoid a single missed object. By applying Conformal Risk Control (CRC) to a frozen YOLOv8x detector, the authors provide a way to maintain strict safety guarantees across varying weather conditions like fog, rain, and snow without the need for expensive model retraining.

What Problem Does This Paper Address?

Camera-based object detectors are notoriously sensitive to adverse weather. In fog or heavy rain, a detector’s confidence scores often drop, leading to missed detections (lower recall). While you can recover these misses by lowering the confidence threshold, doing so typically results in a flood of false alarms—ghost cars or pedestrians that cause dangerous "phantom braking."

Most research focuses on retraining models to be more robust, but this doesn't solve the deployment problem: at what specific threshold should the detector operate? Furthermore, a single global threshold that works in clear weather often fails catastrophically in fog. The authors address the need for a decision rule that adapts to weather conditions while providing a statistical guarantee on the maximum number of objects the system is allowed to miss.

Methods

The authors propose a system that wraps a frozen, pre-trained detector (YOLOv8x) with a secondary calibration layer.

  1. Feature Extraction: For every candidate box the detector produces, the authors tap into internal signals beyond just the final score. This includes class probabilities, "distribution-focal-loss" (which indicates how certain the model is about the box edges), and "neck" features from the neural network.
  2. Calibrated Probability Estimation: These features are fed into a Gradient-Boosted Tree (GBT) model. This model doesn't predict a class; instead, it estimates a highly calibrated probability $p(d)$ that the detection is a real object.
  3. The Trade-off Dial ($\lambda$): The user provides a single parameter, $\lambda$, representing the number of false alarms they will accept to avoid one miss. The system then applies a cost-optimal decision rule: keep a detection if $p(d) \geq 1/(1+\lambda)$.
  4. Conformal Risk Control (CRC): To ensure safety, the authors use CRC to calculate a per-condition threshold. This provides a finite-sample guarantee that the expected miss rate will stay below a user-defined budget (e.g., 20%), even as weather changes.
The full pipeline: from raw camera frame to the calibrated trade-off dial.

Datasets

  • COCO: Used for the initial reproduction of baseline conformal methods.
  • BDD100K: The primary dataset for fitting the trade-off dial. It contains over 21,000 labeled training frames across various times of day and weather conditions.
  • DAWN: A specialized dataset for adverse weather (fog, rain, snow, sand) used exclusively for evaluation to test how well the calibration transfers to real-world shifts.

Key Insights and Ideas

  • Safety has a variable price: The authors found that maintaining a 20% miss budget is relatively "cheap" in clear weather but incredibly "expensive" in adverse conditions. In DAWN’s snow bin, it costs 1.6 false alarms per image, but in BDD100K’s night scenes, that price jumps to 117 false alarms per image.
  • Frozen detectors are underutilized: You don't always need to retrain. The internal features of a "frozen" model (like box-edge certainty) contain rich information that can be used to recalibrate its outputs for new environments.
  • The Batching Bug: During their reproduction of previous work (SeqCRC), the authors discovered that YOLOv8's official wrapper rescales boxes differently depending on batch size. This subtle engineering detail significantly impacted localization accuracy, highlighting the difficulty of reproducing "state-of-the-art" conformal detection results.
Comparison of the baseline SeqCRC approach vs. the proposed trade-off dial.

Key Results

  • Baseline Performance: Using a per-weather CRC threshold (Phase 1) successfully held the miss fraction within a 20% budget across all DAWN weather bins, whereas a single global threshold failed significantly in fog.
  • Cost-Optimal Decisions: The proposed per-detection rule (Option B) was shown to be more efficient than a simple score threshold.
  • Reproduction Accuracy: The authors successfully reproduced the confidence and classification steps of SeqCRC on COCO, with test risks sitting at or near their 0.1 budgets (e.g., 0.085 to 0.0997).
Reproduction results for SeqCRC on COCO, showing confidence and classification risks near the 0.1 budget.

Limitations and Future Work

  • The "Miss Floor": The authors acknowledge a hard limit: if the frozen detector never sees an object at all (zero confidence), no amount of threshold tuning can recover it. In some conditions, 11–18% of objects were simply never detected, making certain low-miss budgets impossible to certify.
  • Exchangeability: The CRC guarantees rely on the assumption that images within a weather "bin" (e.g., "rain at night") are exchangeable. If the rain suddenly gets much heavier than what was seen in the calibration set, the guarantee may weaken.
  • Future Work: The authors plan to fully integrate the CRC guarantee directly onto the $\lambda$ dial (Phase 4), allowing the system to automatically "slow down" or stop the vehicle if the required safety budget cannot be met at a reasonable false-alarm price.

Summary updated

Abstract

Camera object detectors lose recall in rain, fog, snow and sand, and the confidence threshold at which a detector is deployed decides how many objects it misses and how many false alarms it raises. We study how a frozen detector should make this decision in adverse weather. We first reproduce SeqCRC, a conformal object detector whose first step calibrates the confidence threshold by conformal risk control (CRC), and use that step, calibrated separately in each weather condition, as our baseline. With YOLOv8x it keeps the held-out miss fraction within a 20% budget in every DAWN weather bin, where a single global threshold fails in fog, but the false alarms it needs are an output rather than a choice: from 1.6 to 117 per image across DAWN and BDD100K conditions. We then propose a calibrated trade-off dial. One setting, $λ$, the number of false alarms accepted to avoid one missed object, selects the expected-cost-optimal decision rule: either one score threshold, or a per-detection rule that keeps a box when its calibrated probability of being a real object, estimated by gradient-boosted trees from the detector's own per-box outputs, is at least $1/(1+λ)$. F-score preferences map onto the same dial, and CRC applied to $λ$ itself is planned to restore a per-condition miss guarantee. The dial is fitted once on BDD100K train and graded only on BDD100K val and DAWN.
BibTeX
@misc{taylor2026weather,
  author = {Nicholas Taylor and Alexander Picon and Sudip Dhakal},
  title = {{Weather Adaptive Autonomous Driving}},
  year = {2026},
  eprint = {2611.04305},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.04305},
}
Figure