Introduction
Robots and autonomous vehicles increasingly rely on multimodal sensor suites, such as RGB cameras and depth sensors, to perceive their environments. While fusing these modalities typically enhances performance, current fusion architectures are notoriously brittle when faced with unanticipated, out-of-distribution (OOD) degradations. Environmental factors like sudden snow, heavy fog, or hardware-induced impulse noise can cause a model to blindly trust corrupted data, leading to catastrophic failures in semantic segmentation. This paper introduces UNO, a framework designed to dynamically recognize and ignore unreliable sensor data by leveraging spatial uncertainty and a novel Bayesian fusion gate.
The UNO Framework
The core philosophy of UNO is to handle OOD degradation without requiring explicit training on every possible noise type. The authors propose a three-part pipeline to calibrate sensor confidence on-the-fly:
- Deviation Ratio for Uncertainty: The system evaluates multiple uncertainty metrics, including predictive entropy and mutual information via Monte Carlo Dropout. It introduces a "deviation ratio" that measures how much a test sample's uncertainty differs from the training set's baseline. High deviation triggers a "softening" of the network's softmax probabilities, signaling low confidence.
- Spatial Temperature Network (TempNet): Unlike standard temperature scaling that applies a single scalar to an entire image, TempNet is a shallow network that generates a pixel-wise spatial temperature map. This allows the model to account for regional degradations, such as fog that obscures distant objects more than those in the foreground.
- Noisy-Or Fusion: Traditional fusion often uses multiplicative or additive operations, where a single noisy sensor can degrade the final output. UNO employs a Bayesian "Noisy-Or" gate. This logic preserves confident predictions from clean sensors while effectively filtering out the flattened, uncertain distributions from degraded ones.
Experimental Evaluation
The authors evaluated the framework using a custom RGB-D semantic segmentation dataset generated in the AirSim simulator. Models were trained on clean data and mild fog, then tested against severe OOD conditions including frost, snow, motion blur, and various synthetic noises (Gaussian, impulse, shot noise) applied to individual channels.
Key Results
UNO++ demonstrated remarkable robustness compared to existing state-of-the-art methods. While standard fusion models like SSMA (Self-Supervised Model Adaptation) often collapse when a sensor is severely degraded, UNO++ maintains high performance by successfully identifying the corrupted modality.
- Average Performance: UNO++ achieved an average mIoU of 80.00 across all unseen degradations, representing a 28% relative improvement over the SSMA baseline (62.49 mIoU).
- Extreme Robustness: In scenarios like a total Depth channel blackout, SSMA's performance plummeted to 33.02 mIoU, whereas UNO++ maintained 79.13 mIoU by relying entirely on the functional RGB camera.
Insights and Future Work
A critical finding of the study is that no single uncertainty metric is universal; for instance, Monte Carlo Dropout often fails to flag impulse noise, while TempNet identifies it easily. By taking the minimum deviation ratio across metrics, UNO ensures comprehensive coverage of edge cases. Furthermore, the spatial awareness provided by TempNet proves essential for non-uniform environmental degradations.
Despite its success, the reliance on Monte Carlo Dropout introduces computational overhead due to multiple forward passes, which may limit real-time application. Future research will focus on optimizing these uncertainty metrics to reduce inference time while maintaining the framework's robust performance in the wild.
Summary updated