UNO: Uncertainty-aware Noisy-Or Multimodal Fusion for Unanticipated Input Degradation

Junjiao Tian, Wesley Cheung, Nathaniel Glaser, Yen-Cheng Liu, Zsolt Kira

Introduction

Robots and autonomous vehicles increasingly rely on multimodal sensor suites, such as RGB cameras and depth sensors, to perceive their environments. While fusing these modalities typically enhances performance, current fusion architectures are notoriously brittle when faced with unanticipated, out-of-distribution (OOD) degradations. Environmental factors like sudden snow, heavy fog, or hardware-induced impulse noise can cause a model to blindly trust corrupted data, leading to catastrophic failures in semantic segmentation. This paper introduces UNO, a framework designed to dynamically recognize and ignore unreliable sensor data by leveraging spatial uncertainty and a novel Bayesian fusion gate.

The UNO Framework

The core philosophy of UNO is to handle OOD degradation without requiring explicit training on every possible noise type. The authors propose a three-part pipeline to calibrate sensor confidence on-the-fly:

  1. Deviation Ratio for Uncertainty: The system evaluates multiple uncertainty metrics, including predictive entropy and mutual information via Monte Carlo Dropout. It introduces a "deviation ratio" that measures how much a test sample's uncertainty differs from the training set's baseline. High deviation triggers a "softening" of the network's softmax probabilities, signaling low confidence.
  2. Spatial Temperature Network (TempNet): Unlike standard temperature scaling that applies a single scalar to an entire image, TempNet is a shallow network that generates a pixel-wise spatial temperature map. This allows the model to account for regional degradations, such as fog that obscures distant objects more than those in the foreground.
  3. Noisy-Or Fusion: Traditional fusion often uses multiplicative or additive operations, where a single noisy sensor can degrade the final output. UNO employs a Bayesian "Noisy-Or" gate. This logic preserves confident predictions from clean sensors while effectively filtering out the flattened, uncertain distributions from degraded ones.
The UNO architecture integrates TempNet and multiple uncertainty metrics to calibrate predictions before Noisy-Or fusion.

Experimental Evaluation

The authors evaluated the framework using a custom RGB-D semantic segmentation dataset generated in the AirSim simulator. Models were trained on clean data and mild fog, then tested against severe OOD conditions including frost, snow, motion blur, and various synthetic noises (Gaussian, impulse, shot noise) applied to individual channels.

Key Results

UNO++ demonstrated remarkable robustness compared to existing state-of-the-art methods. While standard fusion models like SSMA (Self-Supervised Model Adaptation) often collapse when a sensor is severely degraded, UNO++ maintains high performance by successfully identifying the corrupted modality.

  • Average Performance: UNO++ achieved an average mIoU of 80.00 across all unseen degradations, representing a 28% relative improvement over the SSMA baseline (62.49 mIoU).
  • Extreme Robustness: In scenarios like a total Depth channel blackout, SSMA's performance plummeted to 33.02 mIoU, whereas UNO++ maintained 79.13 mIoU by relying entirely on the functional RGB camera.

Insights and Future Work

A critical finding of the study is that no single uncertainty metric is universal; for instance, Monte Carlo Dropout often fails to flag impulse noise, while TempNet identifies it easily. By taking the minimum deviation ratio across metrics, UNO ensures comprehensive coverage of edge cases. Furthermore, the spatial awareness provided by TempNet proves essential for non-uniform environmental degradations.

Despite its success, the reliance on Monte Carlo Dropout introduces computational overhead due to multiple forward passes, which may limit real-time application. Future research will focus on optimizing these uncertainty metrics to reduce inference time while maintaining the framework's robust performance in the wild.

Summary updated

Abstract

The fusion of multiple sensor modalities, especially through deep learning architectures, has been an active area of study. However, an under-explored aspect of such work is whether the methods can be robust to degradation across their input modalities, especially when they must generalize to degradation not seen during training. In this work, we propose an uncertainty-aware fusion scheme to effectively fuse inputs that might suffer from a range of known and unknown degradation. Specifically, we analyze a number of uncertainty measures, each of which captures a different aspect of uncertainty, and we propose a novel way to fuse degraded inputs by scaling modality-specific output softmax probabilities. We additionally propose a novel data-dependent spatial temperature scaling method to complement these existing uncertainty measures. Finally, we integrate the uncertainty-scaled output from each modality using a probabilistic noisy-or fusion method. In a photo-realistic simulation environment (AirSim), we show that our method achieves significantly better results on a semantic segmentation task, as compared to state-of-art fusion architectures, on a range of degradation (e.g. fog, snow, frost, and various other types of noise), some of which are unknown during training.
BibTeX
@inproceedings{tian2020uno,
  author = {Junjiao Tian and Wesley Cheung and Nathaniel Glaser and Yen-Cheng Liu and Zsolt Kira},
  title = {{UNO: Uncertainty-aware Noisy-Or Multimodal Fusion for Unanticipated Input Degradation}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2020},
  doi = {10.1109/icra40945.2020.9197266},
}
Figure