Informative Data Selection with Uncertainty for Multi-modal Object Detection

Xinyu Zhang, Zhiwei Li, Zhenhong Zou, Xin Gao, Yijin Xiong, Dafeng Jin, Jun Li, Huaping Liu

Introduction: The Challenge of "Dirty Data"

Object detection models for autonomous driving typically excel on clean, curated datasets. However, real-world deployment introduces "dirty data"—noise from adverse weather (fog, rain, frost), lighting fluctuations, or sensor hardware malfunctions. While multi-modal fusion (e.g., combining RGB cameras and LiDAR) provides redundancy, many existing systems are "tightly coupled." In these architectures, noise in a single sensor can propagate through the network, "poisoning" the feature space and degrading overall performance.

The impact of noise on multi-modal fusion in practical applications compared to clean dataset evaluation.

The core challenge addressed here is quantifying sensor reliability in real-time to adaptively select the most informative data while filtering out noise.

Methodology: Uncertainty-Aware Fusion

The authors propose a loosely coupled, uncertainty-aware framework. Instead of early feature fusion, RGB images and LiDAR point clouds (projected as 2D depth maps) are processed through parallel, independent pipelines using a modified YOLOv3 backbone.

The proposed multi-modal detection model architecture featuring parallel backbones and an informative data selection module.

Core Innovations

  1. Aleatoric Uncertainty Modeling: The model predicts both bounding box coordinates and their associated variance ($\sigma^2$). A specialized loss function attenuates the penalty for noisy data, allowing the network to learn an internal measure of "informativeness" (the inverse of uncertainty).
  2. Uncertainty-aware Multi-source NMS: Unlike standard Non-Maximum Suppression (NMS) which relies solely on classification scores and IoU, this approach uses predicted uncertainty to perform a weighted average of candidate boxes. Detections from sensors with high uncertainty contribute less to the final output.
  3. Logic-Gated Fusion: A decision logic categorizes box pairs based on their spatial overlap, allowing the system to handle consistent, ambiguous, or unique detections across modalities appropriately.

Experimental Setup

The framework was evaluated on the KITTI 2D Object Detection dataset. To test robustness, the authors created a "Simulated Dirty KITTI" by applying 15 types of corruptions (e.g., Gaussian noise, motion blur, frost) at five severity levels. This setup specifically targets scenarios where sensor failure or environmental interference is likely.

Key Results and Insights

The proposed fusion model demonstrates remarkable resilience. While it performs slightly below the best single-modal model on clean data (by 0.36 mAP), it significantly outperforms them as noise increases. On average across all "dirty" scenarios, the fusion model achieved a 51.61 mAP improvement over RGB-only and a 34.20 mAP improvement over Depth-only models.

A key insight is that the model effectively "switches" its reliance. For instance, when RGB images are heavily obscured by Gaussian noise, the system recognizes the high uncertainty and shifts its focus to the cleaner LiDAR data. This loosely coupled approach prevents one modality's noise from corrupting the other's feature extraction.

Conclusion

This research presents a robust framework for multi-modal object detection that treats uncertainty as a proxy for data informativeness. By delaying fusion until the NMS stage and explicitly modeling sensor noise, the system maintains high performance in unpredictable real-world conditions. Future work will explore extending this uncertainty-aware selection to 3D spatial information and more complex environmental physics.

Summary updated

Abstract

Noise has always been nonnegligible trouble in object detection by creating confusion in model reasoning, thereby reducing the informativeness of the data. It can lead to inaccurate recognition due to the shift in the observed pattern, that requires a robust generalization of the models. To implement a general vision model, we need to develop deep learning models that can adaptively select valid information from multimodal data. This is mainly based on two reasons. Multimodal learning can break through the inherent defects of single-modal data, and adaptive information selection can reduce chaos in multimodal data. To tackle this problem, we propose a universal uncertainty-aware multimodal fusion model. It adopts a multipipeline loosely coupled architecture to combine the features and results from point clouds and images. To quantify the correlation in multimodal information, we model the uncertainty, as the inverse of data information, in different modalities and embed it in the bounding box generation. In this way, our model reduces the randomness in fusion and generates reliable output. Moreover, we conducted a completed investigation on the KITTI 2-D object detection dataset and its derived dirty data. Our fusion model is proven to resist severe noise interference like Gaussian, motion blur, and frost, with only slight degradation. The experiment results demonstrate the benefits of our adaptive fusion. Our analysis on the robustness of multimodal fusion will provide further insights for future research.
BibTeX
@article{zhang2024informative,
  author = {Xinyu Zhang and Zhiwei Li and Zhenhong Zou and Xin Gao and Yijin Xiong and Dafeng Jin and Jun Li and Huaping Liu},
  title = {{Informative Data Selection with Uncertainty for Multi-modal Object Detection}},
  journal = {IEEE Trans. Neural Networks Learn. Syst.},
  year = {2024},
  doi = {10.1109/tnnls.2023.3270159},
}
Figure