RGB-X Object Detection via Scene-Specific Fusion Modules

Sri Aditya Deevi, Connor Lee, Lu Gan, Sushruth Nagesh, Gaurav Pandey, Soon-Jo ChungWACV 2024

TL;DR

This paper introduces a modular and efficient approach to fusing data from multiple sensors (like RGB and thermal cameras) for object detection in autonomous vehicles. By using lightweight, scene-specific fusion modules trained adaptively, their method achieves state-of-the-art performance without requiring massive, end-to-end retraining of entire networks for every sensor combination.

What Problem Does This Paper Address?

Autonomous vehicles rely on a suite of sensors to perceive their surroundings, but no single sensor is perfect in all conditions. Object detection systems need to be robust to varying weather (fog, rain, snow) and lighting (day, night). Deep sensor fusion (DSF) aims to combine data from multiple sensors to overcome individual limitations. However, existing DSF methods often require large, coregistered multimodal datasets and extensive end-to-end retraining, making them inflexible and computationally expensive. This inflexibility is a major hurdle when adding or removing sensors, or when adapting to new environments. The paper addresses the need for a more modular, adaptable, and computationally efficient approach to multimodal sensor fusion for object detection.

Methods

The core of the proposed method is a modular RGB-X fusion network that leverages pretrained single-modality networks. Instead of retraining everything, they use lightweight, scene-specific fusion modules to combine features from different sensors.

Here's a breakdown:

  • Leveraging Pretrained Models: The approach starts with existing, well-trained object detection models for individual modalities (e.g., an RGB detector). This significantly reduces the need for training from scratch.
  • Convolutional Block Attention Module (CBAM): They employ CBAM, a well-known attention mechanism that combines channel and spatial attention, to fuse features from different sensor modalities. This fusion happens at multiple stages within the network (mid-fusion).
  • Scene-Specific Fusion Modules: The key innovation is training separate CBAM fusion modules for different scene categories (e.g., day, night, fog). This allows the fusion process to be tailored to specific environmental conditions.
  • Auxiliary Scene Classifier: During inference, a lightweight scene classifier is used to predict the current scene category. Based on this prediction, the most appropriate pre-trained scene-specific fusion module is selected and applied. This makes the fusion process adaptive.
  • Modularity: The architecture is designed to be modular. The single-modality backbones are frozen, and only the small fusion modules are trained. This drastically reduces the number of trainable parameters and the data requirements for fusion training.

Datasets

The authors evaluated their approach on several publicly available RGB-X datasets:

  • FLIR Aligned: Contains aligned RGB-thermal image pairs with bounding box annotations for person, bicycle, and car. They manually split this into day and night categories.
  • M³FD (Multi-Modal Multi-Scene Dataset): A dataset with coregistered, time-synchronized RGB-thermal image pairs, annotated for various object classes. It's split into day, night, overcast, and challenge categories.
  • Seeing Through Fog (STF): A multispectral dataset that includes synchronized RGB/gated data, along with thermal data. It covers various weather conditions like clear, fog, and snow/rain.

Key Insights and Ideas

  • Scene Adaptivity is Crucial: The most significant insight is that fusion strategies don't need to be one-size-fits-all. Tailoring fusion modules to specific scene conditions (day vs. night, clear vs. foggy) leads to substantial performance gains.
  • Lightweight Fusion is Sufficient: The paper demonstrates that complex, end-to-end trained fusion architectures aren't always necessary. Lightweight attention-based modules, when applied adaptively, can achieve competitive or superior results with far fewer trainable parameters.
  • Pretrained Models are Powerful: The success of their approach hinges on effectively leveraging existing, powerful pretrained single-modality models. This drastically reduces the burden of data collection and training for the fusion component.
  • Interpretability through Attention: The authors suggest that the attention mechanisms within their fusion modules can also offer insights into which sensor modalities are most important for detecting objects in different scenarios.

Key Results

The proposed scene-adaptive CBAM fusion model consistently outperformed existing methods across the evaluated datasets:

  • M³FD: Achieved a 1.4% improvement in [email protected] over the state-of-the-art EAEFNet and a 1% improvement over their own scene-agnostic CBAM model.
  • FLIR Aligned: Showed significant gains, outperforming baselines by a large margin. Qualitative results indicated better detection of challenging objects like bicycles.
  • STF: The scene-adaptive model achieved comparable or slightly better performance than the scene-agnostic version, especially when trained on specific scene splits.
  • Computational Efficiency: The fusion modules are lightweight, adding minimal computational overhead. The scene-adaptive model's inference time was comparable to the scene-agnostic variant and other fusion methods.

Limitations and Future Work

  • Data Alignment: The method relies on aligned RGB-X data, which is not always readily available.
  • Known Scenes for Training: The scene-specific modules require scenes to be known during training. The approach might not perform as well in completely unexpected or novel weather conditions.
  • Future Directions: The authors suggest exploring unsupervised and online learning techniques to adapt to unexpected conditions. They also aim to leverage larger pretrained models and incorporate multitask learning for more robust fusion. Incorporating unsupervised domain adaptation for RGB-to-thermal is also mentioned.

Summary updated

BibTeX
@inproceedings{deevi2024rgb,
  author = {Sri Aditya Deevi and Connor Lee and Lu Gan and Sushruth Nagesh and Gaurav Pandey and Soon-Jo Chung},
  title = {{RGB-X Object Detection via Scene-Specific Fusion Modules}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure