Multispectral Object Detection Enhanced by Cross-Modal Information Complementary and Cosine Similarity Channel Resampling Modules

Junbo Jang, Chanyeong Park, Heegwang Kim, Jiyoon Lee, Joonki PaikWACV 2025

TL;DR

This paper introduces a novel multispectral object detection framework, MCOR, that effectively fuses RGB and infrared (IR) images. By employing two key modules – Cross-modal Information Complementary (CIC) and Cosine Similarity Channel Resampling (CSCR) – MCOR addresses modality imbalance and improves feature fusion, leading to state-of-the-art performance on challenging datasets.

What Problem Does This Paper Address?

Autonomous driving systems often rely on object detection to perceive their environment. However, traditional methods struggle in challenging conditions like low light or glare, where visible (RGB) cameras are compromised. Infrared (IR) cameras excel in these scenarios, offering clear outlines even in darkness. Combining RGB and IR data (multispectral object detection) promises to leverage the strengths of both, leading to more robust detection.

The challenge lies in effectively fusing these modalities. Distinct characteristics of RGB and IR images can lead to "modality imbalance," where one modality's features might dominate or be lost during fusion. This can result in suboptimal performance, increased noise, and computational complexity. Existing fusion methods often treat features from both modalities equally, failing to distinguish valuable information from noise or to effectively capture complementary insights. This paper aims to bridge this gap by proposing a sophisticated fusion mechanism that goes beyond simple addition.

Methods

The proposed framework, MCOR (Multispectral Complementary Object-weighted Resampling), builds upon the YOLOv5 architecture. It introduces two novel modules designed to enhance feature fusion:

  • Cross-modal Information Complementary (CIC) Module: This module aims to capture and leverage the unique information present in each modality. It comprises two sub-modules:
  • Modality Aware Discrepancy Learning (MADL): This sub-module explicitly models the differences between RGB and IR features. By subtracting modality features and using the difference to weight original features, it exaggerates discrepancies and selectively incorporates complementary information.
  • Modality Aware Cross Self-Attention (MACSA): This sub-module uses self-attention mechanisms to understand the interaction between modalities. It prioritizes important features and filters out noise, further enhancing the representation of complementary information.
  • Cosine Similarity Channel Resampling (CSCR) Module: This module focuses on aligning channels and supplementing features, particularly for object-centric information. It also has two sub-modules:
  • Cosine Similarity based Channel Sorting (CS2): This sub-module creates an "object-centered feature map" as a reference. It then calculates the cosine similarity between this map and features from each modality to sort channels, emphasizing object-centric features.
  • Object-weighted Channel Resampling (OCR): This sub-module addresses variability in object-centric and background-centric features. It aligns feature maps based on cosine similarity and strategically resamples features to enhance object identification, especially in complex backgrounds or with occluded objects.

Datasets

The proposed method was evaluated on several standard multispectral object detection datasets:

  • LLVIP: A dataset focused on pedestrian detection in low-light conditions.
  • FLIR: A dataset containing RGB-IR image pairs for day and night scenarios, with common object types like people, cars, and bikes.
  • M³FD: A dataset with RGB and IR image pairs for fusion and detection tasks.
  • VEDAI: A dataset for vehicle detection in aerial imagery.
  • KAIST: A large dataset for multispectral pedestrian detection, known for its challenging annotations and diverse conditions.

Key Insights and Ideas

  • Beyond Simple Fusion: The paper convincingly argues that simple fusion techniques are insufficient for multispectral object detection due to modality imbalance and noise. Sophisticated interaction and complementary learning are crucial.
  • The Power of Cosine Similarity: The CSCR module's reliance on cosine similarity for channel sorting is a key insight. This metric effectively captures structural and directional features, proving robust to lighting and scale variations, which is vital for aligning object-centric information.
  • Explicitly Modeling Differences: The MADL module's approach of directly calculating and utilizing feature differences between modalities is an effective way to highlight complementary information that might otherwise be masked.
  • Object-Centricity is Key: Both CIC and CSCR modules, in different ways, emphasize object-centric features. This focus is critical for improving detection accuracy, especially in cluttered scenes.

Key Results

The MCOR framework demonstrated significant improvements across multiple datasets:

  • FLIR Dataset: MCOR achieved a 9.5%p increase in AP50 compared to the YOLOv5 (RGB) baseline and surpassed YOLOv5 (IR) by 3.4%p.
  • LLVIP Dataset: It outperformed YOLOv5 (RGB) by 6.8%p and YOLOv5 (IR) by 3.0%p in AP50.
  • M³FD Dataset: MCOR showed a 3.7%p improvement over YOLOv5 (RGB) and a 3.0%p improvement over YOLOv5 (IR) in AP50.
  • VEDAI Dataset: MCOR achieved the highest mAP50 (76.2%) among compared models, outperforming several fusion strategies.
  • KAIST Dataset: The method achieved a lower miss rate (MR) of 7.41% (all conditions), 7.83% (day), and 6.76% (night) compared to other state-of-the-art methods, highlighting its robustness in low-light and challenging scenarios.
  • Efficiency: The proposed method achieved 35.4 FPS on an RTX 3090, indicating a good balance between accuracy and speed.

Qualitative results showed that MCOR effectively leverages the complementary nature of RGB and IR images, leading to more precise detections, especially for objects that are challenging for individual modalities.

Limitations and Future Work

  • Computational Complexity: While the paper aims to improve efficiency, multispectral fusion inherently adds computational overhead. The authors mention achieving 35.4 FPS, which is good, but further optimization for real-time applications on resource-constrained platforms might be a future direction.
  • Generalization: Although the paper demonstrates strong generalization across multiple datasets, exploring performance on even more diverse and challenging real-world scenarios (e.g., extreme weather, different sensor types) could be beneficial.
  • Ablation Study Insights: The ablation studies (Tables 6 and 8) clearly show that both CIC and CSCR modules contribute significantly to the overall performance. Future work could explore further synergistic interactions between these modules or investigate alternative ways to combine their strengths. The paper suggests that the two modules complement each other, implying that their combined effect is greater than the sum of their individual contributions.

Summary updated

BibTeX
@inproceedings{jang2025multispectral,
  author = {Junbo Jang and Chanyeong Park and Heegwang Kim and Jiyoon Lee and Joonki Paik},
  title = {{Multispectral Object Detection Enhanced by Cross-Modal Information Complementary and Cosine Similarity Channel Resampling Modules}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
}
Figure