Efficient Layer-Wise Cross-View Calibration and Aggregation for Multispectral Object Detection

Xiao He, Tong Yang, Tingzhou Yan, Hongtao Li, Yang Ge, Zhijun Ren, Zhe Liu, Jiahe Jiang, Cong TangElectronics 2026 · DOI

TL;DR

This paper introduces LCMA, a novel one-stage object detector for RGB-Infrared (IR) imagery that efficiently aligns and fuses features from both modalities. By employing a layer-wise calibration and aggregation strategy, LCMA overcomes the speed limitations of previous two-stage methods, achieving state-of-the-art accuracy and improved computational efficiency for robust multispectral object detection.

What Problem Does This Paper Address?

Object detection in aerial imagery is crucial, but visible (RGB) cameras struggle in low-light or adverse weather conditions. Infrared (IR) cameras offer a complementary view, but fusing RGB and IR data presents challenges. Existing methods often rely on two-stage detectors that require computationally expensive region proposal steps for feature alignment, making them too slow for large-scale applications. Furthermore, these methods often assume perfect pixel-level alignment between RGB and IR images, which is rarely the case due to sensor synchronization issues, leading to inaccuracies. This paper aims to bridge this gap by developing an efficient, one-stage detector that can effectively handle cross-modality alignment and fusion for robust multispectral object detection.

Methods

The core of the proposed LCMA (Layer-wise Cross-Modality Calibration and Aggregation) detector is its novel one-stage architecture. Instead of aligning features at the region proposal stage, LCMA employs a layer-wise strategy for cross-modality calibration and aggregation.

Key components include:

  • Inter-Modality Spatial-Reduction Attention (ISRA): This module is embedded in each layer of the network. It uses a neighborhood-aware aggregation approach to calibrate and align features from RGB and IR modalities. ISRA leverages a spatial-reduction mechanism and shared cross-modal attention weights to reduce computational burden while effectively capturing contextual information within and across modalities. This helps to mitigate the impact of weak pixel-level misalignments.
  • Gated Coupled Filter (GCF): This module is also integrated into each layer. It addresses the issue of redundant or low-quality information (e.g., from poorly illuminated RGB images) by coupling multi-modality features and using an MLP to filter out irrelevant information. This ensures that only semantically meaningful and well-aligned features are passed to the detection head.

By integrating ISRA and GCF within a layer-wise framework, LCMA avoids the need for region proposals for alignment, enabling direct prediction of object categories and bounding boxes in a unified one-stage detector.

Datasets

The authors evaluated their method on three datasets:

  • DroneVehicle: This dataset contains RGB-IR image pairs captured by drones, covering daytime, nighttime, and low-light conditions. It features five categories of automobiles and provides oriented bounding box annotations.
  • LLVIP: This is a recently released RGB-IR paired pedestrian dataset specifically designed for low-light scenarios, captured by surveillance cameras. The images are strictly spatiotemporally aligned.
  • FLIR: This dataset is tailored for object detection, with images captured under various day and night conditions. It has been refined to ensure precisely aligned image pairs and includes categories like person, car, and bicycle.

Key Insights and Ideas

  • Layer-wise alignment is key: The paper demonstrates that performing cross-modality alignment and aggregation layer by layer, rather than in a single fusion step or at the region proposal stage, is crucial for both accuracy and efficiency.
  • Neighborhood-aware attention for robustness: The ISRA module's ability to consider neighboring pixels and aggregate contextual information is vital for handling the inherent pixel-level misalignments between RGB and IR sensors.
  • Filtering redundant information is critical: The GCF module's role in filtering out low-quality or irrelevant features from one modality (e.g., noisy RGB in low light) significantly improves the robustness and accuracy of the fused features.
  • One-stage detectors can be efficient and accurate: LCMA proves that it's possible to achieve high performance without the computational overhead of traditional two-stage detectors, making it suitable for real-time applications.

Key Results

  • DroneVehicle Dataset: LCMA achieved a mAP of 77.19%, outperforming the previous state-of-the-art TSFDet by 7.84%. It also showed significant improvements in classification accuracy for specific categories like trucks (+32%) and vans (+16%) compared to a baseline.
  • LLVIP Dataset: LCMA achieved a 2.5% improvement in [email protected]:.95 over DCMNet, demonstrating its effectiveness in challenging low-light conditions.
  • FLIR Dataset: LCMA achieved state-of-the-art performance, outperforming existing approaches by 1.3% in [email protected] and 0.9% in [email protected]:.95.
  • Efficiency: LCMA-tiny achieved a speed of 43.9 FPS with a [email protected] of 73.89%, highlighting its suitability for real-time applications on resource-constrained devices like UAVs.

Limitations and Future Work

  • Sensitivity to image quality: The paper acknowledges that the layer-wise calibration and attention mechanisms might still be sensitive to variations in image quality and discrepancies between modalities, especially under extreme lighting or weather conditions. This limits generalization in real-world scenarios.
  • Performance in complex scenes: LCMA's performance in complex real-world scenes with heavy occlusions or cluttered backgrounds requires further improvement.
  • Simultaneous modality degradation: The framework struggles when both modalities degrade simultaneously (e.g., thermal diffusion blurring IR and RGB textures lost to occlusion).
  • Semantic conflicts: Resolving semantic conflicts arising from strong infrared backgrounds contradicting RGB cues remains a challenge.

Future work could involve integrating low-light enhancement algorithms to improve performance under challenging lighting. The authors also suggest developing a unified framework that combines multispectral image enhancement and fusion techniques to further enhance robustness and versatility.

Summary updated

Abstract

Multispectral object detection is a fundamental task with an extensive range of practical implications. In particular, combining visible (RGB) and infrared (IR) images can offer complementary information that enhances detection performance in different weather scenarios. However, the existing methods generally involve aligning features across modalities and require proposals for the two-stage detectors, which are often slow and unsuitable for large-scale applications. To overcome this challenge, we introduce a novel one-stage oriented detector for RGB-infrared object detection called the Layer-wise Cross-Modality calibration and Aggregation (LCMA) detector. LCMA employs a layer-wise strategy to achieve cross-modality alignment by using the proposed inter-modality spatial-reduction attention. Moreover, we design Gated Coupled Filter in each layer to capture semantically meaningful features while ensuring that well-aligned and foreground object information is obtained before forwarding them to the detection head. This relieves the need for a region proposal step for the alignment, enabling direct category and bounding box predictions in a unified one-stage oriented detector. Extensive experiments on two challenging datasets demonstrate that the proposed LCMA outperforms state-of-the-art methods in terms of both accuracy and computational efficiency, which implies the efficacy of our approach in exploiting multi-modality information for robust and efficient multispectral object detection.
BibTeX
@inproceedings{he2026efficient,
  author = {Xiao He and Tong Yang and Tingzhou Yan and Hongtao Li and Yang Ge and Zhijun Ren and Zhe Liu and Jiahe Jiang and Cong Tang},
  title = {{Efficient Layer-Wise Cross-View Calibration and Aggregation for Multispectral Object Detection}},
  booktitle = {Electronics},
  year = {2026},
  doi = {10.3390/electronics15030498},
}
Figure