CrossWeaver: Towards Efficient Cross-Modal Interweaving and Decoupling for Weakly-Aligned Multispectral Object Detection

Haitian Yang, Juan Fang, Yiren Zhu, Xudong Zhao, Yufei Guo, Xiaohan Zhang, Xiaoxing Hu, Xue Yang, Qi MingCVPR 2026 Findings 2026 · OpenReview

TL;DR

CrossWeaver is a highly efficient framework for multispectral (RGB-Infrared) object detection that overcomes the challenges of weak spatial alignment and parameter redundancy. By replacing heavy dual-backbone architectures with a parameter-shared encoder and a bidirectional feature refinement network, it achieves state-of-the-art accuracy while remaining compact enough for real-time edge deployment.

What Problem Does This Paper Address?

Fusing visible (RGB) and infrared (IR) data allows object detectors to operate reliably in adverse conditions like night-time darkness or heavy occlusion. However, most existing multispectral object detection models operate under an unrealistic assumption: that the two image modalities are perfectly aligned at the pixel level. In reality, hardware differences and perspective disparities cause weak alignment.

To handle this, mainstream frameworks deploy heavy, dual-backbone networks to extract features from each modality separately, which drastically increases model size and computational cost. Furthermore, when these weakly-aligned images are downsampled through deep neural networks, small objects suffer from quantization errors—their subtle spatial offsets and structural details get flattened into single feature points, making precise localization nearly impossible.

Methods

CrossWeaver addresses these efficiency and alignment bottlenecks through three core architectural components:

  1. Modality-Contrastive Shared Encoder (MCSE): Instead of using two separate backbones, CrossWeaver uses a single, parameter-shared backbone to extract features from both RGB and IR images. To accommodate different sensor statistics, it maintains independent Batch Normalization (BN) layers for each modality. An auxiliary InfoNCE contrastive loss is applied to ensure high-level semantic features are tightly aligned across modalities.
  2. Modality-Conditioned Deformation Adapter (MCDA): To capture unique, modality-specific details without bloating the model, the authors introduce a lightweight adapter based on Low-Rank Adaptation (LoRA). It incorporates lightweight geometric priors (Laplacian filters for RGB textures; Sobel filters and local variance for IR thermal boundaries) and uses Deformable Convolutions (DCN) to explicitly align these unique features.
  3. Cross-Hierarchical Synergistic Network (CHSN): This network handles feature interaction using a bidirectional strategy. A top-down path propagates robust, offset-insensitive semantic guidance from higher layers to improve classification. Concurrently, a bottom-up path injects the fine-grained spatial details captured by the MCDA back into the upper layers, directly compensating for quantization errors and refining object localization.

Datasets

The model was evaluated on three distinct multispectral datasets representing different alignment conditions:

  • DroneVehicle: A massive aerial dataset containing 28,439 weakly-aligned RGB-IR image pairs. It is dominated by small and medium-sized vehicles, making it the primary testbed for quantization and misalignment challenges.
  • M3FD: A well-aligned, vehicle-mounted dataset used to evaluate the model's performance under ideal, synchronous conditions.
  • DVTOD: A challenging dataset featuring severe, non-rigid spatial discrepancies, used to test the absolute limits of the model's alignment capabilities.

Key Insights and Ideas

  • Redundancy in Dual Backbones: The authors discovered that feature maps generated by separate RGB and IR backbones are inherently highly similar, achieving a Structural Similarity Index Measure (SSIM) of 0.4880. This "aha moment" proved that duplicating backbones is inefficient; a single shared backbone is more than sufficient to capture homogeneous cross-modal representations.
Figure 2. Analysis and visualization results of objects in the Dron- eVehicle dataset. (a) The dual backbone produces feature maps with notable inherent similarity. (b) A shared backbone is suf- ficient to effectively exploit cross-modal homogeneous features. (c) The dataset is dominated by small and medium-sized objects. (d) 99.98% of inter-modal spatial offsets are less than 32 pixels.
  • The Nature of Spatial Offsets: Statistical analysis showed that 99.98% of inter-modal spatial offsets in the DroneVehicle dataset are under 32 pixels. Because high-level feature maps (like $P_5$) have large strides, they are naturally insulated from these minor alignment noises. This makes them ideal for pure semantic guidance, leaving the job of spatial refinement to low-level features.
  • Dynamic LoRA Ranking: Rather than using a fixed rank for feature modulation, dynamically tapering the LoRA rank (e.g., from $r=9$ in early layers down to $r=6$ in later layers) yields superior accuracy. This allows the network to capture broad modulations early on and compress them into compact representations later.

Key Results

CrossWeaver achieves a new state-of-the-art trade-off between accuracy, parameter count, and computational speed.

Figure 1. Comparison with state-of-the-art methods on the Dron- eVehicle dataset in terms of mAP50 and model size.
  • State-of-the-Art Accuracy: On the DroneVehicle dataset, the largest variant (CrossWeaver-S) achieves 86.8% $\text{mAP}_{50}$, outperforming heavy, pretrained dual-backbone models like COMO and ICAfusion despite being trained entirely from scratch.
  • Extreme Efficiency: The smallest variant (CrossWeaver-N) achieves a highly competitive 84.4% $\text{mAP}_{50}$ using a mere 5.41 million parameters and 19.5 GFLOPs.
  • Edge Deployment: To test real-world viability, the authors designed a hardware-friendly "lite" version. Deployed on an NVIDIA Jetson NX platform using TensorRT, CrossWeaver-lite-N clocks an impressive 56.39 FPS (at 83.4% $\text{mAP}_{50}$), making it highly practical for drone and edge applications.

Limitations and Future Work

While CrossWeaver excels at handling weakly-aligned data, the authors note that its performance boundaries under extreme spatial misalignments or completely unpaired multispectral data remain unstudied. Future work will focus on extending the framework's robustness to handle these more severe, uncoordinated sensor scenarios in real-world deployments.

Summary updated

Abstract

While single-modal object detection has made significant progress, real-world perception increasingly depends on multimodal data to capture richer visual cues. However, multimodal object detection under weak spatial alignment remains challenging. In this work, we revisit the problem from two key perspectives: (1) quantization error caused by misaligned multimodal fusion, and (2) balancing modality similarity and complementarity for robust feature integration. Building upon these insights, we propose a novel Cross-Modal Semantic Weaving Network (CrossWeaver) for weakly aligned multimodal object detection. Specifically, we design the Modality-Contrastive Shared Encoder (MCSE) to extract shared semantic representations across modalities under contrastive supervision. Then, we propose the Modality-Conditioned Deformation Adapter (MCDA) to dynamically modulate spatial sampling fields for adaptive geometric transformations and inject lightweight modality-specific priors. Finally, the Cross-Hierarchical Synergistic Network (CHSN) establishes bidirectional interactions between shared semantic and modality-aware features to mitigate quantization errors induced by weak alignment. Extensive experiments show that CrossWeaver achieves state-of-the-art performance on weakly aligned multimodal object detection benchmarks. Our method achieves the mAP_50 of 86.8% on DroneVehicle dataset.
BibTeX
@inproceedings{yang2026crossweaver,
  author = {Haitian Yang and Juan Fang and Yiren Zhu and Xudong Zhao and Yufei Guo and Xiaohan Zhang and Xiaoxing Hu and Xue Yang and Qi Ming},
  title = {{CrossWeaver: Towards Efficient Cross-Modal Interweaving and Decoupling for Weakly-Aligned Multispectral Object Detection}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2026},
  url = {https://openreview.net/forum?id=OsL418RkND},
}

Loading related content...

Figure