TL;DR
CrossWeaver is a highly efficient framework for multispectral (RGB-Infrared) object detection that overcomes the challenges of weak spatial alignment and parameter redundancy. By replacing heavy dual-backbone architectures with a parameter-shared encoder and a bidirectional feature refinement network, it achieves state-of-the-art accuracy while remaining compact enough for real-time edge deployment.
What Problem Does This Paper Address?
Fusing visible (RGB) and infrared (IR) data allows object detectors to operate reliably in adverse conditions like night-time darkness or heavy occlusion. However, most existing multispectral object detection models operate under an unrealistic assumption: that the two image modalities are perfectly aligned at the pixel level. In reality, hardware differences and perspective disparities cause weak alignment.
To handle this, mainstream frameworks deploy heavy, dual-backbone networks to extract features from each modality separately, which drastically increases model size and computational cost. Furthermore, when these weakly-aligned images are downsampled through deep neural networks, small objects suffer from quantization errors—their subtle spatial offsets and structural details get flattened into single feature points, making precise localization nearly impossible.
Methods
CrossWeaver addresses these efficiency and alignment bottlenecks through three core architectural components:
- Modality-Contrastive Shared Encoder (MCSE): Instead of using two separate backbones, CrossWeaver uses a single, parameter-shared backbone to extract features from both RGB and IR images. To accommodate different sensor statistics, it maintains independent Batch Normalization (BN) layers for each modality. An auxiliary InfoNCE contrastive loss is applied to ensure high-level semantic features are tightly aligned across modalities.
- Modality-Conditioned Deformation Adapter (MCDA): To capture unique, modality-specific details without bloating the model, the authors introduce a lightweight adapter based on Low-Rank Adaptation (LoRA). It incorporates lightweight geometric priors (Laplacian filters for RGB textures; Sobel filters and local variance for IR thermal boundaries) and uses Deformable Convolutions (DCN) to explicitly align these unique features.
- Cross-Hierarchical Synergistic Network (CHSN): This network handles feature interaction using a bidirectional strategy. A top-down path propagates robust, offset-insensitive semantic guidance from higher layers to improve classification. Concurrently, a bottom-up path injects the fine-grained spatial details captured by the MCDA back into the upper layers, directly compensating for quantization errors and refining object localization.
Datasets
The model was evaluated on three distinct multispectral datasets representing different alignment conditions:
- DroneVehicle: A massive aerial dataset containing 28,439 weakly-aligned RGB-IR image pairs. It is dominated by small and medium-sized vehicles, making it the primary testbed for quantization and misalignment challenges.
- M3FD: A well-aligned, vehicle-mounted dataset used to evaluate the model's performance under ideal, synchronous conditions.
- DVTOD: A challenging dataset featuring severe, non-rigid spatial discrepancies, used to test the absolute limits of the model's alignment capabilities.
Key Insights and Ideas
- Redundancy in Dual Backbones: The authors discovered that feature maps generated by separate RGB and IR backbones are inherently highly similar, achieving a Structural Similarity Index Measure (SSIM) of 0.4880. This "aha moment" proved that duplicating backbones is inefficient; a single shared backbone is more than sufficient to capture homogeneous cross-modal representations.
- The Nature of Spatial Offsets: Statistical analysis showed that 99.98% of inter-modal spatial offsets in the DroneVehicle dataset are under 32 pixels. Because high-level feature maps (like $P_5$) have large strides, they are naturally insulated from these minor alignment noises. This makes them ideal for pure semantic guidance, leaving the job of spatial refinement to low-level features.
- Dynamic LoRA Ranking: Rather than using a fixed rank for feature modulation, dynamically tapering the LoRA rank (e.g., from $r=9$ in early layers down to $r=6$ in later layers) yields superior accuracy. This allows the network to capture broad modulations early on and compress them into compact representations later.
Key Results
CrossWeaver achieves a new state-of-the-art trade-off between accuracy, parameter count, and computational speed.
- State-of-the-Art Accuracy: On the DroneVehicle dataset, the largest variant (CrossWeaver-S) achieves 86.8% $\text{mAP}_{50}$, outperforming heavy, pretrained dual-backbone models like COMO and ICAfusion despite being trained entirely from scratch.
- Extreme Efficiency: The smallest variant (CrossWeaver-N) achieves a highly competitive 84.4% $\text{mAP}_{50}$ using a mere 5.41 million parameters and 19.5 GFLOPs.
- Edge Deployment: To test real-world viability, the authors designed a hardware-friendly "lite" version. Deployed on an NVIDIA Jetson NX platform using TensorRT, CrossWeaver-lite-N clocks an impressive 56.39 FPS (at 83.4% $\text{mAP}_{50}$), making it highly practical for drone and edge applications.
Limitations and Future Work
While CrossWeaver excels at handling weakly-aligned data, the authors note that its performance boundaries under extreme spatial misalignments or completely unpaired multispectral data remain unstudied. Future work will focus on extending the framework's robustness to handle these more severe, uncoordinated sensor scenarios in real-world deployments.
Summary updated