Introduction
Visible-Infrared Object Detection (VI-OD) is a critical technology for 24/7 robust perception in fields like autonomous driving and aerial surveillance. By fusing the high-resolution texture of visible light with the thermal signatures of infrared sensors, systems can overcome lighting challenges. However, a significant bottleneck remains: the inherent misalignment between these two modalities. Traditional methods rely on complex pre-alignment steps that often fail under dynamic conditions. AlignFreeNet proposes a paradigm shift by abandoning explicit pre-alignment in favor of a lightweight, end-to-end network that handles misalignments natively.
The Challenge of Sensor Misalignment
In real-world deployments, visible and infrared sensors are rarely perfectly synchronized. The authors identify three primary types of misalignment that degrade performance:
- Spatial Offsets: Physical distance between sensors or timing lags lead to coordinate shifts.
- Resolution Discrepancies: Sensors often operate at different native resolutions (e.g., 1080p visible vs. VGA infrared).
- Semantic Deficiencies: Objects may be invisible in one modality (e.g., a pedestrian in total darkness) but prominent in the other.
Traditional "align-then-fuse" approaches often introduce structural noise or blurring when warping features to a common space, which ultimately degrades detection accuracy. AlignFreeNet treats these discrepancies as information to be managed rather than errors to be corrected before processing.
AlignFreeNet: An Alignment-Free Paradigm
AlignFreeNet introduces a dual-stream backbone designed to process unaligned inputs directly. It achieves this through three core architectural innovations:
1. Variation-guided Cross-modal Compensation (VCC)
Integrated at the shallow and intermediate stages of the network, the VCC module identifies unique information in each modality. Instead of forcing a spatial match, it uses cross-modal feedback to "compensate" for weak representations. This is particularly effective for addressing semantic deficiencies where one sensor lacks critical target data.
2. Frequency-guided Cross-modal Fusion (FCF)
At deeper network stages, the FCF module employs the Haar Wavelet Transform to decompose features into frequency components. Since low-frequency information is more stable against local spatial shifts, the network uses it as an "anchor" for fusion. Region- and channel-wise frequency gating then suppress redundant noise, preserving sharp foreground contours even when modalities are spatially offset.
3. Cross-Modulation Block (CMB)
To facilitate efficient long-range interaction, AlignFreeNet utilizes the Visual State Space (VSS/Mamba) architecture. The CMB uses a dual-branch design where one branch learns feature representations and the other acts as a soft gating signal, regulating the flow of complementary information without the heavy computational cost of standard Transformers.
Experimental Results and Efficiency
The model was rigorously tested on three benchmarks: DVTOD (severe mixed misalignment), M3FD (semantic deficiency), and DroneVehicle (light spatial offsets).
- State-of-the-Art Accuracy: On the challenging DVTOD dataset, AlignFreeNet achieved 88.0 [email protected], outperforming previous leaders like GAANet.
- Unprecedented Efficiency: The model is remarkably lightweight, with a 70% reduction in parameters (approx. 24M) compared to previous SOTA methods.
- Real-time Performance: It achieves an inference speed of over 100 FPS, roughly 5x faster than competing "align-then-fuse" frameworks.
Conclusion
AlignFreeNet demonstrates that explicit pre-alignment is not only unnecessary but can be a liability in VI-OD. By leveraging wavelet-based frequency analysis and Mamba-based feature interaction, the network provides a robust, real-time solution for multi-modal perception. Future work will explore extending this alignment-free logic to other sensor pairs, such as LiDAR-Visible fusion, and improving robustness against extreme environmental noise.
Summary updated