Wavelet-guided Misalignment-aware Network for Visible-Infrared Object Detection

Haote Zhang, Lipeng Gu, Wuzhou Quan, Fu Lee Wang, Honghui Fan, Jiali Tang, Dingkun Zhu, Haoran Xie, Xiaoping Zhang, Mingqiang WeiarXiv 2025 · DOI · arXiv

Introduction

Visible-Infrared Object Detection (VI-OD) is a critical technology for 24/7 robust perception in fields like autonomous driving and aerial surveillance. By fusing the high-resolution texture of visible light with the thermal signatures of infrared sensors, systems can overcome lighting challenges. However, a significant bottleneck remains: the inherent misalignment between these two modalities. Traditional methods rely on complex pre-alignment steps that often fail under dynamic conditions. AlignFreeNet proposes a paradigm shift by abandoning explicit pre-alignment in favor of a lightweight, end-to-end network that handles misalignments natively.

The Challenge of Sensor Misalignment

In real-world deployments, visible and infrared sensors are rarely perfectly synchronized. The authors identify three primary types of misalignment that degrade performance:

  1. Spatial Offsets: Physical distance between sensors or timing lags lead to coordinate shifts.
  2. Resolution Discrepancies: Sensors often operate at different native resolutions (e.g., 1080p visible vs. VGA infrared).
  3. Semantic Deficiencies: Objects may be invisible in one modality (e.g., a pedestrian in total darkness) but prominent in the other.
Examples of multi-modal misalignment across different scenarios, including aerial views and street-level captures.

Traditional "align-then-fuse" approaches often introduce structural noise or blurring when warping features to a common space, which ultimately degrades detection accuracy. AlignFreeNet treats these discrepancies as information to be managed rather than errors to be corrected before processing.

AlignFreeNet: An Alignment-Free Paradigm

AlignFreeNet introduces a dual-stream backbone designed to process unaligned inputs directly. It achieves this through three core architectural innovations:

1. Variation-guided Cross-modal Compensation (VCC)

Integrated at the shallow and intermediate stages of the network, the VCC module identifies unique information in each modality. Instead of forcing a spatial match, it uses cross-modal feedback to "compensate" for weak representations. This is particularly effective for addressing semantic deficiencies where one sensor lacks critical target data.

2. Frequency-guided Cross-modal Fusion (FCF)

At deeper network stages, the FCF module employs the Haar Wavelet Transform to decompose features into frequency components. Since low-frequency information is more stable against local spatial shifts, the network uses it as an "anchor" for fusion. Region- and channel-wise frequency gating then suppress redundant noise, preserving sharp foreground contours even when modalities are spatially offset.

A detailed look at resolution and spatial misalignment between a 1920x1080 visible image and a 640x512 infrared image.

3. Cross-Modulation Block (CMB)

To facilitate efficient long-range interaction, AlignFreeNet utilizes the Visual State Space (VSS/Mamba) architecture. The CMB uses a dual-branch design where one branch learns feature representations and the other acts as a soft gating signal, regulating the flow of complementary information without the heavy computational cost of standard Transformers.

Experimental Results and Efficiency

The model was rigorously tested on three benchmarks: DVTOD (severe mixed misalignment), M3FD (semantic deficiency), and DroneVehicle (light spatial offsets).

  • State-of-the-Art Accuracy: On the challenging DVTOD dataset, AlignFreeNet achieved 88.0 [email protected], outperforming previous leaders like GAANet.
  • Unprecedented Efficiency: The model is remarkably lightweight, with a 70% reduction in parameters (approx. 24M) compared to previous SOTA methods.
  • Real-time Performance: It achieves an inference speed of over 100 FPS, roughly 5x faster than competing "align-then-fuse" frameworks.

Conclusion

AlignFreeNet demonstrates that explicit pre-alignment is not only unnecessary but can be a liability in VI-OD. By leveraging wavelet-based frequency analysis and Mamba-based feature interaction, the network provides a robust, real-time solution for multi-modal perception. Future work will explore extending this alignment-free logic to other sensor pairs, such as LiDAR-Visible fusion, and improving robustness against extreme environmental noise.

Summary updated

Abstract

Cross-modal misalignments, such as spatial offsets, resolution discrepancies, and semantic deficiencies, frequently occur in visible-infrared object detection (VI-OD). To mitigate this, existing methods are typically adapted into an alignment-based fusion paradigm, in which an explicit pixel- or feature-level alignment module is inserted before cross-modal fusion. However, pixel-level alignment struggles to cope with severe or mixed misalignments, whereas feature-level alignment often introduces undesirable noise into fused representations under such conditions, ultimately limiting detection performance. In this paper, we propose a novel alignment-free network (AlignFreeNet) for VI-OD. Differing from prior methods, AlignFreeNet abandons any explicit alignment and instead adopts an alignment-free fusion paradigm. Specifically, AlignFreeNet comprises two core modules: variation-guided cross-modal compensation (VCC) and frequency-guided cross-modal fusion (FCF). VCC adaptively feeds the compensated information derived from cross-modal discrepancies back into each modality, enhancing visible and infrared representations without the noise caused by explicit alignment. FCF achieves robust cross-modal fusion by suppressing task-irrelevant redundancy via frequency-domain gating, effectively mitigating noise introduced in the process. Moreover, VCC and FCF jointly exploit low- and high-frequency cues to preserve foreground contours in fused representations, effectively mitigating cross-modal blending caused by severe mixed misalignments. Extensive evaluations on DVTOD, M3FD, and DroneVehicle demonstrate that our AlignFreeNet achieves state-of-the-art performance under severe mixed misalignment conditions, highlighting its robustness and generalization.
BibTeX
@misc{zhang2025wavelet,
  author = {Haote Zhang and Lipeng Gu and Wuzhou Quan and Fu Lee Wang and Honghui Fan and Jiali Tang and Dingkun Zhu and Haoran Xie and Xiaoping Zhang and Mingqiang Wei},
  title = {{Wavelet-guided Misalignment-aware Network for Visible-Infrared Object Detection}},
  year = {2025},
  eprint = {2507.20146},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.20146},
}
Figure