Introduction
Visible-Infrared Image Fusion (VIF) is a critical task for enhancing situational awareness in challenging environments, such as autonomous driving and nighttime surveillance. While visible cameras provide rich textural details, they often fail in low-light conditions. Conversely, infrared sensors excel at detecting thermal signatures but lack fine spatial structure. The goal of VIF is to merge these complementary modalities into a single, high-quality image.
However, the absence of "ground truth" fused images necessitates unsupervised learning strategies. Current methods often struggle with two primary issues: the use of simplistic reference images that lead to uneven brightness, and gradient losses that only consider magnitude, ignoring the directional orientation of edges. This often results in structural artifacts and blurring.
Methodology: The AngularFuse Framework
To address these challenges, the authors propose AngularFuse, an unsupervised framework that introduces "angle-aware" perception. The architecture is built upon three core innovations designed to improve feature extraction, reference signal quality, and structural consistency.
1. Complementary Mask Generation (ComMask)
The ComMask module facilitates cross-modal feature learning by randomly masking patches in the infrared image ($I_{ir}$) and applying the inverse mask to the visible image ($I_{vi}$). This setup forces the network to reconstruct missing information in one modality by leveraging features from the other, effectively turning the fusion task into a cross-modal inpainting problem during training.
2. Fine-Grained Reference Image Synthesis (FRIS)
Rather than relying on a simple weighted average of source images, AngularFuse synthesizes a high-quality reference image ($I_{ref}$). This process involves combining Laplacian edge enhancement to sharpen textures with adaptive histogram equalization to balance brightness across the scene, preventing over- or under-exposure.
3. Angle-aware Perceptual Loss
The defining feature of this work is the treatment of image gradients as vectors. The proposed loss function constrains the fused image in the gradient domain using two terms:
- Magnitude Loss ($L_{mag}$): Ensures the edge strength aligns with the most prominent features from the source images.
- Angle Loss ($L_{angle}$): Utilizes cosine similarity between gradient vectors to ensure the orientation of edges in the fused image matches the physical structure of the scene, preventing "ghosting" effects.
Experimental Results
AngularFuse was trained on the MSRS dataset and evaluated across multiple benchmarks, including RoadScene and M3FD, to test its generalization capabilities.
Quantitative Performance
The model achieved state-of-the-art results on the MSRS test set, notably outperforming existing methods in several key metrics:
- Entropy (EN): 5% improvement, indicating higher information content.
- Standard Deviation (SD): 6% improvement, reflecting better contrast.
- Spatial Frequency (SF): 7% improvement, signifying sharper details and edges.
Qualitative Comparison
Visual evaluations demonstrate that AngularFuse produces images with superior clarity and structural integrity. As shown in the comparison below, AngularFuse maintains sharp contours for objects like vehicles and pedestrians even in dark or complex environments, where other methods may produce blurred or low-contrast results.
Efficiency
Beyond visual quality, the model is designed for practical deployment. With approximately 1.623M parameters and an average inference time of 26.42ms, AngularFuse is suitable for near-real-time applications.
Conclusion and Future Work
AngularFuse represents a significant step forward in unsupervised image fusion by emphasizing the importance of gradient directionality. By combining self-supervised masking with a sophisticated reference synthesis and angle-aware loss, it achieves a new level of visual fidelity. Future research directions include integrating the fusion framework directly with downstream tasks like object detection and addressing the challenges of unaligned multi-modal data.
Summary updated