Unsupervised Misaligned Infrared and Visible Image Fusion via Cross-Modality Image Generation and Registration

Di Wang, Jinyuan Liu, Xin Fan, Risheng Liu

Abstract

Infrared and visible image fusion (IVIF) is a critical task for enhancing scene perception in autonomous driving and surveillance. However, real-world applications often face spatial misalignment between sensors, leading to "ghosting" artifacts in the fused output. This paper proposes a Cross-modality Generation-Registration Paradigm (CGRP) that addresses this challenge by translating the visible image into a "pseudo-infrared" version. This translation converts a difficult cross-modality registration task into a simpler mono-modality one. Combined with a dual-path interaction fusion network, the proposed framework produces sharp, ghost-free images that outperform existing state-of-the-art methods.

Introduction

Infrared and visible image fusion (IVIF) leverages the complementary strengths of different spectral bands: visible light captures high-resolution textures and color information, while infrared sensors detect thermal radiation, making them indispensable for low-light or obscured environments. Most deep learning-based fusion models operate under the assumption that input image pairs are perfectly pre-registered.

In practice, this assumption is frequently violated. Hardware constraints, sensor positioning, and environmental factors often result in slight misalignments. When standard fusion models process these misaligned pairs, they generate "ghosting"—blurry, double-image artifacts that obscure critical details. Registering these images is notoriously difficult because of the significant "domain gap" between modalities. This paper introduces a robust framework designed to bridge this gap and achieve pixel-perfect alignment and fusion in an unsupervised manner.

Methodology

The authors propose the Cross-modality Generation-Registration Paradigm (CGRP), which decomposes the problem into three distinct stages: generation, registration, and fusion.

1. Cross-modality Perceptual Style Transfer Network (CPSTN)

To simplify the registration process, the CPSTN translates the visible image into a "pseudo-infrared" image. While inspired by CycleGAN architectures, the authors introduce a specialized Perceptual Style Transfer (PST) loss. This loss function ensures that the generated image prioritizes sharp geometric structures and thermal boundaries over mere texture mimicry, creating a reliable reference for the subsequent registration step.

2. Multi-level Refinement Registration Network (MRRN)

With the pseudo-IR image acting as a bridge, the registration task is transformed from a cross-modality problem into a mono-modality one. The MRRN compares the real (distorted) infrared image with the generated pseudo-IR image. It utilizes a coarse-to-fine strategy to predict a displacement vector field, which is then used to warp the real IR image into precise alignment with the visible source.

3. Dual-path Interaction Fusion Network (DIFN)

Once the images are aligned, they are passed to the DIFN for fusion. A key innovation here is the Interaction Fusion Module (IFM). Unlike standard fusion methods that rely on simple concatenation or averaging—which can lead to feature smoothing—the IFM adaptively selects the most salient features from both paths. This ensures that the final output retains the sharpest textures from the visible light and the most prominent heat signatures from the infrared data.

Experimental Results

The researchers evaluated the CGRP framework on two benchmark datasets: TNO (military/surveillance) and RoadScene (traffic scenarios). To simulate real-world conditions, the authors applied synthetic affine and elastic transformations to create misaligned pairs.

Quantitative Performance

The CGRP method demonstrated significant improvements in registration accuracy, increasing Mutual Information (MI) by approximately 0.10 on TNO and 0.23 on RoadScene compared to established registration algorithms like FlowNet and VoxelMorph. In terms of fusion quality, the model outperformed eight state-of-the-art methods (including DENSE and DIDFuse) across metrics such as Cross Correlation (CC) and Structural Similarity (SSIM).

Qualitative Performance

Visual inspections confirm that the CGRP framework effectively eliminates ghosting artifacts. The resulting fused images exhibit higher contrast, sharper edges, and more distinct thermal targets compared to models that do not account for misalignment or those that use standard cross-modality registration techniques.

Discussion and Conclusion

The core insight of this research is that image translation can serve as an effective bridge for multi-modality registration. By prioritizing geometric structures over textures, the CPSTN provides the necessary "anchors" for the MRRN to achieve high-precision alignment.

While the three-stage process introduces some computational complexity, the authors' efficiency analysis suggests it remains competitive for practical use. Future work may focus on reducing the memory footprint for deployment on edge devices and testing the paradigm's robustness in extreme weather conditions where initial style transfer might be more challenging. Ultimately, the CGRP framework provides a robust solution for real-world IVIF applications where perfect sensor alignment cannot be guaranteed.

Summary updated

BibTeX
@inproceedings{wang2022unsupervised,
  author = {Di Wang and Jinyuan Liu and Xin Fan and Risheng Liu},
  title = {{Unsupervised Misaligned Infrared and Visible Image Fusion via Cross-Modality Image Generation and Registration}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.24963/ijcai.2022/487},
}
Figure