Multimodal Image Matching Based on Cross-Modality Completion Pre-training

Meng Yang, Fan Fan, Jun Huang, Yong Ma, Xiaoguang Mei, Zhanchuan Cai, Jiayi MaIJCAI 2025 · DOI

Abstract

Multimodal image matching, such as aligning visible light and infrared images, remains a significant challenge in computer vision due to extreme radiometric differences and the scarcity of perfectly aligned training data. We introduce XCP-Match, a novel framework that addresses these issues through a two-phase training strategy. In the first phase, the model undergoes self-supervised pre-training via a cross-modality completion task, learning to reconstruct masked image regions by leveraging information from the alternate modality. In the second phase, the model is fine-tuned for geometric precision using a multiscale feature extraction and a coarse-to-fine matching architecture. Experimental results demonstrate that XCP-Match significantly outperforms existing state-of-the-art methods across various benchmarks, including pose and homography estimation.

Introduction

Image matching is a fundamental task for applications like 3D reconstruction and autonomous navigation. While matching images within the same modality (e.g., visible light) is well-established, multimodal matching (e.g., visible vs. thermal infrared) is hindered by two main factors:

  1. Radiometric Differences: Visible and infrared sensors capture different physical properties (reflection vs. heat emission), leading to intensity inversions and texture variations.
  2. Data Scarcity: Large-scale, pixel-aligned multimodal datasets are difficult and expensive to acquire.

XCP-Match overcomes these hurdles by learning shared semantic features through cross-modality completion before fine-tuning for specific matching tasks.

Methodology

The XCP-Match framework utilizes a dual-phase training approach to bridge the modal gap and achieve high-precision matching.

Phase 1: Self-Supervised Cross-Modality Completion

The core innovation is the use of Masked Image Modeling (MIM) for pre-training. The network is presented with a pair of images (visible and infrared) of the same scene. One image is 50% masked, and the model must reconstruct the missing patches using the other modality as a reference.

Architecture of the MM-FEM and the Cross-Modality Completion Pre-training phase.

To handle the imperfect alignment often found in real-world pre-training data (like the KAIST dataset), the authors introduced an Attention-Weighted Contrastive Loss. This mechanism directs the model's focus toward overlapping regions, ensuring it learns meaningful correspondences rather than noise from non-overlapping areas.

Phase 2: Supervised Fine-Tuning and Matching Pipeline

After pre-training, the weights are integrated into a comprehensive matching pipeline. This pipeline consists of several key modules:

  • Multimodal and Multiscale Feature Extraction Module (MM-FEM): Combines VGG for local textures and Vision Transformers (ViT) for global context.
  • Coarse-to-Fine Matching: Matches are first identified at a 1/8 scale to accommodate large geometric shifts, then refined at a 1/2 scale.
  • Subpixel Refinement Module: Finalizes match coordinates with pinpoint accuracy.
The complete XCP-Match matching pipeline, including coarse-to-fine matching and subpixel refinement.

Experimental Results

XCP-Match was evaluated against state-of-the-art (SOTA) methods like XoFTR and GRiD across multiple datasets:

  • METU-VisTIR (Pose Estimation): XCP-Match achieved an AUC@5° of 19.28 in cloudy-to-cloudy scenarios, outperforming XoFTR (18.39).
  • RoadScene (Homography Estimation): The model reached an AUC@20° of 41.20, a significant improvement over XoFTR’s 32.19.
  • TriModalHuman: Demonstrated superior performance in image registration and human body segmentation alignment.

Qualitative analysis shows that XCP-Match generates denser and more accurate match points, particularly in scenes with extreme lighting and thermal variations where traditional methods often fail.

Conclusion and Future Work

XCP-Match represents a significant advancement in multimodal image matching by leveraging cross-modality completion as a powerful pre-training proxy. While highly effective, future work could address the increased computational load of the multi-stage architecture and improve robustness in "cloudy-to-sunny" scenarios where thermal signatures vary drastically. The authors also suggest extending this strategy to other sensor pairings, such as LiDAR and camera data.

Summary updated

BibTeX
@inproceedings{yang2025multimodal,
  author = {Meng Yang and Fan Fan and Jun Huang and Yong Ma and Xiaoguang Mei and Zhanchuan Cai and Jiayi Ma},
  title = {{Multimodal Image Matching Based on Cross-Modality Completion Pre-training}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/246},
}
Figure