Abstract
Multimodal image matching, such as aligning visible light and infrared images, remains a significant challenge in computer vision due to extreme radiometric differences and the scarcity of perfectly aligned training data. We introduce XCP-Match, a novel framework that addresses these issues through a two-phase training strategy. In the first phase, the model undergoes self-supervised pre-training via a cross-modality completion task, learning to reconstruct masked image regions by leveraging information from the alternate modality. In the second phase, the model is fine-tuned for geometric precision using a multiscale feature extraction and a coarse-to-fine matching architecture. Experimental results demonstrate that XCP-Match significantly outperforms existing state-of-the-art methods across various benchmarks, including pose and homography estimation.
Introduction
Image matching is a fundamental task for applications like 3D reconstruction and autonomous navigation. While matching images within the same modality (e.g., visible light) is well-established, multimodal matching (e.g., visible vs. thermal infrared) is hindered by two main factors:
- Radiometric Differences: Visible and infrared sensors capture different physical properties (reflection vs. heat emission), leading to intensity inversions and texture variations.
- Data Scarcity: Large-scale, pixel-aligned multimodal datasets are difficult and expensive to acquire.
XCP-Match overcomes these hurdles by learning shared semantic features through cross-modality completion before fine-tuning for specific matching tasks.
Methodology
The XCP-Match framework utilizes a dual-phase training approach to bridge the modal gap and achieve high-precision matching.
Phase 1: Self-Supervised Cross-Modality Completion
The core innovation is the use of Masked Image Modeling (MIM) for pre-training. The network is presented with a pair of images (visible and infrared) of the same scene. One image is 50% masked, and the model must reconstruct the missing patches using the other modality as a reference.
To handle the imperfect alignment often found in real-world pre-training data (like the KAIST dataset), the authors introduced an Attention-Weighted Contrastive Loss. This mechanism directs the model's focus toward overlapping regions, ensuring it learns meaningful correspondences rather than noise from non-overlapping areas.
Phase 2: Supervised Fine-Tuning and Matching Pipeline
After pre-training, the weights are integrated into a comprehensive matching pipeline. This pipeline consists of several key modules:
- Multimodal and Multiscale Feature Extraction Module (MM-FEM): Combines VGG for local textures and Vision Transformers (ViT) for global context.
- Coarse-to-Fine Matching: Matches are first identified at a 1/8 scale to accommodate large geometric shifts, then refined at a 1/2 scale.
- Subpixel Refinement Module: Finalizes match coordinates with pinpoint accuracy.
Experimental Results
XCP-Match was evaluated against state-of-the-art (SOTA) methods like XoFTR and GRiD across multiple datasets:
- METU-VisTIR (Pose Estimation): XCP-Match achieved an AUC@5° of 19.28 in cloudy-to-cloudy scenarios, outperforming XoFTR (18.39).
- RoadScene (Homography Estimation): The model reached an AUC@20° of 41.20, a significant improvement over XoFTR’s 32.19.
- TriModalHuman: Demonstrated superior performance in image registration and human body segmentation alignment.
Qualitative analysis shows that XCP-Match generates denser and more accurate match points, particularly in scenes with extreme lighting and thermal variations where traditional methods often fail.
Conclusion and Future Work
XCP-Match represents a significant advancement in multimodal image matching by leveraging cross-modality completion as a powerful pre-training proxy. While highly effective, future work could address the increased computational load of the multi-stage architecture and improve robustness in "cloudy-to-sunny" scenarios where thermal signatures vary drastically. The authors also suggest extending this strategy to other sensor pairings, such as LiDAR and camera data.
Summary updated