Abstract
Multimodal image matching—finding correspondences between images from different sensors like SAR and Optical—is a fundamental challenge in computer vision due to non-linear intensity variations and the high cost of acquiring aligned training pairs. This paper presents MIFNet, a novel framework designed to learn modality-invariant features using only widely available single-modality data. By leveraging semantic priors from Stable Diffusion and a Gaussian Mixture Model (GMM)-based aggregation strategy, MIFNet achieves state-of-the-art zero-shot generalization. Our evaluation across medical (retinal) and remote sensing benchmarks shows that MIFNet significantly enhances the performance of standard descriptors, improving the Success Registration Rate (SRR) on CF-OCT data from 46.8% to 90.4%.
Introduction
While single-modality image matching (e.g., RGB-to-RGB) has seen significant progress, multimodal matching remains difficult. Sensors like Synthetic Aperture Radar (SAR), Optical, and Optical Coherence Tomography (OCT) produce images with vastly different noise patterns and geometric distortions. Traditional supervised approaches require large, perfectly aligned multimodal datasets, which are expensive and rare. MIFNet addresses this by asking if modality-invariant features can be learned from single-modality images through self-supervision, enabling zero-shot application to unseen modalities.
Methods
MIFNet employs a three-stage pipeline to bridge the modality gap without requiring paired multimodal training data.
1. Feature Extraction with Diffusion Priors
The framework utilizes a frozen Stable Diffusion v2 (SDv2) model as a feature extractor. The authors identified that the intermediate layers of the diffusion decoder contain high-level semantic information that is naturally robust to modality shifts. Specifically, the second layer of the SDv2 decoder provides an optimal balance between abstract semantics and spatial resolution.
2. Latent Feature Aggregation (LFA)
Raw diffusion features are often coarse. The LFA module uses a Gaussian Mixture Model (GMM) to cluster and refine these features. This "soft clustering" approach effectively separates semantic regions and improves intra-cluster compactness, making the resulting descriptors more distinct and less sensitive to noise.
3. Cumulative Hybrid Aggregation (CHA)
The CHA module fuses refined semantic features with "base" geometric features from standard detectors (e.g., XFeat). Using a 9-layer Graph Neural Network (GNN) with self- and cross-attention, the model progressively integrates geometric detail with semantic invariance.
The system is trained using a self-supervised strategy, where the model learns to match an image to its homographically warped version. By relying on "modality-blind" features from the diffusion model, it generalizes to different modalities at test time.
Experimental Results
MIFNet was evaluated on diverse datasets, including a newly introduced CF-FA (Color Fundus/Fluorescein Angiography) dataset, as well as remote sensing benchmarks (Optical-SAR).
Performance Gains
Integrating MIFNet with existing detectors like XFeat leads to massive performance improvements:
- Medical Imaging: On the CF-OCT dataset, the SRR jumped from 46.8% to 90.4%.
- Remote Sensing: In zero-shot Optical-SAR matching, MIFNet achieved a 36.1% SRR, outperforming methods specifically trained on paired multimodal data.
- Robustness: The framework demonstrates superior stability under extreme rotations (up to 90°), a scenario where traditional lightweight detectors often fail.
Conclusion
MIFNet demonstrates that foundation models like Stable Diffusion can serve as powerful semantic priors for multimodal tasks. By refining these features through GMM-based aggregation, the framework achieves remarkable zero-shot generalization. While the reliance on a large diffusion model introduces computational overhead, the significant gains in registration accuracy across medical and satellite imagery mark a major step forward in generalizable image matching. Future work will focus on optimizing inference latency and exploring more efficient foundation models.
Summary updated