Nonlinear Intensity, Scale and Rotation Invariant Matching for Multimodal Images

Zhongli Fan, Li Zhang, Yuxuan LiuarXiv 2023 · DOI · arXiv

Introduction

Image matching is a fundamental task in computer vision, yet matching multimodal images—such as optical to infrared, MRI to PET, or optical to SAR—remains a significant challenge. These image pairs often exhibit Nonlinear Intensity Distortion (NID), where pixel values do not follow a simple linear relationship. Furthermore, significant differences in scale and arbitrary rotations between sensors frequently cause traditional gradient-based methods like SIFT to fail.

The NISR Framework

To address these challenges, the authors propose NISR (Nonlinear Intensity, Scale, and Rotation invariant matching). The framework leverages structural information rather than raw intensity values to achieve robustness across disparate sensor data.

Feature Detection and Orientation Estimation

The process begins with scale-space construction using an image pyramid to handle zooming differences. Features are detected using Phase Congruency (PC), which is inherently resistant to illumination changes because it operates in the frequency domain. A key innovation is the use of Orientation Index Maps (OIM). By convolving images with multi-scale, multi-orientated log-Gabor filters, each pixel is assigned an index corresponding to the filter with the maximum response.

Visualization of the orientation index maps and feature detection for optical and LiDAR image pairs.

To overcome rotation ambiguity, NISR employs a Double Index Map strategy, using two sets of filter layers (odd and even) to ensure distinct responses even at "half-step" rotation angles. Rotation invariance is achieved through a Weighted Centroid Orientation method, which calculates a stable primary orientation based on the structural landscape of the OIM.

Sparse-to-Dense Refinement

After initial sparse matching, the system performs a dense template match using 3D phase correlation. This transition from sparse features to a dense "cloud" of correspondences significantly improves registration precision and reliability.

Experimental Evaluation

The NISR algorithm was tested on a diverse dataset of 164 multimodal image pairs across computer vision, medical imaging, and remote sensing domains.

Comparison of NISR with state-of-the-art methods across various multimodal image pairs.

Performance Metrics

NISR demonstrated superior performance compared to state-of-the-art methods like RIFT and LNIFT:

  • Success Rate: NISR achieved a 98.8% success rate, significantly outperforming RIFT (75.6%), which often failed on pairs with large rotations.
  • Match Density: NISR found an average of 1,883 correct matches per pair, whereas competitors typically found fewer than 400.
  • Precision: The Root Mean Square Error (RMSE) was reduced to 0.98 pixels, indicating high geometric accuracy.

Conclusion

NISR provides a robust solution for multimodal image matching by prioritizing structural consistency over intensity values. While the use of log-Gabor filters introduces computational overhead, the framework's resilience to NID, scale, and rotation makes it a powerful tool for complex registration tasks. Future work will focus on integrating deep learning to optimize the generation of orientation maps and improve real-time performance.

Summary updated

BibTeX
@misc{fan2023nonlinear,
  author = {Zhongli Fan and Li Zhang and Yuxuan Liu},
  title = {{Nonlinear Intensity, Scale and Rotation Invariant Matching for Multimodal Images}},
  year = {2023},
  eprint = {2302.14239},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2302.14239},
}
Figure