Progressive Multi-Iteration Registration-Fusion Co-Optimization Network for Unregistered Hyperspectral Image Super-Resolution

Jiahui Qu, Xuyao Liu, Wenqian Dong, Yang Liu, Tongzhen Zhang, Yang Xu, Yunsong LiIEEE TGRS 2024 · OpenReview

Abstract

Existing fusion-based hyperspectral image super-resolution (fusion-based HSI-SR) methods usually reconstruct high-resolution hyperspectral image (HR-HSI) by integrating the complementary information of low-resolution hyperspectral image (LR-HSI) and high-resolution multispectral image (HR-MSI). However, most of such methods rely on accurately registered images or consider registration and fusion as a two-stage task, which means that fusion must tolerate the accumulation of errors due to misregistration. In this article, we propose a progressive multi-iteration registration-fusion co-optimization network (PMI-RFCoNet) for unregistered HSI-SR, which progressively refines the registration and fusion result over multiple levels to reconstruct registered HR-HSI. To achieve registration-fusion co-optimization, the registration-fusion co-optimization block (Co-RFB) is designed to iterate continuously over multiple levels. We embed the interactive registration module (IRM) and the spectral recalibration and fusion module (SRFU) in Co-RFB, which can facilitate the network utilizing spatial and spectral features at different levels to generate more accurate HR-HSI. Specifically, IRM generates deformation field based on spatial correlations captured at long distances to repair non-rigid pixel offsets, and SRFU further performs adaptive high-fidelity spectral correction and spatial information fusion on the registration results. We conduct experimental verification on four widely used datasets, and the results show that PMI-RFCoNet can flexibly cope with different types and degrees of non-rigid deformation and achieve superior performance. Code is available at https://github.com/Jiahuiqu/PMI-RFCoNet .
BibTeX
@article{qu2024progressive,
  author = {Jiahui Qu and Xuyao Liu and Wenqian Dong and Yang Liu and Tongzhen Zhang and Yang Xu and Yunsong Li},
  title = {{Progressive Multi-Iteration Registration-Fusion Co-Optimization Network for Unregistered Hyperspectral Image Super-Resolution}},
  journal = {IEEE Trans. Geosci. Remote. Sens.},
  year = {2024},
  url = {https://openreview.net/forum?id=1yyxiwNqLr},
}

Loading related content...

Figure