Unsupervised Hyperspectral and Multispectral Images Fusion Based on the Cycle Consistency

Shuaikai Shi, Lijun Zhang, Yoann Altmann, Jie Chen

Abstract

Hyperspectral images (HSI) provide rich spectral information but often suffer from low spatial resolution due to hardware constraints. Conversely, multispectral images (MSI) offer high spatial resolution but limited spectral detail. Fusing these two sources to create a high-resolution hyperspectral image (HrHSI) is a cost-effective solution. Traditional fusion methods often require known spatial and spectral degradation parameters, while supervised deep learning models necessitate large amounts of ground-truth HrHSI data, which is difficult to acquire. This paper proposes CycFusion, an unsupervised fusion model based on cycle consistency. CycFusion learns domain transformations between low-resolution HSI (LrHSI) and high-resolution MSI (HrMSI), treating the desired HrHSI as an intermediate feature map. By embedding estimated degradation parameters as pre-training weights, CycFusion achieves state-of-the-art performance across multiple benchmarks without requiring ground-truth labels.

Introduction

In remote sensing and computer vision, a fundamental hardware trade-off exists between spectral and spatial resolution. While HSI is essential for precise material identification, MSI provides the fine spatial details necessary for structural analysis. Fusing these modalities to produce High-resolution Hyperspectral Images (HrHSI) is the primary objective of this research.

Existing methodologies face two significant hurdles: traditional algorithms rely on precise knowledge of the Point Spread Function (PSF) and Spectral Response Function (SRF), which are rarely known in real-world scenarios. Meanwhile, supervised deep learning models require "ground truth" HrHSI for training—data that is nearly impossible to capture directly, leading to poor generalization when trained on simulated datasets. CycFusion addresses these challenges by operating in an unsupervised manner, leveraging cycle consistency to bridge the gap between observable domains.

The CycFusion Framework

CycFusion is inspired by the CycleGAN architecture but tailored for the physical constraints of image fusion. Instead of requiring a direct target for supervision, the model learns the relationship between the LrHSI domain and the HrMSI domain through circular transformations.

The proposed CycFusion architecture, illustrating the cycle consistency between LrHSI and HrMSI domains with HrHSI as an intermediate latent bridge.

Architecture and Methodology

The framework consists of four primary modules:

  • Downsampling Modules: These estimate the PSF (spatial degradation) and SRF (spectral degradation) using single-layer CNNs where weights represent physical parameters.
  • Upsampling Modules: These handle spatial super-resolution (via interpolation and refining blocks) and spectral super-resolution.

The core innovation lies in treating the desired HrHSI as a latent bridge. The model is optimized using three loss functions: Marginal Matching to ensure transformed images match target domain distributions; Cycle Consistency to ensure image integrity across circular transformations; and Fusion Identity, which constrains the outputs of both upsampling paths to be identical, unifying spatial and spectral details.

Experimental Results

The model was evaluated on three benchmark datasets: CAVE (indoor scenes), Chikusei (aerial), and Pavia University (remote sensing). CycFusion consistently outperformed other unsupervised methods, such as GDD and CUCaNet, in both quantitative metrics and visual quality.

Qualitative and quantitative comparison on the CAVE dataset. CycFusion achieves superior PSNR and SAM scores compared to other unsupervised methods.

On the CAVE dataset, CycFusion achieved a PSNR of 43.88 dB, outperforming other blind fusion methods. It also maintained high spectral integrity, as evidenced by lower Spectral Angle Mapper (SAM) scores. Qualitatively, the model avoids common artifacts like "bumps" or "ripples" found in deep-decoder-based approaches. Furthermore, because the model estimates PSF and SRF during a pre-training phase, it demonstrates robust "blind fusion" capabilities, making it highly practical for real-world sensors with unknown specifications.

Conclusion and Future Work

CycFusion provides a robust, unsupervised solution for HSI and MSI fusion by leveraging cycle consistency and physics-informed network weights. While the current model demonstrates superior performance, future research will explore the integration of spatial-spectral attention modules to better capture correlations between light bands. Additionally, the authors aim to extend the framework to handle unpaired degraded images, further increasing its utility in field applications where perfectly aligned data is unavailable.

Summary updated

Abstract

Hyperspectral images (HSI) with abundant spectral information reflected materials property usually perform low spatial resolution due to the hardware limits. Meanwhile, multispectral images (MSI), e.g., RGB images, have a high spatial resolution but deficient spectral signatures. Hyperspectral and multispectral image fusion can be cost-effective and efficient for acquiring both high spatial resolution and high spectral resolution images. Many of the conventional HSI and MSI fusion algorithms rely on known spatial degradation parameters, i.e., point spread function, spectral degradation parameters, spectral response function, or both of them. Another class of deep learning-based models relies on the ground truth of high spatial resolution HSI and needs large amounts of paired training images when working in a supervised manner. Both of these models are limited in practical fusion scenarios. In this paper, we propose an unsupervised HSI and MSI fusion model based on the cycle consistency, called CycFusion. The CycFusion learns the domain transformation between low spatial resolution HSI (LrHSI) and high spatial resolution MSI (HrMSI), and the desired high spatial resolution HSI (HrHSI) are considered to be intermediate feature maps in the transformation networks. The CycFusion can be trained with the objective functions of marginal matching in single transform and cycle consistency in double transforms. Moreover, the estimated PSF and SRF are embedded in the model as the pre-training weights, which further enhances the practicality of our proposed model. Experiments conducted on several datasets show that our proposed model outperforms all compared unsupervised fusion methods. The codes of this paper will be available at this address: https: //github.com/shuaikaishi/CycFusion for reproducibility.
BibTeX
@misc{shi2023unsupervised,
  author = {Shuaikai Shi and Lijun Zhang and Yoann Altmann and Jie Chen},
  title = {{Unsupervised Hyperspectral and Multispectral Images Fusion Based on the Cycle Consistency}},
  year = {2023},
  eprint = {2307.03413},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2307.03413},
}
Figure