Abstract
Hyperspectral images (HSI) provide rich spectral information but often suffer from low spatial resolution due to hardware constraints. Conversely, multispectral images (MSI) offer high spatial resolution but limited spectral detail. Fusing these two sources to create a high-resolution hyperspectral image (HrHSI) is a cost-effective solution. Traditional fusion methods often require known spatial and spectral degradation parameters, while supervised deep learning models necessitate large amounts of ground-truth HrHSI data, which is difficult to acquire. This paper proposes CycFusion, an unsupervised fusion model based on cycle consistency. CycFusion learns domain transformations between low-resolution HSI (LrHSI) and high-resolution MSI (HrMSI), treating the desired HrHSI as an intermediate feature map. By embedding estimated degradation parameters as pre-training weights, CycFusion achieves state-of-the-art performance across multiple benchmarks without requiring ground-truth labels.
Introduction
In remote sensing and computer vision, a fundamental hardware trade-off exists between spectral and spatial resolution. While HSI is essential for precise material identification, MSI provides the fine spatial details necessary for structural analysis. Fusing these modalities to produce High-resolution Hyperspectral Images (HrHSI) is the primary objective of this research.
Existing methodologies face two significant hurdles: traditional algorithms rely on precise knowledge of the Point Spread Function (PSF) and Spectral Response Function (SRF), which are rarely known in real-world scenarios. Meanwhile, supervised deep learning models require "ground truth" HrHSI for training—data that is nearly impossible to capture directly, leading to poor generalization when trained on simulated datasets. CycFusion addresses these challenges by operating in an unsupervised manner, leveraging cycle consistency to bridge the gap between observable domains.
The CycFusion Framework
CycFusion is inspired by the CycleGAN architecture but tailored for the physical constraints of image fusion. Instead of requiring a direct target for supervision, the model learns the relationship between the LrHSI domain and the HrMSI domain through circular transformations.
Architecture and Methodology
The framework consists of four primary modules:
- Downsampling Modules: These estimate the PSF (spatial degradation) and SRF (spectral degradation) using single-layer CNNs where weights represent physical parameters.
- Upsampling Modules: These handle spatial super-resolution (via interpolation and refining blocks) and spectral super-resolution.
The core innovation lies in treating the desired HrHSI as a latent bridge. The model is optimized using three loss functions: Marginal Matching to ensure transformed images match target domain distributions; Cycle Consistency to ensure image integrity across circular transformations; and Fusion Identity, which constrains the outputs of both upsampling paths to be identical, unifying spatial and spectral details.
Experimental Results
The model was evaluated on three benchmark datasets: CAVE (indoor scenes), Chikusei (aerial), and Pavia University (remote sensing). CycFusion consistently outperformed other unsupervised methods, such as GDD and CUCaNet, in both quantitative metrics and visual quality.
On the CAVE dataset, CycFusion achieved a PSNR of 43.88 dB, outperforming other blind fusion methods. It also maintained high spectral integrity, as evidenced by lower Spectral Angle Mapper (SAM) scores. Qualitatively, the model avoids common artifacts like "bumps" or "ripples" found in deep-decoder-based approaches. Furthermore, because the model estimates PSF and SRF during a pre-training phase, it demonstrates robust "blind fusion" capabilities, making it highly practical for real-world sensors with unknown specifications.
Conclusion and Future Work
CycFusion provides a robust, unsupervised solution for HSI and MSI fusion by leveraging cycle consistency and physics-informed network weights. While the current model demonstrates superior performance, future research will explore the integration of spatial-spectral attention modules to better capture correlations between light bands. Additionally, the authors aim to extend the framework to handle unpaired degraded images, further increasing its utility in field applications where perfectly aligned data is unavailable.
Summary updated