Abstract
Hyperspectral image fusion (HIF) reconstructs high spatial resolution hyperspectral images from low spatial resolution hyperspectral images and high spatial resolution multispectral images. Previous works usually assume that the linear mapping between the point spread functions of the hyperspectral camera and the spectral response functions of the conventional camera is known. This is unrealistic in many scenarios. We propose a method for blind HIF problem based on deep learning, where the estimation of the observation model and fusion process are optimized iteratively and alternatingly during the super-resolution reconstruction. In addition, the proposed framework enforces simultaneous spatial and spectral accuracy. Using three public datasets, the experimental results demonstrate that the proposed algorithm outperforms existing blind and non-blind methods.
Introduction
Hyperspectral images (HSI) are critical for environmental monitoring and object recognition due to their rich spectral information. However, hardware constraints often force a trade-off between spectral and spatial resolution. Hyperspectral Image Fusion (HIF) aims to overcome this by merging a low-resolution HSI (LR-HSI) with a high-resolution multispectral image (HR-MSI).
Traditional "non-blind" methods rely on known sensor parameters—specifically the Point Spread Function (PSF) and Spectral Response Function (SRF). In practice, these are rarely known precisely. This paper introduces the Deep Blind Iterative Fusion Network (DBIN) to solve the "blind" HIF problem, where the relationship between sensors must be learned during the fusion process.
Methodology: The DBIN Architecture
The DBIN framework treats fusion as an iterative optimization process, moving away from standard single-pass "black box" neural networks. The core of the architecture is a series of Iterative Refinement Units (IRUs) followed by a Dense Fusion (DF) layer.
As shown in the architecture diagram, each IRU consists of two primary components:
- Measurement Module: This module acts as a "virtual sensor." It simulates the degradation of the current high-resolution estimate back into the original low-resolution input space. By using simple convolutional layers, the network implicitly learns the unknown PSF and SRF.
- Fusion Module: This module calculates the residual error between the virtual measurements and the actual inputs. It then uses a ResNet-style structure to refine the high-resolution estimate, gradually correcting spatial and spectral distortions.
The final Dense Fusion mechanism integrates features from all previous iterations, ensuring that information from different depths is preserved and utilized for the final reconstruction.
Experimental Results
The researchers evaluated DBIN against state-of-the-art blind and non-blind methods using the CAVE, Harvard, NTIRE2018, and WorldView-2 (WV2) datasets.
Quantitative Performance
DBIN demonstrated significant improvements across all standard metrics, including Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM). On the CAVE dataset, DBIN achieved a PSNR of 47.51, outperforming the leading non-blind method (DHSIS, 45.59) and drastically surpassing the leading blind method (HySure, 37.35).
Qualitative Analysis
Visually, DBIN produces sharper details and more accurate colors. Unlike traditional optimization methods that often introduce "grid-like" artifacts or over-smoothing, DBIN maintains high fidelity to the ground truth.
As seen in the comparison above, DBIN (Ours) recovers fine details—such as the characters on a license plate—with much greater clarity than other blind methods like HySure, which suffer from significant blurring and artifacts.
Conclusion
The Deep Blind Iterative Fusion Network (DBIN) represents a significant step forward in hyperspectral image processing. By combining the iterative logic of classical optimization with the representative power of deep learning, it effectively solves the blind HIF problem. Remarkably, DBIN often outperforms non-blind methods, suggesting that its learned sensor models are more robust than idealized mathematical assumptions. Future work will focus on reducing computational overhead and testing the model's generalization across diverse sensor types.
Summary updated