Deep Blind Hyperspectral Image Fusion

Wu Wang, Weihong Zeng, Yue Huang, Xinghao Ding, John W. PaisleyICCV 2019 · OpenReview

Abstract

Hyperspectral image fusion (HIF) reconstructs high spatial resolution hyperspectral images from low spatial resolution hyperspectral images and high spatial resolution multispectral images. Previous works usually assume that the linear mapping between the point spread functions of the hyperspectral camera and the spectral response functions of the conventional camera is known. This is unrealistic in many scenarios. We propose a method for blind HIF problem based on deep learning, where the estimation of the observation model and fusion process are optimized iteratively and alternatingly during the super-resolution reconstruction. In addition, the proposed framework enforces simultaneous spatial and spectral accuracy. Using three public datasets, the experimental results demonstrate that the proposed algorithm outperforms existing blind and non-blind methods.

Introduction

Hyperspectral images (HSI) are critical for environmental monitoring and object recognition due to their rich spectral information. However, hardware constraints often force a trade-off between spectral and spatial resolution. Hyperspectral Image Fusion (HIF) aims to overcome this by merging a low-resolution HSI (LR-HSI) with a high-resolution multispectral image (HR-MSI).

Traditional "non-blind" methods rely on known sensor parameters—specifically the Point Spread Function (PSF) and Spectral Response Function (SRF). In practice, these are rarely known precisely. This paper introduces the Deep Blind Iterative Fusion Network (DBIN) to solve the "blind" HIF problem, where the relationship between sensors must be learned during the fusion process.

Methodology: The DBIN Architecture

The DBIN framework treats fusion as an iterative optimization process, moving away from standard single-pass "black box" neural networks. The core of the architecture is a series of Iterative Refinement Units (IRUs) followed by a Dense Fusion (DF) layer.

The DBIN architecture featuring Iterative Refinement Units (IRUs) and the detailed structure of the Measurement and Fusion modules.

As shown in the architecture diagram, each IRU consists of two primary components:

  1. Measurement Module: This module acts as a "virtual sensor." It simulates the degradation of the current high-resolution estimate back into the original low-resolution input space. By using simple convolutional layers, the network implicitly learns the unknown PSF and SRF.
  2. Fusion Module: This module calculates the residual error between the virtual measurements and the actual inputs. It then uses a ResNet-style structure to refine the high-resolution estimate, gradually correcting spatial and spectral distortions.

The final Dense Fusion mechanism integrates features from all previous iterations, ensuring that information from different depths is preserved and utilized for the final reconstruction.

Experimental Results

The researchers evaluated DBIN against state-of-the-art blind and non-blind methods using the CAVE, Harvard, NTIRE2018, and WorldView-2 (WV2) datasets.

Quantitative Performance

DBIN demonstrated significant improvements across all standard metrics, including Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM). On the CAVE dataset, DBIN achieved a PSNR of 47.51, outperforming the leading non-blind method (DHSIS, 45.59) and drastically surpassing the leading blind method (HySure, 37.35).

Qualitative Analysis

Visually, DBIN produces sharper details and more accurate colors. Unlike traditional optimization methods that often introduce "grid-like" artifacts or over-smoothing, DBIN maintains high fidelity to the ground truth.

Qualitative comparison on the WV2 dataset: (a) Ground Truth, (b) HySure (blind), and (c) Proposed DBIN (Ours).

As seen in the comparison above, DBIN (Ours) recovers fine details—such as the characters on a license plate—with much greater clarity than other blind methods like HySure, which suffer from significant blurring and artifacts.

Conclusion

The Deep Blind Iterative Fusion Network (DBIN) represents a significant step forward in hyperspectral image processing. By combining the iterative logic of classical optimization with the representative power of deep learning, it effectively solves the blind HIF problem. Remarkably, DBIN often outperforms non-blind methods, suggesting that its learned sensor models are more robust than idealized mathematical assumptions. Future work will focus on reducing computational overhead and testing the model's generalization across diverse sensor types.

Summary updated

Abstract

Hyperspectral image fusion (HIF) reconstructs high spatial resolution hyperspectral images from low spatial resolution hyperspectral images and high spatial resolution multispectral images. Previous works usually assume that the linear mapping between the point spread functions of the hyperspectral camera and the spectral response functions of the conventional camera is known. This is unrealistic in many scenarios. We propose a method for blind HIF problem based on deep learning, where the estimation of the observation model and fusion process are optimized iteratively and alternatingly during the super-resolution reconstruction. In addition, the proposed framework enforces simultaneous spatial and spectral accuracy. Using three public datasets, the experimental results demonstrate that the proposed algorithm outperforms existing blind and non-blind methods.
BibTeX
@inproceedings{wang2019deep,
  author = {Wu Wang and Weihong Zeng and Yue Huang and Xinghao Ding and John W. Paisley},
  title = {{Deep Blind Hyperspectral Image Fusion}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  url = {https://openreview.net/forum?id=39ctT0wspLv},
}
Figure