U2Net: A General Framework with Spatial-Spectral-Integrated Double U-Net for Image Fusion

Siran Peng, Chenhao Guo, Xiao Wu, Liang-Jian Deng

Introduction

In the field of remote sensing and hyperspectral imaging, a persistent "hardware catch-22" exists: imaging sensors typically capture either high-spatial-resolution images with minimal spectral data (Panchromatic/PAN) or high-spectral-resolution images with low spatial detail (Multispectral/MS). Image fusion is the critical process of combining these disparate sources to produce a single high-resolution, full-color output.

Existing deep-learning approaches often suffer from three primary limitations: uniform treatment of spatial and spectral data in single-branch networks, "blunt" integration via simple concatenation, and single-scale processing that misses hierarchical features. To address these challenges, we introduce U2Net, a general framework utilizing a "Double U-Net" architecture and a sophisticated integration mechanism.

Methodology

The Double U-Net Architecture

U2Net diverges from traditional single-branch models by employing two parallel U-shaped branches, each specialized for a specific data type:

  • Spatial U-Net: This branch focuses on extracting fine structural details from high-resolution PAN or RGB images using ResNet blocks.
  • Spectral U-Net: This branch is dedicated to preserving the rich spectral information from low-resolution MS or hyperspectral images.
The U2Net architecture featuring dual U-shaped branches for spatial and spectral feature extraction.

The S2Block (Spatial-Spectral Integration)

The core of U2Net's integration is the S2Block, which utilizes a Spatial-Spectral Integration Operation (SSIO). Instead of simple stacking, the S2Block calculates self-correlation matrices for both spatial and spectral features. This allows the model to understand how pixels relate spatially and how color bands relate spectrally, facilitating a cross-pollination of information similar to the attention mechanisms found in Transformers.

Experiments and Results

The model was rigorously evaluated on two primary tasks: Pansharpening (using WorldView-3 and WorldView-2 satellite data) and Hyperspectral Image Super-Resolution (HISR) (using the CAVE dataset).

Quantitative Performance

U2Net achieved new state-of-the-art (SOTA) results across all major metrics. In pansharpening tests on the WV3 dataset, U2Net led with a PSNR of 39.117 and a SAM of 2.888, outperforming 14 other competitive methods. Furthermore, a model trained on WV3 data demonstrated exceptional generalization on WV2 data without retraining.

Qualitative Analysis

Visual evaluations using Absolute Error Maps (AEMs) highlight U2Net's precision. As shown in the comparison below, U2Net's error maps are significantly darker than those of competitors like FusionNet and FusFormer, indicating that its predictions are much closer to the ground truth, particularly in complex urban textures and fine spectral transitions.

Qualitative comparison of U2Net against FusionNet and FusFormer, highlighting superior performance in Absolute Error Maps (AEM).

Discussion and Future Work

U2Net demonstrates that discriminative learning—separating the tasks of edge detection and color preservation—prevents feature interference and leads to superior fusion. The success of the S2Block further proves that modeling internal relationships within data is more effective than simple concatenation.

While the full U2Net is a high-performance "heavyweight" model requiring significant computational resources, the authors also introduced U2Net-L, a lightweight version that maintains architectural superiority at a lower cost. Future research may extend this Double U-Net logic to other domains, such as medical imaging (MRI/CT fusion) or autonomous driving (infrared/visible light fusion), and further integrate Transformer-based attention for even more robust feature extraction.

Summary updated

Abstract

In image fusion tasks, images obtained from different sources exhibit distinct properties. Consequently, treating them uniformly with a single-branch network can lead to inadequate feature extraction. Additionally, numerous works have demonstrated that multi-scaled networks capture information more sufficiently than single-scaled models in pixel-level computer vision problems. Considering these factors, we propose U2Net, a spatial-spectral-integrated double U-shape network for image fusion. The U2Net utilizes a spatial U-Net and a spectral U-Net to extract spatial details and spectral characteristics, which allows for the discriminative and hierarchical learning of features from diverse images. In contrast to most previous works that merely employ concatenation to merge spatial and spectral information, this paper introduces a novel spatial-spectral integration structure called S2Block, which combines feature maps from different sources in a logical and effective way. We conduct a series of experiments on two image fusion tasks, including remote sensing pansharpening and hyperspectral image super-resolution (HISR). The U2Net outperforms representative state-of-the-art (SOTA) approaches in both quantitative and qualitative evaluations, demonstrating the superiority of our method. The code is available at https://github.com/PSRben/U2Net.
BibTeX
@inproceedings{peng2023u,
  author = {Siran Peng and Chenhao Guo and Xiao Wu and Liang-Jian Deng},
  title = {{U2Net: A General Framework with Spatial-Spectral-Integrated Double U-Net for Image Fusion}},
  booktitle = {ACM Multimedia},
  year = {2023},
  doi = {10.1145/3581783.3612084},
}
Figure