Pansharpening Model of Transferable Remote Sensing Images Based on Feature Fusion and Attention Modules

Hui Liu, Liangfeng Deng, Yibo Dou, Xiwu Zhong, Yurong QianSensors 2023 · DOI

TL;DR

This paper introduces a novel deep learning model for pansharpening, a technique that fuses low-resolution multispectral (MS) and high-resolution panchromatic (PAN) satellite images to create a high-resolution MS image. Their approach uses a dual-branch network with an attention mechanism to optimize feature fusion, leading to improved spatial and spectral detail retention. The model demonstrates strong performance and transferability, even extending to hyperspectral image sharpening.

What Problem Does This Paper Address?

Remote sensing satellites often capture images with a trade-off between spatial and spectral resolution. Multispectral (MS) images offer rich spectral information but have lower spatial resolution, while panchromatic (PAN) images provide high spatial detail but lack spectral diversity. Pansharpening aims to combine these two types of images to produce a high-resolution multispectral image that benefits from both. Existing methods, including traditional techniques and earlier deep learning approaches, often struggle to fully preserve both spectral fidelity and spatial detail, leading to spectral distortion or loss of fine features. This paper addresses the challenge of achieving a more effective fusion that maximizes the retention of both spatial and spectral information.

Methods

The proposed method, a dual-branch fusion network, operates in the feature domain using convolutional neural networks (CNNs). It consists of three main modules:

  1. Feature Extraction Module: This module employs two subnetworks with identical structures but different weights. One subnetwork processes the PAN image, and the other processes the MS image. Both subnetworks use 3x3 convolutional layers followed by PReLU activation to extract features. Notably, pooling layers are omitted to preserve fine spatial details crucial for pansharpening.
  2. Optimized Feature Fusion Module (OFFM): This module is the core innovation. It first linearly concatenates the features extracted from the PAN and MS subnetworks. Then, a channel attention mechanism (specifically, a SENet-like approach) is applied. This mechanism learns to assign different weights to feature channels, emphasizing important information and suppressing less relevant features. This allows the network to focus on critical information for optimal fusion.
  3. Image Reconstruction Module: This module reconstructs the high-resolution multispectral image from the fused features. It uses a series of 3x3 convolutional layers. A crucial addition is a "long jump connection" that directly passes the input MS image (upsampled to match the PAN resolution) to the output. This connection helps complement spectral information and is shown to improve fusion results.

The model is trained using an L1 loss function, which the authors found to be more effective than L2 loss for preserving image edges and details.

Datasets

The paper utilizes publicly available datasets for training and evaluation:

  • GaoFen-2: A Chinese civilian optical remote sensing satellite with high spatial resolution (1m PAN, 4m MS).
  • SPOT6: A satellite that collects multispectral images with 6m spatial resolution and full-color images with 1.5m spatial resolution.

For hyperspectral image sharpening, the following datasets were used:

  • Pavia Center: A hyperspectral dataset.
  • Botswana: A hyperspectral dataset.

The authors describe a specific data splitting strategy for training and testing, where sub-regions are cropped from the MS and PAN image pairs.

Key Insights and Ideas

  • Feature Domain Fusion: Operating in the feature domain, rather than directly fusing raw pixel values, allows for more sophisticated feature extraction and manipulation, leading to better preservation of both spatial and spectral information.
  • Attention for Feature Optimization: The integration of a channel attention mechanism is key to intelligently weighting and fusing features. This allows the network to dynamically focus on the most informative channels, improving the overall fusion quality.
  • Dual-Branch Architecture: Using separate branches for PAN and MS image feature extraction allows the network to learn distinct characteristics of each input type before fusing them.
  • Long Jump Connection for Spectral Complementarity: The direct connection from the input MS image to the output helps ensure that spectral information is not lost during the fusion process, which is a common issue in pansharpening.
  • Transferability to Hyperspectral Sharpening: The model's architecture and principles are shown to be generalizable, performing well not only on traditional pansharpening but also on the more complex task of hyperspectral image sharpening.

Key Results

  • Quantitative Performance: The proposed model consistently achieved the best or near-best results across multiple evaluation metrics (PSNR, SSIM, SAM, ERGAS, SCC, Q) on both the GaoFen-2 and SPOT6 datasets when compared to classical and state-of-the-art deep learning methods. For example, on the GaoFen-2 dataset, the model achieved the highest PSNR (30.1202), SSIM (0.8659), SCC (0.9601), and Q (0.8092).
  • Qualitative Performance: Visual inspection of the fused images and their residual maps (differences from the ground truth) showed that the proposed model better preserved spatial details and spectral fidelity compared to other methods. Residual maps were predominantly dark blue, indicating minimal differences.
  • Hyperspectral Sharpening: The model also demonstrated superior performance on the Pavia Center and Botswana hyperspectral datasets, outperforming several recent methods in terms of RMSE, PSNR, ERGAS, and SAM.
  • L1 Loss Advantage: Using the L1 loss function resulted in improved training convergence and better final performance metrics compared to L2 loss.
  • Efficiency: While not the fastest, the model's computational cost was competitive, especially considering its superior performance.

Limitations and Future Work

  • Computational Cost: The paper acknowledges that the dual-branch structure increases computational cost, making the model slightly slower than some single-branch architectures.
  • Generalization to Other Sensors: While the model shows good transferability to hyperspectral images, further investigation into its performance across a wider range of remote sensing sensors and image types could be beneficial.
  • Model Complexity: Future work could explore ways to optimize the network architecture for even greater efficiency without sacrificing performance. The authors suggest that the model's ability to handle different tasks indicates its potential for further development in related areas of remote sensing image processing.

Summary updated

Abstract

The purpose of the panchromatic sharpening of remote sensing images is to generate high-resolution multispectral images through software technology without increasing economic expenditure. The specific method is to fuse the spatial information of a high-resolution panchromatic image and the spectral information of a low-resolution multispectral image. This work proposes a novel model for generating high-quality multispectral images. This model uses the feature domain of the convolution neural network to fuse multispectral and panchromatic images so that the fused images can generate new features so that the final fused features can restore clear images. Because of the unique feature extraction ability of convolution neural networks, we use the core idea of convolution neural networks to extract global features. To extract the complementary features of the input image at a deeper level, we first designed two subnetworks with the same structure but different weights, and then used single-channel attention to optimize the fused features to improve the final fusion performance. We select the public data set widely used in this field to verify the validity of the model. The experimental results on the GaoFen-2 and SPOT6 data sets show that this method has a better effect in fusing multi-spectral and panchromatic images. Compared with the classical and the latest methods in this field, our model fusion obtained panchromatic sharpened images from both quantitative and qualitative analysis has achieved better results. In addition, to verify the transferability and generalization of our proposed model, we directly apply it to multispectral image sharpening, such as hyperspectral image sharpening. Experiments and tests have been carried out on Pavia Center and Botswana public hyperspectral data sets, and the results show that the model has also achieved good performance in hyperspectral data sets.
BibTeX
@inproceedings{liu2023pansharpening,
  author = {Hui Liu and Liangfeng Deng and Yibo Dou and Xiwu Zhong and Yurong Qian},
  title = {{Pansharpening Model of Transferable Remote Sensing Images Based on Feature Fusion and Attention Modules}},
  booktitle = {Sensors},
  year = {2023},
  doi = {10.3390/s23063275},
}
Figure