Spatial-Spectral Similarity-Guided Fusion Network for Pansharpening

Jiazhuang Xiong, Yongshan Zhang, Xinxin Wang, Lefei ZhangIJCAI 2025 · DOI

Abstract

Pansharpening is a critical remote sensing task that fuses high-spatial-resolution panchromatic (PAN) images with high-spectral-resolution multispectral (MS) images. Traditional deep learning approaches often fail to fully exploit the complementary information between these modalities. We present the Spatial-Spectral Similarity-Guided Fusion Network ($S^3$FNet), which utilizes a novel triple-branch encoder and a loss function guided by spatial and spectral correlations. Our model achieves state-of-the-art performance across multiple satellite datasets, producing high-fidelity imagery with minimal artifacts.

Introduction

Satellite sensors face a fundamental physical constraint: they can either capture high-spatial-resolution images with limited color (PAN) or high-spectral-resolution images with low spatial resolution (LRMS). Pansharpening is the process of computationally fusing these two inputs to create a high-resolution multispectral (HRMS) image.

Current deep learning models for pansharpening typically suffer from two main issues. First, they often use simplistic single-branch or dual-branch architectures that do not adequately capture the complex interactions between spatial and spectral features. Second, they frequently overlook the specific correlations that exist between the source images and the desired ground truth—specifically, that MS images provide the spectral "DNA" while PAN images provide the spatial "DNA."

Motivation for similarity-guided fusion: (a) Ground Truth, (b) LRMS showing high spectral correlation, (c) PAN showing high spatial correlation. (d-e) Consistency of these correlations across samples.

Methodology

The $S^3$FNet architecture introduces three primary innovations to address these challenges.

Multi-Branch Asymmetric Encoder

Unlike standard models that stack inputs or process them in parallel branches, $S^3$FNet employs a triple-branch structure. It includes a Spatial Branch for textures and edges, a Spectral Branch for color data, and a dedicated Fusion Branch that learns cross-modal interactions at multiple scales. This design allows the model to learn "cross-modal" features that are not present in either input alone.

Comparison of pansharpening architectures: (a) Single-branch, (b) Dual-branch, and (c) the proposed Triple-branch $S^3$FNet.

Cross-Feature Multi-head Attention Fusion Block (CMAFB)

In the decoder, $S^3$FNet utilizes CMAFB to refine the fusion process. This block incorporates Spatial Cross-Attention (SpaCA) and Spectral Cross-Attention (SpeCA) to selectively integrate details from the spatial and spectral branches based on their relevance to the fused features. This ensures the model knows exactly which details to pull from which source.

Similarity-Constrained Loss

To ensure the output respects the physical properties of the sensors, we introduce a loss function based on Spearman correlation. By using correlation as a training signal rather than just standard pixel-wise error, the model is explicitly taught to preserve the high spectral similarity between the output and the LRMS input, and the high spatial similarity between the output and the PAN input.

Experiments and Results

We evaluated $S^3$FNet on the WorldView-3 (WV3), GaoFen-2 (GF2), and WorldView-2 (WV2) datasets. The model was compared against 11 state-of-the-art methods, including Pan-Mamba and BiMPan.

Quantitative results on the WV3 dataset show that $S^3$FNet achieves superior scores across all metrics. For instance, it reached a Spectral Angle Mapper (SAM) score of 2.8194, outperforming the next best model (Pan-Mamba at 2.8444). Visual analysis through absolute error maps confirms that $S^3$FNet produces significantly fewer residuals and artifacts, especially around sharp edges and complex urban textures. The model also demonstrated strong generalization on full-resolution datasets where no ground truth is available.

Conclusion

$S^3$FNet represents a significant advancement in pansharpening by explicitly modeling the asymmetric relationship between spatial and spectral information. By combining a triple-branch encoder with a similarity-guided loss, the model achieves unprecedented fidelity in high-resolution satellite imagery. Future work will focus on optimizing computational efficiency and exploring the adaptation of this multi-branch attention mechanism for other multi-modal remote sensing tasks, such as cloud removal or change detection.

Summary updated

BibTeX
@inproceedings{xiong2025spatial,
  author = {Jiazhuang Xiong and Yongshan Zhang and Xinxin Wang and Lefei Zhang},
  title = {{Spatial-Spectral Similarity-Guided Fusion Network for Pansharpening}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/233},
}
Figure