SSUN-Net: Spatial-Spectral Prior-Aware Unfolding Network for Pan-Sharpening

Shijie Fang, Hongping GanAAAI 2025 · DOI

Abstract

Pan-sharpening is a critical task in remote sensing, aiming to fuse high-resolution panchromatic (PAN) and low-resolution multispectral (MS) images. While Deep Unfolding Networks (DUNs) have shown promise, they often suffer from uninterpretable "black-box" priors and a lack of multi-scale awareness. This paper introduces SSUN-Net, a lightweight unfolding network that explicitly incorporates spatial and spectral physical priors. By unfolding the Split Bregman optimization algorithm, SSUN-Net achieves superior reconstruction quality with a fraction of the computational overhead of existing models.

Introduction

In satellite remote sensing, a fundamental trade-off exists between spatial and spectral resolution. Sensors typically provide high-resolution panchromatic (PAN) images in grayscale and lower-resolution multispectral (MS) images in color. Pan-sharpening is the fusion process required to generate high-resolution multispectral imagery.

While recent Deep Unfolding Networks (DUNs) have improved this process by combining mathematical optimization with neural architectures, they face two major limitations:

  1. Physical Inconsistency: Most models rely on implicit priors, effectively treating the relationship between MS and PAN images as a black box and ignoring established physical constraints.
  2. Scale Inflexibility: Many models operate at a single scale, failing to capture both broad spectral regions and fine-grained structural details simultaneously.

The SSUN-Net Architecture

To address these issues, the authors propose the Spatial-Spectral Prior-aware Framework (SSPF), which is "unfolded" into the SSUN-Net architecture. Unlike traditional black-box models, SSUN-Net formulates pan-sharpening as a constrained minimization problem with two explicit physical priors:

  • Spectral Prior ($E_e$): Maintains the color integrity of the original low-resolution MS image.
  • Spatial Prior ($E_a$): Ensures the final image inherits the sharp structural details and gradients of the high-resolution PAN image.
Comparison between traditional DUNs and the proposed explicit, multi-scale SSUN-Net framework.

Optimization via Split Bregman

The network solves this optimization problem by unfolding the iterations of the Split Bregman algorithm into $N$ stages. Key architectural innovations include:

  • Multiscale Prior Structure (MPS): Utilizing MSEM (spectral) and MSAM (spatial) modules to extract features across multiple scales, ensuring no fine details are lost.
  • Cross-domain Feature Interaction (CFI): Facilitates communication between spatial and spectral features to reduce redundancy.
  • Iterative Stage Fusion (ISF): Fuses information from every unfolding stage to prevent information loss across the network depth.

Performance and Efficiency

SSUN-Net was evaluated on three major satellite datasets: WorldView-II, WorldView-III, and GaoFen 2. The model consistently outperformed state-of-the-art (SOTA) methods in both quantitative metrics and computational efficiency.

Performance (SSIM) vs. Computational Cost (FLOPs) comparison across various pan-sharpening models.

Key Results

  • Quantitative Superiority: On the WorldView-II dataset, SSUN-Net achieved an ERGAS (error index) of 0.8349, significantly lower than competing models like GPPNN (0.9248) or DISP (0.8759).
  • Computational Efficiency: SSUN-Net requires only 2.67G FLOPs, compared to 55.3G for DISP and 118.3G for MDCUN. This makes it highly suitable for resource-constrained satellite hardware.
  • Visual Quality: Qualitative analysis shows that SSUN-Net produces the finest textures and clearest contours, avoiding the color bleeding and blurring common in other deep learning approaches.

Conclusion

By reintroducing explicit physical priors into the deep unfolding process, SSUN-Net demonstrates that "white-box" architectures can be both more interpretable and more efficient than larger black-box models. While performance is sensitive to the number of stages ($N$), the model's ability to achieve SOTA results with minimal computational cost represents a significant advancement for real-time satellite image processing. Future work may involve adapting the model for hyperspectral data and making degradation operators more adaptive to atmospheric conditions.

Summary updated

BibTeX
@inproceedings{fang2025ssun,
  author = {Shijie Fang and Hongping Gan},
  title = {{SSUN-Net: Spatial-Spectral Prior-Aware Unfolding Network for Pan-Sharpening}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i3.32296},
}
Figure