Unfolding-Associative Encoder-Decoder Network with Progressive Alignment for Pansharpening

Shijie Fang, Hongping GanICCV 2025

TL;DR

This paper introduces UED-Net, a novel deep unfolding network for pansharpening that tackles limitations in existing methods by enabling richer cross-stage and cross-modal feature interactions. By employing a new Spatial-Spectral Encoding Module (SSEM) and Unfolding-Associative Attention Module (UAAM), UED-Net achieves state-of-the-art performance with improved efficiency and interpretability.

What Problem Does This Paper Address?

Pansharpening is a crucial technique in remote sensing that aims to fuse the high spatial resolution of a panchromatic (PAN) image with the high spectral resolution of a multispectral (MS) image to produce a high-resolution multispectral (HRMS) image. This is essential because sensors often have trade-offs between spatial and spectral resolution. While Deep Unfolding Networks (DUNs) have shown promise due to their interpretability, existing DUNs suffer from:

  • Limited Cross-Stage Feature Interaction: They process information sequentially, hindering the effective fusion of multi-level features across different stages of abstraction. This can lead to redundant parameters and inefficient feature extraction.
  • Insufficient Cross-Modal Feature Interaction: They struggle to effectively integrate spatial and spectral information, leading to semantic confusion, feature distortion due to misalignment, and critical information loss.

These limitations result in inaccurate degradation estimation and compromised reconstruction accuracy.

Methods

The paper proposes the Unfolding-Associative Encoder-Decoder Network (UED-Net), a novel framework that addresses the aforementioned challenges through several key innovations:

  • Spatial-Spectral Encoding Module (SSEM): This module acts as the encoder, recursively extracting multi-level spatial and spectral degradation features. It incorporates both global degradation awareness (fidelity term) and sparse multi-scale priors to capture degradation patterns in an interpretable manner. It leverages a C-SALSA solver for optimization.
  • Unfolding-Associative Attention Module (UAAM): This module serves as the decoder. It uses recurrent linear attention to aggregate associative patterns and enable cross-stage feature mapping. UAAM captures pixel-wise cross-stage correlations and applies associative attention to multi-level feature sequences, mitigating abstraction-level discrepancies.
  • Progressive Alignment Mechanism (PGAM): This component dynamically estimates inter-modal spatial offsets to compensate for misalignment-induced feature degradation. It coordinates feature distribution and alignment across modalities and stages in an adaptive manner, employing gated feature redistribution for optimized fusion.

The UED-Net architecture unfolds into N iterative stages, where each stage utilizes SSEM for encoding, PGAM for alignment, and UAAM for decoding and aggregation.

Datasets

The paper evaluates UED-Net on the following datasets:

  • Simulated Tests:

    • WorldView-II
    • WorldView-III
    • GaoFen-2 These datasets were generated following previous pansharpening methods using the Wald protocol for reduced-resolution data.
  • Real Tests:

    • GaoFen-2 (full-resolution) For real tests, since ground truth HRMS images are unavailable, the authors use no-reference metrics.

Key Insights and Ideas

  • Interpretable Unfolding for Richer Interactions: The core idea is that by unfolding the optimization process into iterative stages, we can explicitly model and leverage the interactions between different levels of spatial and spectral features, leading to better interpretability and performance than black-box DL models.
  • Synergy of SSEM and UAAM: The SSEM effectively captures degradation patterns, while the UAAM intelligently aggregates these encoded features across stages and modalities, creating a powerful encoder-decoder structure.
  • Alignment is Key for Fusion: The PGAM highlights the critical importance of accurate spatial alignment between PAN and MS images for effective multimodal fusion, especially in complex scenarios.
  • Efficiency Gains: The proposed method achieves state-of-the-art results with competitive or improved computational efficiency (FLOPs) compared to other methods, demonstrating the benefits of its design.

Key Results

  • Simulated Tests: UED-Net achieved state-of-the-art performance across multiple metrics (ERGAS, SSIM, PSNR, SCC, SAM) on WorldView-II, WorldView-III, and GaoFen-2 datasets. It demonstrated significant reductions in ERGAS compared to both conventional DUNs and SOTA pure DL-based methods.
  • Real Tests: On the GaoFen-2 dataset, UED-Net reduced spatial and spectral distortion indices (Dx, Ds) and enhanced the no-reference quality metric (QNR) compared to other methods.
  • Qualitative Results: Visual comparisons showed that UED-Net produced images with the most refined textures, fewest artifacts, and least error, particularly in magnified regions of interest.
  • Efficiency: UED-Net outperforms most pure DL-based methods in terms of computational cost (FLOPs) while achieving superior results.

Limitations and Future Work

  • Limitations: The paper does not explicitly detail limitations in the main text. However, the ablation studies suggest that while UED-Net is effective, further exploration of specific components like the ME-Shift mechanism or the progressive alignment could yield even better results. The reliance on specific optimization solvers (like C-SALSA) might also introduce dependencies.
  • Future Work: The authors suggest that their framework effectively addresses cross-modal and cross-stage feature interactions. Future work could focus on further optimizing the efficiency of the recursive encoding and decoding components, exploring more advanced attention mechanisms, and potentially extending the framework to other image restoration tasks that involve similar fusion challenges. The ablation studies also point towards further investigation into the individual contributions of each module.

Summary updated

BibTeX
@inproceedings{fang2025unfolding,
  author = {Shijie Fang and Hongping Gan},
  title = {{Unfolding-Associative Encoder-Decoder Network with Progressive Alignment for Pansharpening}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2025},
}
Figure