TL;DR
This paper introduces LGTEUN, a novel deep unfolding network for pan-sharpening that enhances interpretability and performance by integrating a Local-Global Transformer (LGT) module. LGTEUN effectively captures both local and global dependencies in remote sensing images, leading to state-of-the-art results with improved efficiency.
What Problem Does This Paper Address?
Pan-sharpening is a crucial task in remote sensing that aims to fuse a low-resolution multispectral (LrMS) image with a high-resolution panchromatic (PAN) image to produce a high-resolution multispectral (HrMS) image. While deep learning (DL) methods have shown great promise, they often suffer from a "black box" nature, limiting interpretability, and struggle to efficiently capture both local and global image dependencies simultaneously. Existing methods either rely on CNNs, which are good at local features but poor at global ones, or global transformers, which are computationally expensive and can miss fine local details. This paper addresses this gap by proposing a method that balances interpretability, local feature extraction, and global context modeling for improved pan-sharpening performance and efficiency.
Methods
The core of the proposed method, LGTEUN, is built upon the concept of deep unfolding. This technique transforms an iterative optimization algorithm (specifically, proximal gradient descent for pan-sharpening) into a fixed-depth neural network. Each "stage" of the network corresponds to one iteration of the optimization algorithm, comprising two key modules:
- Data Module (D): A lightweight CNN designed to approximate the data fidelity subproblem of the optimization. This module handles the direct relationship between the input images and the desired output.
- Prior Module (P): This is the novel component, a Local-Global Transformer (LGT). The LGT is designed to act as a powerful denoiser, approximating the prior subproblem of the optimization. It uniquely combines:
- Local Branch: Utilizes window-based multi-head self-attention (WMSA) to capture local spatial dependencies within defined windows.
- Global Branch: Employs Fourier transformation to extract global contextual features in the frequency domain, providing an image-wide receptive field.
By integrating these two branches, the LGT can simultaneously model both local and global dependencies, overcoming a key limitation of previous methods. The entire LGTEUN network is then trained end-to-end.
Datasets
The paper evaluates LGTEUN on three publicly available satellite datasets:
- WorldView-3: An 8-band multispectral dataset.
- WorldView-2: A 4-band multispectral dataset.
- GaoFen-2: Another 4-band multispectral dataset.
For training and testing, the authors follow the standard practice of generating reduced-resolution data from ground-truth images. They also perform a full-resolution test on the WorldView-3 dataset. No new datasets were introduced.
Key Insights and Ideas
- Deep Unfolding for Interpretability: By unfolding an iterative optimization algorithm, the network's structure becomes more interpretable, offering insights into the pan-sharpening process compared to traditional black-box DL models.
- Synergy of Local and Global Context: The LGT module's dual-branch architecture (local spatial attention + global frequency domain analysis) is a key innovation. This allows for a more comprehensive understanding of image features, leading to better fusion.
- Frequency Domain for Global Context: Leveraging the Fourier transform for global feature extraction is an elegant way to achieve an image-wide receptive field efficiently, complementing the local window-based attention.
- Performance-Efficiency Balance: The proposed method achieves state-of-the-art performance while maintaining competitive efficiency, a crucial aspect for real-world applications.
Key Results
- Quantitative Performance: LGTEUN consistently outperforms state-of-the-art (SOTA) methods across all five standard image quality assessment (IQA) metrics (PSNR, SSIM, Q-index, SAM, ERGAS) on all three datasets for reduced-resolution testing. For example, it achieves a PSNR improvement of up to 0.2687 dB over the second-best method on WorldView-2.
- Full-Resolution Performance: The method also shows competitive results in full-resolution testing, achieving the second-best scores in DS and QNR.
- Efficiency: LGTEUN demonstrates excellent inference efficiency and promising computational cost (Params and FLOPs), achieving a good balance between performance and efficiency.
- Qualitative Results: Visual comparisons show that LGTEUN produces more visually pleasing results with fewer artifacts, particularly in zoomed-in regions, compared to other methods. Stage-wise visualizations also indicate progressive refinement of the image throughout the network's iterations.
Limitations and Future Work
The authors acknowledge two main limitations:
- Full-Resolution Performance: While competitive, the pan-sharpening results on full-resolution scenes still have room for improvement.
- Model Efficiency: Further enhancements to model efficiency could make LGTEUN even more competitive.
Future work could focus on addressing these limitations by exploring more advanced architectures for full-resolution processing and optimizing the network for even greater speed.
Summary updated