Unsupervised Pan-Sharpening via Mutually Guided Detail Restoration

Huangxing Lin, Yuhang Dong, Xinghao Ding, Tianpeng Liu, Yongxiang LiuAAAI 2024 · DOI

TL;DR

This paper introduces PAN-MGDR, a novel unsupervised pan-sharpening method that treats the problem as a blind deblurring task. By mutually guiding detail restoration between multispectral and panchromatic images, it implicitly learns their spatial relationship, achieving state-of-the-art results without requiring ground truth data.

What Problem Does This Paper Address?

Pan-sharpening is a crucial technique in remote sensing that aims to fuse a low-resolution multispectral (LRMS) image with a high-resolution panchromatic (PAN) image to create a high-resolution multispectral (HRMS) image. This HRMS image is vital for various applications like land cover classification and change detection. The core challenge lies in accurately modeling the complex relationship between the LRMS and PAN images to inject spatial details from the PAN image into the LRMS image without distorting its spectral information.

Traditional methods often rely on hand-crafted features, limiting their effectiveness. While supervised deep learning methods have shown great promise, they typically require ground truth data, which is often unavailable in real-world scenarios. Synthesizing training data by downsampling existing images can lead to a "scale variation" problem, where models trained on reduced resolutions perform poorly on actual full-resolution data. Unsupervised methods aim to overcome this by using prior knowledge, but they often make strong assumptions about degradation models or spectral/spatial relationships that don't always hold true, leading to robustness issues. This paper addresses the need for an unsupervised pan-sharpening method that is robust, effective, and doesn't rely on ground truth or strong, potentially inaccurate, assumptions.

Methods

The proposed method, PAN-MGDR, frames pan-sharpening as a blind image deblurring problem. The key innovations are:

  • Blind Blur Kernel Estimation (KENet): Instead of assuming a known blur kernel for the LRMS image, PAN-MGDR uses a lightweight Convolutional Neural Network (CNN) called KENet to adaptively estimate the blur kernel. This allows the method to handle unknown and varying degradation kernels. The KENet is trained with non-negativity and equality constraints to ensure a valid kernel.
  • Mutually Guided Detail Restoration: This is the core of the unsupervised approach.
  • Pan-Sharpening Network (PSNet): This network aims to restore high-resolution spatial details to the LRMS image. It's constrained by the estimated blur kernel from KENet to preserve spectral fidelity.
  • PAN Image Deblurring Network (PDNet): After PSNet generates a fused image (M), the original PAN image (P) is blurred to create Pblur. PDNet then takes the average of the spectral bands of M (Mavg) and Pblur as input to restore the spatial details of Pblur.
  • Implicit Spatial Relationship Learning: The crucial insight is that by minimizing the difference between the restored PAN image (from PDNet) and the original PAN image (P), the PSNet is implicitly forced to inject spatial details into its output (M) that are consistent with the original PAN image. This mutual guidance between the MS and PAN image restoration tasks allows the network to learn the spatial relationship between them without explicit supervision.
  • Loss Function: The overall loss function combines:
  • Lfidelity: Ensures spectral consistency between the fused image and the LRMS image, guided by the estimated blur kernel.
  • Ldetail: Drives the PDNet to restore spatial details, which indirectly guides PSNet.
  • Lgrad: A gradient consistency loss is added to prevent collapsed solutions and stabilize training, though its influence is downplayed by parameter tuning.

Datasets

The paper evaluates PAN-MGDR on three publicly available satellite datasets:

  • GaoFen2 (GF2)
  • QuickBird (QB)
  • WorldView-3 (WV3)

Experiments are conducted on both:

  • Reduced-resolution data: Synthesized following the Wald's protocol, where LRMS and PAN images are downsampled by a factor of 4, and the original LRMS image serves as the ground truth.
  • Full-resolution data: For which reference MS images are unavailable, necessitating the use of no-reference image quality assessment metrics.

No new benchmarks or datasets were introduced in this paper.

Key Insights and Ideas

  • Blind Deconvolution as a Pan-Sharpening Framework: Reframing pan-sharpening as a blind deblurring problem, where the blur kernel is unknown and needs to be estimated, is a powerful way to handle real-world degradation.
  • Mutual Guidance for Unsupervised Learning: The core idea of using two intertwined detail restoration tasks (PSNet for MS, PDNet for PAN) to implicitly learn the spatial relationship between MS and PAN images is highly effective. This avoids the need for explicit ground truth and strong prior assumptions.
  • Adaptive Kernel Estimation is Crucial: The success of the method hinges on KENet's ability to adaptively estimate the blur kernel. This makes the method robust to variations in image acquisition conditions, a significant limitation of previous unsupervised methods.
  • The Power of Indirect Supervision: By using the PAN image as a "ground truth" for the details of the blurred PAN image (via PDNet), the method indirectly supervises the PSNet to inject accurate spatial information.

Key Results

  • State-of-the-Art Performance: PAN-MGDR significantly outperforms existing unsupervised pan-sharpening methods across all three datasets (GF2, QB, WV3) on both reduced-resolution and full-resolution data.
  • Quantitative Improvements:
    • On reduced-resolution data, PAN-MGDR achieves the best SAM, ERGAS, Q4, and PSNR scores. For example, on GF2, it achieves a SAM of 1.428, ERGAS of 1.484, Q4 of 0.924, and PSNR of 35.71 (Table 1).
  • On full-resolution data, it achieves the best spectral distortion index (Dx), spatial distortion index (Ds), and hybrid QNR (HQNR) scores. For instance, on WV3, it achieves Dx of 0.037, Ds of 0.041, and HQNR of 0.905 (Table 2).
  • Qualitative Improvements: Visual results (Figure 4) show that PAN-MGDR produces fused images with clear spatial structures and preserved spectral information, surpassing other methods that exhibit spectral distortions or inject anomalous artifacts. The MSE residual maps also indicate fewer deviations from the ground truth.
  • Ablation Study: The hyperparameter λ, controlling the balance between detail restoration and fidelity losses, shows that λ=3 strikes a favorable balance, leading to superior fusion outcomes.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that the gradient consistency loss (Lgrad) might compromise the quality of the fused image, although they mitigate this by setting its weight (β) to 1 and λ to 3.
  • While the method is robust to unknown degradation kernels, the performance might still be affected by extreme noise or complex artifacts not explicitly modeled.
  • Future Work:
    • The authors suggest that the proposed method can be beneficial not only for pan-sharpening but also for other image fusion tasks, indicating a broader applicability.
  • Further exploration of the mutual guidance mechanism and its application to different fusion scenarios could be a promising direction.

Summary updated

BibTeX
@inproceedings{lin2024unsupervised,
  author = {Huangxing Lin and Yuhang Dong and Xinghao Ding and Tianpeng Liu and Yongxiang Liu},
  title = {{Unsupervised Pan-Sharpening via Mutually Guided Detail Restoration}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i4.28125},
}
Figure