Pansharpening via Detail Injection Based Convolutional Neural Networks

Lin He, Yizhou Rao, Jun Li, Jocelyn Chanussot, Antonio Plaza, Jiawei Zhu, Bo LiIEEE JSTARS 2019 · OpenReview

TL;DR

This paper introduces two novel Convolutional Neural Network (CNN) architectures, DiCNN1 and DiCNN2, for pansharpening. These methods leverage a "detail injection" framework, directly learning multispectral (MS) details from panchromatic (PAN) and MS images to produce high-resolution fused images with improved spectral and spatial quality, while offering better interpretability and faster convergence than previous CNN approaches.

What Problem Does This Paper Address?

Pansharpening is a crucial task in remote sensing that aims to fuse a low-spatial-resolution multispectral (MS) image with a high-spatial-resolution panchromatic (PAN) image. The goal is to create a new image that retains the spectral richness of the MS image and the fine spatial details of the PAN image. Traditional methods, like component substitution (CS) and multi-resolution analysis (MRA), often struggle to balance spectral fidelity with spatial enhancement, leading to spectral distortion or loss of fine details. While recent CNN-based methods have shown promise, they often treat pansharpening as a black-box problem, lacking clear physical interpretations and sometimes suffering from long training times or computational burdens. This paper addresses these limitations by proposing CNN architectures that are grounded in the established "detail injection" paradigm, offering both improved performance and interpretability.

Methods

The authors propose a general "detail injection" framework for pansharpening, termed DiPAN, which unifies existing CS and MRA approaches. Building upon this, they introduce two CNN-based methods:

  • DiCNN1: This architecture takes both the pre-interpolated MS image and the PAN image as input. It uses a stacked convolutional pathway to learn MS details directly, while a shortcut connection passes the pre-interpolated MS image. This design allows the network to learn the "MS details" that complement the MS image, aiming for good initialization and interpretability.
  • DiCNN2: This method is designed for greater efficiency and transfer learning capabilities. It primarily uses the PAN image as input to its convolutional pathway, with the pre-interpolated MS image still fed through a shortcut. This approach simplifies the input, potentially reducing computational load and allowing for easier fine-tuning when dealing with different sensor characteristics or even "bad bands" in test MS images.

Both DiCNNs are trained to minimize a loss function that directly reconstructs the MS details, avoiding the separate estimation of injection gains and PAN details common in traditional methods. This end-to-end learning of MS details is a key novelty.

Datasets

The paper evaluates the proposed methods on several real-world datasets:

  • WorldView-2 Washington Dataset: This dataset provides a PAN image and an 8-band MS image.
  • IKONOS Hobart Dataset: This dataset includes a PAN image and a 4-band MS image.
  • Quickbird Sundarbans Dataset: This dataset features a high-resolution PAN image and a 4-band MS image.

Experiments include reduced-resolution and full-resolution assessments, as well as transfer learning scenarios. No new datasets were introduced.

Key Insights and Ideas

  • Bridging the Gap between Traditional and Deep Learning: The DiPAN framework and the proposed DiCNNs effectively bridge the gap between interpretable traditional methods (CS/MRA) and powerful but often opaque deep learning approaches. By grounding CNNs in the "detail injection" concept, the authors achieve both performance and interpretability.
  • Direct Learning of MS Details: Instead of learning intermediate components like injection gains or PAN details separately, the DiCNNs directly learn the MS details that need to be injected. This simplifies the learning process and reduces model uncertainty.
  • Improved Initialization: Theoretical analysis and experimental results show that DiCNN1 and DiCNN2 achieve lower initial loss values compared to other CNN methods like PNN and DRPNN. This better initialization is crucial for the success of iterative optimization algorithms used in training.
  • Transfer Learning Capability: DiCNN2 demonstrates a significant advantage in its ability to perform transfer learning. By fine-tuning only the final convolutional layers, it can adapt to MS images with different numbers of bands, a highly desirable feature for real-world applications where sensor characteristics might vary.
  • Efficiency and Interpretability: The proposed DiCNNs offer a good balance between computational efficiency and clear physical interpretation, making them practical for various applications.

Key Results

  • Superior Performance: Across multiple datasets (WorldView-2, IKONOS, Quickbird), DiCNN1 and DiCNN2 consistently outperformed traditional CS/MRA methods and other CNN-based methods (PNN, DRPNN) in terms of various quality metrics like Q8, SAM, ERGAS, and SCC.
  • Spectral Fidelity: The proposed methods showed better spectral preservation, especially for small objects, and produced fewer artifacts compared to other methods.
  • Computational Efficiency: DiCNN2, in particular, achieved the least computational time among the CNN-based methods, especially in the transfer learning scenario where only the final layers were fine-tuned.
  • Reduced Initialization Loss: DiCNN1 and DiCNN2 consistently showed lower initial loss values, indicating better starting points for the optimization process.

Limitations and Future Work

  • Limitations: While the paper highlights the interpretability of the DiCNNs, the "black-box" nature of deep learning still exists to some extent. The paper also mentions that CNN formulations are usually non-convex optimization problems with local minima, which is a general challenge in deep learning.
  • Future Work: The authors suggest exploring the possibility of designing pansharpening CNNs with more hidden layers and more complex interconnections among multiple convolution layers. This indicates a direction towards deeper and more sophisticated network architectures for even better performance.

Summary updated

Abstract

Pansharpening aims to fuse a multispectral (MS) image with an associated panchromatic (PAN) image, producing a composite image with the spectral resolution of the former and the spatial resolution of the latter. Traditional pansharpening methods can be ascribed to a unified detail injection context, which views the injected MS details as the integration of PAN details and bandwise injection gains. In this paper, we design a new detail injection based convolutional neural network (DiCNN) framework for pansharpening with the MS details being directly formulated in end-to-end manners, where the first detail injection based CNN (DiCNN1) mines MS details through the PAN image and the MS image, and the second one (DiCNN2) utilizes only the PAN image. The main advantage of the proposed DiCNNs is that they provide explicit physical interpretations and can achieve fast convergence while achieving high pansharpening quality. Furthermore, the effectiveness of the proposed approaches is also analyzed from a relatively theoretical point of view. Our methods are evaluated via experiments on real MS image datasets, achieving excellent performance when compared to other state-of-the-art methods.
BibTeX
@article{he2019pansharpening,
  author = {Lin He and Yizhou Rao and Jun Li and Jocelyn Chanussot and Antonio Plaza and Jiawei Zhu and Bo Li},
  title = {{Pansharpening via Detail Injection Based Convolutional Neural Networks}},
  journal = {IEEE J. Sel. Top. Appl. Earth Obs. Remote. Sens.},
  year = {2019},
  url = {https://openreview.net/forum?id=WMKuZMW2cY},
}
Figure