Variational Pansharpening for Hyperspectral Imagery Constrained by Spectral Shape and Gram–Schmidt Transformation

Zehua Huang, Qi Chen, Xiuguo LiuSensors 2018 · DOI

TL;DR

This paper introduces a new variational method for hyperspectral image pansharpening that improves both spatial detail and spectral accuracy. It achieves this by incorporating novel "spectral shape" and "correlation fidelity" terms into its energy function, which better handle the spectral distortions introduced by upsampling and ensure a stronger correlation with the original high-resolution data.

What Problem Does This Paper Address?

Hyperspectral (HS) images are incredibly rich in spectral information, making them invaluable for applications like environmental monitoring and agriculture. However, HS sensors often have lower spatial resolution compared to panchromatic (PAN) images due to hardware limitations. Pansharpening aims to fuse a low-resolution HS image with a high-resolution PAN image to create a high-resolution HS image that retains both spectral detail and spatial sharpness. Existing methods often struggle with spectral distortion (changes in spectral signatures) and low correlation between the fused image and the true high-resolution data, especially when dealing with the spatial resolution enhancement inherent in pansharpening. This paper tackles these two key challenges.

Methods

The core of the proposed method is a variational framework, which formulates pansharpening as an energy minimization problem. The authors introduce two novel "fidelity terms" to their energy function:

  • Spectral Shape Fidelity Term: This term addresses spectral distortion by considering the "spectral shape" of neighboring pixels. It uses a new weighting strategy to account for how spatial resolution changes can alter spectral information, aiming to reduce errors, particularly at pixel boundaries.
  • Correlation Fidelity Term: This term aims to improve the correlation between the pansharpened image and the actual high-resolution reference. It leverages the results from the Gram-Schmidt adaptive (GSA) method as a constraint, preventing low correlation that can arise from spectral distortions in the original HS image.

These new terms are combined with the classic spatial fidelity term (which ensures geometric consistency with the PAN image) to form a new energy function. The solution to minimizing this function yields the pansharpened HS image, computed using a gradient descent method.

Datasets

The authors evaluated their method on three datasets:

  1. HJ-1A satellite data: A dataset from Wupin, Fujian Province, featuring mountains with vegetation and buildings.
  2. EO-1 satellite data: A dataset from Hong Kong, characterized by urban and marine areas.
  3. HJ-1A satellite data: A dataset from Wuhan City, Hubei Province, including lakes and urban areas.

These datasets represent diverse terrains (natural, urban, complex) and were obtained from the HJ-1A and EO-1 satellites. Simulated high-resolution PAN and low-resolution HS images were used for evaluation.

Key Insights and Ideas

  • Spectral distortion is a consequence of spatial resolution change: The paper's core assumption is that simply preserving spectral signatures isn't enough; the method must account for how upsampling itself alters spectral information, especially at edges.
  • Neighboring pixels hold crucial spectral clues: By analyzing the spectral shape of neighboring pixels and weighting them based on similarity, the method can better reconstruct the spectral information for the target pixel.
  • GSA as a reference for correlation: Using the GSA method's output as a constraint for correlation is a clever way to leverage a known, albeit imperfect, pansharpening technique to improve the overall correlation of the final result.
  • Balancing spatial and spectral is key: The authors emphasize that their method jointly enhances spatial and spectral information, leading to better overall quality than methods that prioritize one over the other.

Key Results

The proposed method demonstrated superior performance across various quantitative metrics and visual comparisons:

  • Spectral Quality: The method significantly reduced the spectral angle (SAM) from 3.9795 to 3.2789 compared to the classic variational method. It also achieved the best SAM, ERGAS, and RMSE scores across all tested datasets.
  • Spatial Quality: The correlation coefficient (CC) increased from 0.9040 to 0.9367 compared to the classic variational method. The mean CC and UIQI values were 0.9367 and 0.8781, respectively, indicating improved spatial detail and quality.
  • Comparison: The proposed method outperformed several advanced methods, including GSA, GFPCA, SFIM, MTF-GLP, the classic variational method, and NLVD, in terms of both spatial and spectral performance.
  • Computational Time: The method's average processing time was 16.84 seconds, which is considered acceptable for most applications and faster than some other variational approaches.

Limitations and Future Work

The authors acknowledge two main limitations:

  • Spectral distortion in mixed pixels: The spectral fidelity terms might produce larger errors when dealing with mixed pixels.
  • Difficulty in perfectly balancing spatial and spectral quality: The mutual promotion between spatial and spectral qualities is challenging to achieve with current constraints.

For future work, they plan to:

  • Develop new constraints that can better integrate spatial and spectral information for improved comprehensive quality.
  • Explore automatic parameter optimization methods.

Summary updated

Abstract

Image pansharpening can generate a high-resolution hyperspectral (HS) image by combining a high-resolution panchromatic image and a HS image. In this paper, we propose a variational pansharpening method for HS imagery constrained by spectral shape and Gram⁻Schmidt (GS) transformation. The main novelties of the proposed method are the additional spectral and correlation fidelity terms. First, we design the spectral fidelity term, which utilizes the spectral shape feature of the neighboring pixels with a new weight distribution strategy to reduce spectral distortion caused by the change in spatial resolution. Second, we consider that the correlation fidelity term uses the result of GS adaptive (GSA) to constrain the correlation, thereby preventing the low correlation between the pansharpened image and the reference image. Then, the pansharpening is formulized as the minimization of a new energy function, whose solution is the pansharpened image. In comparative trials, the proposed method outperforms GSA, guided filter principal component analysis, modulation transfer function, smoothing filter-based intensity modulation, the classic and the band-decoupled variational methods. Compared with the classic variation pansharpening, our method decreases the spectral angle from 3.9795 to 3.2789, decreases the root-mean-square error from 309.6987 to 228.6753, and also increases the correlation coefficient from 0.9040 to 0.9367.
BibTeX
@inproceedings{huang2018variational,
  author = {Zehua Huang and Qi Chen and Qihao Chen and Xiuguo Liu},
  title = {{Variational Pansharpening for Hyperspectral Imagery Constrained by Spectral Shape and Gram–Schmidt Transformation}},
  booktitle = {Sensors},
  year = {2018},
  doi = {10.3390/s18124330},
}
Figure