A New Look at Image Fusion Methods from a Bayesian Perspective

Hankui K. Zhang, Bo HuangRemote 2015 · DOI

TL;DR

This paper reframes existing image fusion techniques (Component Substitution and Multi-Resolution Analysis) as special cases of a unified Bayesian data fusion framework. By introducing a spectral consistency model, they demonstrate that spectral preservation and spatial enhancement are complementary, not competitive, leading to improved fusion results.

What Problem Does This Paper Address?

Remote sensing satellites often capture images with a trade-off between spatial and spectral resolution. Panchromatic (Pan) images offer high spatial detail but limited spectral information, while multispectral (MS) images provide rich spectral data but at a lower spatial resolution. Image fusion aims to combine these to create a high-resolution MS image. However, existing methods, broadly categorized as Component Substitution (CS) and Multi-Resolution Analysis (MRA), have led to contradictory conclusions regarding their performance. Key questions include:

  • Is spatial enhancement better in CS or MRA methods?
  • Is there a fundamental tradeoff between spatial enhancement and spectral preservation?
  • How can spectral consistency, as defined by Wald et al. (requiring a fused image to be identical to the original when degraded), be achieved?

Methods

The core contribution of this paper is the development of a unified Bayesian data fusion framework that encompasses both CS and MRA methods. They model the fusion process in two steps:

  1. Modeling Step:
    • Observation Model: Relates the low-resolution MS image (z) to the desired high-resolution MS image (Z) through a low-pass filtering and down-sampling process (H). This model incorporates the sensor's Point Spread Function (PSF).
  2. Spatial Injection Model: Assumes a linear correlation between the Pan image (X) and the MS images (Z), represented by X = g(Z) + e. This model is crucial for injecting spatial details.
  3. Spectral Preservation Model: Assumes the desired HR MS images (Z) follow a Gaussian distribution, with the up-sampled LR MS images (ž) serving as the mean. This is a key assumption for spectral preservation.
  4. Inversion Step (MAP Estimation):
    • They derive the Maximum A Posteriori (MAP) estimate for Z. Initially, they show that traditional EGIF methods neglect the spectral consistency term p(z|Z).
  5. Novelty: They introduce a spectral consistency model into the Bayesian framework. This model explicitly aims to satisfy Wald's definition of spectral consistency, ensuring the fused image, when degraded, matches the original.
  6. To solve the resulting optimization problem, especially for arbitrary sensor PSFs, they employ a conjugate gradient search algorithm. This avoids the formation and inversion of large matrices, making it computationally feasible.

Datasets

The authors used two datasets for experimental validation:

  • QuickBird: 0.6-m Pan and 2.4-m MS images.
  • WorldView-2: 0.46-m Pan and 1.84-m MS images (commercially available at 0.5-m and 2.0-m resolution).

These datasets were used for both reduced-scale (degrading images before fusion) and full-scale experiments.

Key Insights and Ideas

  • Unified Framework: CS and MRA methods are not fundamentally different but rather special cases of a Bayesian data fusion approach, differing in how they model the correlation between MS and Pan bands.
  • Weight Parameter Control: A single weight parameter (s in CS, sq in MRA) can control the balance between spectral preservation and spatial injection, suggesting that the perceived tradeoff is an artifact of specific implementations rather than an inherent limitation.
  • Spectral Consistency is Key: The traditional EGIF methods' assumption that up-sampled LR MS images are spectrally consistent is flawed. True spectral consistency, as defined by Wald, is achieved by incorporating a dedicated spectral consistency model.
  • Complementary Enhancement: Spectral consistency and spatial enhancement are not competitive but complementary. Adding spectral consistency improves overall fusion quality without significantly sacrificing spatial detail.
  • Bayesian Perspective Solves Puzzles: The Bayesian framework provides a unified lens to answer long-standing questions about pansharpening, including the relationship between CS and MRA, the existence of the tradeoff, and the achievement of spectral consistency.

Key Results

  • Improved Performance: The proposed spectral consistency model significantly improved the performance of traditional EGIF methods (GS, GLP-M3, GLP-ECB) across various quantitative metrics (ERGAS, Q4, SAM, SNR, Da, Ds, QNR).
  • Spatial Enhancement: The GLP-M3 method, with an appropriately adjusted s value (e.g., > 0.75), demonstrated better spatial enhancement than the GS method.
  • Spectral Consistency Achieved: Fused images with the spectral consistency constraint were spectrally consistent with the original LR MS images, as evidenced by Q4 values close to 1 and small ERGAS and SAM values.
  • No Significant Spatial Loss: Adding spectral consistency resulted in only a negligible decrease in spatial enhancement.
  • Computational Efficiency: The conjugate gradient search algorithm, while computationally intensive, was deemed acceptable for typical image sizes and can be further optimized with parallel computing.

Limitations and Future Work

  • Assumptions: The Bayesian framework relies on assumptions of Gaussian distribution and spatial independence. The authors acknowledge these might not be practical in all cases.
  • Future Research:
    • Image Segmentation: Segmenting images into regions to apply more localized assumptions for Gaussian distribution and spatial independence.
  • Non-Gaussian Distributions and Spatial Correlation: Exploring non-Gaussian distribution assumptions and incorporating spatial correlation, potentially using methods like co-kriging downscaling. This may require more complex optimization techniques.
  • Scale Effect: Further research is needed to address the scale effect, which can influence the estimation of variance and covariance.

Summary updated

Abstract

Component substitution (CS) and multi-resolution analysis (MRA) are the two basic categories in the extended general image fusion (EGIF) framework for fusing panchromatic (Pan) and multispectral (MS) images. Despite of the method diversity, there are some unaddressed questions and contradictory conclusions about fusion. For example, is the spatial enhancement of CS methods better than MRA methods? Is spatial enhancement and spectral preservation competitive? How to achieve spectral consistency defined by Wald et al. in 1997? In their definition any synthetic image should be as identical as possible to the original image once degraded to its original resolution. To answer these questions, this research first finds out that all the CS and MRA methods can be derived from the Bayesian fusion method by adjusting a weight parameter to balance contributions from the spatial injection and spectral preservation models. The spectral preservation model assumes a Gaussian distribution of the desired high-resolution MS images, with the up-sampled low-resolution MS images comprising the mean value. The spatial injection model assumes a linear correlation between Pan and MS images. Thus the spatial enhancement depends on the weight parameter but is irrelevant of which category (i.e., MRA or CS) the method belongs to. This paper then adds a spectral consistency model in the Bayesian fusion framework to guarantee Wald’s spectral consistency with regard to arbitrary sensor point spread function. Although the spectral preservation in the EGIF methods is competitive to spatial enhancement, the Wald’s spectral consistency property is complementary with spatial enhancement. We conducted experiments on satellite images acquired by the QuickBird and WorldView-2 satellites to confirm our analysis, and found that the performance of the traditional EGIF methods improved significantly after adding the spectral consistency model.
BibTeX
@inproceedings{zhang2015new,
  author = {Hankui K. Zhang and Bo Huang},
  title = {{A New Look at Image Fusion Methods from a Bayesian Perspective}},
  booktitle = {Remote},
  year = {2015},
  doi = {10.3390/rs70606828},
}
Figure