Pixel-to-Abundance Translation: Conditional Generative Adversarial Networks Based on Patch Transformer for Hyperspectral Unmixing

Li Wang, Xiaohua Zhang, Jinhua Zhang, Hua Dong, Hongyun Meng, Licheng Jiao

TL;DR

This paper introduces HyperGAN, a novel framework for hyperspectral unmixing that treats the process as a "pixel-to-abundance translation" task. By leveraging a Patch Transformer within a Generative Adversarial Network (GAN), HyperGAN effectively captures spatial-spectral information and learns to generate accurate abundance maps without relying on manually defined prior knowledge, outperforming existing methods.

What Problem Does This Paper Address?

Hyperspectral imaging captures detailed spectral information, but individual pixels often contain mixtures of multiple substances. Hyperspectral unmixing (HSU) aims to identify these substances (endmembers) and their proportions (abundances) within each pixel. Traditional HSU methods often struggle with complex mixing scenarios and require careful selection of prior knowledge and regularization terms, making them difficult to generalize. Existing deep learning approaches, while promising, can still be limited by their reliance on specific mixing models or their inability to fully exploit the rich spatial-spectral information present in hyperspectral data. This paper addresses the challenge of developing a robust and generalizable HSU framework that can handle complex mixing and effectively utilize spatial-spectral information without manual prior knowledge engineering.

Methods

The core of this work is the HyperGAN framework, which frames hyperspectral unmixing as a pixel-to-abundance translation problem. This involves two main components:

  • Generator (G): This component takes a hyperspectral image (HSI) patch as input and aims to generate the abundance map for the central pixel of that patch. The key innovation here is the HSI Patch Transformer. This transformer architecture, inspired by natural language processing models, divides the HSI patch into tokens and uses adaptive attention scores to capture correlations between the central pixel and its neighbors. This allows for a fine-grained exploitation of spatial-spectral information, adapting to the specific characteristics of the HSI data.
  • Discriminator (D): This component acts as a classifier, learning to distinguish between true abundance maps and those generated by the generator. Through an adversarial process, the generator is trained to produce abundances that can fool the discriminator, effectively learning the underlying unmixing model implicitly. This replaces the need for manually defined loss functions and regularization terms.

Additionally, the authors propose a novel data synthesis method based on superpixel segmentation and random split. This method generates synthetic HSI data with realistic spatial structures, which is crucial for evaluating unmixing algorithms that leverage spatial information.

Datasets

The paper utilizes the following datasets:

  • Synthetic Datasets: Generated using the proposed superpixel segmentation and random split method, with varying Signal-to-Noise Ratios (SNR) (10dB, 20dB, 30dB).
  • Real Datasets:
    • Urban Dataset: A standard HSI dataset with 210 spectral bands, commonly used for HSU research.
  • Samson Dataset: Another HSI dataset with 156 spectral bands.
  • Cuprite Dataset: Collected by Aviris, this dataset has 188 spectral bands and is used for evaluating unmixing when only endmember labels are available.

Key Insights and Ideas

  • Unmixing as Translation: Reframing HSU as a "pixel-to-abundance translation" task offers a novel perspective, viewing abundance as a manifold representation within the low-dimensional space of hyperspectral pixels.
  • Adaptive Spatial-Spectral Feature Extraction: The HSI Patch Transformer's adaptive attention mechanism is a significant advancement. Unlike fixed convolutional kernels, it dynamically learns the importance of spatial context, leading to more accurate and nuanced feature extraction.
  • GANs for Implicit Model Learning: Using a GAN for unmixing bypasses the need for explicit, hand-crafted regularization terms. The adversarial process allows the generator to implicitly learn complex mixing models tailored to the data.
  • Realistic Synthetic Data Generation: The superpixel segmentation and random split method for synthetic data generation is a valuable contribution, creating datasets with more realistic spatial structures, which is essential for evaluating spatial-aware unmixing methods.

Key Results

The proposed HyperGAN method demonstrated superior performance across various metrics (RMSE, aRMSE, rmsAAD, aSAM) on both synthetic and real datasets compared to state-of-the-art algorithms, including SUnSAL, FCLS, NNet, CAE, and scGAN.

  • Synthetic Data: HyperGAN achieved the best unmixing results, particularly showing excellent performance in preserving the spatial structure of abundance maps and outperforming previous work (scGAN) by effectively exploiting spatial-spectral information.
  • Real Data (Urban, Samson): HyperGAN consistently achieved the best quantitative results, demonstrating robust performance in both highly mixed and homogeneous regions. For instance, on the Urban dataset, HyperGAN achieved an rmsAAD of 1.00e-1, significantly better than other methods.
  • Cuprite Dataset: HyperGAN achieved the optimal unmixing result with an aSAM value of 0.1513, showing a 6.9% improvement over the scGAN method.
  • Robustness: HyperGAN exhibited better robustness to noise compared to other methods, maintaining accuracy across different SNR levels.

Limitations and Future Work

  • Supervised Learning: The current HyperGAN framework is based on supervised learning, requiring labeled data for training.
  • Future Work: The authors suggest extending the framework to unsupervised unmixing scenarios. They also mention that future work will explore extending the proposed framework to unsupervised unmixing scenarios.

Summary updated

Abstract

Spectral unmixing is a significant challenge in hyperspectral image processing. Existing unmixing methods utilize prior knowledge about the abundance distribution to solve the regularization optimization problem, where the difficulty lies in choosing appropriate prior knowledge and solving the complex regularization optimization problem. To solve these problems, we propose a hyperspectral conditional generative adversarial network (HyperGAN) method as a generic unmixing framework based on the following assumption: The unmixing process from pixel to abundance can be regarded as a transformation of two modalities with an internal specific relationship. The proposed HyperGAN is composed of a generator and a discriminator, the former completes the modal conversion from mixed hyperspectral pixel patch to the abundance of corresponding endmember of the central pixel and the latter is used to distinguish whether the distribution and structure of generated abundance are the same as the true ones. In addition, we propose hyperspectral image (HSI) patch transformer as the main component of the generator, which utilizes adaptive attention score to capture the internal pixels correlation of the HSI patch and leverages the spatial-spectral information in a fine-grained way to achieve optimization of the unmixing process. Furthermore, we propose a data synthesis method based on superpixel segmentation and random split to get synthetic data with a spatial structure to further evaluate the effectiveness of our model for unmixing. Experiments on synthetic data and real hyperspectral data achieve impressive results compared with state-of-the-art competitors.
BibTeX
@article{wang2024pixel,
  author = {Li Wang and Xiaohua Zhang and Jinhua Zhang and Hua Dong and Hongyun Meng and Licheng Jiao},
  title = {{Pixel-to-Abundance Translation: Conditional Generative Adversarial Networks Based on Patch Transformer for Hyperspectral Unmixing}},
  journal = {IEEE J. Sel. Top. Appl. Earth Obs. Remote. Sens.},
  year = {2024},
  url = {https://openreview.net/forum?id=NBDG3bYd3E},
}
Figure